LogZilla: Day 1

TL;DR - It was China!!!

I wanted to increase visibility of my network at home through the use of a central syslog server. I decided on trying a COTS product instead of rolling my own. I chose LogZilla as my product to try. I had it downloaded, running and receiving its first logs in less than 30 minutes.

I downloaded the *.ova as I already have a small ESXi server running a few VM's with some spare resources.

I setup the VM's hardisk as thin provisioned and gave the VM 2 GB memory. The VM booted but gave an error that it needed a minimum of 4 GB to run LogZilla, I stepped it up to 4 GB and it booted fine. I know the website suggests 8GB but I'm cheap :) Upon initial boot the console asked you log in, and begin the 'first boot sequence'. I assume this is downloading the latest version and updating the VM before launching the LogZilla service.

Once LogZilla is up and running you should go around to your devices and configure the IP address of LogZilla as your remote syslog server or configure your current syslog server to forward events to LogZilla.

Upon first log in to the Web GUI of LogZilla you'll be presented with a Generic Dashboard full of helpful widgets.

The first of which displays the overall statistic of LogZilla's log ingestion. This shows the Events Per Day in max and average. This is helpful to understand what scale license you'll require for your environment.

Another widget shows a pie chart depicting all of the hosts that have sent logs to LogZilla and shows a comparison by volume.

Under the top widgets are 2 pre-configured "Live Stream" widgets in table format. These update in real-time and provide the live stream view. These are very helpful, one table contains all the logs that contain "Error" and the other contains all the logs which contain "Failed". This is great for quick look. I have this widget set to show the logs for the whole day. Because I have a small home network all of the events that contain "Error" or "Failed" in a day isn't very many.

All the widgets are customizable and you can build your own widgets. You can also utilize the provided or custom widgets to create or modify the dashboards. Being the first day I wasn't too interested in making custom dashboards or widgets, I really just wanted to get it working and pumping some logs through.

The bottom widget table is also a "Live Stream" widget. I have it set to show the logs from the 'last minute'.


After having LogZilla running for less than 12 hours I had enough data to start looking into some of the investigative features. I noticed a couple of TLS errors coming from openvpn for a time of day I wasn't connecting to my own VPN.

I selected an event and right-clicked, there are a bunch of helpful context options. 

Using the "Display Geo IP Information" tool I was able to located the source:

From the event I am able to right-click and create a Trigger based on that event.

A trigger allows me to specify some criteria to match in a log message and make an action. This can be sending an email alert or assigning an actionable item to another LogZilla user. For my purposes I simply want a notification and the item to be marked as "Actionable":

The "Name" is arbitrary but should be something that means something to you. When using the "Create Trigger" feature from the main dashboard LogZilla will pre-fill all this information based on the event you chose from the beginning. I edited out the specifics of the "Event match" because I didn't want it to on trigger on the single IP address. I want it to trigger anytime there is a log message that contains the "Event match" phrase from the screenshot.

Now, if LogZilla see's a log containing the message we specified it will create a notification and put the log entry in the "Actionable" widget as configured by my Trigger.

LogZilla has also helped me notice some anomalies that would otherwise gone unnoticed. After the first day of collecting logs I noticed I have nearly 10K log events coming from my router. If I watch the "Live Stream" I see most of them are duplicates. I was seeing this log almost every 5 seconds.

Note: "Host" column removed to better fit in screenshot.

I looked it up and found this reference. An update and changing the log level from the router would make this go away.

So, for my first day of watching logs I have found China trying to connect to my VPN and an anomalous log that shows up every 5 seconds. 

Awareness is King!

This concludes Day 1 of my Thanksgiving holiday. 

Future = Application Layer Networking

I was having a conversation with a peer about the future of networking. The foundation of the conversation revolved around SDN and the changes that SDN brings to network operators and engineers. The point was raised to me that 'engineers and operators of future networks won't need to have the granular low level understanding of bits, bytes and protocols.' As control of the network becomes more and more software driven the engineer/operator needs only high level understanding. My response to that is: Nothing could be further from the truth! My prediction for the future is: true application layer networking. Have a predictable and deterministic path through the network based on application only.

I think in the future, of application layer routing, we will need to incorporate some level of routing intelligence on each host/end device. I'm not sure exactly what that will look like yet but, I know it is not along the lines of OSPF or EIGRP.

In our current model, for most networks (home networks and small business networks) there is a single egress point where all traffic leaves your LAN to destinations on the internet.

In mid-sized business/enterprise you'll have redundant links a backups. You may have site-to-site tunnels with IPSEC connected remote sites but anything not on your LAN or part of your remote sites still egresses a single point to destinations on the internet.

Large businesses/Enterprises may have multiple egress points to the internet, all managed by lead engineers and operators with oversight from the senior engineers, involving multiple AS's and public IPv4 subnets that span the globe. This is expensive and the bottom line is, even with all the sophistication, the workstations and end devices are still taking the shortest path out of the network based on destination IP address and not on application specific characteristics.

Routers are doing destination based forwarding all over the globe. They are not making routing decisions based on the the type of traffic in the payload of the packet.

One thing I foresee SDN doing for us is bringing dynamic intelligence into routing. Having your controller understand the link requirements of protocols and identify those protocols as they are passing through the routers and forward them based on the application traffic they are carrying not just their destination IP address.

Another thing I believe the future holds for us is true multi-path routing, where end devices, even a common smart phone, can have multiple gateways and not just redundant default gateways, instead they would be application specific gateways. For example I could be connected to my cellular network, wifi and maybe a bunch of ad-hoc networks all at the same time. Perhaps those ad-hoc networks have gateways of their own and we can use them to egress to the internet essentially giving a device like our phone, multiple egress points. Letting our devices participate in the decision making process for routing and forwarding and how to best utilize the links available to it on a per-application basis.

Sorry I went off on a minor futuristic sci-fi routing tangent for a moment.

To bring this full circle, I feel like the engineers and operators of the future will actually need to know more about the inter-workings of each protocol more than just Layer 4. If the future is anything close to application layer networking, we will actually need to be closer to the bits and bytes to understand the protocol of the applications themselves in-order to programmatically and deterministically route them to their destinations.

P.S. - I'm not talking about getting rid of IP addresses but instead introduce more to forwarding than just the destination. I'm sure all the "every packet should be treated equal" people out there are going to have a fit with this.

Comments are welcomed.