Play With Your Log Data

Got some log data? I highly recommend playing with it! We humans are visual creatures and are actually pretty good at spotting anomalies. So before anyone complains "yabbut I don't have and can't afford fancy tools", even with the most rudimentary opensource tools and the most rudimentary charts and graphs you can spot "weird" (tm) in your log data. Honest. I promise.

So I thought I'd show you some examples of some of the sorts of charts I've found useful at work. I used kibana simply because A) I've got it and that's what I use to search through my log data with and B) because it's dead easy to make any sort of chart/graph it supports in seconds. Whenever I think of a new way to represent the data, it's almost always followed by my thinking "Hmm, what's that" or "I wonder what I'd find if I filter that out" or "what will it look like if I make the chart using only a subnet of this data." One visualization quickly leads to another and another and kibana (and tools like mineset also) lend themselves to this.

And when you find some visualizations that help you identify something odd, add them to one or more dashboards. Look at them daily. You'll be surprised at how often they help you find things all those fancy, expensive CyberSecurity tools missed. :-)

Here's one of my DNS dashboards with 4 different types of charts. While looking at these examples, keep in mind that I can mouse-over any part of the chart to see what hostname/ip/threat/whatever that part of the chart is. I can also look at the colored dots on the right and scroll up/down through this key to find out what the different slices/lines/whatever are.

In the upper left you'll see a simple line chart showing the query rates of several of our DNS servers (each server is a different line and the blacked out section next to the dots is where one could see which color/line is for which DNS server). In this chart I look for big changs in volume of DNS queries. Simple. Usually any big spikes here aren't anything security related, just something mis-configured or mis-behaving.

In the upper right corner you can see an area chart split by the resource record type - what sorts of DNS queries are being made. This sounds silly at first, but think about it - a big spike in queries for MX records is probably a spam-bot trying to send spam. A big spike in PTR queries may be a system trying to map your network, looking up hostnames for every IP address. Or it may be trying to map someone else's network (think "bot infection"). Or perhaps it's an internet-facing server that does a lookup on every IP that connects to it to log a hostname (this is bad for scalability, BTW). And maybe it's being beat on by a botnet trying to brute-force logins. When you see any spike in a particular type of query, don't be shy to think "Hmm, what's that" and go dig into the log data to find out! You may just find a new Indicator Of Compromise (IOC) to watch for in the future!

In the bottom left corner I have a pie chart that's split first by source IP, and then each slice is further sub-divided by the top hostnames that IP is querying DNS for. Mostly I use this to just discover mis-configs and systems that really ought to be running a caching DNS client. But ya never know, a system that's suddenly doing a huge number of queries for a huge variety of hostnames might be suspicious (a wide slice of the pie which is then sub-divided by a huge number of slices - visually very different from "normal"). For instance the first slice is just a system querying for it's own hostname repeatedly. Dumb. It should log IPs instead of hostnames or it should have a caching client so it's not repeatedly querying for the same thing. The second slice is actually one of our mail relays so it looks up MX records for lots of hostnames and sub-domains (which is why it's outer slice has a bunch of sub-divisions).

The lower right chart shows our DNS filters. It's first split by source IPs and then each slice is sub-divided by the hostname it's querying for which we blocked. Most of those teeny slices are various web analytics/tracking garbage - yeah I don't trust marketing organizations that try to track my users' every move on the internet. However that big red slice is a system running some spyware/adware which Desktop Support hasn't scraped off this user's system yet.

Here's an email dashboard with 2 pie charts and 2 line charts:

In the upper left corner you can see something interesting right off. It's a pie chart split first by the IP address connecting to our inbound mail relays and then split by Action (blocked, deferred, delivered, etc) and then those are further sub-divided by Reason (rate control, reputation, rbl-match, etc). So right off we see there's two actions that account for most connections, blocked and deferred. The purple color in the second ring is deferred, because of our rate-control rules. The cyan color is "blocked" and the green color in the 3rd ring is "authentication failure". Basically, most of the traffic our barracudas are seeing is IPs trying repeatedly to do SMTP authentication (and failing) until they're finally being rate-controlled. Yup, spam is less of a problem these days than SMTP brute forcing. Do you let your users connect to and authenticate with your mail servers over the internet, not requiring a VPN or 2-factor authentication? You're basically begging someone to compromise your email accounts (I often see big botnets doing this so each IP only tries 3 or 4 passwords per username they try). Sounds useless? But it works. Too many people use too easily guessed passwords. And when you have a botnet with hundreds of thousands of nodes, you can try a lot of passwords without locking the account. Not too long ago I saw a dozen or so /16 subnets all in China doing this.

In the upper right corner the pie chart shows senders connecting to our internal postfix relays and trying to send an email. It's first divided by the IP/host trying to connect and then each slice is further divided by the to address it's trying to send to. So naturally the top 3 slices are the three barracuda spam firewalls trying to send us email (mostly spam/phish). Then, whee I see a color in the outer ring that all 3 cudas have (like that first, light blue, slice), that's usually the email address of an ex employee. This is often a good candidate for making a new spam trap! If it's an address that isn't in use, and hasn't been in use either ever or for at least, say, 5 years, then it'll make a dandy spam trap. Early warning for spam/phish!

The bottom two line charts are just so I can look for big spikes in actions taken by the barracudas or the reasons why. I'll see a big spike in auth failures, which will send me digging in the logs to see what IPs/Nets are causing that spike, and what username they're trying to authenticate as. This lets me check if it's botnet activity or all coming from a specific country. It lets me look to see if they're targeting a specific user or group or if they're just trying any old account and guessing at user names.

Here's a one-off area chart I whipped up:

It's one I did on my development log server at home. This new log server does GeoIP lookups to figure out what country/city an IP is likely in. So I whipped up a pie chart to show what countries are the source of SMTP brute force attacks on my personal email server.

Here's a graph of some firewalls at work

It's just a line chart which I then split up by source interface. So it shows the rate of rejected packets on each interface for each firewall (or at least of the ones I'm monitoring). As you can see, "outside" is, unsurprisingly enough, far more than rejected packets on the "inside" interface and that in turn is more than all of the other interfaces (because we do a fair amount of egress filtering). Kibana make is easy (4 mouse clicks) to filter out these two interfaces yielding:

Nothing hugely interesting here, at least on this 24 hour period. But this would be a good place to look for big spikes in traffic caused by, say, a compromised system in a DMZ now trying to figure out what it has access to and to attack next. And occasionally those jumps in rejected traffic on the guest WiFi shows a guest has an infected system.

Here's a one-off chart I made showing ssh login failures:

As you can see I've split it up by destination username. (yeah, we've got an ssh server but don't bother trying to brute-force accounts on it - we require 2-factor authentication). This chart immediately showed a few interesting things. First, a HUGE spike in attempts to login as root recently (this chart was over a 7 day time span). That would cause me to go audit the logs to see what IP/s were trying to brute-force root. And the other really interesting thing is the small spike on the left-hand side of the chart where all the lines converge to a point.

Zooming in on the time range where that point is, we can see that during a short period of time some IPs were suddenly tryig lots of different usernames. Interesing, eh?

Using the same time range but splitting the lines by IP rather than by username we can see that one IP is responsible for all of those different username attempts. And it also shows something else interesting - see that flat cyan line? I've got the IP responsible highlighted on the right.

So, if we go peek in the logs for just that IP we can see several spikes in activity and also see a bunch of the different usernames that were tried.

Again, using the same log data, let's plot it as a pie chart split first by the username they tried to login to and then split those slices by the IPs that tried them:

This shows a few interesting things. First, during this time period anyway (back to the last 7 days), you can see root was the most often tried. Silly buggers, we don't allow root to login via ssh! Also of note is that only a few IPs were trying root (just repeatedly). Also note that the outer ring is predominantly made up of only a few colors/IPs and they were trying lots of different usernames.

And so taking the same data and ths time split first by IP and then by the username being tried, then filtering out the top 4 IPs from the previous chart:

We can see that there are still a mix of IPs that try to login only as root and some that try all sorts of usernames.

I use these split line and pie charts a lot, but that's mostly just because kibana (at least the version I'm using) doesn't have fancier stuff like parallel coordinate charts. But as you can see we can do a lot with just line, area, and pie charts, and the odd histogram. Here's my threats dashboard:

The upper left pie chart I split first by source IP and next by the "threat alerts" seen for each IP. You can see I've scrolled the key down to the end of listing colors for IPs (the 2nd ring) and the start of the listing of colors for threats (the 3'rd, outermost ring).

The pie chart on the lower left I show the same data but first split by the threat alert and then by the IPs causing each alert. These let me see, at a glance, if an IP is causing lots of different alerts or if a single alert is being seen on a lot of IPs. Both are "interesting". And as before, I can quickly filter out a threat or an IP as I dig through the logs to identify if it's merely suspicious or an indicator of an actual compromise. The upper right line chart lets me see what sensors/interfaces I'm seeing the alerts on over time (looking for spikes in activty) and the lower right pie is just dedicated to only the Palo Alto Firewall related alerts. The other charts show alerts from any/all sensors (snort, cyphort, botsink, PaloAltos, etc).

It's "normal" to see tons of activity like this when looking at traffic to/from any DMZ segments the outside world can touch. Internet facing services are under attack 24x7x365. Filterig out those segments and networks/IPs, we see:

Same dashboard, same visualizations, but a lot less busy (just to show the value both of seeing everything as well as being able to filter out the routine, noisy traffic of people probing every service on every server on every network exposed to the internet. So, what if we add a bit of GeoIP data to some of our threats data?

The one thing that sticks out here is something I discovered one day when graphing snort alerts on a world-map. This pie chart is split first by country and then further divided by the alerts for IPs in that country. The first slice is the US and the top threats are VOIP related. Not sure how critical that is to know but it's interesting. The next country is China (big surprise) and most of it's attacks are ssh related. It's an interesting pattern I've seen many times before.

Anyway, I hope this spurs someone else to play with visualization tools and their own log data. There's always some new way to arrange the data in some graphical way and always something new to discover in the log data. I find that the more I play with viz tools the more new indicators of suspicious activity I find (new patterns to search for in the logs). Then as I search for that activity that often leads to new ideas of data I ought to log and ways to display that. The log analysis feeds the visualizations and the visualizations feed the log analysis. :-)