Play With Your Log Data
Got some log data? I highly recommend playing with it! We humans are visual
creatures and are actually pretty good at spotting anomalies. So before
anyone complains "yabbut I don't have and can't afford fancy tools", even with
the most rudimentary opensource tools and the most rudimentary charts and
graphs you can spot "weird" (tm) in your log data. Honest. I promise.
So I thought I'd show you some examples of some of the sorts of charts I've
found useful at work. I used kibana simply because A) I've got it and that's
what I use to search through my log data with and B) because it's dead easy
to make any sort of chart/graph it supports in seconds. Whenever I think of
a new way to represent the data, it's almost always followed by my thinking
"Hmm, what's that" or "I wonder what I'd find if I filter that out" or
"what will it look like if I make the chart using only a subnet of
this data." One visualization quickly leads to another and another and
kibana (and tools like mineset also) lend themselves to this.
And when you find some visualizations that help you identify something odd,
add them to one or more dashboards. Look at them daily. You'll be surprised
at how often they help you find things all those fancy, expensive
CyberSecurity tools missed. :-)
Here's one of my DNS dashboards with 4 different types of charts. While
looking at these examples, keep in mind that I can mouse-over any part of the
chart to see what hostname/ip/threat/whatever that part of the chart is. I
can also look at the colored dots on the right and scroll up/down through
this key to find out what the different slices/lines/whatever are.
In the upper left you'll see a simple line chart showing the query rates of
several of our DNS servers (each server is a different line and the
blacked out section next to the dots is where one could see which color/line
is for which DNS server). In this chart I look for big changs in volume of
DNS queries. Simple. Usually any big spikes here aren't anything security
related, just something mis-configured or mis-behaving.
In the upper right corner you can see an area chart split by the resource
record type - what sorts of DNS queries are being made. This sounds silly
at first, but think about it - a big spike in queries for MX records is
probably a spam-bot trying to send spam. A big spike in PTR queries may
be a system trying to map your network, looking up hostnames for every IP
address. Or it may be trying to map someone else's network (think "bot
infection"). Or perhaps it's an internet-facing server that does a lookup
on every IP that connects to it to log a hostname (this is bad for
scalability, BTW). And maybe it's being beat on by a botnet trying to
brute-force logins. When you see any spike in a particular type of query,
don't be shy to think "Hmm, what's that" and go dig into the log data to
find out! You may just find a new Indicator Of Compromise (IOC) to watch
for in the future!
In the bottom left corner I have a pie chart that's split first by source
IP, and then each slice is further sub-divided by the top hostnames that
IP is querying DNS for. Mostly I use this to just discover mis-configs and
systems that really ought to be running a caching DNS client. But ya never
know, a system that's suddenly doing a huge number of queries for a huge
variety of hostnames might be suspicious (a wide slice of the pie which is
then sub-divided by a huge number of slices - visually very different from
"normal"). For instance the first slice is just a system querying for it's
own hostname repeatedly. Dumb. It should log IPs instead of hostnames or
it should have a caching client so it's not repeatedly querying for the
same thing. The second slice is actually one of our mail relays so it looks
up MX records for lots of hostnames and sub-domains (which is why it's outer
slice has a bunch of sub-divisions).
The lower right chart shows our DNS filters. It's first split by source IPs
and then each slice is sub-divided by the hostname it's querying for which
we blocked. Most of those teeny slices are various web analytics/tracking
garbage - yeah I don't trust marketing organizations that try to track my
users' every move on the internet. However that big red slice is a system
running some spyware/adware which Desktop Support hasn't scraped off this
user's system yet.
Here's an email dashboard with 2 pie charts and 2 line charts:
In the upper left corner you can see something interesting right off. It's
a pie chart split first by the IP address connecting to our inbound mail relays
and then split by Action (blocked, deferred, delivered, etc) and then those
are further sub-divided by Reason (rate control, reputation, rbl-match, etc).
So right off we see there's two actions that account for most connections,
blocked and deferred. The purple color in the second ring is deferred,
because of our rate-control rules. The cyan color is "blocked" and the green
color in the 3rd ring is "authentication failure". Basically, most of
the traffic our barracudas are seeing is IPs trying repeatedly to do SMTP
authentication (and failing) until they're finally being rate-controlled. Yup,
spam is less of a problem these days than SMTP brute forcing. Do you let your
users connect to and authenticate with your mail servers over the internet,
not requiring a VPN or 2-factor authentication? You're basically begging
someone to compromise your email accounts (I often see big botnets doing this
so each IP only tries 3 or 4 passwords per username they try). Sounds useless?
But it works. Too many people use too easily guessed passwords. And when
you have a botnet with hundreds of thousands of nodes, you can try a lot
of passwords without locking the account. Not too long ago I saw a dozen or
so /16 subnets all in China doing this.
In the upper right corner the pie chart shows senders connecting to our
internal postfix relays and trying to send an email. It's first divided by
the IP/host trying to connect and then each slice is further divided by the
to address it's trying to send to. So naturally the top 3 slices are the three
barracuda spam firewalls trying to send us email (mostly spam/phish). Then,
whee I see a color in the outer ring that all 3 cudas have (like that first,
light blue, slice), that's usually the email address of an ex employee. This
is often a good candidate for making a new spam trap! If it's an address
that isn't in use, and hasn't been in use either ever or for at least, say,
5 years, then it'll make a dandy spam trap. Early warning for spam/phish!
The bottom two line charts are just so I can look for big spikes in actions
taken by the barracudas or the reasons why. I'll see a big spike in auth
failures, which will send me digging in the logs to see what IPs/Nets are
causing that spike, and what username they're trying to authenticate as.
This lets me check if it's botnet activity or all coming from a specific
country. It lets me look to see if they're targeting a specific user or group
or if they're just trying any old account and guessing at user names.
Here's a one-off area chart I whipped up:
It's one I did on my development log server at home. This new log server does
GeoIP lookups to figure out what country/city an IP is likely in. So I whipped
up a pie chart to show what countries are the source of SMTP brute force
attacks on my personal email server.
Here's a graph of some firewalls at work
It's just a line chart which I then split up by source interface. So it shows
the rate of rejected packets on each interface for each firewall (or at least
of the ones I'm monitoring). As you can see, "outside" is, unsurprisingly
enough, far more than rejected packets on the "inside" interface and that in
turn is more than all of the other interfaces (because we do a fair amount of
egress filtering). Kibana make is easy (4 mouse clicks) to filter out these
two interfaces yielding:
Nothing hugely interesting here, at least on this 24 hour period. But this
would be a good place to look for big spikes in traffic caused by, say, a
compromised system in a DMZ now trying to figure out what it has access to and
to attack next. And occasionally those jumps in rejected traffic on the
guest WiFi shows a guest has an infected system.
Here's a one-off chart I made showing ssh login failures:
As you can see I've split it up by destination username. (yeah, we've got an
ssh server but don't bother trying to brute-force accounts on it - we require
2-factor authentication). This chart immediately showed a few interesting
things. First, a HUGE spike in attempts to login as root recently (this chart
was over a 7 day time span). That would cause me to go audit the logs to see
what IP/s were trying to brute-force root. And the other really interesting
thing is the small spike on the left-hand side of the chart where all the
lines converge to a point.
Zooming in on the time range where that point is, we can see that during a
short period of time some IPs were suddenly tryig lots of different
usernames. Interesing, eh?
Using the same time range but splitting the lines by IP
rather than by username we can see that one IP is responsible for all of those
different username attempts. And it also shows something else
interesting - see that flat cyan line? I've got the IP responsible highlighted
on the right.
So, if we go peek in the logs for just that IP we can see several spikes in
activity and also see a bunch of the different usernames that were tried.
Again, using the same log data, let's plot it as a pie chart split first by
the username they tried to login to and then split those slices by the IPs
that tried them:
This shows a few interesting things. First, during this time period anyway
(back to the last 7 days), you can see root was the most often tried. Silly
buggers, we don't allow root to login via ssh! Also of note is that only a
few IPs were trying root (just repeatedly). Also note that the outer ring is
predominantly made up of only a few colors/IPs and they were trying lots of
And so taking the same data and ths time split first by IP and then by the
username being tried, then filtering out the top 4 IPs from the previous
We can see that there are still a mix of IPs that try to login only as root
and some that try all sorts of usernames.
I use these split line and pie charts a lot, but that's mostly just because
kibana (at least the version I'm using) doesn't have fancier stuff like
parallel coordinate charts. But as you can see we can do a lot with just
line, area, and pie charts, and the odd histogram. Here's my threats
The upper left pie chart I split first by source IP and next by the "threat
alerts" seen for each IP. You can see I've scrolled the key down to the end
of listing colors for IPs (the 2nd ring) and the start of the listing of
colors for threats (the 3'rd, outermost ring).
The pie chart on the lower left I show the same
data but first split by the threat alert and then by the IPs causing each
alert. These let me see, at a glance, if an IP is causing lots of
different alerts or if a single alert is being seen on a lot of IPs. Both
are "interesting". And as before, I can quickly filter out a threat or an IP
as I dig through the logs to identify if it's merely suspicious or an
indicator of an actual compromise. The upper right line chart lets me see
what sensors/interfaces I'm seeing the alerts on over time (looking for
spikes in activty) and the lower right pie is just dedicated to only the
Palo Alto Firewall related alerts. The other charts show alerts from
any/all sensors (snort, cyphort, botsink, PaloAltos, etc).
It's "normal" to see tons of activity like this when looking at traffic
to/from any DMZ segments the outside world can touch. Internet facing
services are under attack 24x7x365. Filterig out those segments and
networks/IPs, we see:
Same dashboard, same visualizations, but a lot less busy (just to show the
value both of seeing everything as well as being able to filter out the
routine, noisy traffic of people probing every service on every server on
every network exposed to the internet. So, what if we add a bit of GeoIP data
to some of our threats data?
The one thing that sticks out here is something I discovered one day when
graphing snort alerts on a world-map. This pie chart is split first by country
and then further divided by the alerts for IPs in that country. The first
slice is the US and the top threats are VOIP related. Not sure how critical
that is to know but it's interesting. The next country is China (big
surprise) and most of it's attacks are ssh related. It's an interesting
pattern I've seen many times before.
Anyway, I hope this spurs someone else to play with visualization tools
and their own log data. There's always some new way to arrange the
data in some graphical way and always something new to discover in the log
data. I find that the more I play with viz tools the more new indicators
of suspicious activity I find (new patterns to search for in the logs). Then
as I search for that activity that often leads to new ideas of data I ought
to log and ways to display that. The log analysis feeds the visualizations
and the visualizations feed the log analysis. :-)