New tool for Visualizing Data
I recently attended SecureWorld in Denver and was reminded by one of the
speakers that I'd been intending to look at visualization tools for ages now
such as those mentioned at
www.secviz.org and then it occurred to
me that SGI recently announced a really nifty tool called
mineset. So I took it for
a test drive.
My first go with it, I was intending to feed it DNS query log and filtering
data, possibly to have it show me what queries were leading to queries that
we blocked (ie, what website was some user surfing to seconds before
they landed on some hostname that we're already blocking in our DNS
filters. Well, that yielded some interesting graphs but not what I was
after (more on that below).
So next, I figured "Start Simple" (tm). I grabbed a portion of our
firewall logs (just a part of one day), then filtered out ONLY the outbound
packets that got blocked, and filtered out ONLY the ones coming from an SGI
IP address and going to a non-SGI address (some of our filters are to
block outbound traffic to private IP space for instance which I wasn't
interested in looking at). I fed it to mineset and it immediately noticed
one "outlier" in the data - someone apparently tried to connect to an X11
server outside of SGI. Hmm, I'll look into that. :-)
Then I told mineset to draw a "parallel coordinates" graph of this
data and got:
Interesting, eh? So, the first thing that leaps out is the UDP traffic.
Note that almost all of it is going to one port? (53) and it's coming from
a wide variety of IPs and going to a wide variety of IPs? This is something
I knew I'd find but it's cool that it's so readily apparent on the diagram.
Basically, this is a jillion or so linux systems now (stupidly) running
dnsmasq as their DNS client instead of something sensible like nscd. Why
is this dumb? Because dnsmasq is a full blown proxy - it does it's own
root server lookups and ignores that the DHCP server said "thou shalt
consult these two IPs for DNS" and instead it tries to do it's own root DNS
queries, which we block, and only then does it finally use the IPs DHCP
told it to use. LAME! I'd love to punch the nitwit who thought
this was a neat idea. Once upon a time, blocked outbound DNS queries
was a fine way to detect a compromised system.
Why do we block outbound DNS queries? Cuz I don't want my users sending
their DNS queries to some malicious server in... Well, I won't name the
country but you get the idea. I also go to a lot of work to make our DNS
servers filter out malicious hostnames and I don't want someone bypassing
those filters by using Google's DNS or something stupid. I also rewrite
the responses for stuff like pool.ntp.org so they resolve to our NTP
servers instead. No sense in people using an NTP server run by who knows
who out on the interwobble when we have our own!
Anyway, it's a slick diagram. I can also see a few other ports (ie, UDP to port
161 - SNMP stuff to a network that's no longer active at SGI), some windows
traffic (ports 135-139) to a few IPs that are no longer in service, etc.
There was also some TCP traffic, someone's VPN client trying to do TCP to
port 53 on their ISP's DNS server (rolls eyes), an old server from
an acquisition that's (???) trying to send email for some reason - time
to remind folks to de-comm that system, and a vpn client trying to connect
to HTTPS on a IP that's in our banned list. Hmph, I'll have to look into
that on Monday too. :-)
These graphs are interactive too. You can re-arrange the poles in the
parallel coordinates graph. You can mouse over any line/dot and get the
details from the data (in this case, IPs and dest port). And it can do a
bunch of different visualizations.
For instance, my first go with it, I fed it a bunch of DNS query data and
this was the parallel coords graph I made:
This one was built using a csv file with about 4.8 million rows.
Here's another parallel cordinates graph but using a smaller dataset
(just one client IP) and this time including the date/time as one of the
dimensions to graph:
Another interesting (for this dataset) type of visualization is the bubble
This one shows some of the different queries that were made and the
"BlockedVia" shows which RPZ rule caused this to be blocked. The ? value
is for queries we didn't block.
Anyway, check out mineset! The demo version is only limited in the size
of the dataset you feed it and you can play around with a pile of different
visualizations of the data. Plus it has a whole bunch of other features for
analyzing the dataset (determining what columns "predict" a value in another
column, for instance, looking for correlations), that I haven't even started
to play with. I'll have to fool around with it with some other data and
see what other cool things I can use it for.