Making an Asset Tracking Tool with OpenLDAP

This post will be about how to use OpenLDAP to make an asset tracking tool, suitable for keeping track of hardware, the software/apps being run who owns them, who cares about them, who to notify in case of outages, notes on the care and feeding and troubleshooting, remote console access, and anything else you care to track. Some assembly is required, as is true with most open source solutions, but then again, this one can easily be tailored for whatever your needs are.

If all you want to see is the geeky details, you can skip the next section where I give a little background and my rationale for this tool.

At my job at SGI, we had a tool called DCSi which I think stood for something like "Data Center System Information". Don't ask me why the i was lower-cased - it always was when described to me and it's just habit for me now - grin). I found it invaluable as the FNG (friggin-new-guy) when I was on-call. When I got a page in the wee hours for a server I'd never heard of, I could point my browser at DCSi, search for the server name and get all sorts of useful information like where the machine was (we're on call for servers all over the world), which building, which room, which footprint in that room, what the server was for, who "owned" it, etc. But it had "issues".

It was an internally developed tool - a collection or Perl CGIs with a Sybase back-end - and the person who had developed it was long-since gone. Nobody knew anything about it's guts, and it seemed to lose changes made to the data, especially if more than one person edited the data. When we'd audit the data center floor and note which servers were in which racks at what footprints, noting serial numbers, asset tags, etc, invariably someone's changes to DCSi would go missing. Bummer.

Then the datacenter in Mtn View, CA was closed and all the servers shipped to Chippewa Falls, WI. And DCSi stopped working. And since it'd been troublesome in the past with updates anyway, nobody wanted to spend any time trying to fix it (it didn't help that almost everyone in a position to fix it was getting laid off after the DCO move was done). So everyone took to editing a spreadsheet stored in a common location as semi after semi hauled all the servers to their new digs in WI each weekend. Needless to say, this didn't work out very well and we had multiple versions of this spreadsheet with conflicting info in it. Ugh.

I'd just built a new LDAP infrastructure for SGI and had to make a custom schema for some of the data we put in it, so it occurred to me that I could easily put the same data in LDAP. And instead of making some one-off custom web app that nobody had time to maintain, we could just use any LDAP browser/editor to manage the data! I like Simple. And unlike the old DCSi app, we could have multiple values for some of the items we tracked (ie, multiple applications for a single server). Also unlike the old DCSi, if we wanted to add something to what we tracked, it was a relatively easy change - no application needed rewriting or SQL database changes needed.

So, I banged out a quick LDAP schema to track the same sorts of things we had in the old DCSi application, added some more things I thought would be useful, and presented it to the new boss (yeah, luckily I was kept on after the servers were almost all moved to WI). After several demos to various IT folks and application folks, it became the new DCSi. I also made a very rudimentary PHP script to query it so we could easily hook it into our nagios server. Each server in nagios now has a little red folder icon next to it - click on it and it does an LDAP query and shows you all sorts of useful info about that server - the sorts of things you're likely to want to know when you just got paged about something on that server like how to get remote console access, where it is, what it runs, links to any build notes or documentation for the server or the apps it's running, serial numbers, support contract info, etc.

Cool, eh?

Instructions

We build our LDAP server from source. We do this for several reasons. I don't really want to have to be tied to a specific version of OpenLDAP just because some linux distro chooses it, or to have to upgrade the OS to get a newer version. Plus it means we can build in support for the things we care about and not build in support for the things we don't. It's just reduces the number of possible security exploits a little. Besides, it's pretty darn trivial to build an LDAP server.

I created a directory to build a particular version of our LDAP server, say /data/ldap/build-v1.3 and inside there make a repository directory that contains all the source .tar.gz files I'll be needing. Here's an old version of that repository directory:
cyrus-sasl-2.1.25.tar.gz
openldap-2.4.31.tgz
db-4.8.24.tar.gz
openssl-0.9.8w.tar.gz
zlib-1.2.6.tar.gz

I also have a script that then builds all the binaries in the right order with the right config options and installs everything in one location. The idea here is that if I want to build a new set of binaries, it's easy - just remove an old source tarball from the repository directory, replace it with a newer version, re-run the extract-source script then optionally tweak the build script to have a new install destination and rebuild a fresh set of binaries without clobbering an existing set. We can quickly stop an LDAP instance and re-start it with a new set of binaries and if we don't like what we see, stop it and re-start it with the old binaries.

Anyway, here's my uber-simple extract-source.bash and do-build.bash script.

Note that the do-build.bash script is very simple. It does no error checking. So if you're building for the first time or if you've just updated a package in the repository be sure you run it saving the output to a log file and check it carefully for errors, especially if you're on an x86_64 or itanium platform. Opensource developers are still sometimes releasing code that doesn't compile out of the box on the 64 bit platforms so the next release of openssl or cyrus-sasl, for instance, might not compile right on 64 bit linuxes. Usually this is fixed fairly quickly, but it's worth watching out for.

Also, the do-build script, by default, builds BerkeleyDB with support for their self-test which require tcl and tk development stuff to be installed. If you want to omit this, though I always recommend re-running all the tests when you switch versions of BerkeleyDB before assuming all will be well with OpenLDAP, you can edit the do-build.bash script and comment out the line that says --with-tcl=/usr/local/tcltk-8.4/lib --enable-test

So, once you've got LDAP binaries built (assuming you didn't start with a pre-packaged set of binaries) you'll probably need to setup a config file and a fresh database. There's several ways you can do this and OpenLDAP has a new way of storing the config in a special LDAP DB rather than using a config file. I'll leave all that as an exercise to the reader and show a simple starting config. I'll also ignore things like running N-Way LDAP replication, which you might want to do eventually.

If you're using pre-packaged binaries, you'll need to edit your slapd.conf file to include my local.schema file and stop/restart your slapd daemon. For the purposes of this example, let's assume our install destination was /data/ldap/instance-v1.3 and that we're not keeping our LDAP instances separate from the build instances. If you do want to keep them separate (to make it easy to switch from one version of binaries to another), then you'd use another directory for your LDAP instance like, maybe /data/ldap/myldap and inside it you'd have your own etc and var directories with your own slapd.conf file, schema files, and LDAP DB files).

Here's a very simple slapd.conf file. You'll want to tailor it to your needs and you'll want to install it in your LDAP instance's etc/openldap directory - in this example, /data/ldap/instance-v1.3/etc/openldap. I'll also include an extra schema file named local.schema which should be in /data/ldap/instance-v1.3/etc/openldap/schema/. Finally, here's an initial.ldif file which we'll use to build a fresh DB and a DB_CONFIG file which will tell OpenLDAP about what BerkeleyDB settings we want to use. Put the initial.ldif file in /data/ldap/instance-v1.3 and put DB_CONFIG in /data/ldap/instance-v1.3/var/openldap-data/.

I also usually create a README.runtime file in /data/ldap/instance-v1.3 (or in my separate LDAP instance directories as needed if I'm keeping 'em separate). It serves two purposes - it lets me drop notes in it on the LDAP instance, can be "sourced" by a bash shell to load all the environment variables to use the man pages or binaries of the instance, and also shows a command line or two to start the LDAP server. Here's an example README.runtime file. You'll want to customize it to contain your LDAP server's hostname or IP in the LDAP URL it specifies on the command to start it slapd.

Ok, almost there. You've created your directories, installed the slapd.conf, local.schema, initial.ldif, DB_CONFIG, and README.runtime file. You've customized them for your base dn, initial password for the cn=Manager account, etc. Now, time to create a fresh DB and start slapd. To do this, do:

cd /data/ldap/instance-v1.3/
source README.runtime
cd var/openldap-data
slapadd -f ../../etc/openldap/slapd.conf <../../initial.ldif

It should say it imported those few records and you should now have some DB files. If so, fire up slapd with:
$BTARGET/libexec/slapd -h "ldap://myhostname:389/" -f /data/ldap/ldap.sgi.com/etc/openldap/slapd.conf

If slapd starts, congratulations, you've built your very own LDAP server from scratch. Pat yourself on the back and worry about things like LDAP replication, SSL/TLS support for later. :-)

Now for the fun. You can use any LDAP browser editor tool that you like, but at the time of this writing, I prefer Aapche Directory Studio. It's interface may seem a little peculiar at first, but you can get used to it quickly and it provides a powerful suite of tools for searching, modifying, and reporting on data in an LDAP server (like the DCSi data). Fire up Apache Directory Studio and connect to your brand new LDAP server as the cn=Manager user. You should see there's already an ou=DCSi object. I'll include some example structure beneath it but you can put whatever you'd like in whatever locations you'd like. It's just LDAP - arrange your data however you see fit! :-)

Having said that, here's how we use it. We have ou=Applications. This branch contains dcsiApplication objects. These contain some info about the various apps we run. Other objects will refer to these.

We also have ou=Platforms. This is where we'll put dcsiPlatform objects and notes about them. If a particular platform always has (or even just usually has) certain things in common like clock rates, number of CPUs, etc, you can set them here. But we usually set stuff like that in specific server records too. But like I said, you can put what ya like where ya like.

Then there's ou=Servers. This is where we put records for our live, operating servers. We also have ou=Spares, ou=Available, and ou=Scrapped branches. The idea is that if we take down a server but want to keep it available for other uses (say, we power it off but leave it in the rack), we put it in ou=Available. When someone needs a new server for some purpose, search there first to see what's most quickly available. If we shut down a server and unrack it but we keep it in storage for spare parts, we'll put it's dcsiServer object in the ou=Spares branch. If we eventually scrap hardware, we'll toss it in the ou=Scrapped branch so if someone later comes asking what happened to server XYZ with asset tag ABC we can still find it and say "It was scrapped on this date". But, again, you can layout the structure of the data however you want. If you're gonna interface stuff with nagios, I'd recommend putting all the devices/servers you're tracking under one ou= branch or at least all the active stuff under one branch like ou=Servers. It'll make querying for it from a script much easier.

Wrapping up

Finally, here's a quick video showing me creating a fresh DB, starting OpenLDAP and connecting to it with my LDAP browser/editor of choice, Apache DirectoryStudio, creating a new connection in DirectoryStudio pointing to my new LDAP server and creating a few new records.

Here's another quick video showing some simple ways to make new searches in DirectoryStudio.

And here's another quick video showing how to make searches based on operational attributes like when a record was created in DCSi (LDAP) or when it was last modified.

One thing I forgot to draw notice to in the video is that if a server is known by many different hostnames, say a webserver that hosts many virtual hosts, you can add them all as additional dcsiServerName attribute values. That way if someone searches by any of the virtual hostnames (ie, when they get a ticket saying "website such 'n such is down") they can quickly see which physical system is likely involved.

Also, there's a variety of attributes for platform objects and application objects, not just servers. So you could have some emergency contacts and owners specified for a server, but you could also have them specified for Application objects that a server's dcsiHostedApplication attribute referred to.

If anyone asks, I've also made a few other quickie videos for my co-workers showing some tips 'n tricks on using DirectoryStudio. I could post them here if there was interest so drop me an email if you'd like to see 'em. They're specific to Apache DirectoryStudio, of course. But if you don't like this particular tool, you can use any LDAP editor/browser.

I also made a script to query a server and populate a DCSi record with as much info about a given server name as I can. It's moderately specific to SGI, but it could be used as an example if someone wanted to make their own tool. I also made one for scanning cisco devices and adding their info to DCSi records. Drop me an email if you'd like a copy of either.

Lastly, I made a few simple PHP scripts for querying DCSi and showing the sorts of attributes I figured an on-call person might be most interested in. The intent was to make it easy to glue in this sort of info into nagios as extra information links (which nagios displays as a red folder icon on host pages and next to hostnames on host lists - click on the icon and it opens a new browser tab to the PHP script querying for that server name). If anyone wants it, I can drop an example of it here too.