Day 4: Going Splunking

Getting Started

I decided to branch this off into its own day, since it's kind of removed from what I was doing before.

I had decided to build a FreeRADIUS config from scratch so I could remove any unnecessary options that were included in the (rather large) default config file, and set my own baseline configuration to move forward.

Not too far into the configuration, I encountered log{} directive, which gave me a handful of logging options... including logging to syslog.

It occurred to me at this point that it might be worth actually setting up a syslog server, considering the breadth of all the things I'm trying to do, so I can have a single place to review log files.

So I did some research and settled on the free version of Splunk (partly because that seems like a helpful skillset in the outside world.)

Installation

First minor complication: virtually everything I've installed so far on CentOS has been available through the default yum repositories. Splunk is not - it comes as an RPM (or a .tgz, or a .deb package.) yum does use RPMs, but normally they're downloaded from repositories.

Anyways. I downloaded the RPM from Splunk to /root, and from there, it wasn't much different from a normal yum install - except for specifying a filename instead of a package name.

splunk installs slightly different from most packages, by putting everything under /opt/splunk. Not a huge deal, except that /opt/splunk/bin is not in my search path, so I just symlinked /usr/bin/splunk to /opt/splunk/bin/splunk and started up splunk per the install instructions.

Troubleshooting

At this point, I hit a minor snag: I couldn't access the Splunk web interface. I could ping the machine, and use SSH, I just couldn't use the web interface.

After some digging, I discovered something I was not aware of: specifically, that CentOS ships with its built-in firewall enabled, and the firewall's default settings only permit incoming SSH and DHCPv6 traffic.

Fortunately, an easy fix, once I found the right ports:

firewall-cmd --new-service=splunk --permanent

firewall-cmd --service=splunk --add-port=8000/tcp --permanent

firewall-cmd --service=splunk --add-port=8089/tcp --permanent

firewall-cmd --service=splunk --add-port=9997/tcp --permanent

firewall-cmd --reload

firewall-cmd --add-service=splunk

Perfect.

Configuration

The big "Add Data" button on the homepage seemed enticing, so I started down that path.

I didn't have any preexisting log files I wanted to import, and I didn't want to set up a forwarder - I just wanted Splunk to listen for incoming syslog data, so I went with Monitor.

The next screen continued on with the trend of being helpful:

At this point, I realized that I was going to have to modify my firewall rules again to accept incoming data:

firewall-cmd --service=splunk --add-port=514/tcp --permanent

firewall-cmd --reload

Naturally, when I did this, I couldn't access Splunk any more. Easy fix: I'd forgotten to add the --permanent option when I added the Splunk service to the current zone.

So, now, Splunk said it was ready to start receiving data. Linux has a utility called "logger" for making syslog entries, so I did exactly that:

# logger -T -n 127.0.0.1 -P 514 "test message"

Lo and behold - searching for "*" in Splunk found my message!

Next Up

Now that I have a syslog server working, it's back to configuring FreeRADIUS. Eventually I'll start migrating most of my CentOS services over to Splunk for logging.