Installed Splunk a week ago and it's already proving useful

[Ben Corbett]

So I initially came across Splunk when seeing a banner ad on a blog site (may have been www.techrepublic.com) and I was inquisitive as to what the hell it was. After passing it over to a colleague to check out he informed me that it looked really great and we could definitely benefit from implementing it.

He set up the server but for a variety of reasons we never really embraced it. It wasn't until recently that I decided to dive in and check it out. I decided to ditch the VM that we had been using and start again from scratch (mainly so that I could get my head around it)

Setting it up was just as easy as advertised. I had a spare server that had some reasonable specs (Dual Quadcore, 8GB Ram) relative to the amount of traffic that I was expecting (~30 hosts) and I installed Centos 5.3 x86_64 and the latest version of Splunk 3. Then it was a simple matter of directing the syslogs of the hosts to the server and watch the information flood in.

One thing that I noticed was that some hosts took a bit longer to appear in the interface than others but after setting it up on Friday I saw that when I came in on Monday they were all there as expected.

Each day this week I have been logging in and nosing around the hosts to see what was happening.

This morning I checked as usual and noticed that there were a load of errors on our external server (website). It turned out that these were all ssh login errors from another IP. A pretty standard brute force ssh attack that seem to be commonplace these days. A quick whois showed that the attack originated from the same datacentre as our server (probably a compromised server hitting all machines on the same subnet). A quick call to the provider and the box was shut down. All done within a few minutes of getting into the office. Nice work Splunk!

I am now very interested to learn more about the best practices for following logs, config files etc. and more about the search syntax

[Michael Wilde]

Excellent Post. Recommendations. Upgrade to Splunk 4.x its like a monster truck rally (complete with flames) compared to the 3.x product. Field extraction (making structured sense out of those logs) is really easy to do. Next to each event (under the timestamp), there's a little "Extract Fields" button. That will launch, as Splunk calls it, IFX (interactive field extractor). It lets you give samples of the data you'd like to extract. For example, do this on SSH logins, type in some user names that appear in IFX's sampler, and it will write the regex, allow you to test the field, persist and name the field extraction and then you're golden.

Golden? After you have things like "User" pulled out of SSH logs (for password failure, etc), you can use the search language to count and look for anomalies... like this:

search: password failed | stats count by user | search count > 20

"Password failed" - grabs anything with that text in it
"stats count by user" turns the results in to a table with two columns, user and count.
"search count>20" shows me any user that fails login more than 20 times (for example)

You might "get all ninja'd" and do something with the src_ip, or port, or whatever else. Also, you could figure out who's staying logged in the most, by finding the login event, the logout event, using the Transaction command to link them together, and graph on the "duration" field. (now that sounds video-worthy)

Best...

Michael Wilde
Splunk Ninja.

[Ben Corbett]

Hi Michael,

I initially installed Splunk 4 but realised that it was the enterprise version with a time limit so therefore opted for the free version. I'm certainly looking forward to being able to upgrade to version 4 once the free release comes out! The new features certainly look impressive and from your comment I can certainly appreciate the potential power of Splunk. All very exciting!

Cheers

Ben