Using LogLogic and Splunk together

[Nathan Dornbrook]

Good morning, gentlemen.

 

I have a situation I think is relatively common in my industry.

 

There is a regulatory and legal monitoring challenge - we have to demonstrate that we are "in control" of our IT estate.

 

In order to do that, a previous set of IT consultants selected LogLogic to store the logs, a decision that I wholeheartedly support.

 

Now, before you start shouting at me, here's why this makes sense:  the estate is large (there are >50,000 servers) and heterogenous (there are 28 different platforms, including TANDEM, VMS, iSeries, pSeries, zSeries, Solaris, HP-UX, Linux and every flavour of Windows from NT 3.51 to Windows Server 2008).

 

The organisation itself has ~250,000 employees. There is still some NT 4.0 on the desktop, but most users are on XP.

 

All of this means that the number of different types of logs that are coming in at volume are beyond any demonstrable Splunk installation - and there's no Splunk connector that takes EBC-DIC. LogLogic has proven implementations that are this large and built in connectors for the iSeries (AS/400), VMS, z/OS and TANDEM platforms.

 

We're risk averse, so we won't use something that isn't already proven - and won't use something where the development risk hasn't already been taken by someone else.

 

But now we're talking about correlating the information within the LogLogic installation, and this is where we come upon the difficulty.

 

We already have an ArcSight implementation. ArcSight is too expensive. It would cost us £300 an endpoint. And the Oracle licenses we'd need in order to get it to scale across the entire estate make the proposition even less attractive.

 

LogLogic is already taking in the logs. Parsing is incomplete, but I'm less worried about that - because, once the logs are readable in ASCII, Splunk can do it.

 

We've taken raw logs and passed them through Splunk and are comfortable that Splunk provides a strong platform for future correlation engineering.

 

Politically, we won't displace the current log collection engine (LogLogic) but can probably get the investment to put Splunk in place as a correlation engine.

 

Does anyone on SplunkNinja have experience using Splunk to correlate logs that have already been aggregated by LogLogic?

 

Can LogLogic integrate northbound into Splunk? Can it spit out a Syslog feed after parsing the logs it collects?

 

- Nathan

[Michael Wilde]

In regards to if LogLogic can spit out syslog... I'll ask my buddy who works there.  

 

Splunk can eat EBC-DIC logs, you just have to tell it what the character set is.  Have you considered sending everything to splunk, and then spitting out a syslog feed to LogLogic or Arcsight?  I've seen a handful of customers do that.

 

 

[Nathan Dornbrook]

Hi, Michael!

 

Thanks for the update.

 

The LogLogic part of the equation is actually outside the remit of what I can mess with.

 

The organisation is big and has ended up fairly segmented.

 

The LogLogic implementation is considered part of a monitoring solution, with nothing to do with security. It's before my time, although I have a lot of respect for LogLogic as a solution.

 

The Splunk implementation, if we're able to convince management that it's feasible, will be purely a SIEM. I know it could do other things, but this is what we need.

 

The business case falls apart if we have to touch 50,000 boxes, so there's no danger that we'll actually send our logs directly to Splunk, whether its capable or not. It could be free and come with a gold plated Ferrari with a unicorn on the hood, but if you have to make a configuration change to the entire estate for it to work then it won't be worth the pain and effort.

 

- Nathan

[Michael Wilde]

Nathan.... (its been a while since i answered you, and something made me look at this question again) while "politically" you may not be able to displace Loglogic, the product is designed to scale far beyond that of LogLogic & Arcsight.  In fact, ALL FEATURES work in distributed mode, search, indexing, reporting, and even real-time analytics. So while i'm not going to shout at you for using LogLogic, i want to reassure you it scales like a boss.

 

The largest Social Gaming company that makes all its money on Facebook has a deployment designed for 5-7TB of logs a day--in the cloud.   Splunk's reference customers do far more daily indexing that LogLogic and Arcsight put together

 

...just sayin...

 

Did you figure out the syslog export from LogLogic?  Its easy from what i understand.

[Nathan Dornbrook]

Hey, Michael!

 

We've got both Splunk and LogLogic working on this and I think we've got a solution that works for everyone.

 

LogLogic can message forward the syslog, and that does the trick for us.

 

By the way, the issue isn't scaling to handle EPS or total volume of logs - it's both log management and archiving and having to make configuration changes across a wide estate.

 

LogLogic, in my experience, doesn't have the maturity of parsing to be able to easily correlate across many different log types. In fact, it struggles to parse syslog. But it has a very mature storage and archiving function.

 

Splunk is the Swiss army knife of correlation, IMHO, and it does a great job at correlating absolutely anything.

 

The downside is that it doesn't have particularly good compression, nor does it do a great job of managing and recovering messages from archives.

 

This means that you have to reach for third party tools - and that it isn't a great idea to hold your entire archive in Splunk.

 

Regulatory concerns have driven nearly all of my clients to hold log data for fifteen months. A large estate will generate petabytes of data in that time. LogLogic has built in capability to package up regular archives, hive it off to virtual tape libraries (or actual ones), index them, recover as needed and gets 10x compression.

 

Splunk gets 5x compression on easy to compress logs, 2x on hard to compress logs (like SMF) and doesn't have an easy way to create logical archives and hive them off. Instead, it relies on the underlying operating system functions and third party archiving utilities to provide that functionality.

 

That means that we'd need a system integrator to design a multi-part solution - which would then very likely be unique to us - that we would have to operate. In addition, I'd have to schedule changes to the logging across the entire estate. This was already done for LogLogic and, though it might be a "someday" task, there isn't much appetite for it today.

 

Ideally, Splunk will develop maturity in this space and I'll be able to sell it more frequently.

 

I've sourced, designed, built and implemented more SIEMs at more banks than I care to count (okay, it's four and I'm in the middle of number five, so I guess it's not that many) and Splunk is always the tool that I turn to when I need to work out the correlation rules.

 

After that, I've always used a different solution - Network Intelligence, ArcSight, Intellitactics - and for non-SIEM work have used other tools based on the requirement.

 

I appreciate you getting back to me.

 

I hope that the guys at Splunk pay attention to the community. I'd love to have a conversation about what I'd consider to be the perfect product for this space.