Reformatting the message in an WinEventLog:Application event

[Atul Mistry]

I have a situation where I need to combine events from an older version of an application with a newer one, while both are live in production.

 

The newer version produces Windows Event log events in plain text, and everything works like a champ.

 

The older version produces Windows Event log events in XML, so the event looks like:

 

01/21/10 10:07:06 AM

LogName=Application

SourceName=APP-XXXXXXX

EventCode=0

EventType=1

Type=Error

ComputerName=XXXXXXXXXX

Category=0

CategoryString=none

RecordNumber=15920

Message=<Exception timestamp="01/21/2010 10:07:06" type="xxxxx.xxxxx.ExceptionUtilityValidationUnexpectedType" message="Object &quot;System.Int32&quot; does not support type &quot;xxxxxx.xxxxx.Xml.Type.IXmlType&quot;" source="xxxxxx.xxxxx" machineName="xxxxxxxxx" applicationDomainName="IsolatedAppDomainHost:xxxxxxxxxxxxxxx.dll" lastRethrowFrameIndex="0" lastRethrowFrameIndexAction=""><StackTrace><Frame exceptionType="xxxxxx.xxxxxx.ExceptionUtilityValidationUnexpectedType" declaringType="xxxxxx.xxxxxx.Validation" method="CheckExpectedType" fileName="xxxxxxxxxxxxxxx.cs" lineNumber="327" /></StackTrace></Exception>

 

I would rather not have the users performing searches be concerned with if the event contains xml or not in the message. I was wondering if there was a
way to reformat the Message to remove the XML as it’s consumed by splunk, so
the event will look something like the following when it’s indexed:

 

01/21/10 10:07:06 AM

LogName=Application

SourceName=APP-XXXXXXX

EventCode=0

EventType=1

Type=Error

ComputerName=XXXXXXXXXX

Category=0

CategoryString=none

RecordNumber=15920

Message=Exception type="xxxxx.xxxxx.ExceptionUtilityValidationUnexpectedType" message="Object "System.Int32" does not support type "xxxxxx.xxxxx.Xml.Type.IXmlType"" source="xxxxxx.xxxxx" machineName="xxxxxxxxx" applicationDomainName="IsolatedAppDomainHost:xxxxxxxxxxxxxxx.dll" lastRethrowFrameIndex="0" lastRethrowFrameIndexAction=""

         exceptionType="xxxxxx.xxxxxx.ExceptionUtilityValidationUnexpectedType" declaringType="xxxxxx.xxxxxx.Validation" method="CheckExpectedType" fileName="xxxxxxxxxxxxxxx.cs" lineNumber="327"

 

I was heading down the path of adding the following into transforms.conf:

 

[reformat_message]

REGEX = «some crazy regex that I haven’t figured out yet»

FORMAT = .

DEST_KEY = MetaData:Message (I’m not sure if that is correct)

 

Then updating the inputs.conf to the following:

 

[WinEventLog:Application]

TRANSFORMS-changemessage = reformat_message

 

But I’m concerned I’m heading down a black hole and may end of breaking something else.

 

Thanks,

[Michael Wilde]

If you want to completely reformat a message, you're going to want to have your DEST_KEY = _raw. Is piping to XMLKV not workable -- | xmlkv turns the XML elements in to fields.

[Atul Mistry]

i did start with xmlkv, but didn't get it to work. i assumed it had problems dealing with the mixed format in the event (Windows Event log info is in muliline keyword value pair, and the message is in xml), but it certainly could be just my lack of knowledge with the search tools.

one of the issues is that some of the xml messages are getting split by splunk into two events. this is probably due to the fact of using xml within a windows event log is not normal. i think using xmlkv becomes ineffective since events that are split apart will be missing fields; some fields in one event and the rest in another.


i figured if i could capture the event as it enters splunk and remote the message i could avoid down stream issues.

[Michael Wilde]

upload a sample of those events if you want... but what you will need to do is give Splunk some education on where to break your events.. most of the time its pretty smart, but I like that it lets me take over and tell it what to do when i want to.

You can add $SPLUNK_HOME/etc/system/local/props.conf:

Add a stanza for host, source, or sourcetype
[host::my_badass_server]
AUTO_LINEMERGE = false #.. uuh.. Splunk.. stop being intelligent, go dumb for a bit
SHOULD_LINEMERGE = true # hey splunk, now i'm telling you to definitely merge lines for multiline events
BREAK_ONLY_BEFORE = <?XML
MUST_NOT_BREAK_BEFORE = /XML> # and maybe something else here (not req'd)

stop splunk, clean it (with "splunk clean eventdata -index main"), restart and the multiline breaking problem should be solved.

We can do this..!

[Atul Mistry]

unfortunately the events i'm having a problem with have PHI (protected health information), and I can't post it with out some scrubbing. i'll try what you suggested tonight. i'm going to Splunk Live today! If I still can't get it to work, i'll post the scrubbed event.

Thanks for your help.

[Michael Wilde]

Atul....

Splunk has a scrubber command built in. Dump that stuff to a file, run "/splunk anonymize file -source /path/to/[filename]"

It usually does a pretty good job of getting rid of PHI and other private stuff.

http://www.splunk.com/base/Documentation/4.0.9/Admin/Anonymizedatas...

[Atul Mistry]

I found the reason the message was getting split, it was because it exceeds the 10000 default limit. I set the TRUNCATE = 0 and now the message stays together.

now that the messages are not getting split, xmlkv is working like a champ.

thanks for your help.

[Michael Wilde]

WIN!...