The dojo of Splunk. Learn, share, teach, mentor.
Videos
Latest Activity
Michael Wilde replied to Nikita's discussion Count failures and success via transaction"How are these transactions linked together... by a field called "ID"? If so.. just build them with the field ID, and then use one of the MV commands to extract a field with success or failure in it. Paste some samples and…"
The Search Cheatsheet (or) Field Conversion with Splunk
I'm working on a challenge with some "sendmail_syslog" data. Those are the logs generated by a sendmail mailer daemon. The log format looks like this:
Aug 23 11:42:59 splunk3 sendmail[1394]: n7NIgqtH001374: to=spamme@splunkit.com,
delay=00:00:04, xdelay=00:00:00, mailer=local, pri=30405, dsn=2.0.0, stat=Sent
When you index this type of data with Splunk it reads it just fine (as it does all text data). Conveniently, the search-time field extraction magic also takes those "key=value" pairs and turns them in to fields automatically, so you can do reports, filtering, etc -- like this:
If you notice that field listing, its really just showing a top values in the result set, of messages that have the string "delay=xx:xx:xx". Humans look at this as "hours, minutes, seconds". Splunk (in the current version), sees it as a string and doesn't automatically convert this to time, or something we can work with.
Splunk has a mountain of search operators you can apply to search results that allow you to do math, conversions, statistics, etc. They all appear by taking search results and "piping" them to other operators -- we'll do that in a second.
The Splunk Search Cheatsheet helped me figure out how to convert those "delay/xdelay" fields in to a value (seconds) that I could then work with.
In my case, I would like to track down which recipients are experiencing the longest delays (to me, its anything over 50 hours) -- here's the solution, step by step:
1. Search for data: -- sourcetype="sendmail_syslog" delay="*"
2. Convert the delay field's duration to seconds. -- | convert dur2sec(delay)
3. Now that its a value, convert seconds to hours -- | eval delay = delay / 60 /60
4. Since i'm looking for anything over 50 hours -- | where delay>50
5. Use "stats" with max operator so we can list by recipient, their delay -- | stats max(delay) as "Delay (hours)"by recipient (note: when i type max(delay) as "Delay (hours)" its just renaming it so it looks nice.
6. Sort my result set descending by delay | sort -"Delay (hours)"
The whole search command looks like this when done:
sourcetype="sendmail_syslog" delay="*" | convert dur2sec(delay) | eval delay = delay / 60 /60 | where delay>50 | stats max(delay) as "Delay (hours)" by recipient | sort -"Delay (hours)"
Next step--create some reports! Maybe "showing delay by mailer". Give it a shot.
Aug 23 11:42:59 splunk3 sendmail[1394]: n7NIgqtH001374: to=spamme@splunkit.com,
delay=00:00:04, xdelay=00:00:00, mailer=local, pri=30405, dsn=2.0.0, stat=Sent
When you index this type of data with Splunk it reads it just fine (as it does all text data). Conveniently, the search-time field extraction magic also takes those "key=value" pairs and turns them in to fields automatically, so you can do reports, filtering, etc -- like this:
If you notice that field listing, its really just showing a top values in the result set, of messages that have the string "delay=xx:xx:xx". Humans look at this as "hours, minutes, seconds". Splunk (in the current version), sees it as a string and doesn't automatically convert this to time, or something we can work with.
Splunk has a mountain of search operators you can apply to search results that allow you to do math, conversions, statistics, etc. They all appear by taking search results and "piping" them to other operators -- we'll do that in a second.
The Splunk Search Cheatsheet helped me figure out how to convert those "delay/xdelay" fields in to a value (seconds) that I could then work with.
In my case, I would like to track down which recipients are experiencing the longest delays (to me, its anything over 50 hours) -- here's the solution, step by step:
1. Search for data: -- sourcetype="sendmail_syslog" delay="*"
2. Convert the delay field's duration to seconds. -- | convert dur2sec(delay)
3. Now that its a value, convert seconds to hours -- | eval delay = delay / 60 /60
4. Since i'm looking for anything over 50 hours -- | where delay>50
5. Use "stats" with max operator so we can list by recipient, their delay -- | stats max(delay) as "Delay (hours)"by recipient (note: when i type max(delay) as "Delay (hours)" its just renaming it so it looks nice.
6. Sort my result set descending by delay | sort -"Delay (hours)"
The whole search command looks like this when done:
sourcetype="sendmail_syslog" delay="*" | convert dur2sec(delay) | eval delay = delay / 60 /60 | where delay>50 | stats max(delay) as "Delay (hours)" by recipient | sort -"Delay (hours)"
Next step--create some reports! Maybe "showing delay by mailer". Give it a shot.
Views: 377
Comment
© 2012 Created by Michael Wilde.
You need to be a member of splunkninja to add comments!
Join splunkninja