Timestamping is the bane of my existance

[nicholas Lehman]

I'm working on a quick fix for AS/400 logging and the ability to cleanly report it. I've got all the fields mapped out, but the timestamping gets retarded. I've tried feeding the logfile to splunk for timestamp recognition. Even tried a bit of regex (which I most likely biffed). Nothin'. Pulling my hair out here guys. Below is the first part of a log event Time stamp is "021110070911", or "MMDDYYhhmmss.Any ideas on getting proper timestamp information out of events?

002430000000658TPW021110070911QZSOSIGN QUSER 024306QZSOSIGN

[Michael Wilde]

Nicholas.. can we establish a pattern on what precedes the timestamp... like will it always be some number of digits, followed by a TPW, followed by your date pattern?

If so.. this should be easy..

[nicholas Lehman]

the information preceeding the timestamp will always be different. The beginning of the event log contains record length, sequence number, journal code and entry type. Even the information immediately before and after the timestamp will be different depending on the event type. However, the AS/400 logs have fixed field length, and the timestamp will always start at an offset of 19 characters. I attempted to enter it in as a regex and start 19 characters out, but it still comes back with no real rhyme or reason to the supposed timestamps that splunk picks.

[Michael Wilde]

This will work.

First, create a props.conf in an appropriate directory (system level or app level), I'll make mine in $SPLUNK_HOME/etc/system/local/props.conf

Add this to it:

#my sourcetype will be called "as400"
[as400]
#I want splunk to bypass the first 18 word (digit or letter) characters when looking for a timestamp
TIME_PREFIX = \w{18}
# i want splunk to turn off its intelligence on linemerging for multiline events (this is optional)
AUTO_LINEMERGE = FALSE
# i want to force splunk to not create multiline events (this is optional)
SHOULD_LINEMERGE = FALSE
# I shall apply the following strptime format.
TIME_FORMAT = %m%d%y%H%M%S

Next, when you index the file(s) lets say with Splunk's "monitor a directory" feature, hardcode the sourcetype as "as400"

My inputs.conf looks like this:

[monitor:///Users/michaelwilde/Logs/as400.log]
disabled = false
sourcetype = as400

And the result looks like this:

If your data is already indexed, and timestamping is incorrect, create the props.conf file, edit your inputs.conf (which is probably in $SPLUNK_HOME/etc/apps/search/local/inputs.conf -or- $SPLUNK_HOME/etc/apps/launcher/search/local/inputs.conf) Put the as400 sourcetype in there.

Then stop splunk, clean your index $SPLUNK_HOME/bin/splunk clean eventdata -index main. Then start Splunk again and it should index it properly. If not, check to see if its taking your new as400 sourcetype. If not, just blow away your inputs, start again. You'll get it.

[nicholas Lehman]

Awesome, it worked! One question though, for the monitor option in the inputs.conf, can I just leave the end of the path open ended to look in that directory for any new files? I see you put a file at the end, and am wondering if I'm a bit confused on the capabilities of the monitor directory funciton.

[Michael Wilde]

You are correct. Monitor a directory will work just fine. I user a single file to answer your challenge. Glad it worked for you!

[nicholas Lehman]

haha, I was just about to come back and let you know I answered my own question. Thanks for all the help Michael!

[Michael Wilde]

Question for you:

How'd you get the logs from the AS/400. Arent they in EBCDIC, if so did you convert them? Whats your strategy for AS/400. Please share. (might even be blog post worthy!)

[nicholas Lehman]

Well, this is a down and dirty compliance stop gap. The log journals get spat out to text file every night, and then a cron job runs a script to pull them off the as/400 via sftp along with an MD5 sum file. The script then performs an md5sum on the file that was pulled down and if the sums match, it throws the log file into the folder that Splunk is monitoring. I got a little creative with the script though, and if the as/400 can't be reached or the md5sums don't match it sends an alert e-mail to my phone. As for making the log info intelligible, I took a look at the IBM guide for reading logs and created custom fields for each section of a log event. It was very time consuming, and the regex I used (I'm positive) could have been better for it, but it works. The script I wrote was quite simple and if you want to have a look at it, I'll post it (minor editing done of course).

[Michael Wilde]

If your script is "post-able".. that'd be great... the "AS/400 question" comes up often and users would like to benefit from your experience.

[nicholas Lehman]

Well, if this is a common occurrence, I may want to look into developing an app for AS/400. In the meantime, I can post the script I made for my particular application. Simple bash (cause perl confuses me).

#!/bin/bash
cd ~/
echo accessing AS400
sftp as400user@as400box SFTPDUN
get auditdata
get md5.txt
quit
SFTPDUN

if [ -e ~/auditdata ]; then

filename=as400_$(date +%Y%m%d).log
newsum=`md5sum auditdata |tr ' ' '\n' |head -n 1`
oldsum=`cat md5.txt | tr ' ' '\n' |head -n 1`

if [ "$newsum" = "$oldsum" ]; then
chown as4log:splunk auditdata
cp auditdata /opt/splunk/as400_logs/$filename
cp auditdata ~/as400_logs/$filename
cp md5.txt ~/as400_logs/md5$(date +%Y%m%d)
rm auditdata
rm md5.txt
else
SUBJECT="Bad AS/400 Log File"
EMAIL="[e-mail]"
BODY="A check of the MD5 sum for AS/400 log file $filename, has returned a bad checksum"
echo $BODY | /bin/mail -s "$SUBJECT" "$EMAIL"
cp auditdata ~/as400_logs/$filename
cp md5.txt ~/as400_logs/md5$(date +%Y%m%d)
rm auditdata
rm md5.txt
fi
else
SUBJECT="Unable to access AS/400 for logs"
EMAIL="[e-mail]"
BODY="ALERT! The scheduled log retrieval for $(date +%m-%d-%Y) could not be retrieved. Please verify that access is available."
echo $BODY | /bin/mail -s "$SUBJECT" "$EMAIL"
fi