Splunk skipping input files & Order of precedence

[Dave P]

Hi,

So, I think I'm missing something obvious here. a 2 part question.

1) I have one of several inputs defined as:

monitor:///data/logs]
disabled = false
host_segment = 4
index = default
sourcetype = syslog

There are several directories under /data/logs/${DATE}. It appears that Splunk has "missed" them as files to index and add to the Splunk DB.

Is there a way to kick Splunk to index files that it appears to have missed? Permissions, ownerships are correct... I'm stumped.

2) Order of precedence.

From $SPLUNK/etc/apps/search/local/inputs.conf

monitor:///data/logs]
disabled = false
host_segment = 4
index = default
sourcetype = syslog

[monitor:///data/logs/*/abc*/maillog]
disabled = false
host_segment = 4
index = default
sourcetype = postfix_syslog

It doesn't appear that files that match the 2nd input are getting tagged with the correct sourcetype. /data/logs/2009.11.1/abc783/maillog is showing up in Splunk, but is sourcetyped as syslog, not postfix_syslog.

Does the order of inputs matter? Does it find the most specific match for a particular file and apply it, or does the first match it finds win?

I've looked over the docs for 4.0.5 for inputs and can't find any reference to this question.

Thanks!

[Mike Langhorst]

a bit old on this but since I had an answer, hopefully it'll help you or someone else with this issue.

Regarding question #2:
Correct, due to the first stanza, the second will not be used. What you can do is add an entry to specify this source type in props.conf similar to the following:

[source::/data/logs/*/abc*/maillog]
sourcetype = postfix_syslog

[Dave P]

I'm getting closer, I think.

I did what you suggested, created a separate entry for the maillog stuff in the props.conf, though it ended up looking like this:

[source::/data/logs/...]
sourcetype = syslog

[source::/data/logs/.../full.hostname.here/maillog]
sourcetype = postfix_syslog

Still no luck. The events are showing up in Splunk (can verify that by searching on the hostname, and I see events from the maillog), but still as sourcetype = syslog.

I'm beginning to wonder if order really does matter in the props.conf. IE, will the first line that matches always win, or is it the most specific entry that matches wins (like in a routing table)?

On a side note, apparently, you can't use a * in the middle of the path, so /data/logs/.../host*/maillog is not considered valid. Bummer, it *should* work. :)

[Alexander Szoenyi]

Hello,

Point 1

Go to the Search App -> Status -> Inputs Activity.
There you can find the "Most recently ignored files".

or use this search

index="_internal" source="*splunkd.log" earliest=-24h Component="fileclassifiermanager" Message="invalid file*" | fields Message

Point 2

How monitor works in Splunk ( see : http://www.splunk.com/base/Documentation/latest/admin/MonitorFilesA...)

Set the sourcetype for directories to Automatic. If the directory contains multiple files of different formats, do not set a value for the source type manually. Manually setting a source type forces a single source type for all files in that directory.

regards Alexander

[Dave P]

There are no entries in the "Most recently ignored files" search results for the past 24 hours (or even the past 72 hours). So, at least Splunk is finding everything, even if it is not sourcetyping it correctly.

I found the same documentation page you've got up there. What I can't figure out is if order in the props.conf file matters. I'm changing the order of my entries to see if it makes any difference.

Thanks!

[Dave P]

Aaaaand Splunk Support has come through again. Below is the solution that worked nicely, though it seems there should have been an easier way to do this.

in $SPLUNK/etc/apps/search/local/

inputs.conf:
[monitor::///data/logs/]
sourcetype=syslog

props.conf:
[source::...maillog]
TRANSFORM-setST=setST

transforms.conf:
[setST]
DEST_KEY = MetaData:Sourcetype
REGEX = (.)
FORMAT = sourcetype::postfix_syslog

Hope this helps other folks too.