Handling Inputs - Blacklisting

[Michael Wilde]

While we all love Splunk's ability to just recursively eat entire directories of files in real time, there are some cases where you want to give splunk a "whoooa nelly.. not so fast"

For example: I've decided i want to monitor the /var/log directory on my Mac laptop, but i really don't need those old rolled over GZipped logfiles.

Normally, you just add an entry to the $SPLUNK_HOME/etc/system/local/inputs.conf to eat a directory, like this:

[monitor:///mnt/logs]
disabled=false

In my case, i'd like to have that monitor cruise through my directories and eat everything but files that have the ".gz" extension. To do that, just add a "_blacklist" entry in that stanza that contains the regular expression (regex) that matches the files you want. In our case, my inputs.conf will now look like this:


[monitor:///var/log]
_blacklist = \.gz$

Of course if i had multiple extensions i wanted to block, we could add an "or" regex, like this:
_blacklist = \.(txt|gz|tgz|bz2)$

Note: You can also use the syntax "_whitelist" which will disallow everything BUT what matches the whitelist's regex.

More info is over here on Splunk's docs as well

[Rob Jahn]

I have a related requirement and I struggling with ....

#1. Here are series of logs in same folder but on multiple servers. I would like each file to have a unique sourcetype.

- from one server. note two sets folder names with A and B in the name

/apps/esrapp1/wls10/user_projects/domains/esrsDomainA_prd/drm.log <-- have sourcetype = esrs_drm
/apps/esrapp1/wls10/user_projects/domains/esrsDomainA_prd/esrs-portal.log <-- have sourcetype = esrs-portal.log

/apps/esrapp1/wls10/user_projects/domains/esrsDomainB_prd/drm.log <-- have sourcetype = esrs_drm
/apps/esrapp1/wls10/user_projects/domains/esrsDomainB_prd/esrs-portal.log <-- have sourcetype = esrs-portal.log


Tried this but the only the first log gets indexed

inputs.conf
[monitor:///apps/esrapp1/wls10/user_projects/domains/.../drm.log]
disabled = false
sourcetype = esrs_drm
index = esrs

[monitor:///apps/esrapp1/wls10/user_projects/domains/.../esrs-portal.log]
disabled = false
sourcetype = esrs_portal
index = esrs

Also tried this, but it did not work either

inputs.conf
[monitor:///apps/esrapp1/wls10/user_projects/domains/.../*.log]
disabled = false
index = esrs2

props.conf
[source::.../drm.log]
sourcetype = esrs2_drm

[source::.../esrs-portal.log]
sourcetype = esrs2_portal


#2 series of files that can all be the same sourcetype but both folder and file names vary

- from one server there are 4 files in folders like this.
- folder and file names with A and B in the name and the server number e.g. 03,04,05, etc.

/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm02ms_1/esrsA_core_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm02ms_1/esrsA_vm02ms_1.log

/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm02ms_1/esrsB_core_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm02ms_1/esrsB_vm02ms_1.log

- pattern repeats on other servers, but the number changes by server e.g. 03,04,05, etc.

/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm03ms_1/esrsA_core_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm03ms_1/esrsA_vm02ms_1.log

/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm03ms_1/esrsB_core_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm03ms_1/esrsB_vm02ms_1.log


Tried things like this but never got to work

[monitor:///apps/esrapp1/logs/.../esrs_*ms.log]
_whitelist = esrs_(core_vm\d\d|vm\d\d)ms_1.log
disabled = false
sourcetype = esrs2_weblogic
index = esrs2



#3 simular to #2, but include a file (access.log) from one subfolder but not the other

- from one server there are 4 files in folders like this.
- folder names with A and B in the name and the server number e.g. 03,04,05, etc.
- all files names are access.log

/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm02ms_1/access.log <-- skip this one
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm02ms_1/access.log <-- keep this one

/apps/esrapp1/logs/esrsDomainB_prd/esrsB_core_vm02ms_1/access.log <-- skip this one
/apps/esrapp1/logs/esrsDomainB_prd/esrsB_vm02ms_1/access.log <-- keep this one


- pattern repeats on other servers, but the number changes by server e.g. 03,04,05, etc.

/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm03ms_1/access.log <-- skip this one
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm03ms_1/access.log <-- keep this one

/apps/esrapp1/logs/esrsDomainB_prd/esrsB_core_vm03ms_1/access.log <-- skip this one
/apps/esrapp1/logs/esrsDomainB_prd/esrsB_vm03ms_1/access.log <-- keep this one



This help file indicates that you can't combine file and directory wildcarding and if one monitor with a wildcard finds a folder match then it will ignore subsequent monitor ones that find the same folder.
http://www.splunk.com/base/Documentation/latest/Admin/Specifyinputp...

I've experimented with a variety of settings in inputs.conf and props.conf but still not getting what I want. Any suggestions would be appreciated.