Learning, learning, learning . . . Our Splunk "expert" is gone, and the non-programmer gets to learn the task! How do you spell SPLUNK?
The dojo of Splunk. Learn, share, teach, mentor.
Videos
Latest Activity
Michael Wilde commented on Michael Wilde's video
Sure... When you do group mapping, map them to groups that don't have the domain admins in them. I have a separate OU=Groups that has "Splunk Users, Splunk Admins, Splunk Power Users" as group names, and specific users…
Mike Hartford commented on Michael Wilde's video
I want to give LDAP access to my splunk servcie but I don't want the LDAP users to have admin capabilitys in Splunk. Can I keep the domain admins out of Splunk if I have LDAP authentication???
Mike Hartford left a comment for Jonathan Hawes
Helow Jonathan,
Glad to have another Splunker. I've been useing Splunk for 2 years and am hooked. I leared how to spell splunk and | transaction too. you'll learn that one soon.
Go over to Splunk…
Mike Hartford commented on Mike Hartford's blog post 'tees for the holy day'
Holy Batskins Ninja, zzzzzwap zgruppp kapow a hidden stash, how great is that!!!!
The team that found them must have special bat senses and highly tooned Splunking skills
I like to wear Extra Lovable…
Handling Inputs - Blacklisting
While we all love Splunk's ability to just recursively eat entire directories of files in real time, there are some cases where you want to give splunk a "whoooa nelly.. not so fast"
For example: I've decided i want to monitor the /var/log directory on my Mac laptop, but i really don't need those old rolled over GZipped logfiles.
Normally, you just add an entry to the $SPLUNK_HOME/etc/system/local/inputs.conf to eat a directory, like this:
[monitor:///mnt/logs]
disabled=false
In my case, i'd like to have that monitor cruise through my directories and eat everything but files that have the ".gz" extension. To do that, just add a "_blacklist" entry in that stanza that contains the regular expression (regex) that matches the files you want. In our case, my inputs.conf will now look like this:
[monitor:///var/log]
_blacklist = \.gz$
Of course if i had multiple extensions i wanted to block, we could add an "or" regex, like this:
_blacklist = \.(txt|gz|tgz|bz2)$
Note: You can also use the syntax "_whitelist" which will disallow everything BUT what matches the whitelist's regex.
More info is over here on Splunk's docs as well
For example: I've decided i want to monitor the /var/log directory on my Mac laptop, but i really don't need those old rolled over GZipped logfiles.
Normally, you just add an entry to the $SPLUNK_HOME/etc/system/local/inputs.conf to eat a directory, like this:
[monitor:///mnt/logs]
disabled=false
In my case, i'd like to have that monitor cruise through my directories and eat everything but files that have the ".gz" extension. To do that, just add a "_blacklist" entry in that stanza that contains the regular expression (regex) that matches the files you want. In our case, my inputs.conf will now look like this:
[monitor:///var/log]
_blacklist = \.gz$
Of course if i had multiple extensions i wanted to block, we could add an "or" regex, like this:
_blacklist = \.(txt|gz|tgz|bz2)$
Note: You can also use the syntax "_whitelist" which will disallow everything BUT what matches the whitelist's regex.
More info is over here on Splunk's docs as well
Views: 98
Replies to This Discussion
-
Permalink Reply by Rob Jahn on -
I have a related requirement and I struggling with ....
#1. Here are series of logs in same folder but on multiple servers. I would like each file to have a unique sourcetype.
- from one server. note two sets folder names with A and B in the name
/apps/esrapp1/wls10/user_projects/domains/esrsDomainA_prd/drm.log <-- have sourcetype = esrs_drm
/apps/esrapp1/wls10/user_projects/domains/esrsDomainA_prd/esrs-portal.log <-- have sourcetype = esrs-portal.log
/apps/esrapp1/wls10/user_projects/domains/esrsDomainB_prd/drm.log <-- have sourcetype = esrs_drm
/apps/esrapp1/wls10/user_projects/domains/esrsDomainB_prd/esrs-portal.log <-- have sourcetype = esrs-portal.log
Tried this but the only the first log gets indexed
inputs.conf
[monitor:///apps/esrapp1/wls10/user_projects/domains/.../drm.log]
disabled = false
sourcetype = esrs_drm
index = esrs
[monitor:///apps/esrapp1/wls10/user_projects/domains/.../esrs-portal.log]
disabled = false
sourcetype = esrs_portal
index = esrs
Also tried this, but it did not work either
inputs.conf
[monitor:///apps/esrapp1/wls10/user_projects/domains/.../*.log]
disabled = false
index = esrs2
props.conf
[source::.../drm.log]
sourcetype = esrs2_drm
[source::.../esrs-portal.log]
sourcetype = esrs2_portal
#2 series of files that can all be the same sourcetype but both folder and file names vary
- from one server there are 4 files in folders like this.
- folder and file names with A and B in the name and the server number e.g. 03,04,05, etc.
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm02ms_1/esrsA_core_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm02ms_1/esrsA_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm02ms_1/esrsB_core_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm02ms_1/esrsB_vm02ms_1.log
- pattern repeats on other servers, but the number changes by server e.g. 03,04,05, etc.
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm03ms_1/esrsA_core_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm03ms_1/esrsA_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm03ms_1/esrsB_core_vm02ms_1.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm03ms_1/esrsB_vm02ms_1.log
Tried things like this but never got to work
[monitor:///apps/esrapp1/logs/.../esrs_*ms.log]
_whitelist = esrs_(core_vm\d\d|vm\d\d)ms_1.log
disabled = false
sourcetype = esrs2_weblogic
index = esrs2
#3 simular to #2, but include a file (access.log) from one subfolder but not the other
- from one server there are 4 files in folders like this.
- folder names with A and B in the name and the server number e.g. 03,04,05, etc.
- all files names are access.log
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm02ms_1/access.log <-- skip this one
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm02ms_1/access.log <-- keep this one
/apps/esrapp1/logs/esrsDomainB_prd/esrsB_core_vm02ms_1/access.log <-- skip this one
/apps/esrapp1/logs/esrsDomainB_prd/esrsB_vm02ms_1/access.log <-- keep this one
- pattern repeats on other servers, but the number changes by server e.g. 03,04,05, etc.
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_core_vm03ms_1/access.log <-- skip this one
/apps/esrapp1/logs/esrsDomainA_prd/esrsA_vm03ms_1/access.log <-- keep this one
/apps/esrapp1/logs/esrsDomainB_prd/esrsB_core_vm03ms_1/access.log <-- skip this one
/apps/esrapp1/logs/esrsDomainB_prd/esrsB_vm03ms_1/access.log <-- keep this one
This help file indicates that you can't combine file and directory wildcarding and if one monitor with a wildcard finds a folder match then it will ignore subsequent monitor ones that find the same folder.
http://www.splunk.com/base/Documentation/latest/Admin/Specifyinputp...
I've experimented with a variety of settings in inputs.conf and props.conf but still not getting what I want. Any suggestions would be appreciated.
© 2012 Created by Michael Wilde.
