HDD full issue for indexing

[Andi Susanto]

Hi,

I wanna ask about indexing.

For Example, if I have 10 GB HDD, and have Splunk 500 MB license; I set the max free space for Splunk to stop indexing when the free space of HDD is 2 GB (2000 MB) -- set from Manager - System Settings. 

If one day, the free HDD space is 2GB, Splunk will pause to index.

For my scenario,
Let say the administrator never monitor the space for unknown reasons... :)
How about the new data? they may not indexed by Splunk and return to device, and because the device has limited space for the logs, the logs may be flushed. and the administrator will lost the logs from the splunk index either from the device.

How to set Splunk to automatically delete the old data? say about automatically delete data above 30 days...

Please guide me how to set this?

thanks for the inputs from all you, guys :)

[Michael Wilde]

Retention Policy in splunk can be set on a per index basis, determined by the age of the data, or the size of the index (or i believe a combination of both).

Most users store their data in the default index, known as "main"

To change the retention time of the [main] index, create a local indexes.conf file and set the "frozenTimePeriodInSecs" value to whatever you need.

For example. in $SPLUNK_HOME/etc/system/local/indexes.conf to make the retention policy 30 days (86400 seconds in a day multiplied by 30 days = 2592000 seconds)

[main]
frozenTimePeriodInSecs = 2592000

Lets say you didn't care about the age of the data, but you wanted to set the maximum size of the index (because you have disk space issue), you can do that from Manager-->Indexes, select your index, change the maximum size. But heck, we're already in the config files at this point, so lets just tell Splunk to make the main index no larger than 7GB. There's an attribute we can add to indexes.conf for our [main] index, conveniently known as "maxTotalDataSizeMB". Set that to the number of megabytes you would like the index to be, in our case it would be 7000 MB (7GB). While there is some controversy in computer science whether 1GB is 1000MB or 1024MB. Splunk knows 1GB as 1000MB. I have actually looked in to this (in case you were wondering). Anyway.. back to the config.

[main]
maxTotalDataSizeMB = 7000

We could combine both and say, the index should never grow more than 7GB and no data should ever be older than 30 days. Adding both attributes would accomplish that.

[Andi Susanto]

Thanks for your explanation...

but i have one more question :)

1. when the HDD full, Splunk will stop indexing; How to tell Splunk to index data to another place (for example, to other PC in network or maybe to another HDD partition) ?.

2. if i set frozenTimePeriodInSecs = 2592000 ; means that splunk will automatically delete indexed data that older than 30 days, right?

3. let say that i've set frozenTimePeriodInSecs = 2592000, and maxTotalDataSizeMB = 7000;
in 16 days, the HDD full, and maxTotalDataSizeMB have reached 7000 MB. Splunk will stop indexing. When I search in Splunk, and the data not yet indexed by Splunk (because the HDD full), where Splunk search the data? direct from the source (directly from the logs on device), or only search the old data in index db?

Thanks again...

[Michael Wilde]

1. A single Splunk index sits in a directory path. If you wanted to move Splunk's entire data store (All indexes), to a different Hard drive. Stop Splunk. Move the $SPLUNK_HOME/var/lib/splunk directory to another location. Edit the $SPLUNK_HOME/etc/splunk-launch.conf and change the SPLUNK_DB setting to your new directory. Start Splunk and you should be back in business.

2. Yes, it will automatically roll off (delete) data older than your frozenTimePeriodInSecs. You can, if you desire, setup a script to archive your old data prior to deletion.

3. Splunk ONLY searches whats in its index -- never what is on the filesystem. Indexing is the process of reading, parsing, extracting metadata and storing the raw copies (compressed) of your original data.

[Andi Susanto]

Hi, thanks for responses

for number 1 question, can Splunk set to "automatically" index data to another place when the default storage full? example, in normal condition, Splunk sets to index at C:\Program Files\Splunk\Database\mydb ; when drive C full, it automatically index to mirror place in D:\SplunkSecondStorage\mydb


thanks?

[Michael Wilde]

No. In that case, you might want to consider using a forwarder using splunk's AutoLB (auto load balancing), to send to randomly available Splunk servers and use distributed search (a Licensed feature)

[Andi Susanto]

Oh, i see...it clear now...but i think mirroring will be a good feature too in Splunk :)

thanks.

[Michael Wilde]

Mirroring the database while its being written to wouldn't make sense, however you can have events cloned to backup Splunk server at index time. That has worked for three years.