Need Help with Automate Archiving

[Atul Mistry]

I'm testing out automatic archiving, but i can't seem to get it to work.

Here is what i'm doing:

• i added the following stanza to my etc\system\local\indexes.conf file

[main]

frozenTimePeriodInSecs = 3600 coldToFrozenScript = WindowsCompressedExport.bat %DIR%

• i placed the WindowsCompressedExport.bat file in C:\Program Files\Splunk\bin (i also put it in C:\Program Files\Splunk\bin\scripts and C:\Program Files\Splunk\etc\system\bin to cover all of the bases)
• the script i'm using is a version off the splunk wiki, and i included it below.
• after making the changes i restarted splunk

I'm not getting any backup files or error messages.

What am i missing?

i'm testing on splunk 4.0.8 build 73243 on Windows XP 32bit.

thanks,

------ script ----------

@echo off

:: Windows compressed archive bat file to use with indexes.conf coldToFrozenScript 

::

:: READ THIS!!!

:: Please modify dest_base to be your archive location, please do not

:: put a trailing '\' at end of path

::

set dest_base=c:\backups

 

::

:: EXAMPLE OF FOLLOWING VALUES BELOW:

:: source_path is: C:\Program Files\Splunk\var\lib\splunk\defaultdb\colddb\db_234234232_132342342_3

:: source_base is C:\Program Files\Splunk\var\lib\splunk\defaultdb\colddb\

:: source_leaf is db_234234232_132342342_3

::

set source_path=%1

set source_base=%~dp1

set source_leaf=%~nx1

set dest_final=%dest_base%\%source_leaf%

 

echo source_path is %source_path%

echo source_base is %source_base%

echo source_leaf is %source_leaf%

echo dest_final is %dest_final%

 

for %%i in (%1\*.tsidx) do splunk-compresstool.exe -M "%%i"

 

::

:: we have to manually create the destination directory and provide that

:: as a target to xcopy since it only is able to copy the source directory contents

:: and not the directory itself

::

mkdir %dest_final%

 

xcopy %1 %dest_final% /E /I /C /Y

[Atul Mistry]

i noticed a typo and made the following change.

[main]
frozenTimePeriodInSecs = 3600 coldToFrozenScript = WindowsCompressedExport.bat "$DIR"

and restarted splunk.

still no luck.

is the time period too short? do i need to set a tie period to move from hot->warm->cold?

[Michael Wilde]

Atul...

3600 seconds.. you really only want 1 hour worth of data in your indexer?. Whats splunkd.log saying.. check with this search:

index=_internal source="*splunkd.log"

[Atul Mistry]

i'm just using 1 hour for testing purposes. in production we will be using 45 days.

nothing in the splunkd.log jumps out at me. is there a component i should filter or focus in on?

i attached a 2 hour sampling of the splunkd.log.

[Atul Mistry]

i figured my time span was too short, so i reconfigured the archive settings to one day and hot span to 4 hours:

[main]
frozenTimePeriodInSecs=86400
maxHotSpanSecs=14400
coldToFrozenScript = WindowsCompressedExport.bat "$DIR"

Waited a day, and everything worked like a champ.

Thanks,