Automated Log Archiving with WinRAR and Robocopy

In my day job, one of the things I do is maintenance of a farm of servers. All these boxes have local logging, which is picked up by Splunk.  However, I still need to clean up those log files.  Here is how I automated the process.

First, in my humble opinion, unless a debug mode switch has been thrown, these logs should be going to the Windows application event log.  Only in debug mode should these log files be generated.  All they do is take up space, and make the production of process necessary.

In any event, since I am a paranoid person I don't like to throw things like this away until I am sure they are not needed.  So I've developed a process wherein the various log files are gathered from each server, compressed, and moved to a central storage repository.  They live in the repository for 30 days, then are removed.

Initially, this process could take me up to an hour to accomplish each day since I am working with a farm of ~30 servers.  Pretty simple process, just tedious.  Sounds like a prime candidate for automation.

Let's break down this batch file.  It's pretty straightforward, and I think I've commented it pretty liberally. It makes use of Robocopy and WinRAR to first move the files to a local temp/working directory, create a date-stamped RAR file, which then gets moved to a central repository for later disposition.

The Setup

This section is pretty straighforward.  We are setting up the run with some default values, such as:

• Repository - Where we are going to send the final product.  This can be a UNC path (\\MyServer\MyShare) or a drive\path (L:\MyArchive).
• PathToArchive - This is the path on the target machine where the files live you want to archive. This can be a sharename\path, or you can (like I do) use the admin share.  For example, if you want to archive C:\Logs on the remote machine, you would set this value to c$\logs.
• MyDrive - The local drive on the machine where the script is being run, so the we can later make sure we are running from the appropriate workspace
• MyPath - path on the local machine where the script lives, once again, we need to be in the right place.
• MyWorkingDir - This is a path below MyPath being used as a temp directory.  I added this in the event that multiple runs of this script are happening at the same time. For example, if I am archiving both web servers and app servers at 2am each day, if I use the same directory, the files will stomp on each other.  By specifying a temp directory per server type, I can run multiple instances of the script at the same time.
• DateStamp - Some string handling to get the date from the OS.  We'll use this later to build the filename for the archive.

@echo off

REM ***
REM *** Set some defaults
REM ***

REM Where the final resting place of the archive will be
Set Repository=PathToLogArchiveRepository

REM Path on target to be archived
Set PathToArchive=PathToBeArchived

REM Working Drive and Directories
Set MyDrive=C:
Set MyPath=\scripts
Set MyWorkingDir=%MyDrive%%MyPath%\work

REM ***
REM *** Get the datestamp for the file, based on the current date
REM *** in the format YYYYMMDD
REM ***

echo. | date | FIND "(mm" > NUL
If errorlevel 1,(call :Parsedate DD MM) Else,(call :Parsedate MM DD)
goto :ScriptDone

:Parsedate

REM ***
REM *** The date is available. Parse the output of DATE /T
REM ***

For /F "tokens=1-4 delims=/.- " %%A in ('date /T') do if %%D!==! (set %1=%%A&set %2=%%B&set YYYY=%%C) else (set DOW=%%A&set %1=%%B&set %2=%%C&set YYYY=%%D)
(Set DateStamp=%YYYY%%MM%%DD%)

Let's get some work done!

Nothing to amazing going on in this section.  First, we switch to the working drive, then ensure we are in the right directory.  From there, the ArchiveFiles subroutine is called for each server that we want to run the archiver against.

It's important to note that in the :ScriptDone section, the Exit /b command is used.  This is necessary to prevent the script from "falling through" and executing the subroutines below it.  By adding the /b switch, any errrorlevel codes are preserved.  I haven't used any errorlevel codes in this script, but I plan on implementing some in the future, which can be picked up by the monitoring system.

REM ***
REM *** Make sure we are where we need to be
REM ***

%MyDrive%
cd%MyPath%

REM ***
REM *** Now, let's get some work done. We'll execute the ArchiveFiles Function against each 
REM *** machine that we want to clean up.
REM ***

Call :ArchiveFiles MyServer01
Call :ArchiveFiles MyServer02
Call :ArchiveFiles MyServer03


:ScriptDone

REM ***
REM *** All done, we need to kill this program so we don't fall through to the subroutines below
REM ***

exit /b

The meat of the script

This is the section that really does the work.  Pretty simple, once you dig into it, but let's examine what is going on.

First, Robocopy is called to move the files from the target server (referenced here as %1). Only files older than one day are moved.  Why move the files to a local disk ? Good question.

When you perform a compression of a file or a large group of them, a lot of traffic is generated.  The archive is going back and forth between the source and the target quite a bit.  By moving the files to be compressed to a local drive, I hope to reduce the network traffic, and speed up the compression operation.

Once the files are relocated,WinRar is called on to create the compressed file.  After the file is created, the source files are deleted.

Finally, the compressed file is moved off to the repository.

:ArchiveFiles

REM ***
REM *** Using RoboCopy, *move* the files that are older than one day to a local folder for archiving
REM *** so as not to clog the network with extra traffic from the archiver.
REM ***

Echo *** Copying log files to work directory
robocopy \\%1\%PathToArchive% %MyWorkingDir% /s /e /minage:1 /MOV

REM ***
REM *** Now, use WinRar at the command line to compress the files in a date-encoded filename, deleting
REM *** the files after the compress operation is complete.
REM ***


echo *** Compressing logfiles to %DateStamp%-%1.rar
rar a -r -df -rr3 %DateStamp%-%1.rar %MyWorkingDir%\*.*

REM *** 
REM *** Move the archive file to the designated repository, then exit the subroutine.
REM ***

echo *** Moving archive to %Repository%
move %DateStamp%-%1.rar %Repository%

exit /b

About this post

Posted: 2014-08-15
By: dwirch
Viewed: 3,455 times

Categories

Tip

Tutorials

Scripting

Windows

Windows Server

Attachments

No attachments for this post

Comments

You must be logged in to make a comment.