It’s common to cache logs captured by Syslog-NG to disk, especially when you have an agent, such as a Splunk UF, that reads logs from disk. Over time, these logs can accumulate and consume a significant amount of disk space. In this post, we will explore one way to automatically delete old logs.

This isn’t the only way to delete old logs. There are other methods for log rotation and deletion, such as using logrotate. This method is simple to implement and works well.

Prerequisites

Before we get started, you will need to have Syslog-NG installed on your system. You will also need to have a basic understanding of Syslog-NG configuration.

Take a look at the posts titled “How to Install Syslog-NG on Linux - Ubuntu and Debian” and “How to Make a Syslog-NG Config with Examples” for help with this.

Once you have Syslog-NG installed, make sure it’s configured in a way that allows you to delete old files easily. This means that we need an easy way to identify the files we want to delete. We talked about this further in the How to Make a Syslog-NG Config with Examples post, but the basic idea is to make sure that log files are created with the date and time in the filename. Here is an example destination config for Syslog-NG that accomplishes this:

destination d_cisco_asa {
   file("/syslog-data/cisco-asa/$HOST/$HOST-$YEAR$MONTH$DAY$HOUR.log");
};

Every new hour, a new log file will be created. The older log files won’t be modified once the hour has pasted. This method would work for most use cases. You may need to look into limiting the size of the log files if you have a high volume of logs.

Tip It is safer to cache logs on a dedicated drive. This helps to ensure that the system is able to boot and open SSH sessions without issue even if the logs fill their drive.

Once setup, your directory should look something like this:

/syslog-data/cisco-asa/
├── asa1
│   ├── asa1-2024022100.log
│   ├── asa1-2024022101.log
│   ├── asa1-2024022102.log
│   ├── asa1-2024022103.log
│   ├── asa1-2024022104.log
│   ├── asa1-2024022105.log

Deleting Old Logs

Warning Make sure you test the commands in a safe environment before running them in production. Deleting files can be dangerous if you are not careful.

Once the logs are splitting out by the hour, we can use a simple cron job to delete old logs. Here is an example cron job that deletes logs that are older than 3-days. You can expand the timeframe if you need a longer retention period (just make sure you have enough disk space).

4 */4 * * * find /syslog_data -type f -name "*.log" -mtime +3 -exec rm {} \;

Lets break down the cron job:

  • 4 */4 * * * - This is the schedule for the cron job. This cron job will run every 4 hours at the 4th minute. You can adjust this to your needs.
  • find /syslog_data -type f -name "*.log" -mtime +3 -exec rm {} \;
    • find /syslog_data - This is the directory where the logs are stored. You will need to change this to match your setup.
    • -type f - This tells find to only look for files.
    • -name "*.log" - This tells find to only look for files that end in .log.
    • -mtime +3 - This tells find to only look for files that are older than 3 days. Keep in mind that -mtime is the number of whole days, not 24-hour periods.
    • -exec rm {} \; - This tells find to delete the files that match the above criteria. The {} is a placeholder for the files that match the criteria.

We can setup the cron job by running crontab -e and adding the line to the file. You need to make sure that the user running the cron job has the correct permissions to delete the files. Adding the job to the root user is a simple way to make sure the job has the correct permissions.

Clearing Out Empty Directories

Warning Make sure you test the commands in a safe environment before running them in production. Deleting files can be dangerous if you are not careful.

Occasionally, you may run into empty directories that are left behind after the logs are deleted. This is common in larger environments where the logs are split out by host. You can use the following cron job to clean up these directories:

15 */4 * * * find /syslog_data -type d -empty -exec rmdir {} \;

Lets break down the cron job:

  • 15 */4 * * * - This is the schedule for the cron job. This cron job will run every 4 hours on the 15th minute. You can adjust this to your needs.
  • find /syslog_data -type d -empty -exec rmdir {} \;
    • find /syslog_data - This is the directory where the logs are stored. You will need to change this to match your setup.
    • -type d - This tells find to only look for directories.
    • -empty - This tells find to only look for directories that are empty.
    • -exec rmdir {} \; - This tells find to delete the directories that match the above criteria. The {} is a placeholder for the directories that match the criteria.

Testing the Cron Jobs

There are three ways you can test the cron jobs:

  • Wait for the cron job to run. This is the easiest way to test the cron job, but it can take a while.
  • Manually run the cron job. You can manually run the cron job by running the command in the cron job. This is a good way to test the cron job before you add it to the crontab.
  • Check the cron log. You can check the cron log to see if the cron job ran. The cron log is usually located at /var/log/cron or /var/log/syslog.

Conclusion

In this post, we explored one way to automatically delete old logs when using Syslog-NG. This method is simple to implement and works well. There are other methods for log rotation and deletion, such as using logrotate. You can use this method as a starting point and expand on it to fit your needs. Here are a few things to keep in mind:

  • Disk Space: Make sure you have enough disk space to store the logs for the retention time you need.
  • Permissions: Make sure the user running the cron job has the correct permissions to delete the files.
  • Testing: Make sure you test the commands in a safe environment before running them in production. Deleting files can be dangerous if you are not careful.