In this guide, we will walk you through the process of creating a Syslog-NG configuration that caches and organizes syslog data on disk. Additionally, we will delve deep into the structure of the config, examining each option we utilize. Finally, we will explore configuring “catch-all” entries.

But before we dive into the details, it’s essential to ensure that you have Syslog-NG installed. If you need assistance with the installation process, we recommend checking out our earlier post How to Set Up Syslog-NG on Linux with an Example Configuration for Ubuntu and Debian. Additionally, once Syslog-NG is up and running, it’s crucial to implement a log rotation script to prevent storage overflow. We’ve covered this in our post titled “How to Automatically Delete Old Logs When Using Syslog-NG”.

Overview

Our primary objective with Syslog-NG in this article is to capture and store logs efficiently, creating a cache that can be readily accessed by tools like Splunk. In this section, we will outline the process of creating the Syslog-NG configuration and discuss the key goals we aim to achieve. Keep in mind that your goal and be different and may require adjustments to the config we are creating in this article. Syslog-NG is infinitely customizable, so see this as a guide to get you started. Here are the primary goals for our configuration:

  • Listening for syslog data on UDP port 514.
  • Writing logs to disk for caching.
  • Organizing logs by host to streamline management.
  • Segmenting logs by date and time for enhanced data organization.
  • Implementing an automated process to delete old logs (for more information on this, check out our article on “How to Automatically Delete Old Logs When Using Syslog-NG”).

While building and troubleshooting your configuration, it’s advisable to refer to the Syslog-NG documentation.

Setting Up Directories

Note: While we won’t delve deeply into permissions in this article, it’s essential to be aware of the permissions required for the directories you’ll be creating. Depending on your system’s requirements, you may need to utilize tools like setfacl to configure proper permissions for the relevant applications or accounts.

Before we proceed with configuring Syslog-NG, the first step is to establish the directory where your cached logs will be stored. In our example, we’ll name this directory syslog-data for easy reference:

mkdir /syslog-data

This syslog-data directory will serve as the root location for storing the syslog data that Syslog-NG captures. Properly organizing your directories is essential for effective log management and analysis.

Initial Configuration Setup

Now, let’s dive into setting up the initial configuration for Syslog-NG.

Syslog-NG configurations are typically stored in the /etc/syslog-ng/ directory. The default configuration file that comes bundled with Syslog-NG is located at /etc/syslog-ng/syslog-ng.conf. It’s essential to exercise caution when making modifications to this default configuration, as changes here can impact your custom configurations. If you encounter any issues, it’s wise to inspect this file for potential conflicts or problems.

For our custom configuration, we will create a dedicated file that will reside in the /etc/syslog-ng/conf.d/ directory. This directory serves as a repository for custom configurations. To initiate the process, let’s create a file to house our custom configuration:

touch /etc/syslog-ng/conf.d/custom.conf

Any files within the /etc/syslog-ng/conf.d/ directory with the *.conf extension will automatically be incorporated as part of Syslog-NG’s configuration. This behavior is preconfigured in the /etc/syslog-ng/syslog-ng.conf file.

As you progress in building your custom configuration, remember that you can apply the changes by executing the following command:

syslog-ng-ctl reload

This command ensures that your Syslog-NG configuration updates are implemented without the need to restart the entire service.

Example Config

Here is a full example config you can use as a starting point. Additionally, below is a more detailed explanation of each section. The “Catch-All” config is address under its own section.

# =========================
# === SYSLOG-NG CONFIG ===
# =========================

# ======= OPTIONS =======
options {
   keep-hostname(yes);
   use-dns(no);
   create-dirs(yes);
   owner("splunk");
   dir-owner("splunk");
   group("splunk");
   dir-group("splunk");
   perm(0750);
   dir-perm(0755);
};

# ======= SOURCES =======
source s_udp_514 {
   network(transport("udp") port(514));
};

source s_tcp_514 {
   network(transport("tcp") port(514));
};

# ======= FILTERS =======
filter f_cisco_asa {
   host(192.168.0.2) or host(192.168.1.2);
};

# ======= DESTINATIONS =======
# Write Cisco ASA logs to disk
destination d_cisco_asa {
   file("/syslog-data/cisco-asa/$HOST/$HOST-$YEAR$MONTH$DAY$HOUR.log");
};

# ======= LOG RULE =======
log {
   source(s_udp_514);
   filter(f_cisco_asa);
   destination(d_cisco_asa);
   flags(final);
};

# =========================
# === CATCH-ALL CONFIG ===
# =========================

# ======= DESTINATIONS =======
# Catchall for TCP sources
destination d_catchall_tcp {
   file("/syslog-data/catchall_tcp/$HOST/$HOST-$YEAR$MONTH$DAY-$HOUR.log");
};

# Catchall for UDP sources
destination d_catchall_udp {
   file("/syslog-data/catchall_udp/$HOST/$HOST-$YEAR$MONTH$DAY-$HOUR.log");
};

# ======= LOG CATCH-ALL RULES =======
log {
   source(s_tcp_514);
   destination(d_catchall_tcp);
   flags(final);
};

log {
   source(s_udp_514);
   destination(d_catchall_udp);
   flags(final);
};

Syslog-NG Config Structure

As you start creating your Syslog-NG configuration, it’s crucial to understand the high-level structure that forms the foundation of every configuration. A Syslog-NG configuration is constructed using five primary building blocks: Options, Source, Filter, Destination, and Log.

Let’s look at each of these building blocks:

Options

Options encompass general configurations that can be fine-tuned to suit your needs. For instance, you can utilize options to specify the owner of a particular directory or define other system-specific settings.

Source

The source is the component that instructs Syslog-NG where to monitor for incoming data. A straightforward example involves monitoring a specific port over a specific protocol. It’s common practice to prefix source configurations with s_ for clarity and organization, such as s_udp_514.

Filter

Filters play a critical role in allowing or blocking data based on specific criteria or functions. These filters act as gatekeepers, deciding which data should be processed further and which should be discarded. A comprehensive list of available filter functions can be found in the Syslog-NG documentation. Common practice dictates prefixing filter configurations with f_, as in f_cisco_ise.

Destination

The destination entry directs Syslog-NG on where to send the processed data. This could involve actions like splitting data by host and writing it to disk or transmitting it to another host. Destinations are typically prefixed with d_, for instance, d_cisco_ise.

Log

The log entry serves as the glue that binds sources, filters, and destinations together. Within a log entry, you can specify that data from a specific source (e.g., port 514/UDP), matching specific filters (e.g., Cisco or Palo Alto filters), should be directed to a designated destination (e.g., a specific directory). Furthermore, you can incorporate additional flags and configurations within the log entry to fine-tune Syslog-NG’s behavior.

Example Config Deep-Dive

Now that you understand the general structure of the Syslog-NG config, lets dive deeper into the example config.

Options

The “Options” section of the Syslog-NG configuration file specifies various global settings and behaviors for the syslog service. These settings control how syslog-ng processes log messages and manages log files. Below are explanations for each option included in this section:

  • keep-hostname(yes): This option retains the hostname in log messages. When set to “yes,” it ensures that log messages include the originating host’s hostname. This can be valuable for identifying the source of log entries.

  • use-dns(no): Disabling DNS resolution with use-dns(no) can improve performance by preventing syslog-ng from performing DNS lookups for hostnames mentioned in log messages. If your logs do not require DNS resolution, this setting is recommended for enhanced efficiency.

  • create-dirs(yes): When set to “yes,” this option enables syslog-ng to create directories if they do not already exist when writing log files. It simplifies log file management by automatically generating necessary directories as needed.

  • owner("splunk"): Specifies the owner of log files created by syslog-ng. Replace “splunk” with the desired username if necessary (I use splunk here since I am usually working with Splunk when leveraging Syslog-NG). Ensure that the specified user has the appropriate permissions to manage log files.

  • dir-owner("splunk"): Similar to the owner option, this setting specifies the owner of directories used for log storage. It should match the user responsible for log file management in your environment.

  • group("splunk"): Defines the group associated with log files. Ensure that the specified group is correct for your setup. Modify it if needed to align with your organizational structure.

  • dir-group("splunk"): Similar to group, this setting determines the group ownership of directories related to log storage. Adjust it as necessary to meet your specific requirements.

  • perm(0750): Sets the permissions for log files. The value “0750” represents the file permission mode, ensuring that only the owner has full access, while the group has read and execute permissions. Modify this value according to your security policies.

  • dir-perm(0755): Specifies the permissions for log directories. The mode “0755” grants read, write, and execute permissions to the owner and read and execute permissions to others. Adjust this value based on your security and access control needs.

Review and adjust these options according to your organization’s logging requirements and security policies to ensure syslog-ng behaves as intended in your environment.

Sources

In the Syslog-NG configuration file, the “Sources” section defines the sources of log messages. These sources specify how syslog-ng should collect log data from various input channels. Below is an explanation of the source included in this section:

  • source s_udp_514: This source is named “s_udp_514.” It is configured to receive syslog messages over UDP on port 514. Port 514 is the default port for syslog communication. Messages arriving at this source will be processed further according to the configuration.

The “Sources” section is crucial as it determines where syslog-ng collects log data from. Additional sources can be added to handle logs from different inputs, such as network devices, applications, or system logs.

Filters

The “Filters” section in the Syslog-NG configuration file defines filters that are used to selectively process log messages based on specified criteria. Filters allow you to determine which log messages should be included in the syslog-ng processing pipeline. Below is an explanation of the filter included in this section:

  • filter f_cisco_asa: This filter is named “f_cisco_asa.” It is designed to match log messages originating from specific hosts with IP addresses 192.168.0.2 or 192.168.1.2. Log messages from these hosts will be selected for further processing.

Filters are essential for fine-tuning log message processing. They enable you to include or exclude log messages based on various attributes, such as source, content, or severity level. In this case, the filter is configured to match log messages from specific hosts.

Additional filters can be added to your configuration to cater to different filtering requirements, depending on your environment. Filters help ensure that only relevant log data is forwarded to the specified destinations.

Destinations

In the Syslog-NG configuration file, the “Destinations” section defines where log messages should be sent or stored once they have been processed and filtered. Destinations specify the output destinations for log data. Below is an explanation of the destination included in this section:

  • destination d_cisco_asa: This destination is named “d_cisco_asa.” It is configured to write log messages to a file system location. The path for log storage is defined as /syslog-data/cisco-asa/$HOST/$HOST-$YEAR$MONTH$DAY$HOUR.log.

Destinations play a crucial role in determining where log messages are directed for storage or further processing. In this case, the destination specifies that log messages matching the filter criteria will be written to files on the file system.

Additional destinations can be added to your configuration to manage log data differently based on your needs. Common destination types include file storage, remote log servers, and various message queuing systems.

When configuring destinations, consider factors such as log retention policies, storage capacity, and access controls. The destination in this section is set to store log messages in a directory structure that includes the hostname and a timestamp, which can aid in organizing and managing log files effectively.

Log

The “Log” section in the Syslog-NG configuration file is where you define the rules for processing and forwarding log messages. This section specifies the sources, filters, and destinations to be used in log message processing. Below is an explanation of the log rule included in this section:

  • log { source(s_udp_514); filter(f_cisco_asa); destination(d_cisco_asa); flags(final); };: This log rule combines the previously defined components to form a processing pipeline. It specifies the following:
    • source(s_udp_514): The source named “s_udp_514” is defined as the input source for log messages.
    • filter(f_cisco_asa): The filter named “f_cisco_asa” is applied to the incoming log messages. Messages that match the filter criteria will proceed to the next step.
    • destination(d_cisco_asa): The destination named “d_cisco_asa” is specified as the location where log messages matching the filter will be sent or stored.
    • flags(final): The “final” flag indicates that once a log message matches this rule and is processed, syslog-ng should stop further processing for that message. This is typically used for efficiency when you want to match and handle specific log entries uniquely.

The “Log” section is where you define the flow and logic for how log messages are processed and where they are directed. This log rule combines the source, filter, and destination components to create a complete processing rule.

You can create multiple log rules in this section to handle different types of log messages or to send them to various destinations based on specific criteria. Each log rule helps organize and manage log data according to your requirements.

Catch-All Configuration

In Syslog-NG, the “Catch-All” configuration is a powerful feature that ensures no log messages are lost, even if they don’t match specific filters or rules. It is particularly useful when you want to collect all log messages from various sources for auditing, troubleshooting, or compliance purposes.

Example Catch-All Config

Here is an example catch-all config you can add to your custom config.

# =========================
# === CATCH-ALL CONFIG ===
# =========================

# ======= DESTINATIONS =======
# Catchall for TCP sources
destination d_catchall_tcp {
   file("/syslog-data/catchall_tcp/$HOST/$HOST-$YEAR$MONTH$DAY-$HOUR.log");
};

# Catchall for UDP sources
destination d_catchall_udp {
   file("/syslog-data/catchall_udp/$HOST/$HOST-$YEAR$MONTH$DAY-$HOUR.log");
};

# ======= LOG CATCH-ALL RULES =======
log {
   source(s_tcp_514);
   destination(d_catchall_tcp);
   flags(final);
};

log {
   source(s_udp_514);
   destination(d_catchall_udp);
   flags(final);
};

Catch-All Destinations

Two distinct destinations are defined for the “Catch-All” configuration:

  • destination d_catchall_tcp: This destination is designed for TCP sources. It specifies that log messages received from TCP sources will be stored in separate log files on the file system. The path for log storage is defined as /syslog-data/catchall_tcp/$HOST/$HOST-$YEAR$MONTH$DAY-$HOUR.log. Each log file is organized by the source hostname and timestamp.
  • destination d_catchall_udp: Similarly, this destination is tailored for UDP sources. It dictates that log messages received from UDP sources will be stored in separate log files within the /syslog-data/catchall_udp directory, following a structure based on source hostname and timestamp like the other.

Log Rules for Catch-All

The “Catch-All” configuration includes two log rules, each targeting a specific type of source:

  • TCP Catch-All Rule: This log rule employs the source named “s_tcp_514” to collect log messages over TCP. It then directs these messages to the “d_catchall_tcp” destination for storage. The “flags(final);” attribute ensures that log messages matching this rule are processed and that syslog-ng stops further processing for efficiency.
  • UDP Catch-All Rule: Conversely, this log rule uses the source “s_udp_514” to collect log messages over UDP. It sends these messages to the “d_catchall_udp” destination for storage. Like the TCP rule, it employs the “flags(final);” attribute to stop processing for matching log messages.

Since these catch-all log entries are at the end of the config, any data that does not match any previous entries will be captured. When implementing the “Catch-All” configuration, be mindful of storage requirements, as it may generate a substantial volume of log data. Ensure that your log retention policies align with your organization’s needs and that you have adequate storage capacity.

Next Steps

References