Linux Universal Forwarder Silent Install

This guide will walk you through the process of installing a Linux Splunk Universal Forwarder (UF) with the aim of automating the process. Before we begin, here are some important notes:

  • Ensure that the host has wget.
  • If the host uses SELinux (getenforce to check), you may need to create policies to allow Splunk to function correctly (e.g., reading logs from /var/log).
  • Choose a user for the Splunk UF to run as (e.g., splunk or splunkfwd). As a best practice, avoid running Splunk as the root user to minimize the consequences of a service compromise. Note that running Splunk as a non-root user may require additional configurations, especially if SELinux is active, but it is the recommended approach.
  • Create a DNS record for the Deployment Server (DS) that clients will reach out to. This will make it easier to change the DS’s IP address in the future.

The Process

Switch to the root user or use sudo <command> throughout the guide. The commands will assume that you are running as root, unless explicitly instructed to switch to the Splunk UF user.

sudo su

Create a user for Splunk with no password. We will switch to this user later using the root user or another user with root privileges (su splunk).

adduser splunk --disabled-password

Check if the host has enough space for the UF installation.

df -h /tmp /opt/

Switch to the /tmp directory and download the Splunk UF Tarball. Get the updated tgz link from here.

cd /tmp/

wget -O splunkforwarder.tgz "https://download.splunk.com/products/universalforwarder/releases/9.1.0.1/linux/splunkforwarder-9.1.0.1-77f73c9edb85-Linux-x86_64.tgz"

Extract the package to the /opt/ directory, which is the standard practice.

tar -zxvf /tmp/splunkforwarder.tgz -C /opt/

Update the ownership of the directory to the Splunk user you created earlier.

chown -R splunk: /opt/splunkforwarder/

Optional: If your host is using ACLs, you can use the following commands to allow the Splunk user to read from the specified directories (if you are planning to read from them.) If your host is using SELinux, there are more hands-on steps you will need to take. Use getenforce to see if you are leveraging SELinux and do some research into the configurations/policies you will need to make. I advice against simply setting SELinux to permissive or disabling SELinux as a solution.

setfacl --recursive -m g:splunk:rx /var/log/

setfacl --recursive -d -m g:splunk:r /var/log/

setfacl --recursive -d -m g:splunk:r /home

Create an app to host our local deploymentclient.conf.

My thoughts: This is a bit more advanced topic, but I wanted to note it here so that you can do some research into it. This app will be local to the deployment client and will not be controlled by the DS. The app starts with ZZ_ to give the app a low precedence. This app will allow the deployment client to connect to the DS for the first time. From there, the DS can send out another app with the updated deploymentclient.conf (Ex. company_all_deploymentclient.) We do it this way so that if the company_all_deploymentclient is ever removed from the DS, the client will fall back to using the ZZ_local_deploymentclient config. Obviously the DNS hostname that the ZZ_local_deploymentclient might be outdated, but you have a chance to now use this old DNS hostname to point all the deployment clients to the new DS in the future before having to go through all clients and reconfigure them manually or through a automation platform. Its just somewhat of a failsafe. The second reason for this is that we want to avoid setting this configuration in $SPLUNK_HOME/system/local directory. If you do this, you won’t be able to update the deploymentclient.conf through an app controlled by the DS.

su - splunk -c 'mkdir -p /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/'

Create the deploymentclient.conf and update the hostname with your DS’s hostname. The management port is usually 8089, unless you changed it.

su - splunk -c 'echo -e "[target-broker:deploymentServer]\ntargetUri = splunk.bearlychilly.com:8089" > /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/deploymentclient.conf'

Create a simple app.conf.

su - splunk -c 'echo -e "# Deployment Client local app" > /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/app.conf'

Start Splunk.

My thoughts: This command will start Splunk, generate a random password, and then print it to screen. You can save the password, but you shouldn’t need it for anything unless you expect to troubleshoot this host in the future. You can always go through the password reset process using user-seed.conf to reset the password in the future. You can remove --no-prompt --gen-and-print-passwd if you want to manually set the password. But keep in mind that you will need to interactive with the command-line if you end up scripting this out. You can pass the password in using --passwd, but I would avoid this to prevent the password being in the audit logs. There are ways around this, but I’ll leave that up to you 🙂.

su splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt --gen-and-print-passwd"

Stop Splunk.

su - splunk -c '/opt/splunkforwarder/bin/splunk stop'

Configure Splunk to start at the boot of the host.

/opt/splunkforwarder/bin/splunk enable boot-start -user splunk

With new versions of the Splunk UF, you can now start the UF using SystemCTL.

systemctl start SplunkForwarder

Remove the UF package.

rm -f /tmp/splunkforwarder.tgz

Check to ensure Splunk is running as the user you specified.

ps -aux | grep -i "splunk"

The Splunk Universal Forwarder is now installed! You can check the Deployment Server to ensure that the client is checking in and start pushing apps to it. If you encounter networking issues, you can refer to my Basics of Network Connectivity Troubleshooting guide to kick-start the troubleshooting steps.

Happy Building! ⚒🙂

The Script

We can string together the commands we just ran using && to have a massive command to install Splunk. Compared to ;, && will only run the next command if the the previous command was successful. When doing this, you need to be aware of the differences between the hosts you are working on. For example, making sure you have wget installed or the script will fail. Although this method provides a basic way to automate the process, it’s not as robust as creating a bash script with better error handling and logging. To do that, you’ll need to make the script on the host, give it executable permissions, and then run it.

Here are the commands we ran:

adduser splunk --disabled-password
cd /tmp/
wget -O splunkforwarder.tgz "https://download.splunk.com/products/universalforwarder/releases/9.1.0.1/linux/splunkforwarder-9.1.0.1-77f73c9edb85-Linux-x86_64.tgz"
tar -zxvf /tmp/splunkforwarder.tgz -C /opt/
chown -R splunk: /opt/splunkforwarder/
su - splunk -c 'mkdir -p /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/'
su - splunk -c 'echo -e "[target-broker:deploymentServer]\ntargetUri = splunk.bearlychilly.com:8089" > /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/deploymentclient.conf'
su - splunk -c 'echo -e "# Deployment Client local app" > /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/app.conf'
su splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt --gen-and-print-passwd"
su - splunk -c '/opt/splunkforwarder/bin/splunk stop'
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk
systemctl start SplunkForwarder
rm -f /tmp/splunkforwarder.tgz
ps -aux | grep -i "splunk"

Here are the commands converted into a massive command concatenated using &&. Make sure you do your own testing. Please don’t just take this and run it. 😊 :

adduser splunk --disabled-password && cd /tmp/ && wget -O splunkforwarder.tgz "https://download.splunk.com/products/universalforwarder/releases/9.1.0.1/linux/splunkforwarder-9.1.0.1-77f73c9edb85-Linux-x86_64.tgz" && tar -zxvf /tmp/splunkforwarder.tgz -C /opt/ && chown -R splunk: /opt/splunkforwarder/ && su - splunk -c 'mkdir -p /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/' && su - splunk -c 'echo -e "[target-broker:deploymentServer]\ntargetUri = splunk.bearlychilly.com:8089" > /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/deploymentclient.conf' && su - splunk -c 'echo -e "# Deployment Client local app" > /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/app.conf' && su splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt --gen-and-print-passwd" && su - splunk -c '/opt/splunkforwarder/bin/splunk stop' && /opt/splunkforwarder/bin/splunk enable boot-start -user splunk && systemctl start SplunkForwarder && rm -f /tmp/splunkforwarder.tgz && ps -aux | grep -i "splunk"

This script has not been tested, but its an example of what this could look like if made into a bash script (There are a bunch of hard-coded values in this example script, like the hostname). There is some error handling added and some checks to see if there is space on the disk. You can use this as a starting point for building out your script. You will need to make the script executable and then run it on the host. Again, don’t just take this and run it.

#!/bin/bash

# Function to check if the script is being run as root. This is the assumption that we made during the first step
check_root() {
    if [[ $EUID -ne 0 ]]; then
        echo "This script must be run as root."
        exit 1
    fi
}

# Function to check if wget is installed
check_wget() {
    if ! command -v wget &> /dev/null; then
        echo "wget is not installed. Please install wget and try again."
        exit 1
    fi
}

# Function to check disk space
check_disk_space() {
    local directory=$1
    local required_space=$2
    local available_space=$(df -BG "$directory" | awk 'NR==2{print $4}' | tr -d 'G')
    if [[ $available_space -lt $required_space ]]; then
        echo "Not enough disk space in $directory. Required: ${required_space}GB, Available: ${available_space}GB"
        exit 1
    fi
}

# Check if the script is being run as root
check_root

# Check if wget is installed
check_wget

# Check if there is at least 5GB space in /tmp. You can update this.
check_disk_space "/tmp" 5

# Check if there is at least 20GB space in /opt You can update this.
check_disk_space "/opt" 20

# Add a new user named "splunk" with a disabled password. This can also be something like "splunkfwd" like we dicussed.
adduser splunk --disabled-password

# Change directory to /tmp/
cd /tmp/

# Download the Splunk Universal Forwarder release. Make sure you check for the latest version at splunk.com.
wget -O splunkforwarder.tgz "https://download.splunk.com/products/universalforwarder/releases/9.1.0.1/linux/splunkforwarder-9.1.0.1-77f73c9edb85-Linux-x86_64.tgz"

# Check if wget was successful in downloading the file
if [ $? -ne 0 ]; then
    echo "Failed to download the Splunk Universal Forwarder. Please check the URL or try again later."
    exit 1
fi

# Extract the downloaded tarball to /opt/
tar -zxvf /tmp/splunkforwarder.tgz -C /opt/

# Change ownership of the /opt/splunkforwarder/ directory to the splunk user
chown -R splunk: /opt/splunkforwarder/

# Create necessary directories and configuration files under the splunk user's home directory
su - splunk -c 'mkdir -p /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/'
su - splunk -c 'echo -e "[target-broker:deploymentServer]\ntargetUri = splunk.bearlychilly.com:8089" > /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/deploymentclient.conf'
su - splunk -c 'echo -e "# Deployment Client local app" > /opt/splunkforwarder/etc/apps/ZZ_local_deploymentclient/local/app.conf'

# Start Splunk for the first time and accept the license agreement
su splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt --gen-and-print-passwd"

# Check if Splunk start was successful
if [ $? -ne 0 ]; then
    echo "Failed to start Splunk Universal Forwarder. Please check the installation."
    exit 1
fi

# Stop Splunk to make necessary configurations
su - splunk -c '/opt/splunkforwarder/bin/splunk stop'

# Enable Splunk to start at boot using the "splunk" user
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk

# Start the Splunk Forwarder using systemctl
systemctl start SplunkForwarder

# Clean up by removing the downloaded tarball
rm -f /tmp/splunkforwarder.tgz

# Check running Splunk processes using grep
ps -aux | grep -i "splunk"