Addressing Missing Index Error in Splunk

You might have run into an error like this in Splunk:

Received event for unconfigured/disabled/deleted index=<index> with source="<source>" host="<host>" sourcetype="<sourcetype>". So far received events from 1 missing index(es).

This error is trying to tell you that you have incoming data destined for an index that doesn’t exist. This would be due to the index being not created or disabled. By default, Splunk will delete logs destined for an index that doesn’t exist. You can prevent the logs from being deleted by creating a “lastchance” index.

The Main Idea

The lastchance index is similar to the main index that ships with Splunk. The main index is the default index that Splunk uses to store data. If you don’t specify an index in your inputs.conf, Splunk by default will send the data to the main index. Here is an example of an inputs.conf file that could cause data to be sent to a the main index:

[monitor:///var/log]
disabled = false

If you try to send logs to an index that doesn’t exist (intentionally or not), Splunk will delete the logs by default and pop a message similar to the one above. To prevent this from happening, you can create a lastchance index. With the index configured, instead of deleting the logs, Splunk will send the logs to the lastchance index.

Keep in mind that the lastchance index is not a “fix”. We simply configure it so that we don’t lose logs.

Creating the lastchance Index

First, we will start by creating the lastchance index. This index doesn’t specifically have to be named lastchance, but it is a common name used for this purpose. To create the index, we can add the following config to the indexes.conf file (add additional index configurations as you need):

[lastchance]
homePath = $SPLUNK_DB/lastchance/db
coldPath = $SPLUNK_DB/lastchance/colddb
thawedPath = $SPLUNK_DB/lastchance/thaweddb

From there, we need to update the [default] stanza in the indexes.conf file to point to the lastchance index.

[default]
lastChanceIndex = lastchance

With both of these pieces of configuration in place, we can now restart Splunk.

Testing the lastchance Index

We can test the lastchance index by trying to collect a sample event into an index that doesn’t exist.

| makeresults
| eval message="A shadowy flight into the dangerous world of an index that does not exist."
| eval _time=now()
| collect index=doesnotexist

If you run the above search, you should see the event in the lastchance index.

Conclusion

The lastchance index is a simple way to prevent logs from being deleted when they are destined for an index that doesn’t exist. This is not a fix, but a way to prevent data loss. You should occasionally check the lastchance index or create an alert to notify you when data is being sent to the index so that you can take the appropriate action.