Why Base Searches?

Even when constructing a simple dashboard, you might have multiple panels that independently run their own searches within the dashboard. Splunk executes these searches separately, retrieves the results, and then displays the visualizations. This process could potentially cause the dashboard to take longer to load, as each panel completes its search individually. If these panels are based on the same initial data, utilizing a base search becomes advantageous. With a base search, the search runs once when the dashboard loads, passing its results to the panels. The panels then carry out post-processing before presenting the visualizations.

This guide details the process of transforming a dashboard to leverage a base search. The focus here is on the “Classic” XML dashboard, rather than the new Dashboard Studio. Although a similar approach can be taken in the Dashboard Studio, the process differs slightly. I’m planning a separate guide for the Dashboard Studio setup, but for now, you can consult this Splunk Doc that discusses “chain searches.”

Why Classic over Dashboard Studio? I still like the look of the Classic dashboards better than the new Dashboard Studio. I still occasionally build on the Dashboard Studio when I want to leverage specific features, but there were initially some features missing from Dashboard Studio that the Classic had, so I found it hard to switch. I’m definitely slowing making the switch over.

Original Dashboard

Presented here is a simple dashboard showcasing the event count by log_level in the _internal index over time. The dashboard includes a time picker with the token global_time and a filter allowing users to select desired log_levels. I’ll be referencing XML snippets from this point, but the complete XML for both the base search and non-base search versions of the dashboard can be found at the end of this guide.

In this super simple example, both the “Log Level” filter and the line chart run their own independent searches to populate the results. Here are their corresponding searches:

  • “Log Level” Filter: index=_internal | stats c by log_level
  • Line Chart: index=_internal log_level IN ($log_level$) | timechart c by log_level

As you can see, both searches are processing the same index=_internal base search. Take note that the line chart does do some additional filtering on the index=_internal, which we will need to account for later. From here, we can implement a base search into the dashboard so that the index=_internal search only runs once and the results are passed to the filter and line chart from there to perform their independent post-processing. You can continue to add panels that leverage the base search. Here are a few things to note:

  • The same dashboard can contain panels that use and don’t use the base search. Its not a global setting for the dashboard, so you have flexibility.
  • You can have multiple base searches if you need to work with different data sets.

So lets move on to converting this dashboard to use base searches.

Creating the Base Search

Let’s click Edit on the dashboard and go to the Source tab. This page is showing the raw XML used to generate the dashboard’s UI. It should look like this:

Bare XML

From here, we will add the base search below the <label> tag. Give your base search a unique ID (Ex. my_internal_base_search). In this example, I also have the base search using the global_time time picker that I previously created in the dashboard.

<form version="1.1">
  <label>My Cool Dashboard with Base Search</label>
  <search id="my_internal_base_search">
    <query>(index=_internal) | fields *</query>
    <earliest>$global_time.earliest$</earliest>
    <latest>$global_time.latest$</latest>
  </search>
  ...

📝 Important Note: Base searches have to be a transforming search. An easy way to ensure your base search is producing all the fields required for your panels is to use | fields * at the end of the search, which is what I’m doing here in this example. Simply doing index=_internal would cause issues and would like not work here. Keep in mind that | fields * is a bit of a brute-force way to do this, so if you filter the returns down to just the fields you want, it may make the base search and subsequent querys using the base search load faster.

Converting the Log Level Filter to Use Base Search

Now we can convert our filter to leverage the base search. Here is what our log_level filter currently looks like:

	...
    <input type="multiselect" token="log_level" searchWhenChanged="true">
      <label>Log Level</label>
      <choice value="*">All</choice>
      <valuePrefix>"</valuePrefix>
      <valueSuffix>"</valueSuffix>
      <delimiter> , </delimiter>
      <fieldForLabel>log_level</fieldForLabel>
      <fieldForValue>log_level</fieldForValue>
      <search>
        <query>index=_internal
| stats c by log_level</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    ...

First we will update the <search> tag to reference the base search we just made using the ID we chose.

<search base="my_internal_base_search">

Now Splunk will tell us that the earliest and latest tags are not needed for this element. So we will remove those lines:

Earliest and Latest

Next, lets update the query to account for the base search. Since the base search is already doing the index=_internal search for us, we don’t need that here. So the <query> tag will go from this:

<query>index=_internal
| stats c by log_level</query>

To this:

<query>| stats c by log_level</query>

This is what the updated log_level filter’s XML looks like:

	...
    <input type="multiselect" token="log_level" searchWhenChanged="true">
      <label>Log Level</label>
      <choice value="*">All</choice>
      <valuePrefix>"</valuePrefix>
      <valueSuffix>"</valueSuffix>
      <delimiter> , </delimiter>
      <fieldForLabel>log_level</fieldForLabel>
      <fieldForValue>log_level</fieldForValue>
      <search base="my_internal_base_search">
        <query>| stats c by log_level</query>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    ...

That’s it! The Log Level filter is now using the base search. You can verify by clicking back to the UI tab and ensuring the that filter is loading. Once you ensure the update is working as expected, don’t forget to click Save.

Converting the Line Chart to Use Base Search

Converting the line chart is a similar process, but we will need to add an additional | search line to the line chart’s query so that we can account for the Log Level filter from before. Here is the starting XML for the line chart:

	...
    <panel>
      <chart>
        <search>
          <query>index=_internal log_level IN ($log_level$)
| timechart c by log_level</query>
          <earliest>$global_time.earliest$</earliest>
          <latest>$global_time.latest$</latest>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
      </chart>
    </panel>
    ...

Lets start by adding the base search ID and removing the earliest and latest tags like before.

Remember that the base search is running index=_internal, but this chart needs to also filter on the log_level from before. So to do this, we will add a | search with the filter to the beginning of the query. This is what the final XML should look like:

    ...
    <panel>
      <chart>
        <search base="my_internal_base_search">
          <query>| search log_level IN ($log_level$)
| timechart c by log_level</query>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
      </chart>
    </panel>
    ...

Once again, you can verify in the UI tab and click Save.

Conclusion

There you have it! Base searches are a good way to speed up your dashboards. I encourage you to test it out in some simple dashboards prior to implementing. Below is a template, raw XML from my example dashboard, and additional references.

Happy building! ⚒

Raw XML

Base Search Dashboard Template

<form version="1.1">
  <label>My Cool Dashboard with Base Search</label>
  <search id="my_base_search">
    <query><!--Your search--> | fields *</query>
    <earliest>$global_time.earliest$</earliest>
    <latest>$global_time.latest$</latest>
  </search>
  <fieldset submitButton="false">
    <input type="time" token="global_time" searchWhenChanged="true">
      <label>Time Picker</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="log_level" searchWhenChanged="true">
      <label>Log Level</label>
      <choice value="*">All</choice>
      <valuePrefix>"</valuePrefix>
      <valueSuffix>"</valueSuffix>
      <delimiter> , </delimiter>
      <fieldForLabel>log_level</fieldForLabel>
      <fieldForValue>log_level</fieldForValue>
      <search base="my_internal_base_search">
        <query>| stats c by log_level</query>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search base="my_internal_base_search">
          <query>| search log_level IN ($log_level$)
| timechart c by log_level</query>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
      </chart>
    </panel>
  </row>
</form>

<form version="1.1">
  <label>My Cool Dashboard</label>
  <fieldset submitButton="false">
    <input type="time" token="global_time" searchWhenChanged="true">
      <label>Time Picker</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="log_level" searchWhenChanged="true">
      <label>Log Level</label>
      <choice value="*">All</choice>
      <valuePrefix>"</valuePrefix>
      <valueSuffix>"</valueSuffix>
      <delimiter> , </delimiter>
      <fieldForLabel>log_level</fieldForLabel>
      <fieldForValue>log_level</fieldForValue>
      <search>
        <query>index=_internal
| stats c by log_level</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search>
          <query>index=_internal log_level IN ($log_level$)
| timechart c by log_level</query>
          <earliest>$global_time.earliest$</earliest>
          <latest>$global_time.latest$</latest>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
      </chart>
    </panel>
  </row>
</form>

<form version="1.1">
  <label>My Cool Dashboard with Base Search</label>
  <search id="my_internal_base_search">
    <query>(index=_internal) | fields *</query>
    <earliest>$global_time.earliest$</earliest>
    <latest>$global_time.latest$</latest>
  </search>
  <fieldset submitButton="false">
    <input type="time" token="global_time" searchWhenChanged="true">
      <label>Time Picker</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="log_level" searchWhenChanged="true">
      <label>Log Level</label>
      <choice value="*">All</choice>
      <valuePrefix>"</valuePrefix>
      <valueSuffix>"</valueSuffix>
      <delimiter> , </delimiter>
      <fieldForLabel>log_level</fieldForLabel>
      <fieldForValue>log_level</fieldForValue>
      <search base="my_internal_base_search">
        <query>| stats c by log_level</query>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search base="my_internal_base_search">
          <query>| search log_level IN ($log_level$)
| timechart c by log_level</query>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
      </chart>
    </panel>
  </row>
</form>

Additional Resources and References