Building Splunk Searches Using AI (GitHub Co-Pilot)

A lot of time spent writing Splunk searches goes into formatting, syntax, and remembering commands or patterns for your SPL. With the help of generative AI, specifically GitHub Co-Pilot, we can significantly reduce the time spent writing searches. It also makes it easier to iterate quickly on existing searches and serves as great notes for later.

I’ve edited this article about five times now, and finding the “perfect” balance of detail has been challenging. I’ll keep it simple. I think this approach will work for 90% of people starting with Co-Pilot and Splunk Searches. I think you should be able to identify more complex use cases as you get more comfortable with the tool.

Note: I’m sharing a method that has worked for me, but it’s not the only way to use Co-Pilot for Splunk Searches. I hope you find helpful bits that you can incorporate into your workflow.

The Solution

We will use Microsoft’s Visual Studio Code (VS Code) with the GitHub Co-Pilot extension to write our Splunk searches. This will also serve as a space to track the searches we are working on.

If you’ve never used Co-Pilot before, you might be wondering how it’s different from using ChatGPT (or similar LLMs) to generate SPL. The main difference is that Co-Pilot works inline with your code. It has context on the SPL you’ve already written and can suggest what might come next.

Here’s a simple example of a comment in Python (in green). Co-Pilot suggests the code that should come next (in grey).

How to Use Co-Pilot

To use Co-Pilot, you’ll need to install the GitHub Co-Pilot extension in VS Code. The extension will require you to sign in with your GitHub account. There is a free tier, but you’ll want to get a paid tier to remove some of the rate limits.

Once setup, create a new file named something like “splunk_searches.md”. The “.md” extension stands for “Markdown” files, a common format for documentation. I prefer taking all my notes in Markdown because it makes writing code blocks (for my SPL) easy and keeps things looking neat. I won’t go into all the details of Markdown here, but you can find a good reference here.

Before you can use Co-Pilot, you’ll need to click on the down arrow for the Co-Pilot icon next to the search bar on the top of the VS Code window. From there, go to Configure Code Completions... and enable completions.

Now, open your file, add the following content, and press Enter after failure and wait.

# Count of Failed Logins by User

```spl
index=main sourcetype=access_combined action=failure

You should see Co-Pilot generate some SPL that looks like this:

Note: The suggestion may take a second to appear. It may also be a bit different from what I have here.

While this is a simple example, we’ll explore more complex searches soon. Here are a couple of things to note:

• Be descriptive in your context. In this example, I explained what I’m trying to do in the title.
• You can press Tab to accept the full suggestion or Ctrl + Arrow Right to accept parts of it.

More Complex Searches

I work on all my related searches in the same note file. This way, Co-Pilot has context from my other searches, which helps prevent hallucinations.

Before diving into more complex searches, I like to pull in examples of searches I frequently write at the top of my note file. This gives Co-Pilot context on what I’m working on and how I typically structure my searches.

I’m sure you have some searches you can reuse, but I recommend limiting the examples to around three. You can expand on them as needed. More examples tend to slow down Co-Pilot. Also, aim to cover different types of searches, such as a basic index=, a rest search, and a tstats search.

If you don’t have examples handy, here are a few you can copy and paste. These aren’t necessarily the “best” way to do these searches, but they represent different SPL patterns I often use. When I start writing certain patterns (like stats sum(eval(...)))), Co-Pilot already understands what I’m trying to do and can help me complete it. In your examples section, try to aim for diversity in patterns and commands.

A few additional things to keep in mind:

• You don’t have to use my example searches—they’re just suggestions. Feel free to use your own.
• Try to include as many of the commands you frequently use in your example searches. This helps Co-Pilot understand your search style.
• Consider adding more complex commands, like transaction, join, append, appendpipe, and map (if you use them).
• The more context you add, the better. This helps reduce hallucinations, although suggestions might slow down. You can always remove some examples if performance becomes an issue.
• I keep a section for Common Lookups and Common Macros. It’s a good place to store all the lookups and macros you use frequently. Co-Pilot will have this context and be able to assist you better.
• I use “—” as a separator between my example searches and the searches I’m currently working on. It’s just a personal preference; feel free to use whatever works for you.

# Example Splunk Searches

## List Count of Scheduled Searches by App Using REST

```spl
| rest /servicesNS/-/-/saved/searches splunk_server=local
| fields title is_scheduled eai:acl.app disabled
| rename eai:acl.* AS *
| search disabled=0 is_scheduled=1
| stats dc(title) AS "Scheduled Searches" by app
| rename app AS "App"
```

## Authentication Failures and Successes by User in the Authentication Data Model

```spl
| tstats count summariesonly=true
    from datamodel=Authentication
    where Authentication.action IN ("failure", "success")
    by Authentication.user Authentication.action
| rename Authentication.* AS *
| stats sum(eval(if(action=="failure", count, 0))) AS "Failed Logins",
    sum(eval(if(action=="success", count, 0))) AS "Successful Logins"
    by user
| rename user AS "User"
```

## List of Windows Machines Restarting in the Last 24 Hours

> Note: This list of EventIDs was sourced from Preplexity.

- Event ID 41 indicates an unexpected restart without a proper shutdown.
- Event ID 1074 shows when an application or user initiates a restart.
- Event ID 6005 indicates system startup and event log service start.
- Event ID 6006 shows a clean shutdown.
- Event ID 12 records the exact system startup time.
- Event ID 13 logs when the operating system begins shutting down.

```spl
index=wineventlog earliest=-1d EventCode IN (41, 1074, 6005, 6006, 12, 13)
| fields host EventCode
| stats count as occurrences earliest(_time) as earliest latest(_time) as latest by host EventCode
| eval earliest=strftime(earliest, "%+"), latest=strftime(latest, "%+")
| eval "Event Description" = case(EventCode == 41, "Unexpected Restart",
    EventCode == 1074, "User Initiated Restart",
    EventCode == 6005, "System Startup",
    EventCode == 6006, "Clean Shutdown",
    EventCode == 12, "System Startup Time",
    EventCode == 13, "Operating System Shutdown")
| rename host AS "Host", EventCode AS "Event Code", occurrences AS "Occurrences", earliest AS "Earliest Time", latest AS "Latest Time"
```

## Common Lookups

> Note: Include any common lookups you use here, with field names for easy reference.

### Lookup: `user_lookup`

Fields: `user, email, department, manager`

## Common Macros

> Note: Include any common macros you use here, with an example of arguments populated.

### Macro: `get_user_info`

```spl
| inputlookup user_lookup
| `lookup(user)`
```

---

Things to Try

There is a ton of functionality in Co-Pilot that you can leverage. The in-line suggestions are just the beginning. To start with, we’ll go through some things you can try.

Here is an example search we can use as a starting point:

index=main sourcetype=access_combined action=failure
| stats count, earliest(_time) as firstTime, latest(_time) as lastTime by user action
| table user action count firstTime lastTime
| rename user AS "User"

Try the following:

• Highlight the search and press Ctrl - I. In the prompt, ask Co-Pilot to convert the field names into snake_case.
• Type a , and space after "User" and see what Co-Pilot suggests.
• Build a new search from scratch for “Count of Failed and Successful Logins by User” and see how Co-Pilot helps you.

Conclusion

As I mentioned before, the in-line suggestions are just the beginning. The workflow for using Co-Pilot to build Splunk searches differs from leveraging it for coding. It’s a great alternative to ChatGPT because it’s less likely to hallucinate, serves as a collaborator (as the name suggests), and doubles as a built-in note-taking tool. Since you’re keeping your searches organized, you also save time by not having to explain the context to ChatGPT every time. Happy building! 🙂