Leveraging Splunk MCP and AI for enhanced IT operations and security investigations

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

The rise of agentic AI systems and Large Language Models (LLMs) has created a powerful new way to interact with complex data. However, securely and efficiently connecting these AI tools to data in the Splunk platform can be a challenge.

The Splunk Model Context Protocol (MCP) server for Splunk Cloud Platform can help resolve this challenge. It's a fully-managed service that acts as a secure bridge, allowing AI agents and applications to interact with your Splunk environment using a standardized protocol. This enables you to query your data, get information about your environment, and automate tasks using natural language, without needing to be an expert in Splunk Search Processing Language (SPL).

The MCP server exposes core Splunk functionality as "tools" that your AI agent can call. The foundational tools available are:

run_splunk_query: Run a query and return the results. Use this to retrieve log data, perform aggregations, or search indexed data.
get_splunk_info: Get basic information about your Splunk instance, such as version and server name.
get_indexes: Get a list of the Splunk indexes that you have permission to access.
get_index_info: Get detailed information about a specific Splunk index.
get_metadata: Retrieve metadata (hosts, sources, or source types) for a given index to assist with query building.
get_user_info: Get information about the current, authenticated user.
get_knowledge_objects: Retrieve a list of knowledge objects of a specified type, such as saved searches, data models, macros, and more.

This guide walks you through a number of different stages to get started:

Check prerequisites.
Configuration for Splunk administrators shows you how admins can enable token authentication and create a specific role for MCP users.
Configuration for AI client users shows you how to create an access token and configure your MCP-compatible client to connect your AI client to the MCP server.

After you've completed configuration and setup, you can complete these use cases for IT and security teams:

Automating alert investigations by integrating LLMs with the Splunk platform and Confluence shows you how an LLM can use the Splunk platform and Confluence to automate incident response for IT operations.
Leveraging LLM reasoning and ML capabilities for alert investigations shows you how an LLM can correlate alerts, enrich data, and apply machine learning models to security investigations.

Prerequisites

Access to a Splunk Cloud Platform environment in commercial AWS regions.
AI/LLM orchestration or a workspace software or service which acts as an MCP client, and some familiarity with configuring the connection to an MCP server. There are many free, open source, and commercial third-party software options available. The client must support the Streamable HTTP protocol.

Configuration for Splunk administrators

Splunk administrators must first enable the environment to accept connections from the MCP server. This involves enabling token authentication and creating a specific role for MCP users.

► Click here for full instructions.

Before proceeding, please review the official documentation on how to set up token authentication to understand potential impacts to the identify and access management settings in your Splunk environment.

Step 1: Enable token authentication

The MCP server authenticates to your Splunk instance using JSON Web Tokens (JWT), also known as “bearer tokens”. You must ensure that token authentication is enabled on your Splunk Cloud Platform instance. For instructions, refer to Enable or disable token authentication in Splunk Docs.

Step 2: Create the mcp_user role

You need to create a dedicated role that will be assigned to all users who require access to the MCP server.

In Splunk Web, navigate to Settings > Roles.
Click New Role.
For the Role Name, enter exactly mcp_user.

For this initial controlled availability release, this role should be created with no capabilities. The MCP server enforces access control by passing the user's token directly to Splunk APIs, ensuring the user can only perform actions they are already authorized to do.
Click Save.

Step 3: Assign users to the mcp_user role

Assign the newly created mcp_user role to any user who needs to connect an AI client to Splunk via the MCP server. For more detailed instructions, refer to Create authentication tokens in Splunk Docs.

Step 4 (optional): Configure for SAML authentication

If you use SAML for authentication, you must map the mcp_user role to the appropriate SAML group to ensure users authenticated via SAML can use the MCP server.

Navigate to Settings > Authentication Methods.
Click SAML Settings.
Select Configure Splunk to use SAML.
In the Role alias section, add the mcp_user role to the necessary SAML groups.

For a future general availability (GA) release, Splunk admins will be able to grant MCP access with a capability, allowing for more flexible authorization models and better compatibility with existing security groups and external directory services. This guide will be updated when that change is implemented.

Configuration for AI client users

After the administrator has configured the Splunk environment, you can create an access token and configure your MCP-compatible client.

► Click here for full instructions.

Step 1: Generate a bearer token

You need a token to authenticate your requests.

In your Splunk user profile, navigate to the token generation section.
When creating your token, you must set the Audience field to mcp. Authentication to the MCP server will fail if this step is omitted.
Set an appropriate expiration date for your token.
Copy the generated token and store it securely.

Step 2: Identify your MCP server endpoint

The MCP server endpoint follows a standard format based on your Splunk Cloud Services (SCS) tenant name, which is likely the same as your stack URL prefix. The URL template is: https://<scs_tenant>.api.scs.splunk.com/<scs_tenant>/mcp/v1/ , including the trailing slash.

Replace <scs_tenant> with your specific tenant name. For example, if your Splunk Cloud Platform stack is available at https://mycompany.splunkcloud.com, then your API endpoint will be:

https://mycompany.api.scs.splunk.com/mycompany/mcp/v1/

Step 3: Configure your MCP client.

In your MCP client, you will need to configure the following:

MCP Endpoint: The URL from Step 2.
Authorization Header: The client must send the bearer token in an authorization header with every request.
- Example Header: Authorization: Bearer <your_generated_token>

Your client is now ready to make calls to the Splunk MCP server.

Troubleshooting

If you encounter issues connecting to the MCP server, check the following common problems.

► Click here for full instructions.

Configuration nuances: Getting the configuration right can be a process of trial and error, as the MCP protocol is still new, and different MCP clients and operating systems may have unique requirements. Be prepared to adjust settings based on your specific environment.
Missing dependencies: MCP clients like Claude Desktop have additional dependencies, like Node.js and mcp-remote, that need to be installed and present in the system path.
Connection errors: Confirm that the user account you generated the token for has the mcp_user role assigned. Check that the bearer token was created with the audience set exactly to mcp. You can use tools like Postman or the MCP Inspector to test basic connectivity.
- Error code: “400 tenant in host does not match tenant in path” indicates that the API endpoint is not correctly configured in your client. Check the stack name and trailing slash are present.
SAML authentication issues: If you are using SAML, ensure your administrator has correctly mapped the mcp_user role to your SAML group as described in Configuration for Splunk administrators Step 4.
Invalid token errors: If a tool call fails with a 401 Unauthorized error after a session has been established, your Splunk token might have expired or been revoked. You will need to generate a new token and re-configure your client.
Splunk query error: Ensure the SPL query conforms to proper syntax. If you are running a saved search, check the permissions of the user running the requested saved search.
MCP remote command auth: If you need to reset any authentication that might be saved for any of your MCP servers, you can always reset the connections. You can do this by running rm -rf ~/.mcp-auth in your machine terminal, then reauthenticate with your auth method.

Use cases

When your AI client is connected to the MCP server, you can action these use cases:

Automating alert investigations by integrating LLMs with the Splunk platform and Confluence: Learn how to unify disparate tools, transforming multi-step manual IT investigations into automated, conversational workflows.
Leveraging LLM reasoning and ML capabilities for Jira alert investigations: Learn how LLMs can correlate security alerts, create enriched datasets, and apply ML models. Utilize advanced ML for investigations, improving the impact of ML models through better data association and enabling the detection of anomalies across various data streams.