Leveraging Splunk MCP and AI for enhanced IT operations and security investigations

How to use Splunk software for this use case 

This guide walks you through a number of different stages to get started:

1. Check prerequisites.
2. Follow configuration guidance.
3. Follow troubleshooting guidance if you encounter issues connecting to the MCP server.

After you've completed configuration and setup, you can complete these use cases for IT and security teams:

• Automating alert investigations by integrating LLMs with the Splunk platform and Confluence shows you how an LLM can use the Splunk platform and Confluence to automate incident response for IT operations.
• Leveraging LLM reasoning and ML capabilities for alert investigations shows you how an LLM can correlate alerts, enrich data, and apply machine learning models to security investigations.

Prerequisites

• Access to a Splunk Cloud Platform environment in commercial AWS regions.
• AI/LLM orchestration or a workspace software or service which acts as an MCP client, and some familiarity with configuring the connection to an MCP server. There are many free, open source, and commercial third-party software options available. The client must support the Streamable HTTP protocol.

Configuration

Follow the guidance in Configure the server and Connect and use an MCP client to set up your MCP server.

Troubleshooting

If you encounter issues connecting to the MCP server, check the following common problems.

► Click here for full instructions.

• Configuration nuances: Getting the configuration right can be a process of trial and error, as the MCP protocol is still new, and different MCP clients and operating systems may have unique requirements. Be prepared to adjust settings based on your specific environment.
• Missing dependencies: MCP clients like Claude Desktop have additional dependencies, like Node.js and mcp-remote, that need to be installed and present in the system path.
• Connection errors: Confirm that the user account you generated the token for has the mcp_user role assigned. Check that the bearer token was created with the audience set exactly to mcp. You can use tools like Postman or the MCP Inspector to test basic connectivity.
  • Error code: “400 tenant in host does not match tenant in path” indicates that the API endpoint is not correctly configured in your client. Check the stack name and trailing slash are present.
• SAML authentication issues: If you are using SAML, ensure your administrator has correctly mapped the mcp_user role to your SAML group as described in Configuration for Splunk administrators Step 4.
• Invalid token errors: If a tool call fails with a 401 Unauthorized error after a session has been established, your Splunk token might have expired or been revoked. You will need to generate a new token and re-configure your client.
• Splunk query error: Ensure the SPL query conforms to proper syntax. If you are running a saved search, check the permissions of the user running the requested saved search.
• MCP remote command auth: If you need to reset any authentication that might be saved for any of your MCP servers, you can always reset the connections. You can do this by running rm -rf ~/.mcp-auth in your machine terminal, then reauthenticate with your auth method.

Next steps

When your AI client is connected to the MCP server, you can action these use cases:

• Automating alert investigations by integrating LLMs with the Splunk platform and Confluence: Learn how to unify disparate tools, transforming multi-step manual IT investigations into automated, conversational workflows.
• Leveraging LLM reasoning and ML capabilities for Jira alert investigations: Learn how LLMs can correlate security alerts, create enriched datasets, and apply ML models. Utilize advanced ML for investigations, improving the impact of ML models through better data association and enabling the detection of anomalies across various data streams.