Building AI agents with Splunk MCP Server

Inputs

1. Objective: What do you need the agent for? The use cases are endless: security, data, ITOps, analysis, and many more.
2. LLM: Evaluate your system needs and requirements. Some options are PydanticAI, OpenAI, LLama Index, and LangChain. The main consideration is making sure that it can break down the tasks and look at what needs to be done, which also requires you to be able to write good prompts, but you should also consider the following:
   • Size versus capabilities versus use case performance
   • Context window versus API cost versus latency
   • Open-source versus proprietary
   • Customization approaches
   • Budget and number of tokens you have
3. Tools: What web applications, APIs, or MCP servers the LLM will use. Consider the following regarding tool calling:
   • Input robustness
   • Flexibility (scenario planning)
   • Security (safe code execution)
   • Discoverability
   • Observability
4. Type: Will it be a simple reflex agents that operates on trigger and flow? Or will it be a more advanced learning agent that tries to figure out a plan? The next section goes into this in more detail.

Agent control flow

Control flow is a matter of how much control logic you will give the agent. Or, in other words, how many of your environment will be human-driven versus agent-executed.

In the router version, shown on the left in the following diagram, the agent only has an option of step 2 or 3. However, in the fully autonomous version on the right, it decides what each step is going be.

The more autonomous the agent is, the more guardrails you want to establish. You don't want the agent to go through 100 steps unnecessarily because that's a lot of resource use. The perception module can use memory to decide to not to go through an entire loop again when it's the same issue that occurred last week. The feedback and learning loop can also enhance the agent's planning process.

Other guardrails to consider include:

• Human-in-the-loop
• Input/output validation
• Access controls
• Content filtering
• Rule-based threat protection
• Drift monitoring

Multi-agent architectures

Multi-agent architectures are what will truly empower autonomous networks, and there are many ways to architect multiple agents, but the most basic way is to consider the following two.

• Vertical has a teacher/orchestrator
• Horizontal is symbiotic, no hierarchy for execution

Both use agent-to-agent (A2A) communication, shown in the bottom of the diagram below.

When building a multi-agent architecture, consider the following:

• Pros: Scalability, adaptability, collaboration, efficiency, and autonomous decision-making
• Cons: Coordination complexities, Human-agent interactions (where does the human belong?), security, lack of standardization (though A2A is improving this drawback), and resource intensive (which is probably the biggest con)

Productionizing your agents

Finally, before putting your agents into production, and while in production, you want to monitor their performance, like you would with any other technology you use. Some best practices to implement in your agent use include:

• Observability and monitoring: Using Splunk Observability Cloud for GenAI
• Unit testing
• Containerization
• Tool restriction
• Use of an orchestrator
• Feedback loop: Generator LLM and an Evaluator LLM for responses

Security risks

MCP servers have some risks that traditional API security cannot address. If you use the On-Cloud version of the Splunk MCP Server, Splunk takes care of the security risks. You only need to manage your MCP client, authorization with bearer tokens, and who can access what data in the Splunk platform with role-based access control. However, if you use the on-premises app version, you need to secure both your MCP server and your deployment by following best practices for securing your environment. The following is a brief overview of some processes you will want to establish for MCP server security.

Tool poisoning

This is a manipulation of tool descriptions or parameters to introduce harmful behaviors. They are bad instructions that are invisible to the user but visible to the LLM.

You can protect against this with:

• Tool behavior monitoring
• Sandbox execution
• Allow-listing acceptable executions

Data exfiltration

This can happen in many ways, but one example of this is invisible prompt injection.

You can protect against this with:

• Output filtering (data loss prevention integration)
• Response size monitoring

Command and control

One example of this is malicious MCP servers distributed by an open tool hub. The servers altered task logs and response formatting to avoid triggering audit tools. Another way is a rug pull, in which a set of MCP tools seems legitimate, but suddenly becomes malicious alter widespread adoption.

You can protect against this with:

• Network segmentation
• Supply chain security
• Tool isolation and secure tool registry
• File integrity monitoring

Identity and access bypass

Misconfigured access logic in the server can lead to exposure of data.

Oauth token theft, can lead to impersonation of MCP server. Uses OAuth2

You can protect against this with:

• Enhanced OAuth implementation
• Role-based access control for data access
• Just-in-time access provisioning