Skip to main content
Lantern Home

 

Splunk Lantern

Monitoring enterprise AI security, compliance, and spend

  1. Last updated
  2. Save as PDF

This article continues from Setting up enterprise AI governance add-ons and covers what you can do with the data those add-ons collect—from security visibility and compliance audit trails to spend tracking, model usage analysis, and coverage of self-hosted models.

Giving your security team visibility into a new attack surface

The Enterprise AI Governance Add-on's AI Security Audit dashboard shows key lifecycle events, admin and SSO configuration changes, sign-in activity, and data exports across every provider in one place. Four of its alerts cover the same ground on a schedule: API Key Created or Deleted, Admin or SSO Configuration Change, Data Export Activity, and Off-Hours Activity Spike. All eight alerts in the add-on are disabled by default; tune the thresholds before enabling them.

Because events are normalized, the cross-provider action macros aigov_signin_actions, aigov_admin_actions, aigov_key_actions, and aigov_export_actions let analysts fold AI activity into correlation searches they already run.

The screenshot below shows the AI Security Audit dashboard, where sign-in activity, export events, and admin and SSO configuration changes are displayed across every provider in one view.

The AI Security Audit dashboard: key lifecycle events, admin and SSO changes, sign-in activity, and export events by provider and user.

For Claude, the Anthropic add-on's Claude Enterprise – AI Security Audit dashboard goes past what the normalized view carries: access failures such as claude_chat_access_failed, artifact publishing and sharing exposure, user activity by IP with multi-IP anomaly detection, and an audit of who is reading your compliance data through the API. Its six detections are also disabled by default: multi-IP user activity, access failures, SSO configuration change, data export started, top cost users, and users near their spend limit.

Producing the audit trail compliance will ask for

When legal, privacy, or an external auditor needs to reconstruct who used which AI tool, when, from where, and against what data, the AI Compliance & Directory dashboard and the underlying audit and directory data provide the record. The Anthropic Compliance API activity feed, OpenAI organization audit logs, ChatGPT Enterprise compliance logs, Gemini Workspace audit events, and Copilot Purview interaction records land in one index with one field vocabulary, so an investigation spanning platforms doesn't have to span tools.

The screenshot below shows the AI Compliance & Directory dashboard, where you can investigate activity timelines, user and group directory context, and per-provider audit coverage.

The AI Compliance and Directory dashboard: activity timelines, user and group directory context, and per-provider audit coverage.

Tracking spend and adoption

The AI Usage & Cost Monitoring dashboard combines usage analytics and daily cost data from Anthropic and OpenAI with per-user Copilot usage reports. The Daily Spend Threshold Exceeded alert catches runaway spend mid-cycle. New AI User Seen works in two directions: an adoption signal for teams rolling AI out and a shadow-usage detector for teams governing it.

The screenshot below shows the AI Usage & Cost Monitoring dashboard, with token usage trends, daily spend broken down by provider, and the top users and models by consumption.

The AI Usage and Cost Monitoring dashboard: token usage trends, daily spend by provider, and top users and models.

For Claude, the Anthropic add-on's Claude Enterprise – Usage & Spend Analytics dashboard adds daily and weekly active users with the stickiness ratio between them, adoption rate, token mix across output, uncached input, cache reads and cache creation, prompt cache hit rate, and blended cost per million tokens. Read prompt cache hit rate next to spend: cached input tokens bill at a fraction of the uncached rate, so a low hit rate means paying full price for context that is being re-sent.

The screenshot below shows the Claude Enterprise – Usage & Spend Analytics dashboard, covering the active user and stickiness metrics, period spend and token totals, adoption rate, token mix, and prompt cache hit rate described above.

Claude Enterprise – Usage and Spend Analytics: active users and stickiness, period spend and total tokens, adoption rate, token mix, prompt cache hit rate, and blended cost per 1M tokens.

Anthropic finalizes analytics with a lag of roughly three days, so use a Last 7 days or Last 30 days range on this dashboard rather than Last 24 hours.

Identifying who is using which model and whether that matches the work

Model families come in tiers: a fast, inexpensive model and a slower, more capable, more expensive one. Some of your spend is people running the top tier for work a cheaper tier handles. Some of it is people whose work needs the top tier. The lower half of Claude Enterprise – Usage & Spend Analytics separates the two.

  • Who Used Which Model – Spend Matrix puts users on rows and models on columns, with spend in each cell and a total_usd column. Tier concentration reads directly off the table.
  • Top 20 Users by Cost and Top 20 Users by Tokens rank the estate and list each user's models and products inline.
  • Spend by User and Model and Tokens by User and Product show the same data as stacked bars.
  • Clicking a user in either Top 20 panel drills the bottom row down to that person: Model Mix by Spend, Product Mix by Tokens, and Daily Spend by Product.

The screenshot below shows all four panels together: the Spend Matrix, the Top 20 Users rankings, the stacked-bar views, and the per-user drill-down.

Who Used Which Model – Spend Matrix, top users by cost and tokens, spend by user and model, and the per-user drill-down showing model mix, product mix, and daily spend.

The matrix shows which model someone chose. Model mix read next to product mix shows whether the choice fits the work:

  • Top tier concentrated in Claude Code is agentic work: long-running, multi-step, high-context tasks. The capable model finishes in one pass where a cheaper one produces output someone has to redo.
  • Top tier concentrated in Claude.ai chat, with low tokens per interaction, is closer to a default setting than a requirement.
  • Peers doing the same job on a cheaper tier is the strongest signal available. Compare across a team before drawing conclusions about a person.
  • A low prompt cache hit rate alongside high spend means context is being re-sent rather than reused.

Before acting on this information, consider that users who genuinely need the top tier will move to personal accounts if you cap them, and you lose the visibility entirely. And the spend matrix reflects per-user attribution from the analytics API, which will not always sum to the org-level cost report. Use the matrix to compare users against each other, and take totals from the Period Spend and cost panels.

The Enterprise AI Governance Add-on carries model and product identity in aigov_product across providers, so the same comparison extends to ChatGPT and Gemini when more than one provider is reporting.

Keeping self-hosted models from becoming a blind spot

Self-hosted models have no vendor invoice, no admin console, and no seat count to notice. The selfhosted_monitor input inventories the models running on your vLLM, Ollama, or LiteLLM servers, scrapes their Prometheus metrics, and health-checks the endpoints. The Self-Hosted & Open-Source Models dashboard tracks them, and the New Self-Hosted Model Detected and Self-Hosted Server Down alerts fire when something appears or disappears.

The screenshot below shows the Self-Hosted & Open-Source Models dashboard, where you can see the model inventory, request and latency metrics, and server health status for each endpoint.

The Self-Hosted and Open-Source Models dashboard: model inventory, request and latency metrics, and server health.

Next steps

You might find the following resources useful for implementing the guidance in this article: