Detecting per-entity security anomalies with the Cisco Deep Time Series Model
Detecting security anomalies with the Cisco Deep Time Series Model showed you how to apply CDTSM to a single environment-wide series: total PDNS blocks per hour, one number for the whole estate. That detection answers "is something wrong?" but not necessarily "where?"
This article tracks each entity against its own baseline instead of one series for the environment. That makes it a good candidate for risk-based alerting (RBA), because you can attribute risk to the specific user or host that behaved unusually.
The signal changes too. PDNS blocks worked well across the whole environment, but a typical user might generate only a handful of blocks a month, which isn't enough to build a baseline from. This article uses web request volume per user instead, looking for activity outside each user's normal pattern. The same approach works for any per-entity metric you can trend hourly.
Why this signal
The detection built in this article flags a user whose web activity is well above their normal level for that time of day or week. That could be a compromised account being used while its owner is asleep, or it could be someone catching up on work at the weekend. Since the detection can't tell you which, its output feeds risk scoring rather than creating findings on its own.
Telling those two apart depends on knowing what's normal for that user at that specific hour, not just what's normal for them overall. A flat threshold can't do this. 200 requests is unremarkable at 2pm and unprecedented at 2am. CDTSM can, because it builds its forecast from each user's own recent history and picks up their daily and weekly rhythm without you defining working hours anywhere.
That also means it generalizes across a workforce spread over timezones, where a fixed set of office hours would be wrong for most of the population. But it only works if the entity has a rhythm to pick up in the first place. Flat, round-the-clock activity gives the seasonality handling nothing to do, and a simple threshold would perform just as well.
Why per-entity detection is a different problem
There are three constraints that don't exist with a single series.
Sparsity: CDTSM needs at least 60 data points, and with fill_null=0 every series is padded to the full window, so that's easy to meet. The problem is that most users are only active a third of the time or less, so most of those points are zeros. The model has little genuine activity to learn from, and the thresholds it derives for very sparse users are unstable.
Heterogeneous scale: A service account might make 10,000 requests an hour and a part-time user 5. The single-series detection above used method=quantile, which suits a single, well-shaped series. For a population, switch to method=iqr_residual. It measures how far each point sits from the forecast relative to that series' own normal spread, so one multiplier works sensibly across users operating at very different volumes.
No visual tuning loop: That earlier detection was tuned by looking at an Anomaly Detection Chart and adjusting the multiplier until the detection rate looked right. That doesn't scale to thousands of users. Tuning becomes statistical, and the chart returns later, at triage.
Prerequisites
- Splunk AI Toolkit version 6.1.0 or later.
- An accelerated Web data model, or equivalent proxy data.
Key tuning parameters
Parameters that differ from the single-series detection or behave differently per entity.
| Parameter | Description |
|---|---|
by |
The field to group on. Must come immediately after the forecast field, comma separated and quoted if grouping on more than one field. |
method |
iqr_residual. See Heterogeneous scale above. |
multiplier |
Your sensitivity dial, as in the single-series detection. |
metric_resolution |
The expected spacing of your series, for example 1h. Required because tstats ... by _time span=1h skips hours with no events, and it tells fill_null where the gaps are. |
fill_null |
What to put in the missing buckets, once metric_resolution has told the model where they are. Set it to 0 for this use case. The default, interpolate, draws a straight line across gaps, which invents traffic for a user who was idle overnight. |
threshold_direction |
Set to upper. iqr_residual scores the size of the error in either direction, so without this, unusually low volume would also flag. Low web volume usually means someone took the afternoon off. |
How to use Splunk software for this use case
Choosing your entity population
Don't apply CDTSM to every user you have. Work out who you can model first.
Count how many hourly buckets each user appears in. Over a 30-day window that's up to 743 possible buckets, and a conventional 9-to-5 worker will populate roughly 176 of them.
| tstats summariesonly=t count AS requests from datamodel=Web where Web.user=* earliest="-30d@d" latest="@h" by Web.user _time span=1h | rename Web.* AS * | stats count AS n BY user | eval band=case(n<100,"1. <100",n<150,"2. 100-150",n<200,"3. 150-200", n<300,"4. 200-300",n<400,"5. 300-400",true(),"6. 400+") | stats dc(user) AS users BY band | sort band
In one enterprise environment, for a population of around 20,000 users:
| Populated buckets | Users |
|---|---|
| under 100 | 10,307 |
| 100 to 150 | 2,432 |
| 150 to 200 | 4,392 |
| 200 to 300 | 2,547 |
| 300 to 400 | 376 |
| 400 or more | 339 |
A gate of 150 buckets leaves 7,654 users, about 38% of the population.
150 was a judgment call for this experiment rather than a measured cutoff. It corresponds to roughly 7 hours of activity per working day. Your population will differ, so run the band search above and pick the threshold that fits what you see.
Building the detection
| tstats summariesonly=t count AS requests from datamodel=Web where Web.user=* earliest="-30d@d" latest="@h" by Web.user _time span=1h | rename Web.* AS * | eventstats count AS n BY user | where n>=150 | fields _time user requests | apply CDTSM requests by "user" mode=anomaly method=iqr_residual conf_interval=60 multiplier=25 on_span=1 off_span=1 threshold_direction=upper metric_resolution=1h fill_null=0 detection_window_earliest=-4h detection_window_latest=-1h | rename requests.* AS * | where requests>threshold_upper | eval dow=strftime(_time,"%a"), hod=strftime(_time,"%H"), over_threshold=round(requests/threshold_upper,1) | table _time user dow hod requests forecast threshold_upper over_threshold

Each row is one user, one hour. forecast is what the model expected for that user at that hour, and threshold_upper is the line it had to cross. over_threshold shows how far past it went, which is how you prioritize when several users flag at once.
The | where requests>threshold_upper line filters on the value rather than on anomaly_state. Segments don't always close when you expect, so filtering on state alone would inflate your counts. See the caveats at the end.
Capacity and scale
apply loads its entire input into memory before processing, so what limits scale is the number of rows: one per entity per hourly bucket with activity. Empty buckets produce no row.
In the environment tested, 7,654 qualified users over 30 days came to 1.9 million rows and completed in about three minutes. The same population over 45 days is around 2.9 million rows and fails. So the lever that matters is how much history you feed it, not how many users qualify: lowering the gate from 150 buckets to 130 adds 537 users but only 4% more rows, because users near the gate are sparse by definition. If the search won't complete, shorten the window, widen the span, or split the population across several scheduled detections. Raising the memory limit isn't an option on Splunk Cloud Platform.
Runtime is a separate question. Three minutes suits an hourly schedule. If your search takes too long for its schedule, write the hourly counts to a summary index with | collect and point the detection at that instead of recomputing 30 days of tstats each run. That reduces runtime but not memory, since apply still receives the same rows.
The model API, documented at 50 requests per minute, is a separate limit again, but it wasn't the constraint during testing. The toolkit batched 7,654 users into around 16 requests, and inference took about 52 seconds of the three-minute run. The Splunk platform handling the data accounts for the rest.
Tuning without charts
With one series you tune by eye. With thousands, you count instead.
Run the detection at several multiplier values and record how many distinct users are flagged.
| Multiplier | Users flagged |
|---|---|
| 10 | 88 |
| 15 | 72 |
| 25 | 41 |
The single-series detection's multiplier=10 and the iqr_residual default of 3 are both too sensitive here. The value doesn't transfer between use cases. At 25, 41 users flag, about 0.5% of the qualified population over the detection window.
Because the output is a risk contribution rather than a finding, decide what proportion of the population should contribute risk from this signal in any given window. Between 0.5% and 1% is a reasonable starting point: low enough that the signal means something when it fires, high enough that you're not tuning away real deviations. Adjust the multiplier until you land near that, then let the risk framework decide what accumulates into a finding.
Check more than one detection window before settling on a multiplier. A value that produces a sensible rate in an evening might behave differently in the morning, so tuning against a single point in time can mislead.
Contributing risk rather than creating findings
Dozens of flagged users per run is unworkable as findings but workable as intermediate findings (risk events).
One anomalous hour for one user is weak evidence on its own. A user who is anomalous on web volume, and also authenticating from a new location, and also triggering a DLP rule on the same day, is worth an analyst's time. Accumulating weak indicators like this one is what RBA is designed for.
Contribute risk against the user identity and let a risk finding fire on accumulated score.
Investigating the anomaly after it fires
When the detection fires, the analyst sees a table of numbers rather than a chart, which makes it hard to see why a result is anomalous. A drilldown that replays the Anomaly Detection Chart for the flagged user solves that.
Where the drilldown appears depends on how the detection is configured. If it creates a finding directly, the drilldown is available from that finding. If it contributes an intermediate finding instead, the drilldown defined on the event-based detection carries through to the intermediate finding timeline, so analysts reach the same chart from the intermediate finding that contributed to the finding they're investigating.
Either way, the drilldown search itself needs one adjustment. It can't use relative offsets like -4h, because they would be evaluated whenever the analyst opens it rather than when the detection ran. The window needs pinning at detection time, so add this to the detection before the | table, and include the four fields in that table so they're available as tokens:
| eval search_earliest=round(relative_time(now(),"-30d@d"),0), search_latest=round(now(),0), dw_earliest=round(relative_time(now(),"-4h"),0), dw_latest=round(relative_time(now(),"-1h"),0) | table _time user dow hod requests forecast threshold_upper over_threshold search_earliest search_latest dw_earliest dw_latest
The drilldown then reproduces the same window the detection used, which matters because window width changes the forecast, as covered in Detecting security anomalies with the Cisco Deep Time Series Model.
| tstats summariesonly=t count AS requests from datamodel=Web where Web.user="$user$" earliest="$search_earliest$" latest="$search_latest$" BY _time span=1h | rename Web.* AS * | apply CDTSM requests mode=anomaly method=iqr_residual conf_interval=60 multiplier=25 on_span=1 off_span=1 threshold_direction=upper metric_resolution=1h fill_null=0 detection_window_earliest="$dw_earliest$" detection_window_latest="$dw_latest$"
Configure this as a drilldown search on the detection, passing user and the four time fields as tokens. During tuning, you can also run it manually by replacing $user$ with a username from the detection output. Either way, switch to the Visualization tab and select the Anomaly Detection Chart.

The blue line is the observed request count. The dashed blue line is what the model expected, and the red dashed line is the threshold that a value has to exceed to be flagged. The shaded region marks where an anomaly was found.
The expected value through the overnight hours sits near zero, because the model has learned this user is inactive then.
Caveats
- Relative detection window offsets count back from the current time, not from the end of your search range. If you point the search at a historical window for tuning, use absolute timestamps or the detection window will be empty.
- Set both window bounds or neither. Supplying only
detection_window_earliestfails with a message about latest being before earliest. - Anomaly segments don't always close as expected. With
off_span=1, rows can still be flaggedANOMALOUSwhere the value is below the threshold, and in testing one such row had a value of zero. Filter onrequests>threshold_upperrather than trustinganomaly_state, or your flagged counts will be inflated. anomaly_scoreis the absolute difference between the actual value and the forecast. It isn't comparable across users of different volumes, so don't use it to scale risk contribution.- Entities with no data in the detection window are skipped rather than scored, so the number of entities evaluated will be far lower than your qualified population.
- The model has no concept of public holidays. It learns hourly, daily and weekly cycles, so it treats a public holiday as an ordinary weekday and expects ordinary weekday volumes.
- Volume is a blunt instrument. It won't catch a low-and-slow beacon, which generates too few requests to register, or a single large upload, which is a handful of requests rather than hundreds.
Next steps
Detecting security anomalies with the Cisco Deep Time Series Model gave you a detection that tells you something is wrong in the environment. This one tells you which account. Neither needs model fitting, scheduled retraining or per-entity models to maintain.
Start with the qualification search. It tells you quickly whether this approach will work on your data.
If this is useful but you would rather not build and maintain your own detections, Splunk Enterprise Security Premier edition includes UEBA with pre-built anomaly detection use cases developed and maintained by Splunk. Speak to your account team if you want that capability out of the box.
You might find the following resources useful for implementing the guidance in this article:
- Splunk Lantern Article: Detecting insider threats with RBA in Enterprise Security 8.x
- Splunk Lantern Article: Tuning anomaly rules
- Splunk Lantern Article: Implementing risk-based alerting
- Splunk Help Docs: Cisco Deep Time Series Model
- Splunk Help Docs: Risk scoring in Splunk Enterprise Security

