SecOps ES Benchmark
An evaluation harness that measures whether an AI agent can actually run a security investigation in Elasticsearch — labeled attack telemetry, a graded exam, and a score.
- 239,000 ECS-normalized documents across 12+ Elastic data streams
- Five Linux intrusions forming one kill chain
- 27 ATT&CK techniques across 12 tactics
- Five investigation tasks, 54 auto-graded questions
- Endpoint, Zeek, nginx, Suricata, and detection-rule alerts
- Hosted read-only copy — point an agent at it with no setup
- Scoring your own investigation agent
- Comparing LLMs on ES|QL reasoning
- Evaluating MCP tool surfaces for security work
- Reproducible research on agentic SecOps
Everyone selling AI for security operations claims their agent can investigate. Almost nobody publishes a way to check.
This is that check. It is an evaluation harness for investigation agents working over Elastic security data: a labeled dataset of real attack telemetry, an exam built from it, a grader, and a one-command runner. You point your agent at a live read-only SIEM, give it a single alert or indicator, and see whether it can pivot across hosts and data sources, reconstruct the intrusion, and recommend a proportionate response. Then you get a number back.
The baseline scores we publish are a reference point. The point is to score yours.
Why it runs on Elasticsearch
Every question is answered by querying the store, on the same surface a working analyst uses. The data is ECS-normalized and organized as Elastic data streams, so an agent has to understand the schema and write ES|QL to get anywhere. There is no path around it — no summarized context, no pre-chewed answer key.
The intrusions span Elastic Endpoint process, file and network events, Zeek flows and TLS and DNS and SSH, nginx access logs carrying the web probing and the webshell hits, Suricata IDS alerts over the same traffic, and Elastic detection-rule and threat-intelligence matches. An agent that only reads alerts scores badly. That is the intended result.
The tool surface is deliberately small
An agent can reach the data however it likes — an MCP server, a CLI, an SDK, an agent skill. The reference surface is four read-only tools:
esql_query · es_search · get_mappings · list_indicesThe benchmark cares about the investigation, not the plumbing.
Try it without installing anything
A hosted copy of the full dataset is online, read-only, login benchmark /
benchmark. Writes and deletes return 403.
ES="https://secops-benchmark-es.k8s.tocharian.eu"
curl -u benchmark:benchmark "$ES/_cat/indices/logs-*-bench,benchmark-*?v"Kibana Discover over the same data is at
secops-benchmark.k8s.tocharian.eu.
The instance is rate-limited and periodically reset. Secrets and business data
are removed; real attacker IPs are kept deliberately, which is discussed in the
repository’s ETHICS.md.
Licensing and citation
Code is Apache 2.0, data is CC BY 4.0, and the dataset has a DOI
(10.5281/zenodo.21770551) so it can be cited in published work. Mirrors are on
Hugging Face Datasets and on Kaggle.
Released as v0.1.1. Later versions are in testing.