The problem with rules in a wiki

Most detection content dies the same death. An analyst writes a Sigma rule during an incident, pastes it into the SIEM console, and moves on. Six months later nobody knows why the rule exists, who owns it, or whether it still fires. When it starts paging on-call at 3 a.m., someone disables it in the UI and the knowledge evaporates.

Detection as code fixes this by applying the boring, proven machinery of software engineering to detection content: version control, peer review, automated testing, controlled deployment, and observability. Your rules are software. They have bugs, dependencies, performance characteristics, and a lifecycle. Treat them accordingly.


The repository is the source of truth

Everything starts with a git repository. Not a shared drive, not the SIEM's built-in editor. If a rule is not in the repo, it does not exist in production.

A layout that works in practice:

detections/
  sigma/
    windows/
      proc_creation_encoded_powershell.yml
      cred_access_lsass_handle.yml
    linux/
  yara/
    families/
      win_cobaltstrike_beacon.yar
    hunting/
  tests/
    fixtures/
      encoded_powershell_events.jsonl
    samples/            # hashes only, binaries live in a store
  pipelines/
    validate.yml

Two conventions matter more than the folder structure:

  • One rule per file. Diffs stay readable, ownership stays clear, reverts stay surgical.
  • Metadata is mandatory. Every rule carries an author, a creation date, a reference (ticket, report, incident ID), an ATT&CK technique, and a status field (experimental, test, stable, deprecated). Sigma has these fields natively. For YARA, enforce them in the meta section with a linter.

Code review for detections

Every rule change goes through a pull request. No direct pushes to main, including for senior engineers, especially during incidents. The review checklist is different from application code:

Question Why it matters
What behavior does this detect, in one sentence? If the author cannot say it, the rule is a guess
What legitimate activity looks identical? False positive cost is paid by the SOC, not the author
Is the logic robust to trivial evasion? Case changes, quoting, path variations
What is the expected alert volume? "Unknown" is an acceptable answer only for experimental
Is there a test proving it fires? A rule with no test is a hope, not a detection

Review also catches the quiet killers: a Sigma contains where endswith was intended, a YARA string short enough to match half the filesystem, a condition that inverts under not.


CI validation: fail fast, fail cheap

The CI pipeline runs on every pull request and blocks the merge on failure. A minimal pipeline has three stages.

Stage 1: syntax and schema. Compile every YARA file, validate every Sigma rule against the specification, lint the metadata:

# YARA: compilation catches syntax errors and undefined identifiers
yara --fail-on-warnings -w rules/index.yar /dev/null

# Sigma: schema validation and conversion checks
sigma check sigma/
sigma convert --target splunk --without-pipeline sigma/windows/

If a rule cannot compile or convert to your SIEM backend, it fails here, in seconds, instead of at deploy time.

Stage 2: unit tests. Rules are functions. Inputs go in, matches come out. Test both directions:

  • Positive fixtures: for Sigma, replayed log events (JSONL exports from a lab detonation or a past incident) that the rule must match. For YARA, known-bad samples referenced by hash and pulled from a private sample store.
  • Negative fixtures: a corpus of known-clean logs and files that the rule must not match. This is your false positive regression suite, and it grows every time an analyst closes an alert as benign.
# Fail the build if the rule misses its own incident data
sigma convert -t splunk detections/sigma/windows/proc_creation_encoded_powershell.yml
python tests/replay.py \
  --rule proc_creation_encoded_powershell \
  --must-match tests/fixtures/encoded_powershell_events.jsonl \
  --must-not-match tests/fixtures/baseline_powershell_7d.jsonl

Stage 3: performance. For YARA, scan a fixed corpus and fail on regression beyond a threshold. A rule that doubles scan time across your fleet is a bug even if it detects perfectly. For SIEM queries, check estimated search cost where the backend exposes it.

An in-browser workbench shortens this loop. hunt.mlab.sh lets you author and validate YARA and Sigma rules and convert Sigma to SIEM backends before the rule ever reaches a pull request, which keeps CI failures rare and boring.


Shipping: staged rollout beats big bang

Merging to main should trigger deployment automatically, but never straight to paging alerts. A sane promotion path:

  1. Silent mode. The rule runs in production but only logs matches to a metrics index. No alerts, no tickets. Let it soak for one to two weeks.
  2. Review the soak. How many hits, on which hosts, at what times? Compare against the volume estimate from the pull request. This is where you discover the backup agent that spawns encoded PowerShell every night.
  3. Promote or tune. Move status from test to stable and route to the alert queue, or send it back with the soak data attached.

Deployment itself is a pipeline job: convert Sigma to the backend format, push via the SIEM API, tag the deployment with the git commit hash. That last part is what makes rollback trivial.


Metrics and rollback

Every deployed rule gets a dashboard row: alerts fired, true positive rate, time to triage, last true positive date. Three signals demand action:

  • Zero hits in 90 days. Either the behavior is gone, telemetry broke, or the rule never worked. Test it with a red team exercise or retire it.
  • True positive rate collapsing. Something in the environment changed. Tune it in a pull request, with the noisy events added as negative fixtures so the regression cannot return.
  • Alert storm. Roll back first, diagnose second. git revert on the rule file, pipeline redeploys the previous version, on-call goes back to sleep. The revert commit documents the failure forever.

Notice the pattern: every failure mode turns into a commit, a fixture, or a test. The system learns.


Start smaller than you think

You do not need all of this on day one. A realistic adoption path: put existing rules in git this week, require pull requests next week, add syntax validation in CI the week after. Fixtures and staged rollout come once the basics are habit. Even the first step alone, a diffable history of what changed and why, will pay for itself the first time someone asks "why did this alert stop firing in March?"

A detection you cannot test is an opinion. A detection in version control, with fixtures and a rollback path, is engineering.