How do you measure success?
Pick a few measures and keep them the same across exercises.
| Measure | What it tells you |
|---|---|
| ATT&CK coverage | Which techniques you have a detection for, and which you do not |
| Detection ratio | Of the techniques executed, how many raised an alert |
| Mean time to detect (MTTD) | How long an action took to be noticed |
| Mean time to respond (MTTR) | How long the response took once noticed |
| Retest result | Whether each fix holds when the technique runs again |
The Mandiant datasheet lists a scorecard of metrics on detection of the simulated incidents among its deliverables. See the Mandiant profile. Paranoid Security’s cyber exercises page says its report lists actions that monitoring did not record, with detection recommendations. See the Paranoid Security profile.
What do you do with the gap report?
Turn each gap into a ticket with an owner: add a log source, write or change a rule, or update a playbook. Record the technique, the evidence of the miss and the expected fix, then retest. Tools such as VECTR can hold this, as covered in the tools guide.
How do you prioritise remediation?
Rank by three questions:
- How likely is this technique in your threat picture?
- How much damage follows if it goes unseen?
- How much effort is the fix?
Do the likely, damaging and cheap gaps first. Keep a list of the expensive ones with a decision, even if the decision is to accept the risk.
What are common mistakes?
- Counting techniques tested instead of techniques detected.
- Fixing the easiest gaps first.
- Skipping the retest.
- Changing the measures every time, so no trend shows.
How do you show value to management?
Show the same technique list before and after, the change in time to detect, and the gaps closed against the regimes you answer to. The regime list is in UAE regulations and purple teaming. Keep the report short and name the next exercise.
Where the next exercise sits in a longer plan is covered in continuous vs one-off purple teaming. See also the ranking and the methodology for how providers describe their reporting.