The McNamara Fallacy in Security Metrics

During the Vietnam War, McNamara’s Pentagon judged progress by quantifiable indicators — above all enemy body counts, which were easy to report (and to inflate). By those numbers the U.S. was “winning” while losing the war. The sociologist Daniel Yankelovich later named the pattern the McNamara fallacy: measure what is easy to measure, then treat what can’t be measured as unimportant, and finally as nonexistent. Security dashboards do the same thing: we count vulnerabilities, track patch percentages, and report mean time to detect.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats

from decision_security.synth import make_rng, sample

rng = make_rng(42)

plt.rcParams.update({
    "font.family": "serif",
    "font.size": 10,
    "axes.labelsize": 11,
    "axes.titlesize": 12,
    "xtick.labelsize": 9,
    "ytick.labelsize": 9,
    "legend.fontsize": 9,
    "figure.dpi": 150,
    "axes.spines.top": False,
    "axes.spines.right": False,
})

PRIMARY = "#1A1A1A"
ACCENT = "#E74C3C"
DARK_BG = "#34495E"
LIGHT_GRAY = "#95A5A6"
MED_GRAY = "#7F8C8D"
VERY_LIGHT = "#BDC3C7"

1. Easy Metrics vs Important Metrics

There are two types of security metrics: those that are easy to collect (vulnerability counts, training completion rates, ticket closure times) and those assumed here to predict outcomes (control coverage gaps, config drift rate, access privilege sprawl). The McNamara Fallacy is measuring the first type and treating it as though it were the second.

n_orgs = 500

breach_propensity = rng.beta(2, 8, n_orgs)
breached = rng.binomial(1, breach_propensity)

# Designed illustration, not an empirical finding: easy metrics are pure noise, hard metrics
# are built from breach_propensity, so "hard metrics predict better" is baked in.
easy_metrics = {
    "vuln_count": rng.poisson(120, n_orgs),
    "patch_pct": rng.beta(7, 3, n_orgs),
    "training_completion": rng.beta(8, 2, n_orgs),
    "tickets_closed": rng.poisson(45, n_orgs),
    "scan_frequency": rng.choice([1, 2, 4, 12], n_orgs),
    "policies_documented": rng.poisson(15, n_orgs),
    "incidents_reported": rng.poisson(8, n_orgs),
}

hard_metrics = {
    "config_drift_rate": 0.3 * breach_propensity + 0.1 * rng.random(n_orgs),
    "priv_access_ratio": 0.2 + 0.5 * breach_propensity + 0.1 * rng.random(n_orgs),
    "coverage_gap_pct": 0.1 + 0.6 * breach_propensity + 0.1 * rng.random(n_orgs),
}

df = pd.DataFrame({**easy_metrics, **hard_metrics, "breached": breached})

correlations = df.corr()["breached"].drop("breached").sort_values(key=abs, ascending=False)
print("Correlation with breach outcome:")
print(correlations.to_string())

Correlation with breach outcome:
coverage_gap_pct       0.295176
priv_access_ratio      0.291739
config_drift_rate      0.254735
policies_documented   -0.091274
scan_frequency         0.084455
vuln_count             0.055390
patch_pct             -0.031778
incidents_reported    -0.026884
training_completion   -0.014761
tickets_closed        -0.001462

2. Predictive Power Visualization

The bar chart sorts all metrics by absolute correlation with breach outcomes and color-codes them by measurement difficulty. By construction the three hard-to-measure metrics (config drift rate, privilege sprawl, coverage gaps) occupy the top three ranks, because in this synthetic data-generating process they are the only variables actually tied to breach propensity – they are computed as readouts of it, while the easy metrics are activity proxies with no built-in signal.

This is a designed illustration, not evidence: if the informative signal lives in the hard-to-measure metrics, a dashboard built from the easy ones will miss it. The easy metrics scatter near zero correlation with random sign, yet they are the ones cheap enough to end up on the dashboard.

fig, ax = plt.subplots(figsize=(9, 5))
colors = [DARK_BG if m in hard_metrics else VERY_LIGHT for m in correlations.index]

# Signed correlation: hard metrics are readouts of breach propensity, so they sit on the
# positive side; noise metrics scatter on both sides.
bars = ax.barh(correlations.index, correlations.values, color=colors, edgecolor="white")
ax.axvline(0, color=MED_GRAY, linewidth=0.8)
ax.set_xlabel("Correlation with Breach Outcome (signed)")
ax.set_title("Which Metrics Actually Predict Breaches?")

for bar, v in zip(bars, correlations.values):
    ax.text(v + (0.004 if v >= 0 else -0.004),
            bar.get_y() + bar.get_height() / 2,
            f"{v:+.2f}", va="center",
            ha="left" if v >= 0 else "right", fontsize=8)

import matplotlib.patches as mpatches
ax.legend(handles=[
    mpatches.Patch(color=DARK_BG, label="Hard to measure"),
    mpatches.Patch(color=VERY_LIGHT, label="Easy to measure"),
], loc="lower right")
ax.margins(x=0.15)
plt.tight_layout()
plt.show()

3. Goodhart’s Law — When a Measure Becomes a Target

“When a measure becomes a target, it ceases to be a good measure” is Marilyn Strathern’s 1997 paraphrase of Goodhart’s law. Goodhart’s 1975 wording: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” If you incentivize patch percentage, teams will prioritize easy-to-patch low-severity vulnerabilities over hard-to-patch critical ones. The metric goes up; the risk stays the same (or gets worse).

n_vulns = 200
severity = rng.choice(["critical", "high", "medium", "low"], n_vulns, p=[0.05, 0.15, 0.40, 0.40])
risk_weight = {"critical": 50, "high": 10, "medium": 2, "low": 0.5}
patch_effort = {"critical": 8, "high": 4, "medium": 2, "low": 1}

vulns = pd.DataFrame({
    "severity": severity,
    "risk": [risk_weight[s] for s in severity],
    "effort": [patch_effort[s] for s in severity],
})

# Same remediation budget (effort-hours) for both strategies; only the order differs.
capacity = 300

by_severity = vulns.sort_values("risk", ascending=False).copy()
by_severity["cum_effort"] = by_severity["effort"].cumsum()
patched_sev = by_severity[by_severity["cum_effort"] <= capacity]

by_ease = vulns.sort_values("effort", ascending=True).copy()
by_ease["cum_effort"] = by_ease["effort"].cumsum()
patched_ease = by_ease[by_ease["cum_effort"] <= capacity]

print("=== Patch by Severity (risk-optimal) ===")
print(f"  Vulns patched: {len(patched_sev)} / {n_vulns} ({100*len(patched_sev)/n_vulns:.0f}%)")
print(f"  Risk eliminated: {patched_sev['risk'].sum():.0f} / {vulns['risk'].sum():.0f} ({100*patched_sev['risk'].sum()/vulns['risk'].sum():.0f}%)")

print(f"\n=== Patch by Ease (metric-optimal) ===")
print(f"  Vulns patched: {len(patched_ease)} / {n_vulns} ({100*len(patched_ease)/n_vulns:.0f}%)")
print(f"  Risk eliminated: {patched_ease['risk'].sum():.0f} / {vulns['risk'].sum():.0f} ({100*patched_ease['risk'].sum()/vulns['risk'].sum():.0f}%)")

print(f"\nGoodhart's Law: {len(patched_ease)-len(patched_sev)} more vulns patched, "
      f"but {patched_sev['risk'].sum()-patched_ease['risk'].sum():.0f} more risk units left unaddressed.")

=== Patch by Severity (risk-optimal) ===
  Vulns patched: 103 / 200 (52%)
  Risk eliminated: 750 / 814 (92%)

=== Patch by Ease (metric-optimal) ===
  Vulns patched: 180 / 200 (90%)
  Risk eliminated: 334 / 814 (41%)

Goodhart's Law: 77 more vulns patched, but 416 more risk units left unaddressed.

4. Goodhart Visualization

Patching by ease produces a higher patch count (the metric looks great) but eliminates less than half the risk. Patching by severity produces a lower count (the metric looks worse) but eliminates nearly all the risk.

This is Goodhart’s Law: the team optimizing for the metric appears to be outperforming the team optimizing for the outcome. Any incentive structure tied to patch percentage will drive rational actors toward the metric-optimal strategy.

fig, axes = plt.subplots(1, 2, figsize=(10, 4))

strategies = ["By Severity\n(risk-optimal)", "By Ease\n(metric-optimal)"]
patch_counts = [len(patched_sev), len(patched_ease)]
risk_reduced = [patched_sev["risk"].sum(), patched_ease["risk"].sum()]

axes[0].bar(strategies, patch_counts, color=[DARK_BG, ACCENT], edgecolor="white")
axes[0].set_ylabel("Vulnerabilities Patched")
axes[0].set_title("The Metric: Patch Count")

axes[1].bar(strategies, risk_reduced, color=[DARK_BG, ACCENT], edgecolor="white")
axes[1].set_ylabel("Risk Units Eliminated")
axes[1].set_title("What Actually Matters: Risk Reduction")

fig.suptitle("Goodhart's Law: Optimizing the Metric \u2260 Optimizing the Outcome",
             fontsize=11, fontweight="bold", y=1.02)
plt.tight_layout()
plt.show()

Check it in your own environment.

1. Put “risk reduced” next to “tickets closed” on the same dashboard, using the severity × exposure weighting you already trust.

2. Look at what was closed last quarter: what share of closures were low severity or not internet-facing?

3. Sort the open backlog by that weighting and compare it with the order work is actually done in.

5. Three Questions for Every Metric

Before adding any metric to a dashboard, ask:

1. So what? — If this number changes, does it change a decision?

2. Can it be gamed? — If someone optimizes this metric directly, does the underlying risk actually improve?

3. What’s the proxy distance? — How many causal steps lie between this metric and the outcome we care about? The more steps, the weaker the signal.

The scorecard below applies these three questions to a handful of common metrics. Unlike the correlations in Section 1, these scores are illustrative, subjective judgments, not a computed result.

metrics_eval = pd.DataFrame([
    {"metric": "Patch %", "so_what": 1, "gameable": 2, "proxy_distance": 2},
    {"metric": "MTTR", "so_what": 2, "gameable": 1, "proxy_distance": 1},
    {"metric": "Training completion", "so_what": 0, "gameable": 2, "proxy_distance": 2},
    {"metric": "Config drift rate", "so_what": 2, "gameable": 0, "proxy_distance": 0},
    {"metric": "Vuln count (open)", "so_what": 1, "gameable": 2, "proxy_distance": 2},
    {"metric": "Privileged access ratio", "so_what": 2, "gameable": 1, "proxy_distance": 0},
])

scores = metrics_eval.set_index("metric")[["so_what", "gameable", "proxy_distance"]].copy()
# so_what is scored 2 = drives a decision (good); flip it so every column reads 0 = good, 2 = bad.
scores["so_what"] = 2 - scores["so_what"]
scores.columns = ["Drives no decision\n(2=never)", "Gameable\n(2=easily)", "Proxy distance\n(2=far)"]

from matplotlib.colors import ListedColormap, BoundaryNorm
cmap = ListedColormap(["#F4F6F6", "#F5B7B1", ACCENT])       # good, mixed, bad
norm = BoundaryNorm([-0.5, 0.5, 1.5, 2.5], cmap.N)

fig, ax = plt.subplots(figsize=(8, 4))
im = ax.imshow(scores.values, cmap=cmap, norm=norm, aspect="auto")
ax.set_xticks(range(3))
ax.set_xticklabels(scores.columns, fontsize=8)
ax.set_yticks(range(len(scores)))
ax.set_yticklabels(scores.index, fontsize=9)
ax.set_xticks(np.arange(-0.5, 3), minor=True)
ax.set_yticks(np.arange(-0.5, len(scores)), minor=True)
ax.grid(which="minor", color="white", linewidth=2)
ax.tick_params(which="minor", length=0)

for i in range(len(scores)):
    for j in range(3):
        v = scores.values[i, j]
        ax.text(j, i, str(v), ha="center", va="center", fontsize=11, fontweight="bold",
                color="white" if v == 2 else PRIMARY)

ax.set_title("Metric Quality Scorecard (illustrative judgment; lower is better)", fontsize=11)
cbar = plt.colorbar(im, ax=ax, shrink=0.8, ticks=[0, 1, 2])
cbar.ax.set_yticklabels(["0 good", "1", "2 bad"])
plt.tight_layout()
plt.show()

6. Pitfalls

• Volume metrics are almost always McNamara metrics. Vulnerability counts, alert counts, ticket counts — on their own, they measure activity. A team that closes 500 low-severity tickets while ignoring 3 critical ones looks great on the dashboard.

• “We reduced MTTR by 40%” means nothing without knowing what changed. Did you get faster at responding, or did you redefine what counts as an incident? Goodhart’s Law applies to process metrics too.

• The hardest metrics to collect can be the most informative. Config drift rate, privilege sprawl, control coverage gaps — these require engineering effort to measure, which also makes them harder to game.

• Dashboards that optimize for green create incentives against honesty. If reporting a risk as “red” creates work and reporting it as “green” doesn’t, the rational response is to find reasons it’s green.