Calculations vs Decisions in Security

Calculations vs Decisions in Security#

Most security work produces calculations, not decisions. A risk register entry, a vulnerability scan report, a Monte Carlo estimate — none of these are decisions until someone irrevocably allocates resources based on them. This notebook explores the distinction and why it matters.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

from decision_security.synth import make_rng, sample
from decision_security.montecarlo import simulate_aggregate_losses, make_lognormal_severity

rng = make_rng(42)

plt.rcParams.update({
    "font.family": "serif",
    "font.size": 10,
    "axes.labelsize": 11,
    "axes.titlesize": 12,
    "xtick.labelsize": 9,
    "ytick.labelsize": 9,
    "legend.fontsize": 9,
    "figure.dpi": 150,
    "axes.spines.top": False,
    "axes.spines.right": False,
})

PRIMARY = "#1A1A1A"
ACCENT = "#E74C3C"
DARK_BG = "#34495E"
LIGHT_GRAY = "#95A5A6"
MED_GRAY = "#7F8C8D"
VERY_LIGHT = "#BDC3C7"

1. The Calculation-Decision Boundary#

A calculation is revocable — you can change your mind without committing resources. A decision is irrevocable — budget is spent, people are hired, contracts are signed. Most “risk-based decisions” in security are calculations dressed up as decisions: the risk register was updated, but nothing changed.

activities = pd.DataFrame([
    {"activity": "Run a vulnerability scan", "type": "Calculation", "why": "No resources committed; you can ignore the results"},
    {"activity": "Buy and deploy EDR ($200K)", "type": "Decision", "why": "Budget irrevocably allocated"},
    {"activity": "Estimate annual loss expectancy", "type": "Calculation", "why": "A number on a slide; no action follows automatically"},
    {"activity": "Terminate a vendor relationship", "type": "Decision", "why": "Switching costs are real and immediate"},
    {"activity": "Update the risk register to 'High'", "type": "Calculation", "why": "The label changed; the exposure didn't"},
    {"activity": "Accept residual risk (signed by CISO)", "type": "Decision", "why": "The acceptance is itself the irrevocable commitment: you forgo mitigation and own the exposure"},
])

def _shade(row):
    c = ACCENT if row["type"] == "Decision" else DARK_BG
    return [f"background-color: {c}; color: white"] * len(row)

(activities.style
    .hide(axis="index")
    .apply(_shade, axis=1)
    .set_properties(**{"text-align": "left", "padding": "6px 10px"})
    .set_table_styles([{"selector": "th", "props": [("text-align", "left"), ("padding", "6px 10px")]}]))
activity type why
Run a vulnerability scan Calculation No resources committed; you can ignore the results
Buy and deploy EDR ($200K) Decision Budget irrevocably allocated
Estimate annual loss expectancy Calculation A number on a slide; no action follows automatically
Terminate a vendor relationship Decision Switching costs are real and immediate
Update the risk register to 'High' Calculation The label changed; the exposure didn't
Accept residual risk (signed by CISO) Decision The acceptance is itself the irrevocable commitment: you forgo mitigation and own the exposure

2. Decision Quality vs Outcome Quality#

Good decisions can produce bad outcomes (you deployed MFA and still got breached). Bad decisions can produce good outcomes (you ignored the audit findings and nothing happened — this time). Evaluating decisions by their outcomes is called “resulting” (Annie Duke). The right question is: given what you knew at the time, was the process sound?

n_trials = 1000

breach_prob = 0.15
breach_cost = 2_000_000
control_cost = 80_000
control_reduction = 0.70

ev_deploy = control_cost + breach_prob * (1 - control_reduction) * breach_cost
ev_skip = breach_prob * breach_cost

# Common random numbers: both policies face the *same* breach realizations.
# One uniform draw decides whether a breach happens; the control only scales its impact.
breach = rng.random(n_trials) < breach_prob

good_process_costs = control_cost + breach * (1 - control_reduction) * breach_cost
gut_feel_costs = breach * breach_cost

gut_wins = np.sum(gut_feel_costs < good_process_costs)
overrun = (gut_feel_costs.sum() - good_process_costs.sum()) / good_process_costs.sum()

print(f"EV(deploy control): ${ev_deploy:,.0f}")
print(f"EV(skip control):   ${ev_skip:,.0f}")
print(f"Good process is cheaper in EV by ${ev_skip - ev_deploy:,.0f}/year")
print(f"Gut feel 'wins' on {gut_wins}/{n_trials} trials ({100*gut_wins/n_trials:.0f}%)")
print(f"Cumulative after {n_trials} trials:")
print(f"  Good process total: ${good_process_costs.sum():,.0f}")
print(f"  Gut feel total:     ${gut_feel_costs.sum():,.0f}")
print()
print(f"Expected: gut feel wins {1 - breach_prob:.0%} of bets yet costs "
      f"{ev_skip / ev_deploy - 1:.0%} more on average.")

EV(deploy control): $170,000
EV(skip control):   $300,000
Good process is cheaper in EV by $130,000/year
Gut feel 'wins' on 835/1000 trials (84%)
Cumulative after 1000 trials:
  Good process total: $179,000,000
  Gut feel total:     $330,000,000

Expected: gut feel wins 85% of bets yet costs 76% more on average.

3. Cumulative Advantage of Good Process#

On any single trial, gut feel usually “wins” – with common random numbers, both policies face the same breach draws, and on the ~85% of periods with no breach the control cost is pure overhead the skipper avoids. This is why anecdotal reasoning (“we didn’t deploy X and nothing happened”) is so persuasive: it is right most of the time.

But decisions compound. The same breaches that gut feel absorbs at full cost, good process absorbs at 30% of cost. Over 1,000 trials the gut-feel line accumulates the $2M spikes that the good-process line flattens, ending well above it.

cum_good = np.cumsum(good_process_costs)
cum_gut = np.cumsum(gut_feel_costs)

fig, ax = plt.subplots(figsize=(10, 4))
ax.plot(cum_good, color=DARK_BG, linewidth=1.5, label="Good process (deploy control)")
ax.plot(cum_gut, color=ACCENT, linewidth=1.5, label="Gut feel (skip control)")

breach_idx = np.where(gut_feel_costs > 0)[0]
ax.scatter(breach_idx, cum_gut[breach_idx], color=ACCENT, s=10, zorder=3)
spike = breach_idx[len(breach_idx) // 3]
ax.annotate("each spike = one uninsured\n\\$2M breach (+\\$2M)",
            xy=(spike, cum_gut[spike]),
            xytext=(spike + 60, cum_gut[spike] - 60e6),
            fontsize=8, color=ACCENT,
            arrowprops=dict(arrowstyle="->", color=ACCENT, lw=0.8))

ax.set_xlabel("Decision #")
ax.set_ylabel("Cumulative Cost ($)")
ax.set_title("Decision Quality vs Outcome Quality Over 1,000 Trials")
ax.legend(loc="upper left")
ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda x, _: f"${x/1e6:.1f}M"))
plt.tight_layout()
plt.show()
../_images/93a7e26a3f6b7f91e76965f979277226dc187a0e2959dcabfdbb8dcf71207959.png

Check it in your own environment.

  1. For the next control decision, write down before the outcome is known what you knew, what you expected, and why you chose.

  2. When reviewing past decisions, judge them against that record, not against whether an incident happened.

  3. For controls that have “never fired”, ask what they are meant to prevent and how often that would be expected to occur, before calling them unnecessary.

4. Anxiety Reduction vs Decision Quality#

Many people don’t really care about making better bets. They care about reducing anxiety about their bets. The hidden optimization function in many security decisions is not minimizing expected loss — it’s minimizing the discomfort of uncertainty. This leads to over-investing in what is easy to defend — what peers buy, what a board can see — and under-investing in work that is harder to show, like configuration management, access reviews and segmentation.

controls = pd.DataFrame([
    {"control": "Next-gen firewall", "cost": 150_000, "risk_reduction": 30_000, "visibility": "High"},
    {"control": "EDR platform", "cost": 200_000, "risk_reduction": 120_000, "visibility": "High"},
    {"control": "Config management", "cost": 50_000, "risk_reduction": 95_000, "visibility": "Low"},
    {"control": "Access review automation", "cost": 40_000, "risk_reduction": 70_000, "visibility": "Low"},
    {"control": "Security awareness training", "cost": 30_000, "risk_reduction": 15_000, "visibility": "High"},
    {"control": "Network segmentation", "cost": 80_000, "risk_reduction": 85_000, "visibility": "Low"},
])
controls["roi"] = controls["risk_reduction"] / controls["cost"]

budget = 300_000

def greedy_pick(df):
    picks, spent = [], 0
    for _, row in df.iterrows():
        if spent + row["cost"] <= budget:
            picks.append(row["control"])
            spent += row["cost"]
    return picks

# ROI-greedy: highest ROI first. Not a knapsack solve: it skips any control that
# won't fit and continues, so it may leave budget unspent.
ev_portfolio = greedy_pick(controls.sort_values("roi", ascending=False))

# Anxiety-driven: only visible controls considered, most expensive first.
anx_portfolio = greedy_pick(
    controls[controls["visibility"] == "High"].sort_values("cost", ascending=False)
)

print("=== ROI-Greedy Portfolio ===")
ev_sub = controls[controls["control"].isin(ev_portfolio)]
print(f"Controls: {ev_portfolio}")
print(f"Total cost: ${ev_sub['cost'].sum():,} (of ${budget:,} budget; "
      f"${budget - ev_sub['cost'].sum():,} left unspent)")
print(f"Total risk reduction: ${ev_sub['risk_reduction'].sum():,}")

print(f"\n=== Anxiety-Driven Portfolio ===")
anx_sub = controls[controls["control"].isin(anx_portfolio)]
print(f"Controls: {anx_portfolio}")
print(f"Total cost: ${anx_sub['cost'].sum():,}")
print(f"Total risk reduction: ${anx_sub['risk_reduction'].sum():,}")

ev_rr, anx_rr = ev_sub["risk_reduction"].sum(), anx_sub["risk_reduction"].sum()
ev_cost, anx_cost = ev_sub["cost"].sum(), anx_sub["cost"].sum()
rr_gap, cost_gap = ev_rr - anx_rr, ev_cost - anx_cost

if rr_gap > 0:
    print(f"\nDifference: ${rr_gap:,} in risk reduction the anxiety portfolio gives up")
elif rr_gap < 0:
    print(f"\nDifference: ${-rr_gap:,} in risk reduction the ROI-greedy set gives up")
else:
    print("\nDifference: none -- both portfolios buy the same risk reduction")

if cost_gap < 0:
    spend = f"spends ${-cost_gap:,} less"
elif cost_gap > 0:
    spend = f"spends ${cost_gap:,} more"
else:
    spend = "spends the same"
print(f"The ROI-greedy set {spend}.")

=== ROI-Greedy Portfolio ===
Controls: ['Config management', 'Access review automation', 'Network segmentation', 'Security awareness training']
Total cost: $200,000 (of $300,000 budget; $100,000 left unspent)
Total risk reduction: $265,000

=== Anxiety-Driven Portfolio ===
Controls: ['EDR platform', 'Security awareness training']
Total cost: $230,000
Total risk reduction: $135,000

Difference: $130,000 in risk reduction the anxiety portfolio gives up
The ROI-greedy set spends $30,000 less.

5. Visualization: ROI by Visibility#

High-visibility controls (firewalls, EDR, training) appear in board decks and vendor pitches. Low-visibility controls (config management, access reviews, segmentation) require operational discipline and produce no screenshots for the quarterly report.

Yet the unglamorous controls often deliver more risk reduction per dollar. When portfolio selection is driven by “what can the board see” rather than “what reduces risk most efficiently,” the organization pays a premium for comfort.

fig, ax = plt.subplots(figsize=(9, 4))
colors_map = {"High": ACCENT, "Low": DARK_BG}
bar_colors = [colors_map[v] for v in controls["visibility"]]
order = controls.sort_values("roi", ascending=True).index

ax.barh([controls.loc[i, "control"] for i in order],
        [controls.loc[i, "roi"] for i in order],
        color=[bar_colors[i] for i in order],
        edgecolor="white")
ax.set_xlabel("ROI = risk reduction / cost (a benefit-cost ratio; above 1 pays for itself)")
ax.set_title("Control ROI: Red = High Visibility, Blue = Low Visibility")

high_patch = mpatches.Patch(color=ACCENT, label="High visibility")
low_patch = mpatches.Patch(color=DARK_BG, label="Low visibility")
ax.legend(handles=[high_patch, low_patch])
plt.tight_layout()
plt.show()
../_images/91ed622c8260ed6dc074e838f3176e9f6e65909de5b51ec0a4b1fdbc75ebfff0.png

6. Pitfalls#

  • “We did a risk assessment” is not a decision. Until resources move, nothing happened. Track decisions (what was deployed, decommissioned, or accepted), not assessments produced.

  • Resulting is everywhere in security. “We got breached therefore our security was bad” ignores that good process reduces probability, not eliminates it.

  • Anxiety-driven portfolios are expensive. Visible controls feel better but often have worse ROI than unglamorous operational improvements. If your portfolio is all “things the board can see,” question whether you optimized for risk reduction or for comfort.

  • The do-nothing baseline is a decision too. Choosing not to act is an irrevocable allocation — of risk acceptance. Make it explicit.