Normalization of Deviance – When Exceptions Become the Rule

Diane Vaughan coined “normalization of deviance” studying the Challenger disaster: repeated boundary violations that produce no immediate consequence get reinterpreted as acceptable. The O-ring erosion was a known anomaly, but because previous flights survived it, the anomaly became the new baseline.

The same pattern plays out in security. Firewall exceptions that accumulate. “Temporary” admin access that becomes permanent. Alert thresholds raised to reduce noise until the alerts no longer detect anything meaningful. Each decision is locally rational – the system kept working, so the deviation must be tolerable.

This notebook simulates the drift, quantifies the hidden risk it creates, and shows why resetting to policy – though politically expensive – is almost always cheaper than the alternative.

Setup

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from decision_security.synth import make_rng
from decision_security.montecarlo import simulate_aggregate_losses, make_lognormal_severity

rng = make_rng(42)

plt.rcParams.update({
    "font.family": "serif",
    "font.size": 10,
    "axes.labelsize": 11,
    "axes.titlesize": 12,
    "xtick.labelsize": 9,
    "ytick.labelsize": 9,
    "legend.fontsize": 9,
    "figure.dpi": 150,
    "axes.spines.top": False,
    "axes.spines.right": False,
})

PRIMARY = "#1A1A1A"
ACCENT = "#E74C3C"
DARK_BG = "#34495E"
LIGHT_GRAY = "#95A5A6"

1. The Drift

Consider a firewall exception policy: at most 5 active exceptions. Each week there is a 55% chance a new exception is requested (and granted – because it is “just one more”) and an 8% chance that an existing exception is reviewed and closed. No incident fires for many weeks. The count drifts upward, and because nothing bad happens, the drift is reinterpreted as normal.

The unexpected becomes expected, and the expected becomes accepted. – paraphrasing Vaughan (1996)

n_weeks = 52
policy_max = 5
p_new = 0.55        # weekly probability of a new exception request
p_close = 0.08      # weekly probability one exception is reviewed and closed

rng_drift = make_rng(42)
exceptions = np.zeros(n_weeks, dtype=int)
exceptions[0] = 3

for w in range(1, n_weeks):
    current = exceptions[w - 1]
    added = 1 if rng_drift.random() < p_new else 0
    closed = 1 if (current > 0 and rng_drift.random() < p_close) else 0
    exceptions[w] = max(0, current + added - closed)

weeks = np.arange(n_weeks)

first_breach_week = int(np.argmax(exceptions > policy_max))

fig, ax = plt.subplots(figsize=(10, 4.5))
ax.fill_between(weeks, 0, exceptions, alpha=0.15, color=DARK_BG)
ax.plot(weeks, exceptions, color=DARK_BG, linewidth=1.8, label="Active exceptions")
ax.axhline(policy_max, color=ACCENT, linestyle="--", linewidth=1.5,
           label=f"Policy threshold ({policy_max})")

ax.annotate(f"Week {first_breach_week}: first threshold breach",
            xy=(first_breach_week, exceptions[first_breach_week]),
            xytext=(first_breach_week + 6, exceptions[first_breach_week] + 6),
            arrowprops=dict(arrowstyle="->", color=ACCENT, lw=1.2),
            fontsize=8, color=ACCENT)
ax.annotate("Week 15: nobody notices",
            xy=(15, exceptions[15]),
            xytext=(19, exceptions[15] + 5),
            arrowprops=dict(arrowstyle="->", color=LIGHT_GRAY, lw=1.2),
            fontsize=8, color=LIGHT_GRAY)
ax.annotate("Week 30: 'this is just\nhow we operate'",
            xy=(30, exceptions[30]),
            xytext=(34, exceptions[30] + 5),
            arrowprops=dict(arrowstyle="->", color=PRIMARY, lw=1.2),
            fontsize=8, color=PRIMARY)

ax.set_xlabel("Week")
ax.set_ylabel("Active firewall exceptions")
ax.set_title("The Drift: Firewall Exceptions Over One Year")
ax.legend(loc="upper left")
ax.set_xlim(0, n_weeks - 1)
ax.set_ylim(0, None)
plt.tight_layout()
plt.show()

print(f"\nPolicy limit: {policy_max}")
print(f"Exceptions at week 26: {exceptions[26]}")
print(f"Exceptions at week {n_weeks - 1}: {exceptions[-1]}")
print(f"Peak: {exceptions.max()} (week {exceptions.argmax()})")

Policy limit: 5
Exceptions at week 26: 13
Exceptions at week 51: 26
Peak: 26 (week 50)

2. Near Misses as Normalization Fuel

The reason normalization of deviance persists is that bad outcomes are probabilistic. Each week without an incident reinforces the belief that the current state is safe. But the probability of an incident is climbing – we just haven’t sampled the bad outcome yet.

Near misses – weeks where the true probability was high but nothing happened – are the most dangerous data points. They teach the organization the wrong lesson: that the current level of deviance is tolerable.

In this model we treat high-risk incident-free weeks as near misses; in practice a near miss is an observable event that almost caused harm.

# Weekly incident probability grows quadratically with active exceptions: each new
# exception adds attack paths that combine with the ones already open. WEEKLY_C is an
# illustrative anchor: 25 exceptions gives ~19% per week. Section 4 reuses it.
WEEKLY_C = 0.0003
p_incident = WEEKLY_C * exceptions**2
p_incident_pct = p_incident * 100

rng_incidents = make_rng(29)
incident_rolls = rng_incidents.random(n_weeks)
incidents = incident_rolls < p_incident
incident_weeks = weeks[incidents]

fig, ax = plt.subplots(figsize=(10, 4.5))

ax.plot(weeks, p_incident_pct, color=ACCENT, linewidth=2,
        label="True incident probability (%)")
ax.fill_between(weeks, 0, p_incident_pct, alpha=0.1, color=ACCENT)

if len(incident_weeks) > 0:
    ax.scatter(incident_weeks, p_incident_pct[incidents],
               color=ACCENT, s=80, zorder=5, marker="X",
               edgecolors=PRIMARY, linewidth=0.8,
               label=f"Actual incident ({len(incident_weeks)} total)")

# Near misses: p > 2% but no incident
near_miss = (p_incident > 0.02) & ~incidents
near_miss_weeks = weeks[near_miss]
if len(near_miss_weeks) > 0:
    ax.scatter(near_miss_weeks, p_incident_pct[near_miss],
               color=LIGHT_GRAY, s=40, zorder=4, marker="o",
               edgecolors=DARK_BG, linewidth=0.6,
               label=f"Near miss (p>2%, no incident) [{len(near_miss_weeks)}]")

# Shade the incident-free stretch of this run
first_incident_week = int(incident_weeks.min()) if len(incident_weeks) > 0 else n_weeks
ax.axvspan(0, first_incident_week, alpha=0.05, color=DARK_BG)
ax.text(first_incident_week / 2, max(p_incident_pct) * 0.35,
        f"In this run, the first incident\nlands at week {first_incident_week}\n"
        f"(one realization -- not a guarantee)",
        ha="center", fontsize=8, color=DARK_BG, style="italic")

ax.set_xlabel("Week")
ax.set_ylabel("Incident probability (%)")
ax.set_title("Rising Incident Probability vs Observed Incidents")
ax.legend(loc="upper left", fontsize=8)
ax.set_xlim(0, n_weeks - 1)
plt.tight_layout()
plt.show()

print(f"Weeks with p(incident) > 2%: {(p_incident > 0.02).sum()}")
print(f"Weeks with p(incident) > 5%: {(p_incident > 0.05).sum()}")
print(f"Actual incidents: {incidents.sum()} (first at week {first_incident_week})")
print(f"Near misses (p>2%, no incident): {near_miss.sum()}")
print("\nAbsence of incidents is not proof of safety -- it is sampling from a distribution.")

Weeks with p(incident) > 2%: 34
Weeks with p(incident) > 5%: 28
Actual incidents: 4 (first at week 33)
Near misses (p>2%, no incident): 30

Absence of incidents is not proof of safety -- it is sampling from a distribution.

3. Threshold Erosion

A SOC team sets an alert threshold at 100 events/day. It looks defensible at first, but even on day one it sits below the attack-signal median: about a quarter of real attacks (~25%) fall beneath it from the start. Then alert fatigue creeps in. Each month, the team raises the threshold by 10% to reduce noise. After 12 months, the threshold has more than tripled (3.1x).

The problem: real attack signals do not grow with the threshold. As the threshold rises, the fraction of actual attacks that fall below it – the detection blindspot – grows.

from scipy import stats

n_months = 12
initial_threshold = 100
monthly_increase = 0.10

months = np.arange(n_months + 1)
thresholds = initial_threshold * (1 + monthly_increase) ** months

# Attack signals: lognormal with median 150 events/day (median = exp(mu))
attack_mu = np.log(150)
attack_sigma = 0.6
attack_dist = stats.lognorm(s=attack_sigma, scale=np.exp(attack_mu))

# Blindspot: fraction of attacks below the threshold
blindspot = np.array([attack_dist.cdf(t) for t in thresholds])

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

ax = axes[0]
ax.plot(months, thresholds, "o-", color=ACCENT, linewidth=2,
        markersize=5, label="Alert threshold")
ax.axhline(150, color=DARK_BG, linestyle=":", linewidth=1,
           label="Attack median (150 events/day)")
ax.fill_between(months, attack_dist.ppf(0.1), attack_dist.ppf(0.9),
                alpha=0.1, color=DARK_BG, label="Attack signal 10th-90th %ile")
ax.set_xlabel("Month")
ax.set_ylabel("Events / day")
ax.set_title("Threshold Erosion Over 12 Months")
ax.legend(fontsize=8)

ax = axes[1]
ax.bar(months, blindspot * 100, color=DARK_BG, alpha=0.7, edgecolor="none")
ax.plot(months, blindspot * 100, "o-", color=ACCENT, linewidth=1.5, markersize=4)
for i in [0, 6, 12]:
    ax.text(i, blindspot[i] * 100 + 2, f"{blindspot[i]:.0%}",
            ha="center", fontsize=8, color=ACCENT, fontweight="bold")
ax.set_xlabel("Month")
ax.set_ylabel("Detection blindspot (%)")
ax.set_title("Fraction of Attacks Below Threshold")
ax.set_ylim(0, 100)

plt.tight_layout()
plt.show()

threshold_growth = thresholds[12] / thresholds[0]
print(f"Month 0: threshold = {thresholds[0]:.0f}, blindspot = {blindspot[0]:.1%}")
print(f"Month 6: threshold = {thresholds[6]:.0f}, blindspot = {blindspot[6]:.1%}")
print(f"Month 12: threshold = {thresholds[12]:.0f}, blindspot = {blindspot[12]:.1%}")
print(f"\nThe threshold more than tripled ({threshold_growth:.1f}x). The blindspot grew"
      f" from {blindspot[0]:.0%} to {blindspot[12]:.0%}.")

Month 0: threshold = 100, blindspot = 25.0%
Month 6: threshold = 177, blindspot = 60.9%
Month 12: threshold = 314, blindspot = 89.1%

The threshold more than tripled (3.1x). The blindspot grew from 25% to 89%.

Check it in your own environment.

1. Pull the change history of your alert thresholds and exceptions for the past year, and compare today’s values with the original ones, not with last month’s.

2. For each threshold, write down what attack activity it is meant to catch, and test whether it still would.

3. Count exceptions that were granted as temporary and are still open, and how old they are.

4. The Cost of Resetting

Resetting to policy is politically expensive. Re-approving exceptions means paperwork, disruption, and difficult conversations. But deferred resets compound risk.

We model the decision: reset firewall exceptions to the policy limit of 5 (at a known cost of ~$50K in disruption) versus continue operating at the peak exception count the drift reached in Section 1 (26 in this run). We use a compound Poisson model where frequency scales quadratically with exception count, with the same weekly coefficient as Section 2.

We assume the reset is paired with enforcement that holds the count at the policy limit for the year; without it, the Section 1 dynamics would rebuild the drift.

n_sims = 10_000
reset_cost = 50_000

# meanlog=12: median around $160K; sdlog=1.5: heavy right tail
sev = make_lognormal_severity(meanlog=12, sdlog=1.5)

# Inaction keeps the realized peak from the Section 1 drift
drift_peak = int(exceptions.max())

# Annual frequency: Section 2's weekly rate x 52 weeks
lam_5 = WEEKLY_C * policy_max**2 * 52
lam_peak = WEEKLY_C * drift_peak**2 * 52

losses_5 = simulate_aggregate_losses(n_sims, lam_5, sev, rng=make_rng(42))
losses_peak = simulate_aggregate_losses(n_sims, lam_peak, sev, rng=make_rng(43))

total_reset = reset_cost + losses_5
total_inaction = losses_peak

fig, ax = plt.subplots(figsize=(10, 5))

bins = np.linspace(0, np.percentile(total_inaction, 99), 80)
ax.hist(total_reset, bins=bins, alpha=0.6, color=DARK_BG,
        label=f"Reset to policy ({policy_max} exc.) + ${reset_cost/1e3:.0f}K cost", edgecolor="none")
ax.hist(total_inaction, bins=bins, alpha=0.6, color=ACCENT,
        label=f"Continue with {drift_peak} exceptions", edgecolor="none")

med_reset = np.median(total_reset)
med_inaction = np.median(total_inaction)
ax.axvline(med_reset, color=DARK_BG, linestyle="--", linewidth=1.5,
           label=f"Median (reset): ${med_reset:,.0f}")
ax.axvline(med_inaction, color=ACCENT, linestyle="--", linewidth=1.5,
           label=f"Median (inaction): ${med_inaction:,.0f}")

ax.set_xlabel("Annual cost ($)")
ax.set_ylabel("Frequency")
ax.set_title("Cost of Resetting vs Cost of Inaction")
ax.legend(fontsize=8)
plt.tight_layout()
plt.show()

print(f"Annual frequency at {policy_max} exceptions:  lambda = {lam_5:.1f}")
print(f"Annual frequency at {drift_peak} exceptions: lambda = {lam_peak:.1f}")
print()
print(f"{'Metric':<20} {f'Reset + ${reset_cost/1e3:.0f}K':>16} {'Inaction':>16}")
print(f"{'-'*52}")
for label, q in [("Mean", None), ("Median (p50)", 0.5), ("p90", 0.9), ("p95", 0.95)]:
    if q is None:
        v_r, v_i = total_reset.mean(), total_inaction.mean()
    else:
        v_r = np.quantile(total_reset, q)
        v_i = np.quantile(total_inaction, q)
    print(f"{label:<20} ${v_r:>14,.0f} ${v_i:>14,.0f}")

# Independent draws, so this estimates an unpaired P(inaction > reset)
pct_inaction_worse = (total_inaction > total_reset).mean()
print(f"\nInaction costs more than resetting in {pct_inaction_worse:.1%} of simulated years.")
print(f"In the other {1 - pct_inaction_worse:.1%}, inaction was cheaper: "
      f"the lucky years that keep the drift going.")

mean_reset = total_reset.mean()
mean_inaction = total_inaction.mean()
print(f"\nExpected annual cost: ~${mean_inaction/1e6:.2f}M under inaction vs "
      f"~${mean_reset/1e6:.2f}M after resetting (~{mean_inaction/mean_reset:.0f}x),")
print("driven almost entirely by frequency, since both share the same severity curve.")

Annual frequency at 5 exceptions:  lambda = 0.4
Annual frequency at 26 exceptions: lambda = 10.5

Metric                   Reset + $50K         Inaction
----------------------------------------------------
Mean                 $       246,580 $     5,221,274
Median (p50)         $        50,000 $     4,035,767
p90                  $       474,560 $     9,870,049
p95                  $       984,660 $    13,002,077

Inaction costs more than resetting in 98.7% of simulated years.
In the other 1.3%, inaction was cheaper: the lucky years that keep the drift going.

Expected annual cost: ~$5.22M under inaction vs ~$0.25M after resetting (~21x),
driven almost entirely by frequency, since both share the same severity curve.

5. Organizational Indicators

Normalization of deviance is a cultural phenomenon. It shows up in the language people use, the decisions they defer, and the way they frame past outcomes. A diagnostic scorecard: eight warning signs that an organization is deep in the drift. Each is scored on a 1-5 scale (1 = not present, 5 = pervasive).

indicators = [
    '"We\'ve always done it this way"',
    '"It hasn\'t failed yet"',
    '"The policy is unrealistic"',
    '"We don\'t have time to fix it"',
    '"Everyone does this"',
    '"The risk is theoretical"',
    '"We\'ll fix it after the next release"',
    '"The auditors didn\'t flag it"',
]

risk_weights = np.array([3, 5, 4, 3, 4, 5, 2, 3], dtype=float)
risk_weights /= risk_weights.max()  # scale to 0-1

# Hypothetical organization scores (1-5 scale)
org_scores = np.array([4, 5, 3, 4, 3, 5, 2, 4], dtype=float)

weighted = org_scores * risk_weights

sort_idx = np.argsort(weighted)
sorted_indicators = [indicators[i] for i in sort_idx]
sorted_weighted = weighted[sort_idx]
sorted_scores = org_scores[sort_idx]

fig, ax = plt.subplots(figsize=(10, 5))

colors = [ACCENT if w > 3.5 else DARK_BG if w > 2 else LIGHT_GRAY
          for w in sorted_weighted]
bars = ax.barh(range(len(sorted_indicators)), sorted_weighted,
               color=colors, edgecolor="none", alpha=0.8)

for i, (w, s) in enumerate(zip(sorted_weighted, sorted_scores)):
    ax.text(w + 0.1, i, f"{w:.1f}  (score {s:.0f})", va="center", fontsize=8, color=PRIMARY)

ax.set_yticks(range(len(sorted_indicators)))
ax.set_yticklabels(sorted_indicators, fontsize=9)
ax.set_xlabel("Weighted risk score")
ax.set_title("Normalization of Deviance: Diagnostic Scorecard")
ax.set_xlim(0, 6)

from matplotlib.patches import Patch
legend_elements = [
    Patch(facecolor=ACCENT, alpha=0.8, label="High concern"),
    Patch(facecolor=DARK_BG, alpha=0.8, label="Moderate concern"),
    Patch(facecolor=LIGHT_GRAY, alpha=0.8, label="Lower concern"),
]
ax.legend(handles=legend_elements, loc="lower right", fontsize=8)

plt.tight_layout()
plt.show()

total_score = weighted.sum()
max_possible = (5 * risk_weights).sum()
print(f"Total weighted score: {total_score:.1f} / {max_possible:.1f}")
min_possible = risk_weights.sum()  # every indicator scored 1 ("not present")
index = (total_score - min_possible) / (max_possible - min_possible)
print(f"Normalization index: {index:.0%} (0% = all scores 1, 100% = all scores 5)")

Total weighted score: 22.8 / 29.0
Normalization index: 73% (0% = all scores 1, 100% = all scores 5)

6. Pitfalls

Near misses reinforce normalization instead of warning. Near misses should trigger investigation; instead they build confidence.

Success teaches the wrong lesson. The absence of failure is not the same as the presence of safety. A system that has never failed may simply be a system that has not yet been tested by the right conditions.

Resets are politically expensive, so they get deferred. The known, visible cost of resetting is weighed against the unknown, probabilistic cost of continuing. People reliably avoid the certain small cost and gamble on the uncertain large one – even when the expected value clearly favors resetting (Kahneman & Tversky, 1979).

The drift is invisible to those inside it. Each step is small. Nobody decided to have 26 firewall exceptions. It happened one exception at a time, each one locally reasonable.

Dissenting voices get labeled as alarmist. When someone raises the concern, the response is: “We’ve been operating this way for months and nothing has happened.” The data (no incidents) appears to support the status quo. The dissenter cannot point to a concrete failure – only to a probability.

Near misses and small incidents should not be regarded or normalized as they can be indicators of much larger problems.

---

Part 2.2 of the Security Decision Science notebook series.