Exchange Outage + Market Data Gap Incident Playbook (Automated Trading)

Why this matters

Most automated strategies are not killed by “bad alpha,” but by bad state awareness during infrastructure incidents:

• exchange order gateway partially down,
• market-data feed stale or fragmented,
• cancel/replace accepted but not acknowledged,
• post-trade/drop-copy delayed,
• reconnect storms after reopen.

If your system keeps trading with false assumptions, slippage and inventory risk can explode faster than normal model controls can react.

This playbook is a practical framework for detect → classify → contain → recover.

1) Incident taxonomy (keep it operational)

Use a small, unambiguous taxonomy so humans and bots share the same language.

A. Connectivity incidents

• Order-path down: cannot place new orders
• Cancel-path degraded: new orders work, cancels delayed/failing
• Ack ambiguity: requests sent, final exchange state uncertain

B. Market-data incidents

• Stale top-of-book: timestamps lag beyond threshold
• Depth truncation: only partial levels available
• Cross-feed divergence: primary vs backup feed mismatch
• Sequence gaps: packet loss/reorder beyond recovery window

C. Post-trade incidents

• Drop-copy delay: fills arrive late
• Position reconciliation lag: broker/exchange position differs from internal ledger

D. Market-structure incidents

• Volatility interruption / auction mode transitions
• Venue halt / symbol-specific pause
• Reopen auction disorder

2) Four-state operating model

Define one global incident state machine and make every strategy consume it.

1. NORMAL
   • full trading allowed
2. DEGRADED
   • reduced size, reduced aggression, tighter symbol whitelist
3. CONTAINMENT
   • no new risk; cancels-only (or flatten-only)
4. RECOVERY
   • staged re-enable with probation windows

State transition principles

• escalate fast, de-escalate slow,
• any “unknown position” immediately implies CONTAINMENT,
• recovery requires both transport health and position consistency.

3) Health signals and hard thresholds

Avoid vague “looks unstable.” Use explicit numeric gates.

Order channel

• New-order reject/timeout rate (1m, 5m)
• Cancel ack p95/p99 latency
• Orphan-order count (sent but unresolved)

Data channel

• Feed age (ms since last valid update)
• Sequence gap count per minute
• Primary/backup mid-price divergence (bps)

Reconciliation

• Internal vs broker position delta
• Fill-to-drop-copy lag p95
• Unmatched execution reports count

Example thresholds (template)

• Feed age > 500ms for > 3s → DEGRADED
• Cancel ack p99 > 2s for > 30s → CONTAINMENT
• Position delta != 0 for > 10s during incident → CONTAINMENT lock
• Divergence > 5 bps on liquid symbols for > 5s → DEGRADED

Tune per venue and symbol liquidity bucket.

4) Containment controls (ordered by priority)

When incident state >= CONTAINMENT:

1. Block new parent orders
2. Stop child slicing engines
3. Cancel live passive quotes (if cancel path still available)
4. Freeze strategy handoff queues
5. Enable flatten-only mode for pre-approved symbols
6. Cut max notional / inventory caps to emergency profile
7. Pin routing to safest venue set (or single venue)

If cancel path is impaired, prioritize:

• reducing further submissions to zero,
• tracking uncertain live orders as worst-case exposure,
• using hedge venue only if explicitly approved in policy.

5) “Unknown position” protocol (non-negotiable)

Unknown position is the highest-severity operational state.

Trigger conditions

• conflicting fill reports,
• drop-copy gap with active order flow,
• reconnect after prolonged ack blackout.

Required actions

• immediate CONTAINMENT,
• freeze all alpha-driven signals,
• snapshot all order IDs + local state,
• reconcile with broker/exchange authoritative source,
• only allow flattening after delta confidence is restored.

Never resume normal strategy flow before reconciliation confidence clears threshold.

6) Recovery runbook (staged, not binary)

Phase R1: Transport recovery

• stable heartbeats on order + data channels
• no unresolved sequence gaps
• cancel latency back under threshold

Phase R2: Ledger recovery

• position parity validated
• orphan orders resolved or explicitly quarantined
• PnL/book snapshots consistent across systems

Phase R3: Soft restart

• smallest risk limits
• whitelist only top-liquidity symbols
• maker-only or low-aggression tactics
• probation timer (e.g., 15–30 min)

Phase R4: Full resume

• all KPIs stable through probation
• incident commander sign-off
• post-incident checklist opened automatically

7) Architecture patterns that reduce blast radius

• Global kill switch service (independent process, minimal dependencies)
• Strategy-level circuit breakers + global override precedence
• Dual-feed validator (primary/secondary consistency check)
• Write-ahead order intent log for replay/audit
• Deterministic incident snapshots (state dump + timeline)
• Read-only fallback UI for manual supervision during control-plane issues

Design rule: if your kill switch depends on the same failing component, it is not a kill switch.

8) Metrics to review after every incident

• Time to detect (TTD)
• Time to containment (TTC)
• Time to reconcile positions (TTR-pos)
• Incident slippage vs baseline
• Uncertain exposure peak notional
• Number of manual interventions
• Recovery probation violations

Track these as a trend, not one-off anecdotes.

9) Drill cadence (tabletop + simulation)

At least monthly, simulate:

1. stale data while order channel still active,
2. cancel-path degradation with active inventory,
3. drop-copy lag + reconnect storm,
4. halt/reopen auction sequence.

A runbook that was never drilled is just documentation.

10) Minimal implementation checklist

• Incident state machine shared across all executors
• Numeric health thresholds per venue/symbol bucket
• Hard block for new risk in CONTAINMENT
• Unknown-position protocol wired and tested
• Auto-generated incident timeline + snapshots
• Recovery requires transport + ledger dual confirmation
• Post-incident review template linked to metrics

Closing note

In automated trading, outages are not edge cases; they are scheduled surprises. Your competitive edge is not just alpha quality, but how safely and quickly your system degrades under uncertainty.