Link-Flap Recovery & Cold-Cache Warmup Slippage Playbook

Date: 2026-03-26
Category: research
Audience: quant execution operators running low-latency gateways on redundant network links

Why this note

A short link flap (L1/L2 down→up) is often treated as a harmless infra blip if packet loss quickly returns to normal.

In practice, execution cost can stay elevated after link recovery because multiple layers are still cold:

• neighbor cache / ARP-ND re-resolution,
• transport/session reuse miss (TLS/QUIC/FIX reconnect branches),
• queue and pacing state reset,
• per-destination route/cache re-learning,
• and backlog catch-up that bunches child orders.

Result: the system looks “up,” but is still in a costly warmup regime where stale-arrival and forced-urgency tails widen.

1) Cost decomposition with warmup state

For child decision at time (t):

[ \mathbb{E}[C_t] = \mathbb{E}[C_{base} \mid a_t, s_t] + \lambda_w \cdot \mathbb{E}[C_{warmup} \mid w_t] ]

Where:

• (C_{base}): spread + impact + fees + residual opportunity cost
• (w_t): post-flap warmup state
• (C_{warmup}): incremental cost from cold-path effects

Practical branch form:

[ C_{warmup} \approx p_{resolve}L_{resolve} + p_{reconnect}L_{reconnect} + p_{burst}L_{burst} + p_{deadline}L_{deadline} ]

• (p_{resolve}): probability of neighbor/session/path re-resolution delay
• (p_{reconnect}): probability of expensive reconnect branch (full handshake/logon)
• (p_{burst}): probability of catch-up burst after temporary stall
• (p_{deadline}): probability urgency convexity is triggered near schedule end

2) Observability contract (must-have fields)

A) Link and host-network telemetry

• link down/up timestamps with interface id
• carrier events, speed/duplex renegotiation duration
• RX/TX drops, pause/flow-control counters
• ARP/ND miss and resolution latency
• retransmit spikes during first N seconds after link-up

B) Session and gateway telemetry

• reconnect branch tag (warm reuse / resumed / full)
• logon/handshake latency bucket
• first-ACK-after-recovery latency
• first-fill-after-recovery latency
• backlog depth at flap start/end

C) Execution timeline telemetry

Per child order:

• decision timestamp
• send timestamp
• ack/reject timestamp
• fill timestamp
• feature age at decision and estimated age at arrival

Without explicit “post-flap epoch” tagging, postmortems usually blame market volatility for what was actually warmup-state slippage.

3) State machine for post-flap execution

Define four states:

1. STABLE — normal latency/queue/session behavior
2. RECOVER_COLD — link up but caches/sessions/path not yet stabilized
3. RECOVER_WARMING — key metrics improving, but tails still elevated
4. DEGRADED_REFLAP — repeated flaps or unstable recovery; high risk

Use hysteresis + minimum dwell time. Avoid STABLE re-entry on a single good 5s window.

4) Warmup Readiness Score (WRS)

Build an online score (WRS_t \in [0,1]):

[ WRS_t = \sigma(\beta_0 - \beta_1 NRL_t - \beta_2 RBL_t - \beta_3 HFL_t - \beta_4 BSI_t) ]

Where:

• NRL: neighbor-resolution latency percentile (p95)
• RBL: reconnect-branch loss (full/slow branch rate)
• HFL: first-hop latency inflation vs stable baseline
• BSI: backlog shock index (pending-queue and catch-up pressure)

Interpretation:

• (WRS < 0.4): RECOVER_COLD
• (0.4 \le WRS < 0.75): RECOVER_WARMING
• (WRS \ge 0.75) with dwell pass: STABLE

5) Execution policy by state

STABLE (GREEN)

• normal child sizing/routing policy

RECOVER_COLD (RED)

• cap child size and participation
• tighten signal-age admission (short-half-life alpha gates)
• avoid aggressive catch-up bursts
• prefer venues/sessions with confirmed warm branch
• enforce stricter timeout/retry budget ceilings

RECOVER_WARMING (AMBER)

• gradual size ramp-up (stepwise, not jump)
• maintain burst caps and spacing controls
• keep stale-arrival guard until dwell criteria met

DEGRADED_REFLAP (HARD RED)

• suspend fragile tactics
• switch to conservative completion-only mode
• throttle parent launch concurrency
• raise incident event + containment runbook

6) KPIs for production control

• WDI (Warmup Duration Index): link-up → STABLE dwell-passed time
• FAI (First-Action Inflation): p95(decision→ack) post-flap / stable baseline
• SAB (Stale-Arrival Burden): fraction of arrivals past alpha-age budget
• CBR (Catch-up Burst Ratio): burst volume in first recovery window / planned volume
• WCS (Warmup Cost Share): incremental IS attributable to non-STABLE states

Alert examples:

• WDI above weekly p95 for 3 events ⇒ investigate network/session warm paths
• SAB + CBR spike together ⇒ likely bursty re-entry causing self-impact
• high WCS with flat spread regime ⇒ infra warmup tax, not market toxicity

7) Validation ladder

1. Historical relabeling
   Label link-flap episodes and compare state-conditional slippage.

2. Counterfactual replay
   Replay with RED/AMBER controls to estimate avoided q95/q99 cost.

3. Drill-based chaos tests
   Controlled interface bounce in staging; verify WRS, state transitions, and rollback logic.

4. Shadow then canary
   Start with monitoring-only, then enable policy on small flow slice.

Success criterion: lower tail slippage and lower catch-up burst intensity without unacceptable completion-risk increase.

8) 10-day implementation plan

Days 1-2
Add explicit flap epoch tags and reconnect branch fields.

Days 3-4
Compute WDI/FAI/SAB/CBR dashboards.

Days 5-6
Ship first WRS model + hysteresis state machine.

Days 7-8
Integrate RED/AMBER controls in shadow mode.

Day 9
Canary with strict rollback threshold.

Day 10
Finalize runbook + postmortem template and ownership map.

Common mistakes

1. Declaring recovery at “packet loss back to zero.”
   Warmup cost can persist well after link-up.

2. No reconnect branch labels.
   Without warm/resumed/full tags, model attribution collapses.

3. Instant full-capacity re-entry.
   This often creates self-inflicted catch-up bursts.

4. Single-metric health gates.
   Mean latency alone misses stale-arrival and burst dynamics.

5. Ignoring repeated-flap clustering.
   Reflap episodes should escalate to hard containment quickly.

Bottom line

Link flaps are not binary outages; they are state transitions with warmup economics.

If you explicitly model post-flap warmup (cold → warming → stable), attach execution controls to that state, and track warmup-attributed cost, you can remove a recurring hidden bps leak from live operations.