NVMe Thermal-Throttling Journal-Latency Slippage Playbook

When Storage Heat Turns Into Queue Loss, Stale Decisions, and Tail-Bps Bleed

Why this note: Many execution stacks assume storage is “fast enough.” In production, NVMe thermal throttling can inflate fsync/log-commit latency, create dispatch gaps, and silently leak slippage through stale arrivals and late catch-up aggression.

1) Failure Mode in One Sentence

When NVMe enters thermal-throttle states, persistence latency becomes bursty; order decisions reach the wire later, passive queue edge decays, and routers overpay in final buckets.

2) Where the Hidden Tax Appears

Typical low-latency path:

1. market data / model decision
2. risk + intent ledger write (WAL/journal)
3. child-order submit
4. ACK/fill reconciliation

If step (2) stretches from sub-ms to multi-ms tails during throttle windows, step (3) shifts right in time. That delay compounds through:

• stale quote interaction (missed passive or worse aggressive fill)
• queue-age decay (lost priority from later placement)
• deadline convexity (late residual -> panic participation)

3) Practical Branch Model

Let decision-time action be (a_t) and thermal state (z_t \in {\text{cool}, \text{warm}, \text{throttle}}).

Expected slippage:

[ \mathbb{E}[IS_t(a_t)] = \sum_{z_t} P(z_t \mid x_t);\mathbb{E}[IS_t(a_t) \mid z_t] ]

Split conditional cost into explicit parts:

[ IS = C_{spread/impact} + C_{delay}(\Delta\tau) + C_{queue_loss}(\Delta\tau) + C_{deadline}(r_t) ]

• (\Delta\tau): added decision->wire latency from storage path
• (r_t): remaining inventory fraction

A compact delay penalty approximation:

[ C_{delay}(\Delta\tau) \approx \lambda_t \cdot \Delta\tau ]

where (\lambda_t) is short-horizon alpha-decay / adverse-drift slope (bps per ms).

4) Thermal Hazard Nowcast (Operator-Friendly)

Start with a logistic hazard for entering active throttling in the next horizon (H):

[ P(z_{t+H}=\text{throttle}) = \sigma(\beta_0 + \beta_1 T_{comp} + \beta_2 \dot{T} + \beta_3 qd + \beta_4 bw_{write} + \beta_5 fsync_{p99}) ]

Features:

• T_comp: NVMe composite temperature
• dT_dt: short-window temperature slope
• queue_depth / write bandwidth
• fsync_p99_ms and journal_commit_p99_ms
• optional: ambient temperature / chassis fan state

This is enough to trigger pre-throttle controls before hard slowdown.

5) Telemetry Contract (Must Have)

Storage + thermal

• nvme_composite_temp_c
• nvme_temp_sensor_[n]_c
• thermal_throttle_status
• thermal_throttle_time_ms (or cumulative counter)
• device_write_bw_mb_s, queue_depth

Persistence path

• wal_append_ms_p50/p95/p99
• fsync_ms_p50/p95/p99
• journal_backlog_bytes
• log_flush_interval_ms

Execution linkage

• decision_to_send_ms
• ack_latency_ms
• queue_age_at_entry_ms
• realized_is_bps, markout_1s/5s
• deadline_residual_ratio

Without explicit decision->send timing, storage heat stays invisible and gets mislabeled as "market noise."

6) KPIs for This Specific Failure Class

1. TTAR (Thermal Throttle Active Ratio)
   • fraction of wall time with active throttle state
2. JTL (Journal Tail Lift)
   • fsync_p99 / fsync_p50 (or vs cool-state baseline)
3. DGI (Dispatch Gap Inflation)
   • decision_to_send_p99 / cool_baseline_p99
4. QLD (Queue Loss Delta)
   • passive fill-rate drop conditional on similar spread/imbalance
5. LCP (Late Catch-up Premium)
   • incremental bps paid in final schedule bucket vs baseline

If JTL and DGI rise before IS tails widen, you have actionable early warning.

7) Live Control States

STORAGE_CLEAN

• normal policy
• full tactic set enabled

STORAGE_WARM (pre-throttle risk rising)

• reduce non-critical sync writes on hot path
• cap message burst size
• tighten passive timeout (avoid stale resting intent)

THROTTLE_ACTIVE

• prioritize completion reliability over queue-gambling
• shift scoring weight from passive-edge to delay-sensitive expected cost
• lower maximum order-amend churn (avoid additional journal pressure)

SAFE_CONTAIN

• hard guard when DGI/JTL exceed limits
• activate simplified execution mode (fewer tactics, deterministic pacing)
• emit explicit incident reason code for post-trade attribution

Use hysteresis and minimum dwell times; otherwise systems flap between warm/active states.

8) Mitigation Ladder (Infra + Model + Policy)

Infra layer

1. Separate WAL/log device from high-throughput scratch I/O
2. Ensure sustained airflow/heatsink margin for NVMe controllers
3. Keep safe write cache + flush semantics explicit (no accidental durability drift)
4. Preemptively rebalance write-heavy background jobs away from trading windows

Modeling layer

1. Add thermal-state features and interaction terms to slippage model
2. Calibrate conditional tails by thermal regime, not global average
3. Track coverage separately for cool/warm/throttle strata

Policy layer

1. Add thermal-aware penalty to action score: [ score(a)=\mathbb{E}[IS]+\lambda,CVaR_q + \gamma,P(\text{throttle})\cdot \text{delay_sensitivity}(a) ]
2. Disable tactics that depend on ultra-fresh queue position during active throttle
3. Increase urgency gradually with bounded acceleration to avoid burst-panic loops

9) Practical On-Host Checks (Linux)

# NVMe health / temperature
nvme smart-log /dev/nvme0

# Optional vendor-neutral SMART view
smartctl -a /dev/nvme0

# Observe storage latency + queue pressure
iostat -x 1
pidstat -d 1

# Correlate with app-level decision->send + fsync histograms
# (export from your tracing/metrics stack)

Treat these as correlation tools; final attribution should come from synchronized app + device timelines.

10) Rollout Plan

1. Shadow phase (1–2 weeks)
   • log thermal hazard and derived control state, but do not change routing
2. Canary phase
   • enable STORAGE_WARM/THROTTLE_ACTIVE rules on limited symbols/notional
3. Promotion gate
   • require lower tail IS and stable completion under heat stress
4. Kill-switch
   • instant fallback to baseline policy if completion risk increases beyond threshold

11) Fast Checklist

[ ] Wire NVMe thermal + fsync/journal tails into execution telemetry
[ ] Model throttle hazard and conditional slippage tails
[ ] Add STORAGE_CLEAN/WARM/THROTTLE_ACTIVE/SAFE_CONTAIN states
[ ] Penalize delay-sensitive tactics when throttle probability rises
[ ] Separate infra fixes (cooling/write-path isolation) from policy fixes
[ ] Gate rollout on q95/q99 IS and completion, not mean-only improvement

References

• NVM Express Base Specification (Thermal Management features, HCTM/TMT controls).
• linux-nvme nvme-cli documentation (smart-log, health/temperature counters).
• Dean, J., Barroso, L. A. (2013), The Tail at Scale.
• Yan, M. et al. (FAST'17), Tiny-Tail Flash: Near-Perfect Elimination of Garbage-Collection Tail Latencies in NAND SSDs.

TL;DR

NVMe heat is not just a hardware concern: it changes persistence latency, which changes order timing, which changes execution cost. If thermal state is absent from your slippage model and control loop, you are probably paying hidden tail bps during hot periods.