Introduction

Your code is perfect. Your network is perfect. But once every hour, your trading algorithm stalls for 200 microseconds. You check top. You check perf. You see nothing. You have been hit by the BIOS.

This lesson explores the “Hidden Physics” of the x86 platform-events that happen underneath the Operating System.

---

System Management Interrupts (SMI)

The CPU has a “God Mode” called High Priority Interrupts. Usually used for thermal throttling or ECC Ram error correction. When an SMI fires, the CPU pauses the OS entirely (SMM Mode) to run firmware code. Linux does not even know it happened.

Diagnosis:

# Check turbostat for SMI count
sudo turbostat --show Core,CPU,SMI
# If SMI > 0 on your isolated cores, you have a BIOS problem.

Fix: Disable Power Management in BIOS.

---

The Physics: C-States

To save power, the CPU shuts down parts of the Core when idle.

• C0: Awake.
• C1: Halted (Clock stopped).
• C6: Deep Sleep (Cache flushed, Voltage off).

Wakeup Physics: Waking from C6 takes 100µs - 200µs. If your strategy waits for a market packet (Idle), and the packet arrives, you are 200µs late.

Fix: Force C0 Boot Parameter: processor.max_cstate=1 intel_idle.max_cstate=0 or use cpupower idle-set -D 0.

---

Turbo Boost: Frequency Jitter

Intel CPUs boost frequency when thermal headroom is available (e.g. 2.4GHz -> 5.0GHz). Problem: The frequency changes dynamically. A 100-cycle operation takes $X$ ns at 5GHz, but $2X$ ns at 2.4GHz. Jitter = Unpredictability.

Fix: Lock Frequency. HFT servers run at a static frequency (e.g., 4.5GHz Constant).

# Set Performance Governor
cpupower frequency-set -g performance

---

The isolcpus Trap

Adding isolcpus=2-10 is not enough. The Kernel still schedules Per-CPU Workqueues and Managed IRQs on them.

The Fix: You must check /proc/interrupts and /sys/devices/virtual/workqueue to ensure absolutely nothing is bound to your core.

---

Practice Exercises

Exercise 1: The SMI Hunt (Beginner)

Task: Run turbostat while compiling a kernel. Observation: Watch the SMI counter increment. Action: Go to BIOS -> Disable “Dell System Management” (or equivalent).

Exercise 2: C-State Latency (Intermediate)

Task: Use wult (Wake Up Latency Tracer) by Intel. Action: Measure wakeup time from C1 vs C6. Result: Expect ~1-2µs for C1, ~100µs for C6.

Exercise 3: Frequency Locking (Advanced)

Task: Run a tight loop benchmarking RDTSC (Time Stamp Counter). Action: Enable/Disable Turbo Boost. Observation: Variance drops to near zero when frequency is locked.

---

Knowledge Check

1. What happens to the OS during an SMI?
2. Why is C6 sleep bad for HFT?
3. Does isolcpus prevent Workqueues?
4. How do you detect SMIs from Linux?
5. Why lock CPU frequency?

Answers

1. It Pauses. The OS is suspended while Firmware runs.
2. Wakeup Latency. It takes >100µs to restore power/cache.
3. No. You need rcu_nocbs and to physically move workqueues.
4. MSR Registers. Tools like turbostat read the SMI counter MSR.
5. Determinism. Variable frequency = Variable Execution Time.

---

Summary

• SMI: The Ghost in the machine.
• C-States: Sleep is for the weak (and slow).
• Turbo: Consistency > Raw Speed.
• isolcpus: The start of the journey.

---

---

Pro Version: For production-grade implementation details, see the full research article: hidden-linux-settings-latency

---

Pro Version: See the full research: Hidden Linux Settings for Latency