https://shermanhayes.com/ Notes on storage, databases, and the parts of running systems that only show up at three in the morning. en-us https://shermanhayes.com/posts/the-fsync-you-didnt-call https://shermanhayes.com/posts/the-fsync-you-didnt-call Thu, 18 Jun 2026 08:00:00 -0500 A successful write() means almost nothing. Here is the rest of the path, including the directory entry that everyone forgets. storage https://shermanhayes.com/posts/a-retry-storm-start-to-finish https://shermanhayes.com/posts/a-retry-storm-start-to-finish Sat, 02 May 2026 08:00:00 -0500 A downstream service got 200 ms slower. Twenty minutes later we were sending it four times our normal traffic. incidentreliability https://shermanhayes.com/posts/p99-is-lying-to-you https://shermanhayes.com/posts/p99-is-lying-to-you Wed, 11 Mar 2026 08:00:00 -0500 Your load generator waits for a response before sending the next request. That single fact deletes the worst latency from your data. latencymeasurement https://shermanhayes.com/posts/winter-bike-commuting-year-six https://shermanhayes.com/posts/winter-bike-commuting-year-six Sat, 07 Feb 2026 08:00:00 -0600 Studded tires, three layers, and the one piece of advice that actually matters: it's your hands. not software https://shermanhayes.com/posts/autovacuum-is-not-optional https://shermanhayes.com/posts/autovacuum-is-not-optional Wed, 14 Jan 2026 08:00:00 -0600 The table was 41 GB holding 3 GB of live rows. Autovacuum had been running the whole time. It had simply never finished. postgres https://shermanhayes.com/posts/the-log-line-that-wasnt-there https://shermanhayes.com/posts/the-log-line-that-wasnt-there Thu, 20 Nov 2025 08:00:00 -0600 We spent most of a day looking for the cause of a bad export. The clue was a log line that should have been printed and wasn't. debugging https://shermanhayes.com/posts/timeouts-are-a-contract https://shermanhayes.com/posts/timeouts-are-a-contract Tue, 02 Sep 2025 08:00:00 -0500 A timeout isn't a safety net you sprinkle on a client. It's a promise about how long the whole call may take, and it has to be passed down. reliability