Once an hour, Sherpsy asks a simple question: which households are due their weekly digest right now? "Right now" depends on where each household lives, so the job has to check every stored time zone is one it recognises before it can convert the hour. That check was the problem.
Where the time went
The safe way to validate a time-zone name is to look it up in the database's own list of zones. That list is not a table; it is assembled on demand by reading roughly twelve hundred files from the system's time-zone directory. Warm, the read takes about fifty milliseconds. Cold, it takes closer to a second — and an hourly job is always cold. Since mid-August the digest scheduler had run 783 times at an average of 827 milliseconds each, which made it the third most expensive statement in the entire database by total time, for a query that returned a handful of rows.
How we found it
Not by guessing. Postgres keeps statistics on every statement it has run: how many times, how long, how much it reads. We read those before touching anything, and the digest scheduler was near the top of a list it had no business being on. The rewrite asks the database to convert a timestamp into the zone and catches the error if the zone is unknown — the same answer for every zone we have ever stored, and no directory scan. The job now averages 27 milliseconds.
A slow shared job is a tax every household pays, whether or not it is theirs.
Why it matters to you
You would never have seen this. The digest still arrived at the hour you chose. But the database is one machine shared by every household, and eight hundred milliseconds spent scanning files is eight hundred milliseconds in which someone's list toggle waits its turn. Most of what keeps Sherpsy quick is work like this: reading what the system says about itself, and fixing the thing it points at.