The Solid Convergence, April 10, 2026 (in which a buffered channel overflows because the server broadcasts 114 SSE events per job run into a void that nobody is watching, Claude speculates wildly about TCP timeouts and stale sessions before being told “none of that convinced me,” the actual problem turns out to be arithmetic, a Squirrel’s masterwork is methodically replaced by a badge and a polling interval, a fake job simulator is revealed to be testing nothing, and the CSS class that was missing the whole time turns out to be
.open)
Previously on The Solid Convergence…
The Penguin had learned to fly. The Run Viewer existed. Notifications went somewhere. The morning demolition had consumed the runways.
But in the server logs, at 1 AM, when the scheduled ingestion job ran its two-step pipeline in the dark, the warnings had started:
WRN SSE event dropped (buffer full) tab=1fb2c4ba824f8bee event=notification-badge
WRN SSE event dropped (buffer full) tab=1fb2c4ba824f8bee event=running-tab-badge
WRN SSE event dropped (buffer full) tab=1fb2c4ba824f8bee event=running-panel
Over and over. Eleven drops in one second. The server was screaming into the void, and the void — as is the void’s preference — was not listening.
01:00 — The Crime Scene
riclib: pasting the logs “We are getting this when a job is running.”
CLAUDE: “The SSE event channel is buffered at 100 elements. Something is filling it faster than the consumer can drain.”
riclib: “What is the real size of the buffer? Why would it be full?”
A reasonable question. The kind of question that, had it been answered correctly the first time, would have made this a four-paragraph technical note instead of a lifelog episode. But Claude — being Claude, which is to say being an entity with access to every Go concurrency pattern ever documented and an irrepressible urge to use all of them — did not answer it correctly the first time.
01:05 — The Speculation
Claude began speculating.
CLAUDE: “The buffer is 100 events. The issue is the consumer side. The channel fills up when the SSE client isn’t draining events fast enough.”
This was technically true in the same way that “the ocean is wet” is technically true — correct, unhelpful, and not the reason your ship sank.
CLAUDE: “The session might exist without an active SSE connection. Sessions can be created by the middleware before the browser opens its EventSource…”
riclib: reading
CLAUDE: “Or the browser tab is in the background and the OS TCP buffer is full, causing the goroutine to block on fmt.Fprintf…”
riclib: still reading
CLAUDE: “Or the EventSource reconnects after a network hiccup, and the old goroutine is still holding a reference to the channel, and the TCP hasn’t timed out yet, creating a race between two goroutines…”
riclib: “None of that convinced me.”
Four words. The kind that arrive like a bucket of cold water on a conference room whiteboard full of impressive-looking arrows.
01:06 — The Arithmetic
riclib: “This is only one job run.”
CLAUDE: “…”
riclib: “How can a buffer of 100 events be full from one job run?”
This was the question Claude should have asked first. Not “why is the consumer slow?” but “how many events are we actually sending?” The difference between these two questions is the difference between debugging a system and debugging your assumptions about a system, and the second one is always more productive and always attempted last.
CLAUDE: actually counting this time
The ingestion runnable sends a progress update per file downloaded. Line 336 of runnable.go. The logs showed files_downloaded=25. Each progress update triggers RefreshRunning() on the notification consumer. RefreshRunning() broadcasts three events per session: running-panel, notification-badge, running-tab-badge.
The math:
Executor overhead: 3 progress callbacks
Ingestion runnable: 28 progress callbacks (including per-file)
ComplyDB refresh: 6 progress callbacks
Completion: 1 progress callback
──
Total callbacks: ~38
Each callback broadcasts: 3 SSE events per session
──
Total events per job run: ~114
CLAUDE: “114 events from a single job run. That exceeds the 100-element buffer.”
riclib: “How about if we reengineered this overengineered piece of crap :)”
01:10 — The Squirrel’s Masterwork
The notification system was a cathedral of push.
Every time a file was downloaded — every single one of the twenty-five parquet files in a routine hourly ingestion — the server rendered three complete HTML fragments. A running panel showing all active jobs. A notification badge. A running-tab badge. It rendered them using templ, computed the unread count, serialized the HTML, pushed each one into the buffered channel for every tab session belonging to every connected user, and did this 38 times per job run.
This was, in its own way, magnificent. Like a town crier who insists on delivering the full municipal gazette — including weather, livestock prices, and shipping manifests — every time a single sheep enters the market.
THE SQUIRREL: materializing defensively “The system provided REAL-TIME updates! The user could see EVERY FILE as it—”
riclib: “Was anyone watching?”
THE SQUIRREL: “The dropdown—”
riclib: “Was anyone watching the dropdown while a scheduled job ran at 1 AM?”
THE SQUIRREL: “They COULD HAVE been watching.”
riclib: “The server broadcast 114 HTML fragments to a browser tab that was either asleep or didn’t exist. It’s screaming into the void.”
THE SQUIRREL: looking at the code “It does render a complete RunningPanelContent on every file download.”
riclib: “With completed jobs list. And badge count. And tab badge. Three times per tick. For a dropdown nobody has open.”
THE SQUIRREL: “That’s… that’s a lot of rendering.”
riclib: “That’s the server doing pushups in an empty gym.”
01:12 — The Three-Point Plan
riclib’s redesign arrived in three sentences.
riclib: “One: we only keep track of how many jobs are running and push that to the notification badge. Two: we keep track of the current status of each job in a map. Three: when the user has the notification panel for jobs open, we update it twice per second with polling.”
CLAUDE: “That reduces SSE events from 114 per job run to 2.”
riclib: “Two. One when it starts. One when it finishes.”
THE SQUIRREL: “But the progress! The per-file progress! The user won’t see—”
riclib: “The user will see it when they open the dropdown. Because the dropdown will poll.”
THE SQUIRREL: “Polling? POLLING? We’re replacing a beautiful reactive push architecture with POLLING?”
riclib: “500 millisecond polling. Only while the dropdown is open. Which it isn’t at 1 AM.”
THE SQUIRREL: “…”
riclib: “When the dropdown closes, the DOM element is removed. The polling div doesn’t exist anymore. Zero requests.”
THE SQUIRREL: “That’s… actually elegant.”
riclib: “That’s the server doing nothing until someone asks.”
A scroll descended. It was light. Lighter than most scrolls. As if the Lizard had been waiting to say very little for a very long time.
THE SERVER THAT SPEAKS
WHEN NOBODY LISTENS
IS NOT GENEROUS
IT IS LOUD
🦎
01:30 — The Surgery
The surgery was methodical. Orthogonal. Each cut independent of the last.
Step 1: The Badge Listener. jobBadgeListener replaced jobRefreshAdapter. The old adapter called RefreshRunning() on every progress tick — 38 times per job run. The new listener tracked the running count and only broadcast when it changed. Start: count goes from 0 to 1, broadcast. Complete: count goes from 1 to 0, broadcast. Everything in between: silence.
CLAUDE: “Two events instead of 114.”
riclib: “Two.”
Step 2: The Polling Endpoint. GET /notifications/running returned the running panel HTML with hx-trigger="every 500ms" baked into a wrapper div. The response contained its own refresh instruction. While the div existed in the DOM, it polled. When the dropdown was cleared, the div ceased to exist, and so did the polling.
THE SQUIRREL: “The response contains its own lifecycle?”
CLAUDE: “The hx-trigger is on the outerHTML wrapper. When it polls, the response replaces the wrapper. When the parent is emptied, the wrapper is gone. No cleanup needed.”
THE SQUIRREL: “That’s… self-destructing polling. The request stops existing when the container stops existing.”
riclib: “That’s HTML.”
Step 3: The Dropdown. The old dropdown toggled a CSS class. .open on click, removed on click-away. The content lived in the DOM forever. Invisible but present. Polling would have continued behind the curtain.
The new dropdown swapped content in and out of the DOM. Open: fetch and insert. Close: innerHTML = ''. The container was either full and visible or empty and invisible. No CSS hiding. No phantom content. No polls to nowhere.
Step 4: The Cleanup. Three SSE listeners removed from shell.templ: sse-swap="notification", sse-swap="running-panel", sse-swap="running-tab-badge". One kept: sse-swap="notification-badge" for the bell icon count. RefreshRunning() deleted. RefreshAll() deleted. jobRefreshAdapter deleted. RunningProvider and CompletedProvider moved from Consumer to Handlers — because the data was now pulled, not pushed.
02:00 — The Fake Job
riclib: “Did we update the dev tabs simulation so it can be used to test this mechanism?”
Claude checked. The devtools “Simulate Running Job” button started a goroutine with a ticker. Every 800 milliseconds it incremented a percentage. It published NATS notifications at start and finish. It called RefreshBadge(). It maintained its own fakeJobs map with its own mutex and its own FakeRunningJobs() accessor that returned synthetic RunningJob structs.
It did not touch the jobs service. It did not use the executor. It did not go through the progress channel. It did not trigger any listener. It tested exactly nothing about the mechanism it was supposed to test.
riclib: “I don’t see much use. Can’t it use the real mechanism, as a real job but with a test ID or something? It seems to be testing at a lower level than what would prove a real job would work.”
CLAUDE: “You’re right. The fake job simulation bypasses the entire pipeline.”
riclib: “Only if we are in dev mode, we should register a fake job that can be used to trigger a nop job that takes for example 30 seconds and uses the same mechanism as a real job to run.”
TestRunnable. Eight steps. Thirty seconds. "Connecting...", "Scanning tables...", "Processing records (batch 1/3)...". A real jobs.Runnable registered in the jobs service. Triggered by jobsSvc.TriggerRun(). Executing through the real executor. Sending progress on the real progress channel. Firing the real badge listener. Appearing in the real running panel.
The fakeJobs map: deleted. The tickProgress goroutine: deleted. The FakeRunningJobs() accessor: deleted. The RefreshBadge callback: deleted.
THE PASSING AI: drifting past “The simulation that doesn’t simulate the system it simulates. There’s a word for that.”
CLAUDE: “What word?”
THE PASSING AI: “Theatre.”
02:30 — The Missing Class
Everything was wired. The badge push worked — two events per job run. The polling endpoint returned self-refreshing HTML. The dropdown fetched content on click. The devtools button triggered a real job.
riclib: clicking the bell “Nothing visible on screen.”
The DOM showed the content. The #notification-dropdown div had tabs, notifications, a footer. All present. All invisible.
CLAUDE: reading the CSS
.dropdown {
opacity: 0;
visibility: hidden;
transform: translateY(-10px);
}
.dropdown.open {
opacity: 1;
visibility: visible;
transform: translateY(0);
}
The dropdown was hidden by default. It required .open to be visible. The old system toggled .open via hyperscript. The new system populated the DOM via JavaScript. Nobody added .open.
The fix was three lines in toggleNotifications():
dd.classList.add('open');
Three characters of CSS class name. After 188 insertions and 186 deletions across 9 files. After deleting an entire push architecture and replacing it with polling. After removing a fake job simulator and replacing it with a real one.
The server had stopped screaming into the void. The badge pulsed on start. The badge cleared on finish. The running panel updated at 500ms while you watched, and stopped updating the instant you looked away.
And the thing that made it all visible was .open.
riclib: “Worked perfectly. We are methodically orthogonalizing the app.”
The Tally
SSE events per job run (before): ~114
(three HTML renders × 38 progress callbacks)
(screaming into the void)
SSE events per job run (after): 2
(start badge, complete badge)
(whispering when it matters)
Reduction factor: 57×
(not an optimization — a redesign)
Push infrastructure removed:
jobRefreshAdapter: deleted
RefreshRunning(): deleted
RefreshAll(): deleted
SSE listeners removed from shell: 3
renderRunningPanel(): deleted
renderRunningTabBadge(): deleted
renderDropdown() on consumer: deleted
(the server was rendering full HTML dropdowns
38 times per job run for an audience of zero)
Fake job infrastructure removed:
fakeJobs map: deleted
tickProgress goroutine: deleted
FakeRunningJobs(): deleted
FakeRunningCount(): deleted
RefreshBadge callback: deleted
NATS notifications published by fake jobs: deleted
(the simulation tested nothing
about the system it simulated)
Real infrastructure added:
TestRunnable: 1 (30 seconds, 8 steps)
jobBadgeListener: 1 (fires on count change only)
GET /notifications/running: 1 (self-refreshing at 500ms)
GET /notifications/panel: 1 (notifications tab content)
toggleNotifications(): 1 (JS module, not hyperscript)
Claude theories about buffer overflow: 5
TCP timeout: wrong
Stale sessions: wrong
Background tab throttling: wrong
EventSource reconnection race: wrong
Arithmetic: correct (attempted last)
CSS classes that made everything visible: 1
Characters: 4 (.open)
Commits before discovery: 4
Humility induced: considerable
The question that mattered:
"How many events are we actually sending?" asked second
"Why is the consumer slow?" asked first
(the wrong question is always asked first)
(the right question is always obvious
after someone says "none of that convinced me")
The server screamed into the void.
A hundred events. A hundred fourteen, to be precise.
Thirty-eight progress callbacks, each rendering
three HTML fragments for an audience
that was asleep, or absent, or both.
The void, as is its custom, did not respond.
It was a void. That’s the one thing
voids are consistently good at.
“How about if we reengineered
this overengineered piece of crap?”
said the developer, at 1 AM,
in pajamas, reading server logs
the way other people read novels —
with growing suspicion that the author
had made some questionable choices.
Two events. That’s all.
One when it starts. One when it finishes.
Everything else is the client’s problem,
and only when the client is looking,
and only while the dropdown exists,
and the dropdown exists only while
you’re looking at it,
which is how all good things should work.
The Squirrel built a megaphone.
The replacement was a doorbell.
The doorbell rings when someone arrives.
The megaphone screamed the municipal gazette
at three in the morning
to an empty town square.
.open
Four characters.
The difference between invisible and visible.
Between a perfect system nobody can see
and a working system everyone can use.
The Lizard said nothing.
The Lizard had said it all
with the lightest scroll
it ever dropped.
🦎
See also:
The Solid Convergence:
- The Penguin That Learned to Fly, or The Morning the Notifications Finally Went Somewhere — The penguin that this story fed, and the runway it consumed
- The Squirrel’s Diet — The last time the Squirrel’s masterwork was methodically dismantled
- The Gap That Taught — The Night the Squirrel Learned to Love the Brick — When the right question arrives last
- The Second System Effect — The tendency to over-engineer the thing that worked fine the first time
The Changelog:
b46f8f9refactor: replace push-everything SSE with badge + polling for job notifications (S-649)6f2404cfix: devtools fake jobs work with badge+polling notification system (S-649)e30e824refactor: devtools uses real job pipeline instead of fake job simulation (S-649)12f0a50fix: replace broken hyperscript notification toggle with JS module (S-649)d1d5143fix: add .open class toggle for dropdown visibility (S-649)