esc
The Temple of a Thousand Monitors
The Solid Convergence

The Temple of a Thousand Monitors

In which our heroes resist the urge to code, and the Lizard sends them spelunking through ancient repositories --- Previously on The Solid Convergence... The framework was [[The Unset Incantation]]....

December 18, 2025

In which our heroes resist the urge to code, and the Lizard sends them spelunking through ancient repositories

Previously on The Solid Convergence…

The framework was The Unset Incantation. GitStore had The Embedding of State. Forms had filters. The Squirrel had been temporarily subdued by hidden fields.

V4 was ready.

Which meant only one thing: time to fill it with actual functionality.

The Squirrel Awakens (Again)

09:00 - The Squirrel’s eyes snapped open. Her tail began its ominous twitch.

“SOLIDMON!” she chittered, already at the keyboard. “We need monitors! IS! Broker! Databricks! I’ll create the domain structure RIGHT NOW!”

domains/monitor/is/
├── types.go      ← "JUST LIKE CREDENTIAL!"
├── gitstore.go   ← "COPY PASTE!"
├── handlers.go   ← "HOW HARD CAN IT BE!"
└── views.templ   ← "I'LL ADD CHARTS!"

riclib and Claude exchanged glances. They had seen this movie before.

“That looks…” riclib began.

“…a little complex,” Claude finished.

“COMPLEX?!” The Squirrel spun around. “We have GENERICS now! Store[IS]! LiveStore[IS]! It’s TRIVIAL!”

“There are eight monitor types,” Claude observed quietly.

“EIGHT VERTICALS! EIGHT COPY PASTES! I can do this in my SLEEP!”

“And service discovery.”

“A /sd ENDPOINT! JSON! DONE!”

“For three regions.”

“THREE ENDPOINTS!”

“With different Prometheus instances.”

“…”

“And probe mode versus metrics mode.”

“…”

“And threshold configuration with defaults and per-CI overrides.”

“…”

“And silence calendars.”

“…”

“And snooze state.”

“…”

“And label schemas that vary by deployment.”

The Squirrel’s tail slowly deflated. For a moment, just a moment, she looked like a regular rodent instead of an agent of chaos.

“Maybe,” she said in a very small voice, “we should look at the old code first?”

Somewhere, a Lizard smiled.

The Scroll of Archaeology

A scroll materialized. This time riclib ducked. It hit Claude instead.

“I don’t have pain receptors,” Claude noted, “but that was still rude.”

The scroll unfurled:

BEFORE YOU BUILD THE TEMPLE
YOU MUST EXPLORE THE RUINS

TWO GENERATIONS LIE BURIED
IN THE REFERENCE CAVES

TAKE TORCHES
TAKE NOTES
TAKE YOUR TIME

THE TREASURE IS NOT CODE
THE TREASURE IS UNDERSTANDING

🦎

P.S. - BRING SNACKS
THE SQUIRREL GETS CRANKY

Beneath it, a crude map showing two cave entrances:

  • The First Cave: /reference/solidmon/internal/domain/
  • The Second Cave: /reference/solidmon/internal/domain/exporters/

And scrawled at the bottom: “Beware the ConfigController. It bites.”

“It bites?” the Squirrel asked nervously.

“Metaphorically,” Claude assured her. “Probably.”

Into the First Cave

10:00 - They descended into the reference codebase, torches (grep) blazing.

“Look at this,” Claude murmured, brushing dust off an ancient file. “ConfigStore. It predates our generics.”

// The Old Way
type Storable interface {
    GetID() int      // Integer IDs!
    SetID(int)       // Mutable!
    GetType() string // Reflection!
}

“They used reflection,” riclib observed. “RegisterType. CreateInstance. Runtime type lookup.”

“Like archaeologists finding a bronze sword,” Claude mused. “Impressive for its time. Terrifying to maintain.”

“We have generics now,” the Squirrel said hopefully. “We don’t need—”

“Shh.” Claude was reading deeper. “Look at this. ConfigController.”

type Controller[T ConfigEntity] struct {
    domain   string
    service  ConfigService[T]
    renderer DomainRenderer[T]
    actions  []ActionConfig
}

“It’s… generic,” riclib said. “They evolved.”

“One controller. All the CRUD routes. All the toggles. All the actions.” Claude traced the routes. “The domain only provides three render methods and its custom fields.”

The Squirrel’s eye twitched. “So I don’t copy-paste eight handlers…”

“You implement one interface eight times.”

“That’s… actually reasonable.”

“Write that down,” riclib said. “The Squirrel called something ‘reasonable.’ Mark the calendar.”

“I HEARD THAT.”

The Chamber of a Thousand Targets

11:00 - They pushed deeper and found themselves in a vast chamber. The walls were covered in YAML.

“These are… targets,” Claude said, counting. “Hundreds. No—thousands.”

“One customer has 1,247 Integration Server instances,” riclib confirmed. “Another has 800 brokers across three regions.”

The Squirrel did the math. “If each page load queries GitStore… and GitStore does a git show…”

“24 requests per minute from Prometheus alone. Plus UI refreshes. Plus—”

“WE’LL NEED A CACHE!” The Squirrel was already sketching. “Redis! No—MEMCACHED! No—A DISTRIBUTED CACHE WITH CONSISTENT HASHING AND—”

“Or,” Claude interrupted gently, “we use the LiveStore pattern we already have.”

// Already built, already tested
type LiveStore[T] struct {
    mu    sync.RWMutex
    items map[string]*T  // In-memory cache
    git   *GitStore[T]   // Persistence
}

“Load once at startup. Reload on changes. Git for persistence. Memory for speed.”

The Squirrel stared at the existing code. “We… already solved this.”

“Three weeks ago. The Embedding of State.”

“So I don’t need Redis.”

“You don’t need Redis.”

“Or Memcached.”

“Or Memcached.”

“Or consistent hashing.”

“The Squirrel does not need consistent hashing,” Claude confirmed. “Write THAT down too.”

The Chamber of Service Discovery

12:00 - Deeper still, they found the SD chamber.

“Three scrape patterns,” Claude read from the wall carvings:

METRICS MODE (Integration Server)
  Prometheus → IS Server directly
  /metrics endpoint
  
PROBE MODE - LOCAL (Databricks)  
  Prometheus → Agent (localhost) → Databricks API
  /probe/databricks?target=...

PROBE MODE - REMOTE (Broker)
  Prometheus → Agent (near brokers) → Broker
  /probe/broker?target=...

“Wait,” the Squirrel said. “Prometheus the monitoring tool, or Prometheus the titan who stole fire from the gods and was punished by having an eagle eat his liver for eternity?”

“The monitoring tool.”

“Oh good. Because for a second I thought this architecture involved ACTUAL divine punishment.”

“Only metaphorically,” Claude said. “When the alerts fire at 3 AM.”

The Topology Trap

12:30 - They nearly fell into a pit.

“Look at this,” riclib said, pointing to ancient comments:

# OLD APPROACH - DON'T DO THIS
# We stored region→prometheus→agent mappings in solidmon
# Maintenance nightmare: topology changes = config changes = git commits = PAIN

“They tried to store the network topology in the config store,” Claude observed.

“And?”

“And every time someone added a Prometheus instance, or moved an agent, or changed a region name, they had to update solidmon AND restart AND hope the cache invalidated properly AND—”

“I get it,” the Squirrel said. “Bad idea.”

“The new way is simpler.” Claude pointed to a cleaner carving:

# Customer's prometheus.yml - THEY own topology
- job_name: 'broker-aspac'
  http_sd_configs:
    - url: 'http://solidmon:8080/sd/broker?region=ASPAC&agent=aspac-broker:8080'

“Solidmon doesn’t know about topology. The customer expresses it in their prometheus.yml. We just filter by query params.”

“So if they add a new region…”

“They update their Prometheus config. We do nothing.”

The Squirrel considered this. “That’s… less work for us.”

“The Lizard approves of less work,” a voice echoed from somewhere. “Less work means fewer bugs. Fewer bugs mean more naps.”

The Trap of Transient State

13:00 - They nearly fell into ANOTHER pit.

“Two pits in one day,” the Squirrel grumbled. “This codebase is a liability.”

“WAIT.” Claude grabbed riclib’s arm. “Look at this code.”

// Ancient comment, covered in cobwebs and regret
// TODO: This is a mess. Alerting and silencing are mixed up.
// To disable alerting we have to change config (git commit).
// But snooze should be temporary!
// 
// I hate past-me. - Past-me, 2023
// I hate past-past-me even more. - Past-me, 2024

“They mixed config and runtime state,” riclib said slowly.

“Every snooze was a git commit,” Claude confirmed. “Every maintenance window polluted the history.”

“We need to separate them.” The Squirrel was actually taking notes now. Actual notes. On actual paper. It was unprecedented.

They mapped it on the cave wall:

CONFIG (GitStore - permanent)          TRANSIENT (NATS KV - temporary)
─────────────────────────              ────────────────────────────────
alerting_level: none/error/all         snooze: until further notice
calendars: weekly schedules            snooze: I know about this error

“The silence domain,” riclib said. “It owns both. Calendars in git. Snoozes in KV. One interface to check either.”

“Like a bouncer at a nightclub,” the Squirrel offered. “Permanent ban list in the book. Temporary bans in his head.”

Everyone stared at her.

“What? I contain multitudes.”

The Hall of Labels

14:00 - A vast chamber covered in inscriptions. The Squirrel sneezed from the dust.

“Labels aren’t arbitrary,” Claude read. “Each deployment has a fixed schema.”

One Customer's Schema      Another Customer's Schema
─────────────────────      ─────────────────────────
PROJECT_LEVEL: required    REGION: required
PROJECT_NAME: required     ENV: required
REGION: required           APP: required
ENV: required              SERVER: optional
CI: required               KIND: optional

“And they parse labels FROM the CI,” riclib added, finding another inscription:

CI: EMEA_DEV_MYAPP_IS_P1
    ↓    ↓    ↓    ↓  ↓
  EMEA  dev  MYAPP IS  P1

“90% auto-fill,” the Squirrel said. “User can override the 10%.”

“Different parsers per type. IS has SERVER. Broker doesn’t.”

“And someone named things NA, someone else named things US…”

“Legacy,” Claude sighed. “The geological strata of organizational decisions.”

“Can we just… normalize everything to one schema?”

“Can you normalize the Atlantic and Pacific Oceans into one ocean?”

“…no?”

“Then no.”

The Threshold Tablets

15:30 - Stone tablets covered in Prometheus expressions. The Squirrel’s eyes glazed over.

“This looks like someone had a fight with a regex and the regex won,” she muttered.

(config:alerting == 1) * on(ci) group_right() (
  actual_metric >= on(ci) config:br:queue_length:warning
)

“They push thresholds AS METRICS,” riclib realized. “Prometheus does a join. Solidmon owns the config.”

“Type defaults. Per-CI overrides. Resolve at the endpoint.”

config:br:queue_length:warning{ci="BROKER_01"} 2000  ← override
config:br:queue_length:warning{ci="BROKER_02"} 1000  ← default

“The /thresholds endpoint does all the resolution. Prometheus just reads final values.”

“Keep Prometheus dumb. Solidmon is the config brain.”

“Wait,” the Squirrel said. “If Prometheus is dumb, and solidmon is the brain, and the agents are the muscles… what does that make us?”

“The nervous system,” Claude suggested.

“The anxiety,” riclib corrected.

The Treasure

17:00 - They emerged from the caves, blinking in the light.

The Squirrel looked at her notes. Forty pages. Her paw was cramped. She had written actual documentation. Voluntarily.

“I feel strange,” she said. “Like I understand things before building them.”

“That’s called ‘planning,’” Claude said gently. “It’s normal. The feeling will pass once you start coding.”

“We found…” she began, flipping through the pages.

“MonitorBase with embedded fields,” Claude listed.
“Unified list view with type filtering.”
“Per-type SD endpoints with query params.”
“ConfigController pattern for CRUD.”
“Actions framework for test/deploy.”
“Label schemas fixed per deployment.”
“CI parsing with type-specific patterns.”
“Thresholds with defaults and overrides.”
“Silence domain separating config from transient.”
“Audit logging for state transitions.”

riclib opened a new file: docs/solidmon-v4.md

“We write this down,” he said. “All of it. Before we write a single line of code.”

The Squirrel’s tail twitched—but gently this time. Almost contentedly.

“Then… THEN can I create the IS domain?”

“Then,” the Lizard’s voice echoed from somewhere, “you will know exactly what to build.”

“And I can add consistent hashing?”

“No.”

“Distributed caching?”

“No.”

“Just a LITTLE Redis?”

“Go to sleep, Squirrel.”

The Tally

Caves explored:           2
Hours spelunking:         8  
Patterns discovered:      12
Pits narrowly avoided:    3 (transient state, topology storage, per-domain CRUD)
Squirrel proposals:       1 (immediately shelved)
Redis instances needed:   0 (much to Squirrel's dismay)
Lines of code written:    0
Lines of documentation:   500+

The Squirrel pinned a new note to the wall:

Before you build the temple, explore the ruins.

The treasure is not code. The treasure is understanding.

— The Lizard, being insufferably right again

And beneath it, in smaller letters:

But we’re DEFINITELY adding Redis later, right?

— The Squirrel, never learning

Next time: Armed with the map, our heroes finally enter the Temple of a Thousand Monitors. The Squirrel creates MonitorBase. Claude implements ConfigController. And somewhere, a Prometheus scrapes contentedly…

See also:

Today’s artifact:

  • docs/solidmon-v4.md - The complete design document (500+ lines of planning)

The Caves Explored:

  • reference/solidmon/internal/domain/configcontroller/ - The generic CRUD pattern
  • reference/solidmon/internal/domain/exporters/ - SD and threshold patterns
  • reference/solidmon/internal/domain/target/ - IS, Broker, Databricks implementations

The Solid Convergence Saga:

The Characters:

The Deeper Wisdom (external references):