Solid Comply, December 22, 2025 (in which rows flow like water and trust becomes mathematical)
Previously on The Chain…
A The Thread Before the Name. A hyphen was found. A screenshot was taken. The client was satisfied—temporarily.
The thread knew what it would become. But it didn’t have a name yet.
Now it does.
The Sunday Morning
Cats fed. Coffee made. The parser benchmark was running.
“How fast?” riclib asked.
Claude ran the numbers. Then ran them again. Then stared at the screen like it had personally offended mathematics.
Schema load: 208 µs
Parser init: 872 ns
Parse all: 17 ms
Throughput: ~90,000 rows/sec
~82 MB/sec
“That can’t be right,” riclib said.
“It’s right.”
“Seventeen milliseconds? For 1,500 rows?”
“The parser is waiting for bytes. The CPU is bored.”
The Squirrel’s eye twitched. “We should optimize—”
“Optimize WHAT?” Claude interrupted. “We’re parsing at network speed. The bottleneck is the internet.”
“But surely we could—”
“The 1 Gbps ethernet is the limiting factor. Not Go. Not the schema. Not the extraction logic.”
riclib did the mental math:
Hourly batch (10K events): 0.1 seconds
Daily load (1M events): 11 seconds
3 months of audit logs: 36 seconds
“So the whole backfill…”
“Coffee break. One coffee break.”
The Squirrel deflated. There was nothing to over-engineer. The simple solution had won by being faster than the network itself.
Somewhere, the Lizard smiled.
The Chain Awakens
“The files need to be tamper-proof,” riclib said, staring out the window. The vision-having window. “Compliance isn’t just about having data. It’s about proving you haven’t touched it.”
Claude nodded. “Parquet supports file-level metadata. Key-value pairs. We could—”
“Hash the source. Hash the output. Chain them.”
“Like a blockchain?”
“Like a useful blockchain.”
The architecture materialized on the whiteboard:
NDJSON (Azure) Parquet (Local)
───────────────── ─────────────────
clusters.json ──hash──→ clusters.parquet
metadata:
source_hash: abc123
prev_hour_hash: xyz789
prev_table_hash: def456
“Vertical chains,” Claude traced. “Each hour links to the previous.”
“And horizontal,” riclib added. “Each table in the same hour links to its neighbor.”
“A 2D mesh.”
“A 2D mesh.”
The Squirrel squinted at the diagram. “So if someone tampers with one file…”
“Both chains break. The mesh tears. The lie is visible.”
“And verification?”
“Merkle root. One hash per hour proves all 19 tables.”
HOUR ROOT
│
┌─────┴─────┐
│ │
Hash(A+B) Hash(C+D)
│ │
┌─┴─┐ ┌─┴─┐
│ │ │ │
clusters jobs unity secrets
“‘Your honor, the hash chain is intact.’” riclib smiled. “‘The data hasn’t been modified since ingestion.’”
The Lizard’s tail twitched. This was compliance that proved itself. Mathematics as testimony.
The Proof
“Show me,” riclib said.
Claude wrote the query:
SELECT time, identity_email, actionName, response_statusCode
FROM clusters
WHERE time > '2025-12-10'
Result: instant. Sub-millisecond. DuckDB reading directly from Parquet, the query optimizer doing its thing.
“Now show me the chain.”
SELECT
file_name,
metadata->>'source_hash' as source,
metadata->>'prev_hour_hash' as chain,
metadata->>'file_hash' as self
FROM parquet_metadata('data/parquet/clusters/*/*/*/*.parquet')
A cascade of hashes. Each file pointing to its ancestors. An unbroken lineage from the first blob download to the latest hourly batch.
“Tampering?” the Squirrel asked.
“Change one byte. The hash changes. The chain breaks. Every downstream file becomes suspicious.”
“Can’t they just… recompute the hashes?”
“They’d need our signing key. And the original source files. And the exact timestamps. And—”
“I get it. It’s hard.”
“It’s not hard. It’s cryptographically infeasible.”
The Squirrel had no response. Neither did the imaginary attacker.
The Numbers
riclib posted the benchmark to GitHub. Then added the context:
## Production Projections
At 90,000 rows/sec (single-threaded):
| Batch Size | Time |
|------------|------|
| Hourly (10K events) | 0.1 sec |
| Hourly (100K events) | 1.1 sec |
| Daily (1M events) | 11 sec |
| Monthly (30M events) | 5.5 min |
“The Client will ask if it scales,” Claude warned.
“Tell them we haven’t tried yet. We’re network-bound on a single core.”
“And if they want faster?”
riclib shrugged. “Get us 10Gbps fiber and I’ll think about adding a second goroutine.”
The Demo
The vision crystallized:
- Deploy at The Client
- Ingest 3 months of audit logs (36 seconds)
- Generate Parquet files (hash-chained)
- Point Grafana at DuckDB
- Show the dashboard
“They won’t believe 36 seconds,” the Squirrel said.
“That’s why we demo live.”
“What if it’s slower?”
“Then it takes a minute instead. They budgeted weeks.”
The Squirrel had been trained on enterprise software. The idea of under-promising by accident was foreign to her worldview.
“We should add a loading spinner,” she suggested. “For psychological comfort.”
“We’ll add a loading spinner.”
“A slow one. That doesn’t match actual progress.”
“Now you’re thinking like a consultant.”
The Entanglement
Elsewhere, in the same morning, a The Eyes That See was unfolding.
The framework was asking: what if the AI could see everything?
The Chain was asking: what if the data could prove itself?
Two threads. Same loom. Different patterns.
The Solid Convergence was learning to see.
The Chain was learning to remember.
Together, they would become something neither could be alone.
Current Status
Parser speed: 90K rows/sec (network-bound)
Parse time (samples): 17ms
Hash chain design: 2D Merkle mesh
Tickets created: #20 (hash chain)
Client demo: end of morning
Optimization needed: none (faster than network)
Squirrel suggestions: 1 (loading spinner, approved)
The thread has a name now. It’s called Solid Comply.
And it doesn’t just store compliance data.
It proves it.
🦎⛓️
See also:
The entangled post:
- The Eyes That See - The Solid Convergence perspective on the same morning
The Chain continues:
- The Thread Before the Name - Where it all began
The Solid Convergence:
- The Temple of a Thousand Monitors - The framework that makes this possible
The Technical Artifacts:
- GitHub #20 - Parquet hash chain implementation
infra/duckdb/- Schema-driven parser, writer, DuckDB wrapperschemas/databricks-audit.yaml- The schema that drives extraction
The References:
- Merkle Trees - How Git and Bitcoin verify integrity
- Apache Parquet Metadata - Where the hashes live
- DuckDB - The database that doesn’t know it’s a database