Core Problem
- Current AI memory has Write (real-time ingestion) and Read (real-time retrieval) but NO reprocessing
- No way to discover implicit links between seemingly unrelated memories
- Can't compress old detailed memories into higher-level abstractions
- Can't generate new patterns or inferences from accumulated knowledge
- Like human memory without REM sleep consolidation
Solution — Dream Pipeline
An orthogonal processing layer added to the existing Write-Index-Read path:
Write (real-time) ──→ Memory Store ──→ Read (real-time)
↑↓
Dream (offline batch)
- Pattern discovery
- Link creation
- Summary/compression
- Inference generationKey Features — Dream's 4 Roles
1. Pattern Discovery
Extract recurring themes, behavior patterns, and preferences from accumulated memories. Over time, individual data points coalesce into recognizable tendencies.
2. Link Creation
Discover implicit connections between seemingly unrelated memories. Two facts stored months apart may share a causal or contextual relationship that only becomes visible in aggregate.
3. Summary / Compression
Compress old detailed memories into higher-level concepts. This improves memory efficiency — 15 granular records become 1 abstract summary while the originals are deprioritized.
4. Inference Generation
Derive new knowledge from existing memories. This is the true identity of source: auto_inferred — knowledge that was never explicitly stated but can be confidently deduced from what is already stored.
Message Log ≠ Memory
An important distinction that underpins the entire architecture:
| Message Log | Memory | |
|---|---|---|
| Nature | Raw material | Extracted knowledge |
| Structure | Append-only archive | Structured store |
| Searchability | Not searchable (noisy) | Searchable (precise) |
| Mutability | Immutable | Confidence changes / merges / expires |
Self-Referential Design (Isomorphism)
The Spark → Idea system and Dream Pipeline share an identical structure:
| Spark → Idea System | Dream Pipeline |
|---|---|
| Spark accumulation | Memory accumulation |
| Pattern analysis | Pattern discovery |
| Idea promotion | Inference generation |
| Idea index update | Knowledge graph update |
This means improvements to the Spark System directly hint at Dream Pipeline design — the two systems are isomorphic.
User Scenarios
Preference Pattern
3 months of accumulated memories → Dream discovers "this user always chooses the simpler option" → a new auto_inferred preference is created and surfaced in future recommendations.
Memory Compression
15 detailed memories on the same topic → Dream creates 1 summary memory and lowers the confidence scores of the originals, keeping them accessible but deprioritized in search results.
Implicit Link
"Kim is security team lead" + "Security audits are quarterly" → Dream infers "Kim likely leads quarterly audits" and stores this as a linked inference with appropriate confidence.