pegas-memory/ARCHITECTURE.md

# Pegas Memory — Architecture

Pegas Memory is the long-term memory and knowledge base for the **Pegas** research agent.
It is a Python service exposed to the agent as an **MCP server** (18 tools).

---

## 1. High-level design

```
                 ┌──────────────────────────────────────────┐
                 │  Pegas agent (dsh Research preset)       │
                 │  calls MCP tools: mcp__pegas-memory__*   │
                 └───────────────────┬──────────────────────┘
                                     │  JSON-RPC 2.0 / stdio
                 ┌───────────────────▼──────────────────────┐
                 │  mcp_server.py  (tool layer, 18 tools)   │
                 └───────────────────┬──────────────────────┘
                                     │  Python API
        ┌────────────────────────────┼────────────────────────────┐
        │                            │                            │
┌───────▼────────┐          ┌────────▼────────┐          ┌────────▼────────┐
│  PostgreSQL    │  outbox  │     Qdrant      │          │    Memgraph     │
│ SOURCE OF TRUTH│─────────▶│ vector slice    │          │  graph slice    │
│ documents      │   (CDC)  │ EmbeddingGemma2 │          │  CodeSymbol     │
│ atoms, links   │          │ dim = 768       │          │  Atom + RELATES │
│ citations      │          └─────────────────┘          └─────────────────┘
│ code_symbols   │
│ code_dependency│
│ episodic traces│
│ research gaps  │
│ outbox         │
└────────────────┘
```

**The one rule:** PostgreSQL is the single source of truth. Qdrant and Memgraph are
**rebuildable projections** — if either is lost, it is rebuilt from PostgreSQL via the outbox.

---

## 2. Storage layers

| Layer | Engine | Role | Rebuildable |
|---|---|---|---|
| Canonical | **PostgreSQL** (16) | all authoritative data, provenance, transactions | — (source of truth) |
| Vector | **Qdrant** | semantic search over atom embeddings (dim 768) | yes, from `pm_atom` |
| Graph | **Memgraph** | relations, code dependencies, logical inference | yes, from `pm_atom_link` + `pm_code_dependency` |

**Embedding model:** `embeddinggemma-2-740m-Q8_0` (Google EmbeddingGemma 2, **dim = 768**),
served by Ollama. Chosen over bge-m3 (dim 1024) for better Russian recall (R@1 0.922 vs 0.882)
and lower GPU footprint. **Not interchangeable** with the old 1024-dim collections — different
vector size means a different collection.

Deliberate non-choices: **pgvector** and **Apache AGE** are *not* used — dedicated engines
(Qdrant for vectors, Memgraph for graphs) are faster and cleaner, while PostgreSQL keeps the
canonical relational role.

---

## 3. Data model (PostgreSQL)

```
pm_document         source registry (web|pdf|epub|text|code_repo), uri, authority, content_hash
pm_chunk            raw parsed segments (evidence), locator; NOT used for retrieval directly
pm_atom             TYPED KNOWLEDGE UNIT — the retrieval unit
                      type ∈ {text, definition, logical_rule, code, table, fact}
                      claim, content, content_hash (dedup), confidence,
                      status ∈ {candidate, verified, stale, contested}, doc_id, metadata
pm_atom_link        relations between atoms: src → dst, kind, confidence, source, inferred
                      kind ∈ {inference, cause, reference, contradicts, depends_on, calls, defines}
pm_citation         evidence: atom → document → locator → quoted_text
pm_code_symbol      tree-sitter symbols (repo, path, language, name, kind, signature, line range)
pm_code_dependency  code graph edges: src → dst, kind ∈ {calls, imports, inherits}
pm_episodic_trace   agent's own traces (plan/action/observation/reflection) — SEPARATE from curated KB
pm_research_gap     detected knowledge gaps (question, reason, priority, status)
pm_outbox           CDC queue: (entity_type, entity_id, op, processed) → Qdrant + Memgraph
```

**Agent memory (episodic) and the knowledge base are deliberately separate**: traces are the
agent's experience, atoms are curated facts.

---

## 4. Modules

| Module | Responsibility |
|---|---|
| `config.py` | service endpoints, model names, thresholds, `PROXY_URL` |
| `embedding.py` | EmbeddingGemma 2 client (single + batch) |
| `db.py` | PostgreSQL: upsert document/atom/link, content hashing, outbox enqueue |
| `qdrant_store.py` | vector slice: `ensure_collection`, upsert, search (with payload filters) |
| `memgraph_store.py` | graph slice: atom/CodeSymbol nodes, RELATES edges, rule application |
| `ingest.py` | parsers: text (chunking), PDF, EPUB, web (trafilatura + httpx, proxy-aware) |
| `distill.py` | LLM distillation of a chunk into typed atoms; `verify_atom` novelty scoring |
| `pipeline.py` | orchestrator: document → chunks → atoms → PostgreSQL (+ dedup, outbox) |
| `outbox_sync.py` | CDC: PostgreSQL outbox → Qdrant and Memgraph |
| `dedup.py` | semantic deduplication by embedding cosine (threshold 0.90) |
| `verify.py` | self-critique (LLM verdict) + cross-confirmation (multiple sources) |
| `research.py` | research-gap management (PostgreSQL only): add / list / get / update / delete |
| `code_symbols.py` | tree-sitter symbol extraction |
| `code_graph.py` | tree-sitter dependency extraction (calls/imports/inherits) |
| `inference.py` | forward-chaining logical rules over the graph until fixpoint |
| `mcp_server.py` | MCP tool layer (JSON-RPC 2.0 over stdio) |

---

## 5. Data flows

### 5.1 Ingest (document → knowledge)

```
source (text|pdf|epub|web)
  → ingest.py            split into chunks
  → distill.py           LLM → typed atoms
  → dedup.py             skip near-duplicates (cosine ≥ 0.90)
  → db.upsert_atom()     INSERT ... ON CONFLICT (content_hash)
  → db.enqueue_outbox()  mark for projection
  → outbox_sync.py       ──▶ Qdrant (embedding + payload)
                         └─▶ Memgraph (node)
```

### 5.2 Research (gap → verified knowledge)

```
memory_stats / memory_list_gaps  → detect gap
memory_add_gap(question)         → register
web_search                       → URLs
memory_ingest(url)               → atoms   (full page; snippet fallback if blocked)
memory_verify                    → cross-confirm → verified
```

### 5.3 Verification

Two independent steps:

1. **Self-critique (LLM)** — verdict against the *nearest* atoms (by embedding, not by id):
   `new | duplicate | corroborates | contradicts`. `contradicts` → `contested`; `duplicate` → `stale`.
2. **Cross-confirmation (embeddings)** — the same fact found via ≥ 1 other `doc_id`
   (cosine ≥ 0.90) raises confidence and marks the atom `verified`.

### 5.4 Logical inference (graph)

Rules are atoms of type `logical_rule` whose content is JSON:

```json
{"if": "imports", "then": "depends_on"}
{"if": ["depends_on", "depends_on"], "then": "depends_on"}
```

`inference.py` applies them to Memgraph edges repeatedly until fixpoint. Inferred edges are
marked `inferred = true, source = 'inference'` so they are never confused with extracted facts.
Only single-token kinds (no spaces) are accepted — human-readable rules from distillation are ignored.

---

## 6. MCP tools (16)

| Tool | Purpose |
|---|---|
| `memory_search` | semantic search inside the base (Qdrant) |
| `memory_get_context` | gather relevant context before answering |
| `memory_stats` | counts: atoms / documents / open gaps |
| `memory_store` | store one typed atom (with dedup) |
| `memory_ingest` | ingest text/URL/file → atoms |
| `memory_verify` | self-critique + cross-confirmation |
| `memory_add_gap` | register a research gap |
| `memory_list_gaps` | list gaps (optional status filter) |
| `memory_get_gap` | read one gap by id |
| `memory_update_gap` | edit question / reason / priority / status |
| `memory_delete_gap` | delete a gap |
| `memory_store_code` | extract + store code symbols |
| `memory_search_code` | lexical search over code symbols |
| `memory_code_graph` | build symbol + dependency graph |
| `memory_infer` | run logical inference |
| `memory_query_graph` | query graph edges/counters |

---

## 7. Deployment

| Component | Endpoint |
|---|---|
| PostgreSQL | `127.0.0.1:5434` (atom-db container) |
| Qdrant | `127.0.0.1:6333` (collection `alicorn_atoms`) |
| Memgraph | `127.0.0.1:7687` (Bolt) |
| Ollama | `127.0.0.1:11434` |

The MCP server runs from the project venv and is registered in the dsh profile:

```yaml
- id: mcp-pegas-memory
  name: '@deepseek-ai/dsh-mcp-client'
  config:
    serverName: pegas-memory
    transport: stdio
    command: /home/neo/workspace/pegas-memory/.venv/bin/python
    args: [/home/neo/workspace/pegas-memory/mcp_server.py]
    env:
      PROXY_URL: socks5://127.0.0.1:1080   # optional, for geo-blocked sources
```

**Hardware budget:** 1× RTX A2000 12 GB (24/7). EmbeddingGemma 2 (~0.4 GB) coexists with the
agent LLM; large models do not. Memory is designed so embedding load stays low and background.