# Pegas Memory — Architecture Pegas Memory is the long-term memory and knowledge base for the **Pegas** research agent. It is a Python service exposed to the agent as an **MCP server** (16 tools). --- ## 1. High-level design ``` ┌──────────────────────────────────────────┐ │ Pegas agent (dsh Research preset) │ │ calls MCP tools: mcp__pegas-memory__* │ └───────────────────┬──────────────────────┘ │ JSON-RPC 2.0 / stdio ┌───────────────────▼──────────────────────┐ │ mcp_server.py (tool layer, 16 tools) │ └───────────────────┬──────────────────────┘ │ Python API ┌────────────────────────────┼────────────────────────────┐ │ │ │ ┌───────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐ │ PostgreSQL │ outbox │ Qdrant │ │ Memgraph │ │ SOURCE OF TRUTH│─────────▶│ vector slice │ │ graph slice │ │ documents │ (CDC) │ EmbeddingGemma2 │ │ CodeSymbol │ │ atoms, links │ │ dim = 768 │ │ Atom + RELATES │ │ citations │ └─────────────────┘ └─────────────────┘ │ code_symbols │ │ code_dependency│ │ episodic traces│ │ research gaps │ │ outbox │ └────────────────┘ ``` **The one rule:** PostgreSQL is the single source of truth. Qdrant and Memgraph are **rebuildable projections** — if either is lost, it is rebuilt from PostgreSQL via the outbox. --- ## 2. Storage layers | Layer | Engine | Role | Rebuildable | |---|---|---|---| | Canonical | **PostgreSQL** (16) | all authoritative data, provenance, transactions | — (source of truth) | | Vector | **Qdrant** | semantic search over atom embeddings (dim 768) | yes, from `pm_atom` | | Graph | **Memgraph** | relations, code dependencies, logical inference | yes, from `pm_atom_link` + `pm_code_dependency` | **Embedding model:** `embeddinggemma-2-740m-Q8_0` (Google EmbeddingGemma 2, **dim = 768**), served by Ollama. Chosen over bge-m3 (dim 1024) for better Russian recall (R@1 0.922 vs 0.882) and lower GPU footprint. **Not interchangeable** with the old 1024-dim collections — different vector size means a different collection. Deliberate non-choices: **pgvector** and **Apache AGE** are *not* used — dedicated engines (Qdrant for vectors, Memgraph for graphs) are faster and cleaner, while PostgreSQL keeps the canonical relational role. --- ## 3. Data model (PostgreSQL) ``` pm_document source registry (web|pdf|epub|text|code_repo), uri, authority, content_hash pm_chunk raw parsed segments (evidence), locator; NOT used for retrieval directly pm_atom TYPED KNOWLEDGE UNIT — the retrieval unit type ∈ {text, definition, logical_rule, code, table, fact} claim, content, content_hash (dedup), confidence, status ∈ {candidate, verified, stale, contested}, doc_id, metadata pm_atom_link relations between atoms: src → dst, kind, confidence, source, inferred kind ∈ {inference, cause, reference, contradicts, depends_on, calls, defines} pm_citation evidence: atom → document → locator → quoted_text pm_code_symbol tree-sitter symbols (repo, path, language, name, kind, signature, line range) pm_code_dependency code graph edges: src → dst, kind ∈ {calls, imports, inherits} pm_episodic_trace agent's own traces (plan/action/observation/reflection) — SEPARATE from curated KB pm_research_gap detected knowledge gaps (question, reason, priority, status) pm_outbox CDC queue: (entity_type, entity_id, op, processed) → Qdrant + Memgraph ``` **Agent memory (episodic) and the knowledge base are deliberately separate**: traces are the agent's experience, atoms are curated facts. --- ## 4. Modules | Module | Responsibility | |---|---| | `config.py` | service endpoints, model names, thresholds, `PROXY_URL` | | `embedding.py` | EmbeddingGemma 2 client (single + batch) | | `db.py` | PostgreSQL: upsert document/atom/link, content hashing, outbox enqueue | | `qdrant_store.py` | vector slice: `ensure_collection`, upsert, search (with payload filters) | | `memgraph_store.py` | graph slice: atom/CodeSymbol nodes, RELATES edges, rule application | | `ingest.py` | parsers: text (chunking), PDF, EPUB, web (trafilatura + httpx, proxy-aware) | | `distill.py` | LLM distillation of a chunk into typed atoms; `verify_atom` novelty scoring | | `pipeline.py` | orchestrator: document → chunks → atoms → PostgreSQL (+ dedup, outbox) | | `outbox_sync.py` | CDC: PostgreSQL outbox → Qdrant and Memgraph | | `dedup.py` | semantic deduplication by embedding cosine (threshold 0.90) | | `verify.py` | self-critique (LLM verdict) + cross-confirmation (multiple sources) | | `research.py` | research-gap management (PostgreSQL only): add / list / get / update / delete | | `code_symbols.py` | tree-sitter symbol extraction | | `code_graph.py` | tree-sitter dependency extraction (calls/imports/inherits) | | `inference.py` | forward-chaining logical rules over the graph until fixpoint | | `mcp_server.py` | MCP tool layer (JSON-RPC 2.0 over stdio) | --- ## 5. Data flows ### 5.1 Ingest (document → knowledge) ``` source (text|pdf|epub|web) → ingest.py split into chunks → distill.py LLM → typed atoms → dedup.py skip near-duplicates (cosine ≥ 0.90) → db.upsert_atom() INSERT ... ON CONFLICT (content_hash) → db.enqueue_outbox() mark for projection → outbox_sync.py ──▶ Qdrant (embedding + payload) └─▶ Memgraph (node) ``` ### 5.2 Research (gap → verified knowledge) ``` memory_stats / memory_list_gaps → detect gap memory_add_gap(question) → register web_search → URLs memory_ingest(url) → atoms (full page; snippet fallback if blocked) memory_verify → cross-confirm → verified ``` ### 5.3 Verification Two independent steps: 1. **Self-critique (LLM)** — verdict against the *nearest* atoms (by embedding, not by id): `new | duplicate | corroborates | contradicts`. `contradicts` → `contested`; `duplicate` → `stale`. 2. **Cross-confirmation (embeddings)** — the same fact found via ≥ 1 other `doc_id` (cosine ≥ 0.90) raises confidence and marks the atom `verified`. ### 5.4 Logical inference (graph) Rules are atoms of type `logical_rule` whose content is JSON: ```json {"if": "imports", "then": "depends_on"} {"if": ["depends_on", "depends_on"], "then": "depends_on"} ``` `inference.py` applies them to Memgraph edges repeatedly until fixpoint. Inferred edges are marked `inferred = true, source = 'inference'` so they are never confused with extracted facts. Only single-token kinds (no spaces) are accepted — human-readable rules from distillation are ignored. --- ## 6. MCP tools (16) | Tool | Purpose | |---|---| | `memory_search` | semantic search inside the base (Qdrant) | | `memory_get_context` | gather relevant context before answering | | `memory_stats` | counts: atoms / documents / open gaps | | `memory_store` | store one typed atom (with dedup) | | `memory_ingest` | ingest text/URL/file → atoms | | `memory_verify` | self-critique + cross-confirmation | | `memory_add_gap` | register a research gap | | `memory_list_gaps` | list gaps (optional status filter) | | `memory_get_gap` | read one gap by id | | `memory_update_gap` | edit question / reason / priority / status | | `memory_delete_gap` | delete a gap | | `memory_store_code` | extract + store code symbols | | `memory_search_code` | lexical search over code symbols | | `memory_code_graph` | build symbol + dependency graph | | `memory_infer` | run logical inference | | `memory_query_graph` | query graph edges/counters | --- ## 7. Deployment | Component | Endpoint | |---|---| | PostgreSQL | `127.0.0.1:5434` (atom-db container) | | Qdrant | `127.0.0.1:6333` (collection `alicorn_atoms`) | | Memgraph | `127.0.0.1:7687` (Bolt) | | Ollama | `127.0.0.1:11434` | The MCP server runs from the project venv and is registered in the dsh profile: ```yaml - id: mcp-pegas-memory name: '@deepseek-ai/dsh-mcp-client' config: serverName: pegas-memory transport: stdio command: /home/neo/workspace/pegas-memory/.venv/bin/python args: [/home/neo/workspace/pegas-memory/mcp_server.py] env: PROXY_URL: socks5://127.0.0.1:1080 # optional, for geo-blocked sources ``` **Hardware budget:** 1× RTX A2000 12 GB (24/7). EmbeddingGemma 2 (~0.4 GB) coexists with the agent LLM; large models do not. Memory is designed so embedding load stays low and background.