Pegas Memory — Architecture

Pegas Memory is the long-term memory and knowledge base for the Pegas research agent. It is a Python service exposed to the agent as an MCP server (13 tools).

1. High-level design

                 ┌──────────────────────────────────────────┐
                 │  Pegas agent (dsh Research preset)       │
                 │  calls MCP tools: mcp__pegas-memory__*   │
                 └───────────────────┬──────────────────────┘
                                     │  JSON-RPC 2.0 / stdio
                 ┌───────────────────▼──────────────────────┐
                 │  mcp_server.py  (tool layer, 13 tools)   │
                 └───────────────────┬──────────────────────┘
                                     │  Python API
        ┌────────────────────────────┼────────────────────────────┐
        │                            │                            │
┌───────▼────────┐          ┌────────▼────────┐          ┌────────▼────────┐
│  PostgreSQL    │  outbox  │     Qdrant      │          │    Memgraph     │
│ SOURCE OF TRUTH│─────────▶│ vector slice    │          │  graph slice    │
│ documents      │   (CDC)  │ EmbeddingGemma2 │          │  CodeSymbol     │
│ atoms, links   │          │ dim = 768       │          │  Atom + RELATES │
│ citations      │          └─────────────────┘          └─────────────────┘
│ code_symbols   │
│ code_dependency│
│ episodic traces│
│ research gaps  │
│ outbox         │
└────────────────┘

The one rule: PostgreSQL is the single source of truth. Qdrant and Memgraph are rebuildable projections — if either is lost, it is rebuilt from PostgreSQL via the outbox.

2. Storage layers

Layer	Engine	Role	Rebuildable
Canonical	PostgreSQL (16)	all authoritative data, provenance, transactions	— (source of truth)
Vector	Qdrant	semantic search over atom embeddings (dim 768)	yes, from `pm_atom`
Graph	Memgraph	relations, code dependencies, logical inference	yes, from `pm_atom_link` + `pm_code_dependency`

Embedding model: embeddinggemma-2-740m-Q8_0 (Google EmbeddingGemma 2, dim = 768), served by Ollama. Chosen over bge-m3 (dim 1024) for better Russian recall (R@1 0.922 vs 0.882) and lower GPU footprint. Not interchangeable with the old 1024-dim collections — different vector size means a different collection.

Deliberate non-choices: pgvector and Apache AGE are not used — dedicated engines (Qdrant for vectors, Memgraph for graphs) are faster and cleaner, while PostgreSQL keeps the canonical relational role.

3. Data model (PostgreSQL)

pm_document         source registry (web|pdf|epub|text|code_repo), uri, authority, content_hash
pm_chunk            raw parsed segments (evidence), locator; NOT used for retrieval directly
pm_atom             TYPED KNOWLEDGE UNIT — the retrieval unit
                      type ∈ {text, definition, logical_rule, code, table, fact}
                      claim, content, content_hash (dedup), confidence,
                      status ∈ {candidate, verified, stale, contested}, doc_id, metadata
pm_atom_link        relations between atoms: src → dst, kind, confidence, source, inferred
                      kind ∈ {inference, cause, reference, contradicts, depends_on, calls, defines}
pm_citation         evidence: atom → document → locator → quoted_text
pm_code_symbol      tree-sitter symbols (repo, path, language, name, kind, signature, line range)
pm_code_dependency  code graph edges: src → dst, kind ∈ {calls, imports, inherits}
pm_episodic_trace   agent's own traces (plan/action/observation/reflection) — SEPARATE from curated KB
pm_research_gap     detected knowledge gaps (question, reason, priority, status)
pm_outbox           CDC queue: (entity_type, entity_id, op, processed) → Qdrant + Memgraph

Agent memory (episodic) and the knowledge base are deliberately separate: traces are the agent's experience, atoms are curated facts.

4. Modules

Module	Responsibility
`config.py`	service endpoints, model names, thresholds, `PROXY_URL`
`embedding.py`	EmbeddingGemma 2 client (single + batch)
`db.py`	PostgreSQL: upsert document/atom/link, content hashing, outbox enqueue
`qdrant_store.py`	vector slice: `ensure_collection`, upsert, search (with payload filters)
`memgraph_store.py`	graph slice: atom/CodeSymbol nodes, RELATES edges, rule application
`ingest.py`	parsers: text (chunking), PDF, EPUB, web (trafilatura + httpx, proxy-aware)
`distill.py`	LLM distillation of a chunk into typed atoms; `verify_atom` novelty scoring
`pipeline.py`	orchestrator: document → chunks → atoms → PostgreSQL (+ dedup, outbox)
`outbox_sync.py`	CDC: PostgreSQL outbox → Qdrant and Memgraph
`dedup.py`	semantic deduplication by embedding cosine (threshold 0.90)
`verify.py`	self-critique (LLM verdict) + cross-confirmation (multiple sources)
`research.py`	gap-filling loop: web search → ingest → commit
`code_symbols.py`	tree-sitter symbol extraction
`code_graph.py`	tree-sitter dependency extraction (calls/imports/inherits)
`inference.py`	forward-chaining logical rules over the graph until fixpoint
`mcp_server.py`	MCP tool layer (JSON-RPC 2.0 over stdio)

5. Data flows

5.1 Ingest (document → knowledge)

source (text|pdf|epub|web)
  → ingest.py            split into chunks
  → distill.py           LLM → typed atoms
  → dedup.py             skip near-duplicates (cosine ≥ 0.90)
  → db.upsert_atom()     INSERT ... ON CONFLICT (content_hash)
  → db.enqueue_outbox()  mark for projection
  → outbox_sync.py       ──▶ Qdrant (embedding + payload)
                         └─▶ Memgraph (node)

5.2 Research (gap → verified knowledge)

memory_stats / memory_list_gaps  → detect gap
memory_add_gap(question)         → register
web_search                       → URLs
memory_ingest(url)               → atoms   (full page; snippet fallback if blocked)
memory_verify                    → cross-confirm → verified

5.3 Verification

Two independent steps:

Self-critique (LLM) — verdict against the nearest atoms (by embedding, not by id): new | duplicate | corroborates | contradicts. contradicts → contested; duplicate → stale.
Cross-confirmation (embeddings) — the same fact found via ≥ 1 other doc_id (cosine ≥ 0.90) raises confidence and marks the atom verified.

5.4 Logical inference (graph)

Rules are atoms of type logical_rule whose content is JSON:

{"if": "imports", "then": "depends_on"}
{"if": ["depends_on", "depends_on"], "then": "depends_on"}

inference.py applies them to Memgraph edges repeatedly until fixpoint. Inferred edges are marked inferred = true, source = 'inference' so they are never confused with extracted facts. Only single-token kinds (no spaces) are accepted — human-readable rules from distillation are ignored.

6. MCP tools (13)

Tool	Purpose
`memory_search`	semantic search inside the base (Qdrant)
`memory_get_context`	gather relevant context before answering
`memory_stats`	counts: atoms / documents / open gaps
`memory_store`	store one typed atom (with dedup)
`memory_ingest`	ingest text/URL/file → atoms
`memory_verify`	self-critique + cross-confirmation
`memory_add_gap`	register a research gap
`memory_list_gaps`	list open gaps
`memory_store_code`	extract + store code symbols
`memory_search_code`	lexical search over code symbols
`memory_code_graph`	build symbol + dependency graph
`memory_infer`	run logical inference
`memory_query_graph`	query graph edges/counters

7. Deployment

Component	Endpoint
PostgreSQL	`127.0.0.1:5434` (atom-db container)
Qdrant	`127.0.0.1:6333` (collection `alicorn_atoms`)
Memgraph	`127.0.0.1:7687` (Bolt)
Ollama	`127.0.0.1:11434`

The MCP server runs from the project venv and is registered in the dsh profile:

- id: mcp-pegas-memory
  name: '@deepseek-ai/dsh-mcp-client'
  config:
    serverName: pegas-memory
    transport: stdio
    command: /home/neo/workspace/pegas-memory/.venv/bin/python
    args: [/home/neo/workspace/pegas-memory/mcp_server.py]
    env:
      PROXY_URL: socks5://127.0.0.1:1080   # optional, for geo-blocked sources

Hardware budget: 1× RTX A2000 12 GB (24/7). EmbeddingGemma 2 (~0.4 GB) coexists with the agent LLM; large models do not. Memory is designed so embedding load stays low and background.

9.3 KiB Raw Blame History Unescape Escape