201 lines
9.3 KiB
Markdown
201 lines
9.3 KiB
Markdown
# Pegas Memory — Architecture
|
||
|
||
Pegas Memory is the long-term memory and knowledge base for the **Pegas** research agent.
|
||
It is a Python service exposed to the agent as an **MCP server** (13 tools).
|
||
|
||
---
|
||
|
||
## 1. High-level design
|
||
|
||
```
|
||
┌──────────────────────────────────────────┐
|
||
│ Pegas agent (dsh Research preset) │
|
||
│ calls MCP tools: mcp__pegas-memory__* │
|
||
└───────────────────┬──────────────────────┘
|
||
│ JSON-RPC 2.0 / stdio
|
||
┌───────────────────▼──────────────────────┐
|
||
│ mcp_server.py (tool layer, 13 tools) │
|
||
└───────────────────┬──────────────────────┘
|
||
│ Python API
|
||
┌────────────────────────────┼────────────────────────────┐
|
||
│ │ │
|
||
┌───────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐
|
||
│ PostgreSQL │ outbox │ Qdrant │ │ Memgraph │
|
||
│ SOURCE OF TRUTH│─────────▶│ vector slice │ │ graph slice │
|
||
│ documents │ (CDC) │ EmbeddingGemma2 │ │ CodeSymbol │
|
||
│ atoms, links │ │ dim = 768 │ │ Atom + RELATES │
|
||
│ citations │ └─────────────────┘ └─────────────────┘
|
||
│ code_symbols │
|
||
│ code_dependency│
|
||
│ episodic traces│
|
||
│ research gaps │
|
||
│ outbox │
|
||
└────────────────┘
|
||
```
|
||
|
||
**The one rule:** PostgreSQL is the single source of truth. Qdrant and Memgraph are
|
||
**rebuildable projections** — if either is lost, it is rebuilt from PostgreSQL via the outbox.
|
||
|
||
---
|
||
|
||
## 2. Storage layers
|
||
|
||
| Layer | Engine | Role | Rebuildable |
|
||
|---|---|---|---|
|
||
| Canonical | **PostgreSQL** (16) | all authoritative data, provenance, transactions | — (source of truth) |
|
||
| Vector | **Qdrant** | semantic search over atom embeddings (dim 768) | yes, from `pm_atom` |
|
||
| Graph | **Memgraph** | relations, code dependencies, logical inference | yes, from `pm_atom_link` + `pm_code_dependency` |
|
||
|
||
**Embedding model:** `embeddinggemma-2-740m-Q8_0` (Google EmbeddingGemma 2, **dim = 768**),
|
||
served by Ollama. Chosen over bge-m3 (dim 1024) for better Russian recall (R@1 0.922 vs 0.882)
|
||
and lower GPU footprint. **Not interchangeable** with the old 1024-dim collections — different
|
||
vector size means a different collection.
|
||
|
||
Deliberate non-choices: **pgvector** and **Apache AGE** are *not* used — dedicated engines
|
||
(Qdrant for vectors, Memgraph for graphs) are faster and cleaner, while PostgreSQL keeps the
|
||
canonical relational role.
|
||
|
||
---
|
||
|
||
## 3. Data model (PostgreSQL)
|
||
|
||
```
|
||
pm_document source registry (web|pdf|epub|text|code_repo), uri, authority, content_hash
|
||
pm_chunk raw parsed segments (evidence), locator; NOT used for retrieval directly
|
||
pm_atom TYPED KNOWLEDGE UNIT — the retrieval unit
|
||
type ∈ {text, definition, logical_rule, code, table, fact}
|
||
claim, content, content_hash (dedup), confidence,
|
||
status ∈ {candidate, verified, stale, contested}, doc_id, metadata
|
||
pm_atom_link relations between atoms: src → dst, kind, confidence, source, inferred
|
||
kind ∈ {inference, cause, reference, contradicts, depends_on, calls, defines}
|
||
pm_citation evidence: atom → document → locator → quoted_text
|
||
pm_code_symbol tree-sitter symbols (repo, path, language, name, kind, signature, line range)
|
||
pm_code_dependency code graph edges: src → dst, kind ∈ {calls, imports, inherits}
|
||
pm_episodic_trace agent's own traces (plan/action/observation/reflection) — SEPARATE from curated KB
|
||
pm_research_gap detected knowledge gaps (question, reason, priority, status)
|
||
pm_outbox CDC queue: (entity_type, entity_id, op, processed) → Qdrant + Memgraph
|
||
```
|
||
|
||
**Agent memory (episodic) and the knowledge base are deliberately separate**: traces are the
|
||
agent's experience, atoms are curated facts.
|
||
|
||
---
|
||
|
||
## 4. Modules
|
||
|
||
| Module | Responsibility |
|
||
|---|---|
|
||
| `config.py` | service endpoints, model names, thresholds, `PROXY_URL` |
|
||
| `embedding.py` | EmbeddingGemma 2 client (single + batch) |
|
||
| `db.py` | PostgreSQL: upsert document/atom/link, content hashing, outbox enqueue |
|
||
| `qdrant_store.py` | vector slice: `ensure_collection`, upsert, search (with payload filters) |
|
||
| `memgraph_store.py` | graph slice: atom/CodeSymbol nodes, RELATES edges, rule application |
|
||
| `ingest.py` | parsers: text (chunking), PDF, EPUB, web (trafilatura + httpx, proxy-aware) |
|
||
| `distill.py` | LLM distillation of a chunk into typed atoms; `verify_atom` novelty scoring |
|
||
| `pipeline.py` | orchestrator: document → chunks → atoms → PostgreSQL (+ dedup, outbox) |
|
||
| `outbox_sync.py` | CDC: PostgreSQL outbox → Qdrant and Memgraph |
|
||
| `dedup.py` | semantic deduplication by embedding cosine (threshold 0.90) |
|
||
| `verify.py` | self-critique (LLM verdict) + cross-confirmation (multiple sources) |
|
||
| `research.py` | gap-filling loop: web search → ingest → commit |
|
||
| `code_symbols.py` | tree-sitter symbol extraction |
|
||
| `code_graph.py` | tree-sitter dependency extraction (calls/imports/inherits) |
|
||
| `inference.py` | forward-chaining logical rules over the graph until fixpoint |
|
||
| `mcp_server.py` | MCP tool layer (JSON-RPC 2.0 over stdio) |
|
||
|
||
---
|
||
|
||
## 5. Data flows
|
||
|
||
### 5.1 Ingest (document → knowledge)
|
||
|
||
```
|
||
source (text|pdf|epub|web)
|
||
→ ingest.py split into chunks
|
||
→ distill.py LLM → typed atoms
|
||
→ dedup.py skip near-duplicates (cosine ≥ 0.90)
|
||
→ db.upsert_atom() INSERT ... ON CONFLICT (content_hash)
|
||
→ db.enqueue_outbox() mark for projection
|
||
→ outbox_sync.py ──▶ Qdrant (embedding + payload)
|
||
└─▶ Memgraph (node)
|
||
```
|
||
|
||
### 5.2 Research (gap → verified knowledge)
|
||
|
||
```
|
||
memory_stats / memory_list_gaps → detect gap
|
||
memory_add_gap(question) → register
|
||
web_search → URLs
|
||
memory_ingest(url) → atoms (full page; snippet fallback if blocked)
|
||
memory_verify → cross-confirm → verified
|
||
```
|
||
|
||
### 5.3 Verification
|
||
|
||
Two independent steps:
|
||
|
||
1. **Self-critique (LLM)** — verdict against the *nearest* atoms (by embedding, not by id):
|
||
`new | duplicate | corroborates | contradicts`. `contradicts` → `contested`; `duplicate` → `stale`.
|
||
2. **Cross-confirmation (embeddings)** — the same fact found via ≥ 1 other `doc_id`
|
||
(cosine ≥ 0.90) raises confidence and marks the atom `verified`.
|
||
|
||
### 5.4 Logical inference (graph)
|
||
|
||
Rules are atoms of type `logical_rule` whose content is JSON:
|
||
|
||
```json
|
||
{"if": "imports", "then": "depends_on"}
|
||
{"if": ["depends_on", "depends_on"], "then": "depends_on"}
|
||
```
|
||
|
||
`inference.py` applies them to Memgraph edges repeatedly until fixpoint. Inferred edges are
|
||
marked `inferred = true, source = 'inference'` so they are never confused with extracted facts.
|
||
Only single-token kinds (no spaces) are accepted — human-readable rules from distillation are ignored.
|
||
|
||
---
|
||
|
||
## 6. MCP tools (13)
|
||
|
||
| Tool | Purpose |
|
||
|---|---|
|
||
| `memory_search` | semantic search inside the base (Qdrant) |
|
||
| `memory_get_context` | gather relevant context before answering |
|
||
| `memory_stats` | counts: atoms / documents / open gaps |
|
||
| `memory_store` | store one typed atom (with dedup) |
|
||
| `memory_ingest` | ingest text/URL/file → atoms |
|
||
| `memory_verify` | self-critique + cross-confirmation |
|
||
| `memory_add_gap` | register a research gap |
|
||
| `memory_list_gaps` | list open gaps |
|
||
| `memory_store_code` | extract + store code symbols |
|
||
| `memory_search_code` | lexical search over code symbols |
|
||
| `memory_code_graph` | build symbol + dependency graph |
|
||
| `memory_infer` | run logical inference |
|
||
| `memory_query_graph` | query graph edges/counters |
|
||
|
||
---
|
||
|
||
## 7. Deployment
|
||
|
||
| Component | Endpoint |
|
||
|---|---|
|
||
| PostgreSQL | `127.0.0.1:5434` (atom-db container) |
|
||
| Qdrant | `127.0.0.1:6333` (collection `alicorn_atoms`) |
|
||
| Memgraph | `127.0.0.1:7687` (Bolt) |
|
||
| Ollama | `127.0.0.1:11434` |
|
||
|
||
The MCP server runs from the project venv and is registered in the dsh profile:
|
||
|
||
```yaml
|
||
- id: mcp-pegas-memory
|
||
name: '@deepseek-ai/dsh-mcp-client'
|
||
config:
|
||
serverName: pegas-memory
|
||
transport: stdio
|
||
command: /home/neo/workspace/pegas-memory/.venv/bin/python
|
||
args: [/home/neo/workspace/pegas-memory/mcp_server.py]
|
||
env:
|
||
PROXY_URL: socks5://127.0.0.1:1080 # optional, for geo-blocked sources
|
||
```
|
||
|
||
**Hardware budget:** 1× RTX A2000 12 GB (24/7). EmbeddingGemma 2 (~0.4 GB) coexists with the
|
||
agent LLM; large models do not. Memory is designed so embedding load stays low and background.
|