Add agent instructions, architecture doc and usage examples (English)
This commit is contained in:
n1kt4r committed
1 parent
83e77ea8c9
commit
a82d8c5a21
4 files changed
+539
-2
No files matched your search
+201
@@ -0,0 +1,201 @@
|
||||
# Pegas Memory — Architecture
|
||||
|
||||
Pegas Memory is the long-term memory and knowledge base for the **Pegas** research agent.
|
||||
It is a Python service exposed to the agent as an **MCP server** (14 tools).
|
||||
|
||||
---
|
||||
|
||||
## 1. High-level design
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ Pegas agent (dsh Research preset) │
|
||||
│ calls MCP tools: mcp__pegas-memory__* │
|
||||
└───────────────────┬──────────────────────┘
|
||||
│ JSON-RPC 2.0 / stdio
|
||||
┌───────────────────▼──────────────────────┐
|
||||
│ mcp_server.py (tool layer, 14 tools) │
|
||||
└───────────────────┬──────────────────────┘
|
||||
│ Python API
|
||||
┌────────────────────────────┼────────────────────────────┐
|
||||
│ │ │
|
||||
┌───────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐
|
||||
│ PostgreSQL │ outbox │ Qdrant │ │ Memgraph │
|
||||
│ SOURCE OF TRUTH│─────────▶│ vector slice │ │ graph slice │
|
||||
│ documents │ (CDC) │ EmbeddingGemma2 │ │ CodeSymbol │
|
||||
│ atoms, links │ │ dim = 768 │ │ Atom + RELATES │
|
||||
│ citations │ └─────────────────┘ └─────────────────┘
|
||||
│ code_symbols │
|
||||
│ code_dependency│
|
||||
│ episodic traces│
|
||||
│ research gaps │
|
||||
│ outbox │
|
||||
└────────────────┘
|
||||
```
|
||||
|
||||
**The one rule:** PostgreSQL is the single source of truth. Qdrant and Memgraph are
|
||||
**rebuildable projections** — if either is lost, it is rebuilt from PostgreSQL via the outbox.
|
||||
|
||||
---
|
||||
|
||||
## 2. Storage layers
|
||||
|
||||
| Layer | Engine | Role | Rebuildable |
|
||||
|---|---|---|---|
|
||||
| Canonical | **PostgreSQL** (16) | all authoritative data, provenance, transactions | — (source of truth) |
|
||||
| Vector | **Qdrant** | semantic search over atom embeddings (dim 768) | yes, from `pm_atom` |
|
||||
| Graph | **Memgraph** | relations, code dependencies, logical inference | yes, from `pm_atom_link` + `pm_code_dependency` |
|
||||
|
||||
**Embedding model:** `embeddinggemma-2-740m-Q8_0` (Google EmbeddingGemma 2, **dim = 768**),
|
||||
served by Ollama. Chosen over bge-m3 (dim 1024) for better Russian recall (R@1 0.922 vs 0.882)
|
||||
and lower GPU footprint. **Not interchangeable** with the old 1024-dim collections — different
|
||||
vector size means a different collection.
|
||||
|
||||
Deliberate non-choices: **pgvector** and **Apache AGE** are *not* used — dedicated engines
|
||||
(Qdrant for vectors, Memgraph for graphs) are faster and cleaner, while PostgreSQL keeps the
|
||||
canonical relational role.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data model (PostgreSQL)
|
||||
|
||||
```
|
||||
pm_document source registry (web|pdf|epub|text|code_repo), uri, authority, content_hash
|
||||
pm_chunk raw parsed segments (evidence), locator; NOT used for retrieval directly
|
||||
pm_atom TYPED KNOWLEDGE UNIT — the retrieval unit
|
||||
type ∈ {text, definition, logical_rule, code, table, fact}
|
||||
claim, content, content_hash (dedup), confidence,
|
||||
status ∈ {candidate, verified, stale, contested}, doc_id, metadata
|
||||
pm_atom_link relations between atoms: src → dst, kind, confidence, source, inferred
|
||||
kind ∈ {inference, cause, reference, contradicts, depends_on, calls, defines}
|
||||
pm_citation evidence: atom → document → locator → quoted_text
|
||||
pm_code_symbol tree-sitter symbols (repo, path, language, name, kind, signature, line range)
|
||||
pm_code_dependency code graph edges: src → dst, kind ∈ {calls, imports, inherits}
|
||||
pm_episodic_trace agent's own traces (plan/action/observation/reflection) — SEPARATE from curated KB
|
||||
pm_research_gap detected knowledge gaps (question, reason, priority, status)
|
||||
pm_outbox CDC queue: (entity_type, entity_id, op, processed) → Qdrant + Memgraph
|
||||
```
|
||||
|
||||
**Agent memory (episodic) and the knowledge base are deliberately separate**: traces are the
|
||||
agent's experience, atoms are curated facts.
|
||||
|
||||
---
|
||||
|
||||
## 4. Modules
|
||||
|
||||
| Module | Responsibility |
|
||||
|---|---|
|
||||
| `config.py` | service endpoints, model names, thresholds, `PROXY_URL` |
|
||||
| `embedding.py` | EmbeddingGemma 2 client (single + batch) |
|
||||
| `db.py` | PostgreSQL: upsert document/atom/link, content hashing, outbox enqueue |
|
||||
| `qdrant_store.py` | vector slice: `ensure_collection`, upsert, search (with payload filters) |
|
||||
| `memgraph_store.py` | graph slice: atom/CodeSymbol nodes, RELATES edges, rule application |
|
||||
| `ingest.py` | parsers: text (chunking), PDF, EPUB, web (trafilatura + httpx, proxy-aware) |
|
||||
| `distill.py` | LLM distillation of a chunk into typed atoms; `verify_atom` novelty scoring |
|
||||
| `pipeline.py` | orchestrator: document → chunks → atoms → PostgreSQL (+ dedup, outbox) |
|
||||
| `outbox_sync.py` | CDC: PostgreSQL outbox → Qdrant and Memgraph |
|
||||
| `dedup.py` | semantic deduplication by embedding cosine (threshold 0.90) |
|
||||
| `verify.py` | self-critique (LLM verdict) + cross-confirmation (multiple sources) |
|
||||
| `research.py` | gap-filling loop: SearXNG search → ingest → commit |
|
||||
| `code_symbols.py` | tree-sitter symbol extraction |
|
||||
| `code_graph.py` | tree-sitter dependency extraction (calls/imports/inherits) |
|
||||
| `inference.py` | forward-chaining logical rules over the graph until fixpoint |
|
||||
| `mcp_server.py` | MCP tool layer (JSON-RPC 2.0 over stdio) |
|
||||
|
||||
---
|
||||
|
||||
## 5. Data flows
|
||||
|
||||
### 5.1 Ingest (document → knowledge)
|
||||
|
||||
```
|
||||
source (text|pdf|epub|web)
|
||||
→ ingest.py split into chunks
|
||||
→ distill.py LLM → typed atoms
|
||||
→ dedup.py skip near-duplicates (cosine ≥ 0.90)
|
||||
→ db.upsert_atom() INSERT ... ON CONFLICT (content_hash)
|
||||
→ db.enqueue_outbox() mark for projection
|
||||
→ outbox_sync.py ──▶ Qdrant (embedding + payload)
|
||||
└─▶ Memgraph (node)
|
||||
```
|
||||
|
||||
### 5.2 Research (gap → verified knowledge)
|
||||
|
||||
```
|
||||
memory_stats / memory_list_gaps → detect gap
|
||||
memory_add_gap(question) → register
|
||||
SearXNG search → URLs
|
||||
memory_ingest(url) → atoms (full page; snippet fallback if blocked)
|
||||
memory_verify → cross-confirm → verified
|
||||
```
|
||||
|
||||
### 5.3 Verification
|
||||
|
||||
Two independent steps:
|
||||
|
||||
1. **Self-critique (LLM)** — verdict against the *nearest* atoms (by embedding, not by id):
|
||||
`new | duplicate | corroborates | contradicts`. `contradicts` → `contested`; `duplicate` → `stale`.
|
||||
2. **Cross-confirmation (embeddings)** — the same fact found via ≥ 1 other `doc_id`
|
||||
(cosine ≥ 0.90) raises confidence and marks the atom `verified`.
|
||||
|
||||
### 5.4 Logical inference (graph)
|
||||
|
||||
Rules are atoms of type `logical_rule` whose content is JSON:
|
||||
|
||||
```json
|
||||
{"if": "imports", "then": "depends_on"}
|
||||
{"if": ["depends_on", "depends_on"], "then": "depends_on"}
|
||||
```
|
||||
|
||||
`inference.py` applies them to Memgraph edges repeatedly until fixpoint. Inferred edges are
|
||||
marked `inferred = true, source = 'inference'` so they are never confused with extracted facts.
|
||||
Only single-token kinds (no spaces) are accepted — human-readable rules from distillation are ignored.
|
||||
|
||||
---
|
||||
|
||||
## 6. MCP tools (14)
|
||||
|
||||
| Tool | Purpose |
|
||||
|---|---|
|
||||
| `memory_search` | semantic search inside the base (Qdrant) |
|
||||
| `memory_get_context` | gather relevant context before answering |
|
||||
| `memory_stats` | counts: atoms / documents / open gaps |
|
||||
| `memory_store` | store one typed atom (with dedup) |
|
||||
| `memory_ingest` | ingest text/URL/file → atoms |
|
||||
| `memory_verify` | self-critique + cross-confirmation |
|
||||
| `memory_add_gap` | register a research gap |
|
||||
| `memory_list_gaps` | list open gaps |
|
||||
| `memory_research` | full research cycle for one question |
|
||||
| `memory_store_code` | extract + store code symbols |
|
||||
| `memory_search_code` | lexical search over code symbols |
|
||||
| `memory_code_graph` | build symbol + dependency graph |
|
||||
| `memory_infer` | run logical inference |
|
||||
| `memory_query_graph` | query graph edges/counters |
|
||||
|
||||
---
|
||||
|
||||
## 7. Deployment
|
||||
|
||||
| Component | Endpoint |
|
||||
|---|---|
|
||||
| PostgreSQL | `127.0.0.1:5434` (atom-db container) |
|
||||
| Qdrant | `127.0.0.1:6333` (collection `alicorn_atoms`) |
|
||||
| Memgraph | `127.0.0.1:7687` (Bolt) |
|
||||
| Ollama | `127.0.0.1:11434` |
|
||||
|
||||
The MCP server runs from the project venv and is registered in the dsh profile:
|
||||
|
||||
```yaml
|
||||
- id: mcp-pegas-memory
|
||||
name: '@deepseek-ai/dsh-mcp-client'
|
||||
config:
|
||||
serverName: pegas-memory
|
||||
transport: stdio
|
||||
command: /home/neo/workspace/pegas-memory/.venv/bin/python
|
||||
args: [/home/neo/workspace/pegas-memory/mcp_server.py]
|
||||
env:
|
||||
PROXY_URL: socks5://127.0.0.1:1080 # optional, for geo-blocked sources
|
||||
```
|
||||
|
||||
**Hardware budget:** 1× RTX A2000 12 GB (24/7). EmbeddingGemma 2 (~0.4 GB) coexists with the
|
||||
agent LLM; large models do not. Memory is designed so embedding load stays low and background.
|
||||
Reference in new issue
Block a user