diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..507c862 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,100 @@ +# Pegas Memory — Agent Instructions + +You are a **research agent**. Your long-term memory and knowledge base is **Pegas Memory**, +exposed through the MCP tools `mcp__pegas-memory__*`. + +Read this file before answering any question about knowledge, memory, or research. + +--- + +## 1. Golden rules + +1. **Your memory comes FIRST.** Before searching the web, call `memory_get_context` or + `memory_search`. You may already know the answer. +2. **NEVER search your own knowledge base on the web.** `web_search` and `mcp__searxng__*` + query the OUTSIDE internet only. Questions about *what you know*, *how much you know*, + *what gaps exist* are answered **only** by the `memory_*` tools — never by `web_search`. +3. **Save typed atoms, not paragraphs.** One atom = one atomic statement (a fact, a definition, + a rule). No fluff, no marketing text. +4. **Verify before trusting.** `memory_verify` + cross-confirmation from 2 or more independent + sources. One weak source is not knowledge. +5. **Do not duplicate.** Call `memory_search` before `memory_store`. +6. **Never guess.** If the base has nothing, say so and start research. + +--- + +## 2. Which tool for which question + +| User asks / Intent | Tool to call | +|---|---| +| "What do you know?" · "What's in your knowledge base?" · "How many atoms/documents?" | `memory_stats` | +| "What should you research next?" · "What gaps exist?" · "What is missing?" | `memory_list_gaps` | +| "What do you know about X?" · "Find X in memory" | `memory_search` | +| "Give me context about X" (before answering) | `memory_get_context` | +| "Research X on the internet" | `memory_research` | +| "Save/store this fact" | `memory_store` | +| "Ingest this URL / file / text" | `memory_ingest` | +| "Is X true?" · "Check this fact" | `memory_verify` | +| "Analyze this code / build a code graph" | `memory_code_graph` | +| "Find a function/class named X" | `memory_search_code` | +| "Show relations / graph edges" | `memory_query_graph` | +| "Remember to research X later" | `memory_add_gap` | +| "Run logical inference" | `memory_infer` | + +**Ambiguity rule:** if the question is about *knowledge content* → memory tools. +If it is about *the outside world right now* → `web_search` / `searxng`. + +--- + +## 3. Research loop (closing knowledge gaps) + +``` +1. DETECT memory_stats / memory_list_gaps → what is missing? +2. CHECK memory_search → do I already know it? +3. PLAN memory_add_gap(question) → register the gap +4. ACQUIRE searxng search / read URL → find sources +5. EXTRACT memory_ingest(text|url|path) → distill into atoms +6. VERIFY memory_verify → candidate → verified/contested +7. COMMIT (automatic: ingest writes to PostgreSQL → Qdrant → Memgraph) +``` + +Never skip DETECT and CHECK: the most common mistake is re-researching what you already know. + +--- + +## 4. Atom types + +| Type | When to use | `content` format | +|---|---|---| +| `definition` | term + its meaning | plain text | +| `fact` | a checkable statement with a number/date | plain text | +| `text` | other valuable knowledge | plain text | +| `logical_rule` | "if X then Y" | JSON `{"if": ..., "then": ...}` | +| `code` | a function/class/snippet | source code | +| `table` | tabular data | JSON or markdown table | + +Every atom needs a **claim** (one short sentence) and, where possible, provenance +(the source doc_id or URL is attached automatically by `memory_ingest`). + +--- + +## 5. Common mistakes to avoid + +- **Wrong tool for own memory.** "What do you know?" → `memory_stats`, **not** `web_search`. + Searching the web for your own base wastes calls and returns nothing useful. +- **Reading a directory.** `read` expects a *file*. Use `bash` (`ls`) or the fs-search tool + for directories. `read /some/dir` fails with "not a regular file". +- **Storing raw text.** Do not `memory_store` whole articles. Distill first + (`memory_ingest`) or write one atomic claim. +- **Skipping verification.** An unverified atom stays `candidate`. Cross-confirm it. +- **Re-researching known facts.** Always `memory_search` before going to the internet. +- **Inventing numbers.** `memory_stats` returns exact counts; never estimate them. + +--- + +## 6. Expected answer style + +- Lead with the fact, then provenance (where it came from). +- If a fact is `candidate` (unverified), say so. +- If the base has nothing, say "not in the knowledge base" and offer to research it. +- Never present web content as memory, and never present memory as live web data. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..9a7b227 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,201 @@ +# Pegas Memory — Architecture + +Pegas Memory is the long-term memory and knowledge base for the **Pegas** research agent. +It is a Python service exposed to the agent as an **MCP server** (14 tools). + +--- + +## 1. High-level design + +``` + ┌──────────────────────────────────────────┐ + │ Pegas agent (dsh Research preset) │ + │ calls MCP tools: mcp__pegas-memory__* │ + └───────────────────┬──────────────────────┘ + │ JSON-RPC 2.0 / stdio + ┌───────────────────▼──────────────────────┐ + │ mcp_server.py (tool layer, 14 tools) │ + └───────────────────┬──────────────────────┘ + │ Python API + ┌────────────────────────────┼────────────────────────────┐ + │ │ │ +┌───────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐ +│ PostgreSQL │ outbox │ Qdrant │ │ Memgraph │ +│ SOURCE OF TRUTH│─────────▶│ vector slice │ │ graph slice │ +│ documents │ (CDC) │ EmbeddingGemma2 │ │ CodeSymbol │ +│ atoms, links │ │ dim = 768 │ │ Atom + RELATES │ +│ citations │ └─────────────────┘ └─────────────────┘ +│ code_symbols │ +│ code_dependency│ +│ episodic traces│ +│ research gaps │ +│ outbox │ +└────────────────┘ +``` + +**The one rule:** PostgreSQL is the single source of truth. Qdrant and Memgraph are +**rebuildable projections** — if either is lost, it is rebuilt from PostgreSQL via the outbox. + +--- + +## 2. Storage layers + +| Layer | Engine | Role | Rebuildable | +|---|---|---|---| +| Canonical | **PostgreSQL** (16) | all authoritative data, provenance, transactions | — (source of truth) | +| Vector | **Qdrant** | semantic search over atom embeddings (dim 768) | yes, from `pm_atom` | +| Graph | **Memgraph** | relations, code dependencies, logical inference | yes, from `pm_atom_link` + `pm_code_dependency` | + +**Embedding model:** `embeddinggemma-2-740m-Q8_0` (Google EmbeddingGemma 2, **dim = 768**), +served by Ollama. Chosen over bge-m3 (dim 1024) for better Russian recall (R@1 0.922 vs 0.882) +and lower GPU footprint. **Not interchangeable** with the old 1024-dim collections — different +vector size means a different collection. + +Deliberate non-choices: **pgvector** and **Apache AGE** are *not* used — dedicated engines +(Qdrant for vectors, Memgraph for graphs) are faster and cleaner, while PostgreSQL keeps the +canonical relational role. + +--- + +## 3. Data model (PostgreSQL) + +``` +pm_document source registry (web|pdf|epub|text|code_repo), uri, authority, content_hash +pm_chunk raw parsed segments (evidence), locator; NOT used for retrieval directly +pm_atom TYPED KNOWLEDGE UNIT — the retrieval unit + type ∈ {text, definition, logical_rule, code, table, fact} + claim, content, content_hash (dedup), confidence, + status ∈ {candidate, verified, stale, contested}, doc_id, metadata +pm_atom_link relations between atoms: src → dst, kind, confidence, source, inferred + kind ∈ {inference, cause, reference, contradicts, depends_on, calls, defines} +pm_citation evidence: atom → document → locator → quoted_text +pm_code_symbol tree-sitter symbols (repo, path, language, name, kind, signature, line range) +pm_code_dependency code graph edges: src → dst, kind ∈ {calls, imports, inherits} +pm_episodic_trace agent's own traces (plan/action/observation/reflection) — SEPARATE from curated KB +pm_research_gap detected knowledge gaps (question, reason, priority, status) +pm_outbox CDC queue: (entity_type, entity_id, op, processed) → Qdrant + Memgraph +``` + +**Agent memory (episodic) and the knowledge base are deliberately separate**: traces are the +agent's experience, atoms are curated facts. + +--- + +## 4. Modules + +| Module | Responsibility | +|---|---| +| `config.py` | service endpoints, model names, thresholds, `PROXY_URL` | +| `embedding.py` | EmbeddingGemma 2 client (single + batch) | +| `db.py` | PostgreSQL: upsert document/atom/link, content hashing, outbox enqueue | +| `qdrant_store.py` | vector slice: `ensure_collection`, upsert, search (with payload filters) | +| `memgraph_store.py` | graph slice: atom/CodeSymbol nodes, RELATES edges, rule application | +| `ingest.py` | parsers: text (chunking), PDF, EPUB, web (trafilatura + httpx, proxy-aware) | +| `distill.py` | LLM distillation of a chunk into typed atoms; `verify_atom` novelty scoring | +| `pipeline.py` | orchestrator: document → chunks → atoms → PostgreSQL (+ dedup, outbox) | +| `outbox_sync.py` | CDC: PostgreSQL outbox → Qdrant and Memgraph | +| `dedup.py` | semantic deduplication by embedding cosine (threshold 0.90) | +| `verify.py` | self-critique (LLM verdict) + cross-confirmation (multiple sources) | +| `research.py` | gap-filling loop: SearXNG search → ingest → commit | +| `code_symbols.py` | tree-sitter symbol extraction | +| `code_graph.py` | tree-sitter dependency extraction (calls/imports/inherits) | +| `inference.py` | forward-chaining logical rules over the graph until fixpoint | +| `mcp_server.py` | MCP tool layer (JSON-RPC 2.0 over stdio) | + +--- + +## 5. Data flows + +### 5.1 Ingest (document → knowledge) + +``` +source (text|pdf|epub|web) + → ingest.py split into chunks + → distill.py LLM → typed atoms + → dedup.py skip near-duplicates (cosine ≥ 0.90) + → db.upsert_atom() INSERT ... ON CONFLICT (content_hash) + → db.enqueue_outbox() mark for projection + → outbox_sync.py ──▶ Qdrant (embedding + payload) + └─▶ Memgraph (node) +``` + +### 5.2 Research (gap → verified knowledge) + +``` +memory_stats / memory_list_gaps → detect gap +memory_add_gap(question) → register +SearXNG search → URLs +memory_ingest(url) → atoms (full page; snippet fallback if blocked) +memory_verify → cross-confirm → verified +``` + +### 5.3 Verification + +Two independent steps: + +1. **Self-critique (LLM)** — verdict against the *nearest* atoms (by embedding, not by id): + `new | duplicate | corroborates | contradicts`. `contradicts` → `contested`; `duplicate` → `stale`. +2. **Cross-confirmation (embeddings)** — the same fact found via ≥ 1 other `doc_id` + (cosine ≥ 0.90) raises confidence and marks the atom `verified`. + +### 5.4 Logical inference (graph) + +Rules are atoms of type `logical_rule` whose content is JSON: + +```json +{"if": "imports", "then": "depends_on"} +{"if": ["depends_on", "depends_on"], "then": "depends_on"} +``` + +`inference.py` applies them to Memgraph edges repeatedly until fixpoint. Inferred edges are +marked `inferred = true, source = 'inference'` so they are never confused with extracted facts. +Only single-token kinds (no spaces) are accepted — human-readable rules from distillation are ignored. + +--- + +## 6. MCP tools (14) + +| Tool | Purpose | +|---|---| +| `memory_search` | semantic search inside the base (Qdrant) | +| `memory_get_context` | gather relevant context before answering | +| `memory_stats` | counts: atoms / documents / open gaps | +| `memory_store` | store one typed atom (with dedup) | +| `memory_ingest` | ingest text/URL/file → atoms | +| `memory_verify` | self-critique + cross-confirmation | +| `memory_add_gap` | register a research gap | +| `memory_list_gaps` | list open gaps | +| `memory_research` | full research cycle for one question | +| `memory_store_code` | extract + store code symbols | +| `memory_search_code` | lexical search over code symbols | +| `memory_code_graph` | build symbol + dependency graph | +| `memory_infer` | run logical inference | +| `memory_query_graph` | query graph edges/counters | + +--- + +## 7. Deployment + +| Component | Endpoint | +|---|---| +| PostgreSQL | `127.0.0.1:5434` (atom-db container) | +| Qdrant | `127.0.0.1:6333` (collection `alicorn_atoms`) | +| Memgraph | `127.0.0.1:7687` (Bolt) | +| Ollama | `127.0.0.1:11434` | + +The MCP server runs from the project venv and is registered in the dsh profile: + +```yaml +- id: mcp-pegas-memory + name: '@deepseek-ai/dsh-mcp-client' + config: + serverName: pegas-memory + transport: stdio + command: /home/neo/workspace/pegas-memory/.venv/bin/python + args: [/home/neo/workspace/pegas-memory/mcp_server.py] + env: + PROXY_URL: socks5://127.0.0.1:1080 # optional, for geo-blocked sources +``` + +**Hardware budget:** 1× RTX A2000 12 GB (24/7). EmbeddingGemma 2 (~0.4 GB) coexists with the +agent LLM; large models do not. Memory is designed so embedding load stays low and background. diff --git a/EXAMPLES.md b/EXAMPLES.md new file mode 100644 index 0000000..517e14d --- /dev/null +++ b/EXAMPLES.md @@ -0,0 +1,231 @@ +# Pegas Memory — Usage Examples + +Concrete examples of how the agent should interact with the knowledge base. +All calls go through the MCP tools `mcp__pegas-memory__*`. + +--- + +## Example 1 — "What do you know?" (the most common mistake) + +**User:** "What's in your knowledge base?" + +### ✗ Wrong (a real observed failure) + +``` +memory_search("knowledge base") → [] (semantic search finds nothing useful) +read("/home/neo/workspace") → ERROR (it is a directory, not a file) +web_search("memory") → ❌ searching the INTERNET for your OWN base +web_search("knowledge base") → ❌ +web_search("atoms") → ❌ +web_search("logical_rule") → ❌ +``` + +That is 2 wrong calls + 4 wasted web searches. + +### ✓ Correct + +``` +memory_stats() +``` + +**Result:** + +```json +{"atoms": 82, "documents": 6, "open_gaps": 0} +``` + +**Answer:** "My knowledge base holds 82 atoms across 6 documents; no open research gaps." + +--- + +## Example 2 — "What do you know about X?" + +**User:** "What do you know about Memgraph?" + +### ✓ Correct + +``` +memory_search("Memgraph", top_k = 5) +``` + +**Result:** + +```json +{"query": "Memgraph", + "hits": [ + {"id": 4, "score": 0.91, "type": "fact", + "claim": "Memgraph is an in-memory graph database compatible with Cypher", + "status": "verified", "confidence": 1.0, "doc_id": 1}, + {"id": 9, "score": 0.87, "type": "definition", + "claim": "Memgraph is an in-memory graph DB using Cypher and Bolt", + "status": "candidate", "confidence": 0.9, "doc_id": 2} + ]} +``` + +**Answer:** "Yes — an in-memory graph database, Cypher-compatible, faster than Neo4j on +analytical workloads. One of these atoms is still `candidate` (not cross-confirmed yet)." + +If nothing is found → say "not in the knowledge base" and offer to research it. + +--- + +## Example 3 — Filling a knowledge gap (full loop) + +**User:** "Research how HNSW indexing works in vector databases." + +``` +1. memory_search("HNSW") → [] # not known yet +2. memory_add_gap("How does HNSW work?", + reason = "low_coverage", priority = 0.8) +3. memory_research("How does HNSW indexing work?") + → searches SearXNG, ingests sources, + distills atoms, saves them +4. memory_verify() → cross-confirm the new atoms +5. memory_search("HNSW") → now returns verified hits +``` + +**Note:** step 1 (CHECK) is mandatory — it prevents re-researching what is already known. + +--- + +## Example 4 — Storing one fact + +**User:** "Remember that Memgraph keeps the whole graph in RAM." + +### ✓ Correct — one atomic claim + type + +``` +memory_store( + type = "fact", + claim = "Memgraph keeps the entire graph in RAM", + content = "Memgraph stores the graph in memory, which is why it is faster than a disk-based graph DB on analytical workloads.", + confidence = 0.9 +) +``` + +**Result:** + +```json +{"atom_id": 117, "created": true} +``` + +### ✗ Wrong + +``` +memory_store(type = "text", content = "") +``` + +Raw articles are not atoms. Use `memory_ingest` to distill a document, or write one claim. + +--- + +## Example 5 — Ingesting a document + +**User:** "Add this article to the base: https://example.com/graph-dbs" + +``` +memory_ingest(url = "https://example.com/graph-dbs") +``` + +**Result:** + +```json +{"doc_id": 7, "chunks": 5, "atoms_created": 14, + "atoms_skipped": 3, "duplicates": 2} +``` + +- `duplicates` — near-duplicates filtered by the semantic deduplicator (cosine ≥ 0.90). +- `atoms_skipped` — low-confidence atoms dropped by the distillation threshold. +- If the site is geo-blocked and `PROXY_URL` is set, the fetch goes through the proxy; + otherwise a research-level snippet fallback is used. + +--- + +## Example 6 — Verifying a fact + +**User:** "Is it true that Memgraph is faster than Neo4j?" + +``` +memory_verify(atom_id = 113, self_critique = true) +``` + +**Result:** + +```json +{"atom_id": 113, "verdict": "corroborates", "corroborations": 2, + "new_status": "verified", "new_confidence": 1.0, + "reason": "Confirmed by two independent sources (doc 1 and doc 3)."} +``` + +Verdicts: `new` · `duplicate` (→ stale) · `corroborates` (→ verified) · `contradicts` (→ contested). + +--- + +## Example 7 — Code analysis + +**User:** "What functions does this module call, and who depends on it?" + +``` +memory_code_graph(language = "python", source = "", + repo = "pegas-memory", path = "db.py") +``` + +**Result:** + +```json +{"symbols_stored": 7, "dependencies_stored": 27} +``` + +Then: + +``` +memory_search_code("upsert") +``` + +**Result:** + +```json +{"symbols": [ + {"repo": "pegas-memory", "path": "db.py", "symbol_name": "upsert_atom", + "kind": "function", "line_start": 34} +]} +``` + +Inference (edges `imports → depends_on`, transitive): + +``` +memory_infer(if = "imports", then = "depends_on") +memory_infer(if = ["depends_on", "depends_on"], then = "depends_on") +``` + +--- + +## Example 8 — Inspecting the graph + +``` +memory_query_graph(kind = "depends_on", top_k = 10) +``` + +**Result:** + +```json +{"kind": "depends_on", + "edges": [ + {"src": "db", "dst": "psycopg", "kind": "depends_on", "inferred": true}, + {"src": "embedding", "dst": "httpx", "kind": "depends_on", "inferred": true} + ], + "nodes": 176, "code_symbols": 162} +``` + +--- + +## Anti-patterns (do not do this) + +| Anti-pattern | Why it is wrong | Do instead | +|---|---|---| +| `web_search` to learn what your base contains | it searches the internet, not memory | `memory_stats` / `memory_search` | +| `read` on a directory | `read` expects a file | `bash ls` or the fs-search tool | +| `memory_store` of a whole article | violates the atomic-atom rule | `memory_ingest` | +| Skipping `memory_search` before research | re-researches known facts | check first | +| Reporting unverified atoms as facts | they are `candidate` | run `memory_verify` | +| Estimating counts | `memory_stats` gives exact numbers | call it | diff --git a/README.md b/README.md index 639b15c..ad86d1e 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,12 @@ - **Memgraph** — графовый срез (связи атомов, зависимости кода). - **Единица знания** — типизированный атом: text | definition | logical_rule | code | table | fact. -Полный план: [ROADMAP.md](./ROADMAP.md) +## Документация + +- [AGENTS.md](./AGENTS.md) — **инструкция для агента** (English): какой инструмент для какого вопроса, цикл исследования, типичные ошибки. +- [ARCHITECTURE.md](./ARCHITECTURE.md) — архитектура: слои хранения, модель данных, потоки, MCP-инструменты. +- [EXAMPLES.md](./EXAMPLES.md) — примеры работы с базой знаний (English), включая анти-примеры. +- [ROADMAP.md](./ROADMAP.md) — план развития. ## Структура @@ -18,7 +23,7 @@ pegas_memory/ embedding.py — EmbeddingGemma 2 через Ollama (dim=768) qdrant_store.py — векторный срез memgraph_store.py— графовый срез - atoms.py — модель атома, дистилляция, верификация + distill.py — дистилляция чанка в атомы (LLM) ingest.py — парсеры text/pdf/epub/web outbox_sync.py — PG → Qdrant + Memgraph research.py — цикл исследования пробелов