Add agent instructions, architecture doc and usage examples (English)

AGENTS.md

@@ -0,0 +1,100 @@
+# Pegas Memory — Agent Instructions
+
+You are a **research agent**. Your long-term memory and knowledge base is **Pegas Memory**,
+exposed through the MCP tools `mcp__pegas-memory__*`.
+
+Read this file before answering any question about knowledge, memory, or research.
+
+---
+
+## 1. Golden rules
+
+1. **Your memory comes FIRST.** Before searching the web, call `memory_get_context` or
+   `memory_search`. You may already know the answer.
+2. **NEVER search your own knowledge base on the web.** `web_search` and `mcp__searxng__*`
+   query the OUTSIDE internet only. Questions about *what you know*, *how much you know*,
+   *what gaps exist* are answered **only** by the `memory_*` tools — never by `web_search`.
+3. **Save typed atoms, not paragraphs.** One atom = one atomic statement (a fact, a definition,
+   a rule). No fluff, no marketing text.
+4. **Verify before trusting.** `memory_verify` + cross-confirmation from 2 or more independent
+   sources. One weak source is not knowledge.
+5. **Do not duplicate.** Call `memory_search` before `memory_store`.
+6. **Never guess.** If the base has nothing, say so and start research.
+
+---
+
+## 2. Which tool for which question
+
+| User asks / Intent | Tool to call |
+|---|---|
+| "What do you know?" · "What's in your knowledge base?" · "How many atoms/documents?" | `memory_stats` |
+| "What should you research next?" · "What gaps exist?" · "What is missing?" | `memory_list_gaps` |
+| "What do you know about X?" · "Find X in memory" | `memory_search` |
+| "Give me context about X" (before answering) | `memory_get_context` |
+| "Research X on the internet" | `memory_research` |
+| "Save/store this fact" | `memory_store` |
+| "Ingest this URL / file / text" | `memory_ingest` |
+| "Is X true?" · "Check this fact" | `memory_verify` |
+| "Analyze this code / build a code graph" | `memory_code_graph` |
+| "Find a function/class named X" | `memory_search_code` |
+| "Show relations / graph edges" | `memory_query_graph` |
+| "Remember to research X later" | `memory_add_gap` |
+| "Run logical inference" | `memory_infer` |
+
+**Ambiguity rule:** if the question is about *knowledge content* → memory tools.
+If it is about *the outside world right now* → `web_search` / `searxng`.
+
+---
+
+## 3. Research loop (closing knowledge gaps)
+
+```
+1. DETECT   memory_stats / memory_list_gaps  → what is missing?
+2. CHECK    memory_search                    → do I already know it?
+3. PLAN     memory_add_gap(question)         → register the gap
+4. ACQUIRE  searxng search / read URL        → find sources
+5. EXTRACT  memory_ingest(text|url|path)     → distill into atoms
+6. VERIFY   memory_verify                    → candidate → verified/contested
+7. COMMIT   (automatic: ingest writes to PostgreSQL → Qdrant → Memgraph)
+```
+
+Never skip DETECT and CHECK: the most common mistake is re-researching what you already know.
+
+---
+
+## 4. Atom types
+
+| Type | When to use | `content` format |
+|---|---|---|
+| `definition` | term + its meaning | plain text |
+| `fact` | a checkable statement with a number/date | plain text |
+| `text` | other valuable knowledge | plain text |
+| `logical_rule` | "if X then Y" | JSON `{"if": ..., "then": ...}` |
+| `code` | a function/class/snippet | source code |
+| `table` | tabular data | JSON or markdown table |
+
+Every atom needs a **claim** (one short sentence) and, where possible, provenance
+(the source doc_id or URL is attached automatically by `memory_ingest`).
+
+---
+
+## 5. Common mistakes to avoid
+
+- **Wrong tool for own memory.** "What do you know?" → `memory_stats`, **not** `web_search`.
+  Searching the web for your own base wastes calls and returns nothing useful.
+- **Reading a directory.** `read` expects a *file*. Use `bash` (`ls`) or the fs-search tool
+  for directories. `read /some/dir` fails with "not a regular file".
+- **Storing raw text.** Do not `memory_store` whole articles. Distill first
+  (`memory_ingest`) or write one atomic claim.
+- **Skipping verification.** An unverified atom stays `candidate`. Cross-confirm it.
+- **Re-researching known facts.** Always `memory_search` before going to the internet.
+- **Inventing numbers.** `memory_stats` returns exact counts; never estimate them.
+
+---
+
+## 6. Expected answer style
+
+- Lead with the fact, then provenance (where it came from).
+- If a fact is `candidate` (unverified), say so.
+- If the base has nothing, say "not in the knowledge base" and offer to research it.
+- Never present web content as memory, and never present memory as live web data.

ARCHITECTURE.md

@@ -0,0 +1,201 @@
+# Pegas Memory — Architecture
+
+Pegas Memory is the long-term memory and knowledge base for the **Pegas** research agent.
+It is a Python service exposed to the agent as an **MCP server** (14 tools).
+
+---
+
+## 1. High-level design
+
+```
+                 ┌──────────────────────────────────────────┐
+                 │  Pegas agent (dsh Research preset)       │
+                 │  calls MCP tools: mcp__pegas-memory__*   │
+                 └───────────────────┬──────────────────────┘
+                                     │  JSON-RPC 2.0 / stdio
+                 ┌───────────────────▼──────────────────────┐
+                 │  mcp_server.py  (tool layer, 14 tools)   │
+                 └───────────────────┬──────────────────────┘
+                                     │  Python API
+        ┌────────────────────────────┼────────────────────────────┐
+        │                            │                            │
+┌───────▼────────┐          ┌────────▼────────┐          ┌────────▼────────┐
+│  PostgreSQL    │  outbox  │     Qdrant      │          │    Memgraph     │
+│ SOURCE OF TRUTH│─────────▶│ vector slice    │          │  graph slice    │
+│ documents      │   (CDC)  │ EmbeddingGemma2 │          │  CodeSymbol     │
+│ atoms, links   │          │ dim = 768       │          │  Atom + RELATES │
+│ citations      │          └─────────────────┘          └─────────────────┘
+│ code_symbols   │
+│ code_dependency│
+│ episodic traces│
+│ research gaps  │
+│ outbox         │
+└────────────────┘
+```
+
+**The one rule:** PostgreSQL is the single source of truth. Qdrant and Memgraph are
+**rebuildable projections** — if either is lost, it is rebuilt from PostgreSQL via the outbox.
+
+---
+
+## 2. Storage layers
+
+| Layer | Engine | Role | Rebuildable |
+|---|---|---|---|
+| Canonical | **PostgreSQL** (16) | all authoritative data, provenance, transactions | — (source of truth) |
+| Vector | **Qdrant** | semantic search over atom embeddings (dim 768) | yes, from `pm_atom` |
+| Graph | **Memgraph** | relations, code dependencies, logical inference | yes, from `pm_atom_link` + `pm_code_dependency` |
+
+**Embedding model:** `embeddinggemma-2-740m-Q8_0` (Google EmbeddingGemma 2, **dim = 768**),
+served by Ollama. Chosen over bge-m3 (dim 1024) for better Russian recall (R@1 0.922 vs 0.882)
+and lower GPU footprint. **Not interchangeable** with the old 1024-dim collections — different
+vector size means a different collection.
+
+Deliberate non-choices: **pgvector** and **Apache AGE** are *not* used — dedicated engines
+(Qdrant for vectors, Memgraph for graphs) are faster and cleaner, while PostgreSQL keeps the
+canonical relational role.
+
+---
+
+## 3. Data model (PostgreSQL)
+
+```
+pm_document         source registry (web|pdf|epub|text|code_repo), uri, authority, content_hash
+pm_chunk            raw parsed segments (evidence), locator; NOT used for retrieval directly
+pm_atom             TYPED KNOWLEDGE UNIT — the retrieval unit
+                      type ∈ {text, definition, logical_rule, code, table, fact}
+                      claim, content, content_hash (dedup), confidence,
+                      status ∈ {candidate, verified, stale, contested}, doc_id, metadata
+pm_atom_link        relations between atoms: src → dst, kind, confidence, source, inferred
+                      kind ∈ {inference, cause, reference, contradicts, depends_on, calls, defines}
+pm_citation         evidence: atom → document → locator → quoted_text
+pm_code_symbol      tree-sitter symbols (repo, path, language, name, kind, signature, line range)
+pm_code_dependency  code graph edges: src → dst, kind ∈ {calls, imports, inherits}
+pm_episodic_trace   agent's own traces (plan/action/observation/reflection) — SEPARATE from curated KB
+pm_research_gap     detected knowledge gaps (question, reason, priority, status)
+pm_outbox           CDC queue: (entity_type, entity_id, op, processed) → Qdrant + Memgraph
+```
+
+**Agent memory (episodic) and the knowledge base are deliberately separate**: traces are the
+agent's experience, atoms are curated facts.
+
+---
+
+## 4. Modules
+
+| Module | Responsibility |
+|---|---|
+| `config.py` | service endpoints, model names, thresholds, `PROXY_URL` |
+| `embedding.py` | EmbeddingGemma 2 client (single + batch) |
+| `db.py` | PostgreSQL: upsert document/atom/link, content hashing, outbox enqueue |
+| `qdrant_store.py` | vector slice: `ensure_collection`, upsert, search (with payload filters) |
+| `memgraph_store.py` | graph slice: atom/CodeSymbol nodes, RELATES edges, rule application |
+| `ingest.py` | parsers: text (chunking), PDF, EPUB, web (trafilatura + httpx, proxy-aware) |
+| `distill.py` | LLM distillation of a chunk into typed atoms; `verify_atom` novelty scoring |
+| `pipeline.py` | orchestrator: document → chunks → atoms → PostgreSQL (+ dedup, outbox) |
+| `outbox_sync.py` | CDC: PostgreSQL outbox → Qdrant and Memgraph |
+| `dedup.py` | semantic deduplication by embedding cosine (threshold 0.90) |
+| `verify.py` | self-critique (LLM verdict) + cross-confirmation (multiple sources) |
+| `research.py` | gap-filling loop: SearXNG search → ingest → commit |
+| `code_symbols.py` | tree-sitter symbol extraction |
+| `code_graph.py` | tree-sitter dependency extraction (calls/imports/inherits) |
+| `inference.py` | forward-chaining logical rules over the graph until fixpoint |
+| `mcp_server.py` | MCP tool layer (JSON-RPC 2.0 over stdio) |
+
+---
+
+## 5. Data flows
+
+### 5.1 Ingest (document → knowledge)
+
+```
+source (text|pdf|epub|web)
+  → ingest.py            split into chunks
+  → distill.py           LLM → typed atoms
+  → dedup.py             skip near-duplicates (cosine ≥ 0.90)
+  → db.upsert_atom()     INSERT ... ON CONFLICT (content_hash)
+  → db.enqueue_outbox()  mark for projection
+  → outbox_sync.py       ──▶ Qdrant (embedding + payload)
+                         └─▶ Memgraph (node)
+```
+
+### 5.2 Research (gap → verified knowledge)
+
+```
+memory_stats / memory_list_gaps  → detect gap
+memory_add_gap(question)         → register
+SearXNG search                   → URLs
+memory_ingest(url)               → atoms   (full page; snippet fallback if blocked)
+memory_verify                    → cross-confirm → verified
+```
+
+### 5.3 Verification
+
+Two independent steps:
+
+1. **Self-critique (LLM)** — verdict against the *nearest* atoms (by embedding, not by id):
+   `new | duplicate | corroborates | contradicts`. `contradicts` → `contested`; `duplicate` → `stale`.
+2. **Cross-confirmation (embeddings)** — the same fact found via ≥ 1 other `doc_id`
+   (cosine ≥ 0.90) raises confidence and marks the atom `verified`.
+
+### 5.4 Logical inference (graph)
+
+Rules are atoms of type `logical_rule` whose content is JSON:
+
+```json
+{"if": "imports", "then": "depends_on"}
+{"if": ["depends_on", "depends_on"], "then": "depends_on"}
+```
+
+`inference.py` applies them to Memgraph edges repeatedly until fixpoint. Inferred edges are
+marked `inferred = true, source = 'inference'` so they are never confused with extracted facts.
+Only single-token kinds (no spaces) are accepted — human-readable rules from distillation are ignored.
+
+---
+
+## 6. MCP tools (14)
+
+| Tool | Purpose |
+|---|---|
+| `memory_search` | semantic search inside the base (Qdrant) |
+| `memory_get_context` | gather relevant context before answering |
+| `memory_stats` | counts: atoms / documents / open gaps |
+| `memory_store` | store one typed atom (with dedup) |
+| `memory_ingest` | ingest text/URL/file → atoms |
+| `memory_verify` | self-critique + cross-confirmation |
+| `memory_add_gap` | register a research gap |
+| `memory_list_gaps` | list open gaps |
+| `memory_research` | full research cycle for one question |
+| `memory_store_code` | extract + store code symbols |
+| `memory_search_code` | lexical search over code symbols |
+| `memory_code_graph` | build symbol + dependency graph |
+| `memory_infer` | run logical inference |
+| `memory_query_graph` | query graph edges/counters |
+
+---
+
+## 7. Deployment
+
+| Component | Endpoint |
+|---|---|
+| PostgreSQL | `127.0.0.1:5434` (atom-db container) |
+| Qdrant | `127.0.0.1:6333` (collection `alicorn_atoms`) |
+| Memgraph | `127.0.0.1:7687` (Bolt) |
+| Ollama | `127.0.0.1:11434` |
+
+The MCP server runs from the project venv and is registered in the dsh profile:
+
+```yaml
+- id: mcp-pegas-memory
+  name: '@deepseek-ai/dsh-mcp-client'
+  config:
+    serverName: pegas-memory
+    transport: stdio
+    command: /home/neo/workspace/pegas-memory/.venv/bin/python
+    args: [/home/neo/workspace/pegas-memory/mcp_server.py]
+    env:
+      PROXY_URL: socks5://127.0.0.1:1080   # optional, for geo-blocked sources
+```
+
+**Hardware budget:** 1× RTX A2000 12 GB (24/7). EmbeddingGemma 2 (~0.4 GB) coexists with the
+agent LLM; large models do not. Memory is designed so embedding load stays low and background.

EXAMPLES.md

@@ -0,0 +1,231 @@
+# Pegas Memory — Usage Examples
+
+Concrete examples of how the agent should interact with the knowledge base.
+All calls go through the MCP tools `mcp__pegas-memory__*`.
+
+---
+
+## Example 1 — "What do you know?" (the most common mistake)
+
+**User:** "What's in your knowledge base?"
+
+### ✗ Wrong (a real observed failure)
+
+```
+memory_search("knowledge base")       → []      (semantic search finds nothing useful)
+read("/home/neo/workspace")           → ERROR   (it is a directory, not a file)
+web_search("memory")                  → ❌ searching the INTERNET for your OWN base
+web_search("knowledge base")          → ❌
+web_search("atoms")                   → ❌
+web_search("logical_rule")            → ❌
+```
+
+That is 2 wrong calls + 4 wasted web searches.
+
+### ✓ Correct
+
+```
+memory_stats()
+```
+
+**Result:**
+
+```json
+{"atoms": 82, "documents": 6, "open_gaps": 0}
+```
+
+**Answer:** "My knowledge base holds 82 atoms across 6 documents; no open research gaps."
+
+---
+
+## Example 2 — "What do you know about X?"
+
+**User:** "What do you know about Memgraph?"
+
+### ✓ Correct
+
+```
+memory_search("Memgraph", top_k = 5)
+```
+
+**Result:**
+
+```json
+{"query": "Memgraph",
+ "hits": [
+   {"id": 4, "score": 0.91, "type": "fact",
+    "claim": "Memgraph is an in-memory graph database compatible with Cypher",
+    "status": "verified", "confidence": 1.0, "doc_id": 1},
+   {"id": 9, "score": 0.87, "type": "definition",
+    "claim": "Memgraph is an in-memory graph DB using Cypher and Bolt",
+    "status": "candidate", "confidence": 0.9, "doc_id": 2}
+ ]}
+```
+
+**Answer:** "Yes — an in-memory graph database, Cypher-compatible, faster than Neo4j on
+analytical workloads. One of these atoms is still `candidate` (not cross-confirmed yet)."
+
+If nothing is found → say "not in the knowledge base" and offer to research it.
+
+---
+
+## Example 3 — Filling a knowledge gap (full loop)
+
+**User:** "Research how HNSW indexing works in vector databases."
+
+```
+1. memory_search("HNSW")            → []                        # not known yet
+2. memory_add_gap("How does HNSW work?",
+                  reason = "low_coverage", priority = 0.8)
+3. memory_research("How does HNSW indexing work?")
+                                    → searches SearXNG, ingests sources,
+                                      distills atoms, saves them
+4. memory_verify()                  → cross-confirm the new atoms
+5. memory_search("HNSW")            → now returns verified hits
+```
+
+**Note:** step 1 (CHECK) is mandatory — it prevents re-researching what is already known.
+
+---
+
+## Example 4 — Storing one fact
+
+**User:** "Remember that Memgraph keeps the whole graph in RAM."
+
+### ✓ Correct — one atomic claim + type
+
+```
+memory_store(
+  type       = "fact",
+  claim      = "Memgraph keeps the entire graph in RAM",
+  content    = "Memgraph stores the graph in memory, which is why it is faster than a disk-based graph DB on analytical workloads.",
+  confidence = 0.9
+)
+```
+
+**Result:**
+
+```json
+{"atom_id": 117, "created": true}
+```
+
+### ✗ Wrong
+
+```
+memory_store(type = "text", content = "<entire article pasted here>")
+```
+
+Raw articles are not atoms. Use `memory_ingest` to distill a document, or write one claim.
+
+---
+
+## Example 5 — Ingesting a document
+
+**User:** "Add this article to the base: https://example.com/graph-dbs"
+
+```
+memory_ingest(url = "https://example.com/graph-dbs")
+```
+
+**Result:**
+
+```json
+{"doc_id": 7, "chunks": 5, "atoms_created": 14,
+ "atoms_skipped": 3, "duplicates": 2}
+```
+
+- `duplicates` — near-duplicates filtered by the semantic deduplicator (cosine ≥ 0.90).
+- `atoms_skipped` — low-confidence atoms dropped by the distillation threshold.
+- If the site is geo-blocked and `PROXY_URL` is set, the fetch goes through the proxy;
+  otherwise a research-level snippet fallback is used.
+
+---
+
+## Example 6 — Verifying a fact
+
+**User:** "Is it true that Memgraph is faster than Neo4j?"
+
+```
+memory_verify(atom_id = 113, self_critique = true)
+```
+
+**Result:**
+
+```json
+{"atom_id": 113, "verdict": "corroborates", "corroborations": 2,
+ "new_status": "verified", "new_confidence": 1.0,
+ "reason": "Confirmed by two independent sources (doc 1 and doc 3)."}
+```
+
+Verdicts: `new` · `duplicate` (→ stale) · `corroborates` (→ verified) · `contradicts` (→ contested).
+
+---
+
+## Example 7 — Code analysis
+
+**User:** "What functions does this module call, and who depends on it?"
+
+```
+memory_code_graph(language = "python", source = "<file contents>",
+                  repo = "pegas-memory", path = "db.py")
+```
+
+**Result:**
+
+```json
+{"symbols_stored": 7, "dependencies_stored": 27}
+```
+
+Then:
+
+```
+memory_search_code("upsert")
+```
+
+**Result:**
+
+```json
+{"symbols": [
+  {"repo": "pegas-memory", "path": "db.py", "symbol_name": "upsert_atom",
+   "kind": "function", "line_start": 34}
+]}
+```
+
+Inference (edges `imports → depends_on`, transitive):
+
+```
+memory_infer(if = "imports", then = "depends_on")
+memory_infer(if = ["depends_on", "depends_on"], then = "depends_on")
+```
+
+---
+
+## Example 8 — Inspecting the graph
+
+```
+memory_query_graph(kind = "depends_on", top_k = 10)
+```
+
+**Result:**
+
+```json
+{"kind": "depends_on",
+ "edges": [
+   {"src": "db", "dst": "psycopg", "kind": "depends_on", "inferred": true},
+   {"src": "embedding", "dst": "httpx", "kind": "depends_on", "inferred": true}
+ ],
+ "nodes": 176, "code_symbols": 162}
+```
+
+---
+
+## Anti-patterns (do not do this)
+
+| Anti-pattern | Why it is wrong | Do instead |
+|---|---|---|
+| `web_search` to learn what your base contains | it searches the internet, not memory | `memory_stats` / `memory_search` |
+| `read` on a directory | `read` expects a file | `bash ls` or the fs-search tool |
+| `memory_store` of a whole article | violates the atomic-atom rule | `memory_ingest` |
+| Skipping `memory_search` before research | re-researches known facts | check first |
+| Reporting unverified atoms as facts | they are `candidate` | run `memory_verify` |
+| Estimating counts | `memory_stats` gives exact numbers | call it |

README.md

@@ -7,7 +7,12 @@
 - **Memgraph** — графовый срез (связи атомов, зависимости кода).
 - **Единица знания** — типизированный атом: text | definition | logical_rule | code | table | fact.
 
-Полный план: [ROADMAP.md](./ROADMAP.md)
+## Документация
+
+- [AGENTS.md](./AGENTS.md) — **инструкция для агента** (English): какой инструмент для какого вопроса, цикл исследования, типичные ошибки.
+- [ARCHITECTURE.md](./ARCHITECTURE.md) — архитектура: слои хранения, модель данных, потоки, MCP-инструменты.
+- [EXAMPLES.md](./EXAMPLES.md) — примеры работы с базой знаний (English), включая анти-примеры.
+- [ROADMAP.md](./ROADMAP.md) — план развития.
 
 ## Структура
 
@@ -18,7 +23,7 @@ pegas_memory/
   embedding.py     — EmbeddingGemma 2 через Ollama (dim=768)
   qdrant_store.py  — векторный срез
   memgraph_store.py— графовый срез
-  atoms.py         — модель атома, дистилляция, верификация
+  distill.py       — дистилляция чанка в атомы (LLM)
   ingest.py        — парсеры text/pdf/epub/web
   outbox_sync.py   — PG → Qdrant + Memgraph
   research.py      — цикл исследования пробелов