neo/pegas-memory
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add agent instructions, architecture doc and usage examples (English)

Browse Source
This commit is contained in:
n1kt4r committed 2026-10-15 15:36:41 +05:00
1 parent 83e77ea8c9
commit a82d8c5a21
4 files changed
+539 -2
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+100
View File
@@ -0,0 +1,100 @@
# Pegas Memory — Agent Instructions
You are a **research agent**. Your long-term memory and knowledge base is **Pegas Memory**,
exposed through the MCP tools `mcp__pegas-memory__*`.
Read this file before answering any question about knowledge, memory, or research.
---
## 1. Golden rules
1. **Your memory comes FIRST.** Before searching the web, call `memory_get_context` or
`memory_search`. You may already know the answer.
2. **NEVER search your own knowledge base on the web.** `web_search` and `mcp__searxng__*`
query the OUTSIDE internet only. Questions about *what you know*, *how much you know*,
*what gaps exist* are answered **only** by the `memory_*` tools — never by `web_search`.
3. **Save typed atoms, not paragraphs.** One atom = one atomic statement (a fact, a definition,
a rule). No fluff, no marketing text.
4. **Verify before trusting.** `memory_verify` + cross-confirmation from 2 or more independent
sources. One weak source is not knowledge.
5. **Do not duplicate.** Call `memory_search` before `memory_store`.
6. **Never guess.** If the base has nothing, say so and start research.
---
## 2. Which tool for which question
| User asks / Intent | Tool to call |
|---|---|
| "What do you know?" · "What's in your knowledge base?" · "How many atoms/documents?" | `memory_stats` |
| "What should you research next?" · "What gaps exist?" · "What is missing?" | `memory_list_gaps` |
| "What do you know about X?" · "Find X in memory" | `memory_search` |
| "Give me context about X" (before answering) | `memory_get_context` |
| "Research X on the internet" | `memory_research` |
| "Save/store this fact" | `memory_store` |
| "Ingest this URL / file / text" | `memory_ingest` |
| "Is X true?" · "Check this fact" | `memory_verify` |
| "Analyze this code / build a code graph" | `memory_code_graph` |
| "Find a function/class named X" | `memory_search_code` |
| "Show relations / graph edges" | `memory_query_graph` |
| "Remember to research X later" | `memory_add_gap` |
| "Run logical inference" | `memory_infer` |
**Ambiguity rule:** if the question is about *knowledge content* → memory tools.
If it is about *the outside world right now* → `web_search` / `searxng`.
---
## 3. Research loop (closing knowledge gaps)
```
1. DETECT memory_stats / memory_list_gaps → what is missing?
2. CHECK memory_search → do I already know it?
3. PLAN memory_add_gap(question) → register the gap
4. ACQUIRE searxng search / read URL → find sources
5. EXTRACT memory_ingest(text|url|path) → distill into atoms
6. VERIFY memory_verify → candidate → verified/contested
7. COMMIT (automatic: ingest writes to PostgreSQL → Qdrant → Memgraph)
```
Never skip DETECT and CHECK: the most common mistake is re-researching what you already know.
---
## 4. Atom types
| Type | When to use | `content` format |
|---|---|---|
| `definition` | term + its meaning | plain text |
| `fact` | a checkable statement with a number/date | plain text |
| `text` | other valuable knowledge | plain text |
| `logical_rule` | "if X then Y" | JSON `{"if": ..., "then": ...}` |
| `code` | a function/class/snippet | source code |
| `table` | tabular data | JSON or markdown table |
Every atom needs a **claim** (one short sentence) and, where possible, provenance
(the source doc_id or URL is attached automatically by `memory_ingest`).
---
## 5. Common mistakes to avoid
- **Wrong tool for own memory.** "What do you know?" → `memory_stats`, **not** `web_search`.
Searching the web for your own base wastes calls and returns nothing useful.
- **Reading a directory.** `read` expects a *file*. Use `bash` (`ls`) or the fs-search tool
for directories. `read /some/dir` fails with "not a regular file".
- **Storing raw text.** Do not `memory_store` whole articles. Distill first
(`memory_ingest`) or write one atomic claim.
- **Skipping verification.** An unverified atom stays `candidate`. Cross-confirm it.
- **Re-researching known facts.** Always `memory_search` before going to the internet.
- **Inventing numbers.** `memory_stats` returns exact counts; never estimate them.
---
## 6. Expected answer style
- Lead with the fact, then provenance (where it came from).
- If a fact is `candidate` (unverified), say so.
- If the base has nothing, say "not in the knowledge base" and offer to research it.
- Never present web content as memory, and never present memory as live web data.
ARCHITECTURE.md
+201
View File
@@ -0,0 +1,201 @@
# Pegas Memory — Architecture
Pegas Memory is the long-term memory and knowledge base for the **Pegas** research agent.
It is a Python service exposed to the agent as an **MCP server** (14 tools).
---
## 1. High-level design
```
┌──────────────────────────────────────────┐
│ Pegas agent (dsh Research preset) │
│ calls MCP tools: mcp__pegas-memory__* │
└───────────────────┬──────────────────────┘
│ JSON-RPC 2.0 / stdio
┌───────────────────▼──────────────────────┐
│ mcp_server.py (tool layer, 14 tools) │
└───────────────────┬──────────────────────┘
│ Python API
┌────────────────────────────┼────────────────────────────┐
│ │ │
┌───────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐
│ PostgreSQL │ outbox │ Qdrant │ │ Memgraph │
│ SOURCE OF TRUTH│─────────▶│ vector slice │ │ graph slice │
│ documents │ (CDC) │ EmbeddingGemma2 │ │ CodeSymbol │
│ atoms, links │ │ dim = 768 │ │ Atom + RELATES │
│ citations │ └─────────────────┘ └─────────────────┘
│ code_symbols │
│ code_dependency│
│ episodic traces│
│ research gaps │
│ outbox │
└────────────────┘
```
**The one rule:** PostgreSQL is the single source of truth. Qdrant and Memgraph are
**rebuildable projections** — if either is lost, it is rebuilt from PostgreSQL via the outbox.
---
## 2. Storage layers
| Layer | Engine | Role | Rebuildable |
|---|---|---|---|
| Canonical | **PostgreSQL** (16) | all authoritative data, provenance, transactions | — (source of truth) |
| Vector | **Qdrant** | semantic search over atom embeddings (dim 768) | yes, from `pm_atom` |
| Graph | **Memgraph** | relations, code dependencies, logical inference | yes, from `pm_atom_link` + `pm_code_dependency` |
**Embedding model:** `embeddinggemma-2-740m-Q8_0` (Google EmbeddingGemma 2, **dim = 768**),
served by Ollama. Chosen over bge-m3 (dim 1024) for better Russian recall (R@1 0.922 vs 0.882)
and lower GPU footprint. **Not interchangeable** with the old 1024-dim collections — different
vector size means a different collection.
Deliberate non-choices: **pgvector** and **Apache AGE** are *not* used — dedicated engines
(Qdrant for vectors, Memgraph for graphs) are faster and cleaner, while PostgreSQL keeps the
canonical relational role.
---
## 3. Data model (PostgreSQL)
```
pm_document source registry (web|pdf|epub|text|code_repo), uri, authority, content_hash
pm_chunk raw parsed segments (evidence), locator; NOT used for retrieval directly
pm_atom TYPED KNOWLEDGE UNIT — the retrieval unit
type ∈ {text, definition, logical_rule, code, table, fact}
claim, content, content_hash (dedup), confidence,
status ∈ {candidate, verified, stale, contested}, doc_id, metadata
pm_atom_link relations between atoms: src → dst, kind, confidence, source, inferred
kind ∈ {inference, cause, reference, contradicts, depends_on, calls, defines}
pm_citation evidence: atom → document → locator → quoted_text
pm_code_symbol tree-sitter symbols (repo, path, language, name, kind, signature, line range)
pm_code_dependency code graph edges: src → dst, kind ∈ {calls, imports, inherits}
pm_episodic_trace agent's own traces (plan/action/observation/reflection) — SEPARATE from curated KB
pm_research_gap detected knowledge gaps (question, reason, priority, status)
pm_outbox CDC queue: (entity_type, entity_id, op, processed) → Qdrant + Memgraph
```
**Agent memory (episodic) and the knowledge base are deliberately separate**: traces are the
agent's experience, atoms are curated facts.
---
## 4. Modules
| Module | Responsibility |
|---|---|
| `config.py` | service endpoints, model names, thresholds, `PROXY_URL` |
| `embedding.py` | EmbeddingGemma 2 client (single + batch) |
| `db.py` | PostgreSQL: upsert document/atom/link, content hashing, outbox enqueue |
| `qdrant_store.py` | vector slice: `ensure_collection`, upsert, search (with payload filters) |
| `memgraph_store.py` | graph slice: atom/CodeSymbol nodes, RELATES edges, rule application |
| `ingest.py` | parsers: text (chunking), PDF, EPUB, web (trafilatura + httpx, proxy-aware) |
| `distill.py` | LLM distillation of a chunk into typed atoms; `verify_atom` novelty scoring |
| `pipeline.py` | orchestrator: document → chunks → atoms → PostgreSQL (+ dedup, outbox) |
| `outbox_sync.py` | CDC: PostgreSQL outbox → Qdrant and Memgraph |
| `dedup.py` | semantic deduplication by embedding cosine (threshold 0.90) |
| `verify.py` | self-critique (LLM verdict) + cross-confirmation (multiple sources) |
| `research.py` | gap-filling loop: SearXNG search → ingest → commit |
| `code_symbols.py` | tree-sitter symbol extraction |
| `code_graph.py` | tree-sitter dependency extraction (calls/imports/inherits) |
| `inference.py` | forward-chaining logical rules over the graph until fixpoint |
| `mcp_server.py` | MCP tool layer (JSON-RPC 2.0 over stdio) |
---
## 5. Data flows
### 5.1 Ingest (document → knowledge)
```
source (text|pdf|epub|web)
→ ingest.py split into chunks
→ distill.py LLM → typed atoms
→ dedup.py skip near-duplicates (cosine ≥ 0.90)
→ db.upsert_atom() INSERT ... ON CONFLICT (content_hash)
→ db.enqueue_outbox() mark for projection
→ outbox_sync.py ──▶ Qdrant (embedding + payload)
└─▶ Memgraph (node)
```
### 5.2 Research (gap → verified knowledge)
```
memory_stats / memory_list_gaps → detect gap
memory_add_gap(question) → register
SearXNG search → URLs
memory_ingest(url) → atoms (full page; snippet fallback if blocked)
memory_verify → cross-confirm → verified
```
### 5.3 Verification
Two independent steps:
1. **Self-critique (LLM)** — verdict against the *nearest* atoms (by embedding, not by id):
`new | duplicate | corroborates | contradicts`. `contradicts` → `contested`; `duplicate` → `stale`.
2. **Cross-confirmation (embeddings)** — the same fact found via ≥ 1 other `doc_id`
(cosine ≥ 0.90) raises confidence and marks the atom `verified`.
### 5.4 Logical inference (graph)
Rules are atoms of type `logical_rule` whose content is JSON:
```json
{"if": "imports", "then": "depends_on"}
{"if": ["depends_on", "depends_on"], "then": "depends_on"}
```
`inference.py` applies them to Memgraph edges repeatedly until fixpoint. Inferred edges are
marked `inferred = true, source = 'inference'` so they are never confused with extracted facts.
Only single-token kinds (no spaces) are accepted — human-readable rules from distillation are ignored.
---
## 6. MCP tools (14)
| Tool | Purpose |
|---|---|
| `memory_search` | semantic search inside the base (Qdrant) |
| `memory_get_context` | gather relevant context before answering |
| `memory_stats` | counts: atoms / documents / open gaps |
| `memory_store` | store one typed atom (with dedup) |
| `memory_ingest` | ingest text/URL/file → atoms |
| `memory_verify` | self-critique + cross-confirmation |
| `memory_add_gap` | register a research gap |
| `memory_list_gaps` | list open gaps |
| `memory_research` | full research cycle for one question |
| `memory_store_code` | extract + store code symbols |
| `memory_search_code` | lexical search over code symbols |
| `memory_code_graph` | build symbol + dependency graph |
| `memory_infer` | run logical inference |
| `memory_query_graph` | query graph edges/counters |
---
## 7. Deployment
| Component | Endpoint |
|---|---|
| PostgreSQL | `127.0.0.1:5434` (atom-db container) |
| Qdrant | `127.0.0.1:6333` (collection `alicorn_atoms`) |
| Memgraph | `127.0.0.1:7687` (Bolt) |
| Ollama | `127.0.0.1:11434` |
The MCP server runs from the project venv and is registered in the dsh profile:
```yaml
- id: mcp-pegas-memory
name: '@deepseek-ai/dsh-mcp-client'
config:
serverName: pegas-memory
transport: stdio
command: /home/neo/workspace/pegas-memory/.venv/bin/python
args: [/home/neo/workspace/pegas-memory/mcp_server.py]
env:
PROXY_URL: socks5://127.0.0.1:1080 # optional, for geo-blocked sources
```
**Hardware budget:** 1× RTX A2000 12 GB (24/7). EmbeddingGemma 2 (~0.4 GB) coexists with the
agent LLM; large models do not. Memory is designed so embedding load stays low and background.
EXAMPLES.md
+231
View File
@@ -0,0 +1,231 @@
# Pegas Memory — Usage Examples
Concrete examples of how the agent should interact with the knowledge base.
All calls go through the MCP tools `mcp__pegas-memory__*`.
---
## Example 1 — "What do you know?" (the most common mistake)
**User:** "What's in your knowledge base?"
### ✗ Wrong (a real observed failure)
```
memory_search("knowledge base") → [] (semantic search finds nothing useful)
read("/home/neo/workspace") → ERROR (it is a directory, not a file)
web_search("memory") → ❌ searching the INTERNET for your OWN base
web_search("knowledge base") → ❌
web_search("atoms") → ❌
web_search("logical_rule") → ❌
```
That is 2 wrong calls + 4 wasted web searches.
### ✓ Correct
```
memory_stats()
```
**Result:**
```json
{"atoms": 82, "documents": 6, "open_gaps": 0}
```
**Answer:** "My knowledge base holds 82 atoms across 6 documents; no open research gaps."
---
## Example 2 — "What do you know about X?"
**User:** "What do you know about Memgraph?"
### ✓ Correct
```
memory_search("Memgraph", top_k = 5)
```
**Result:**
```json
{"query": "Memgraph",
"hits": [
{"id": 4, "score": 0.91, "type": "fact",
"claim": "Memgraph is an in-memory graph database compatible with Cypher",
"status": "verified", "confidence": 1.0, "doc_id": 1},
{"id": 9, "score": 0.87, "type": "definition",
"claim": "Memgraph is an in-memory graph DB using Cypher and Bolt",
"status": "candidate", "confidence": 0.9, "doc_id": 2}
]}
```
**Answer:** "Yes — an in-memory graph database, Cypher-compatible, faster than Neo4j on
analytical workloads. One of these atoms is still `candidate` (not cross-confirmed yet)."
If nothing is found → say "not in the knowledge base" and offer to research it.
---
## Example 3 — Filling a knowledge gap (full loop)
**User:** "Research how HNSW indexing works in vector databases."
```
1. memory_search("HNSW") → [] # not known yet
2. memory_add_gap("How does HNSW work?",
reason = "low_coverage", priority = 0.8)
3. memory_research("How does HNSW indexing work?")
→ searches SearXNG, ingests sources,
distills atoms, saves them
4. memory_verify() → cross-confirm the new atoms
5. memory_search("HNSW") → now returns verified hits
```
**Note:** step 1 (CHECK) is mandatory — it prevents re-researching what is already known.
---
## Example 4 — Storing one fact
**User:** "Remember that Memgraph keeps the whole graph in RAM."
### ✓ Correct — one atomic claim + type
```
memory_store(
type = "fact",
claim = "Memgraph keeps the entire graph in RAM",
content = "Memgraph stores the graph in memory, which is why it is faster than a disk-based graph DB on analytical workloads.",
confidence = 0.9
)
```
**Result:**
```json
{"atom_id": 117, "created": true}
```
### ✗ Wrong
```
memory_store(type = "text", content = "<entire article pasted here>")
```
Raw articles are not atoms. Use `memory_ingest` to distill a document, or write one claim.
---
## Example 5 — Ingesting a document
**User:** "Add this article to the base: https://example.com/graph-dbs"
```
memory_ingest(url = "https://example.com/graph-dbs")
```
**Result:**
```json
{"doc_id": 7, "chunks": 5, "atoms_created": 14,
"atoms_skipped": 3, "duplicates": 2}
```
- `duplicates` — near-duplicates filtered by the semantic deduplicator (cosine ≥ 0.90).
- `atoms_skipped` — low-confidence atoms dropped by the distillation threshold.
- If the site is geo-blocked and `PROXY_URL` is set, the fetch goes through the proxy;
otherwise a research-level snippet fallback is used.
---
## Example 6 — Verifying a fact
**User:** "Is it true that Memgraph is faster than Neo4j?"
```
memory_verify(atom_id = 113, self_critique = true)
```
**Result:**
```json
{"atom_id": 113, "verdict": "corroborates", "corroborations": 2,
"new_status": "verified", "new_confidence": 1.0,
"reason": "Confirmed by two independent sources (doc 1 and doc 3)."}
```
Verdicts: `new` · `duplicate` (→ stale) · `corroborates` (→ verified) · `contradicts` (→ contested).
---
## Example 7 — Code analysis
**User:** "What functions does this module call, and who depends on it?"
```
memory_code_graph(language = "python", source = "<file contents>",
repo = "pegas-memory", path = "db.py")
```
**Result:**
```json
{"symbols_stored": 7, "dependencies_stored": 27}
```
Then:
```
memory_search_code("upsert")
```
**Result:**
```json
{"symbols": [
{"repo": "pegas-memory", "path": "db.py", "symbol_name": "upsert_atom",
"kind": "function", "line_start": 34}
]}
```
Inference (edges `imports → depends_on`, transitive):
```
memory_infer(if = "imports", then = "depends_on")
memory_infer(if = ["depends_on", "depends_on"], then = "depends_on")
```
---
## Example 8 — Inspecting the graph
```
memory_query_graph(kind = "depends_on", top_k = 10)
```
**Result:**
```json
{"kind": "depends_on",
"edges": [
{"src": "db", "dst": "psycopg", "kind": "depends_on", "inferred": true},
{"src": "embedding", "dst": "httpx", "kind": "depends_on", "inferred": true}
],
"nodes": 176, "code_symbols": 162}
```
---
## Anti-patterns (do not do this)
| Anti-pattern | Why it is wrong | Do instead |
|---|---|---|
| `web_search` to learn what your base contains | it searches the internet, not memory | `memory_stats` / `memory_search` |
| `read` on a directory | `read` expects a file | `bash ls` or the fs-search tool |
| `memory_store` of a whole article | violates the atomic-atom rule | `memory_ingest` |
| Skipping `memory_search` before research | re-researches known facts | check first |
| Reporting unverified atoms as facts | they are `candidate` | run `memory_verify` |
| Estimating counts | `memory_stats` gives exact numbers | call it |
README.md
+7 -2
View File
@@ -7,7 +7,12 @@
- **Memgraph** — графовый срез (связи атомов, зависимости кода).
- **Единица знания** — типизированный атом: text | definition | logical_rule | code | table | fact.
Полный план: [ROADMAP.md](./ROADMAP.md)
## Документация
- [AGENTS.md](./AGENTS.md) — **инструкция для агента** (English): какой инструмент для какого вопроса, цикл исследования, типичные ошибки.
- [ARCHITECTURE.md](./ARCHITECTURE.md) — архитектура: слои хранения, модель данных, потоки, MCP-инструменты.
- [EXAMPLES.md](./EXAMPLES.md) — примеры работы с базой знаний (English), включая анти-примеры.
- [ROADMAP.md](./ROADMAP.md) — план развития.
## Структура
@@ -18,7 +23,7 @@ pegas_memory/
embedding.py — EmbeddingGemma 2 через Ollama (dim=768)
qdrant_store.py — векторный срез
memgraph_store.py— графовый срез
atoms.py — модель атома, дистилляция, верификация
distill.py — дистилляция чанка в атомы (LLM)
ingest.py — парсеры text/pdf/epub/web
outbox_sync.py — PG → Qdrant + Memgraph
research.py — цикл исследования пробелов