Systematic Diagnosis and Benchmarking of Memory Systems in Autonomous AI Research Agents: A Low-Resource Framework

A hands-on KDD'26 tutorial — diagnose, benchmark, and improve memory systems for autonomous research agents. Everything runs on a laptop; no GPU required for Phases 1–2.

Tutorial Workflow

┌─────────────────────────────────────────────────────────────────────┐
│                  4-PHASE HANDS-ON TUTORIAL (180 min)                │
├─────────────────────────────────────────────────────────────────────┤
│  Phase 1 (45 min)  30 memory techniques × 6 families (8 books) │
│                    LLM demos (offline fallback)            │
│  Phase 2 (40 min)  Diagnose + benchmark on public data (5 books)    │
│                    3-probe diag · utilization lab · LLM-as-judge    │
│  Phase 3 (35 min)  Real LLM-pretraining sweep on Modal L4 (4 books) │
│                    groups split by depth/batch · memory diagnostic │
│  Phase 4 (50 min)  Cognitive Memory Layer — constraint-aware    │
│                    memory: CML-lite build + diagnose (1 notebook)   │
├─────────────────────────────────────────────────────────────────────┤
│  4 storage providers: verbatim | extracted_facts | episodic | hybrid │
│  + cognitive_constraint (Phase 4) — constraint-aware retrieval   │
│  Honest metrics: util ≠ hit | semantic_pass ≠ hit | 3-arm reader │
└─────────────────────────────────────────────────────────────────────┘

What You'll Learn

🔬 Run in Google Colab

Every notebook below is a clickable Colab link (🔬). Click any name to open that specific notebook in Google Colab — the setup cell auto-clones the repo and installs dependencies.

Individual notebook links (all 18) are in the Phase tables below.

Setup (5 minutes)

Prerequisites

Install

# 1. Clone the repository
git clone -b tutorial-rebuild https://github.com/syaikhipin/kdd26-memdiag kdd-tutorial
cd kdd-tutorial

# 2. Install dependencies (offline-only, lightweight)
pip install numpy matplotlib pyyaml

# 3. Install the CLI package (optional but recommended)
pip install -e source # core package
pip install -e "source[api]" # + openai/tiktoken for LLM demos

# 4. Verify
python -m compileall -q source # should print nothing (success)

export OPENAI_API_KEY=-<your-key>
export OPENAI_BASE_URL=https://api.openai.com/v1

Quick Start

Open any notebook in Jupyter or Colab — each is independent and self-contained:

# Phase 1 (offline or ):
jupyter notebook tutorial/phase1_memory_architectures/01_short_term_memory.ipynb

# Phase 2 (offline, no key):
jupyter notebook tutorial/phase2_public_datasets/01_diagnostic_framework.ipynb

# Or use the CLI directly:
diagnostic-framework diagnose --strategies verbatim,extracted_facts,episodic,hybrid --top_k 5
benchmark-cli compare --providers verbatim,extracted_facts,episodic,hybrid --sample_size 20

Phase 1 Agent Memory Techniques (60 min)

Goal: tour all 30 memory techniques across 6 families, with hands-on demos. LLM-based techniques use an OpenAI-compatible endpoint; offline fallbacks are always available.

Adapted from Agent Memory Techniques by Nir Diamant (Apache-2.0).

The 6 families

Phase 1 notebooks (8)

Phase 2 Public Datasets (60 min, offline)

Goal: diagnose retrieval vs utilization failures on real benchmarks, then benchmark the 4 providers. No API key needed.

The 3-probe diagnostic framework

Phase 2 notebooks (5)

Phase 3 Autoresearch (60 min, Modal L4)

Goal: run a real LLM-pretraining sweep on a cloud GPU, aggregate across groups, and diagnose memory failures.

Group slices (each fits one L4)

Phase 3 notebooks (4)

Phase 4 Cognitive Memory Layer (50 min, offline)

Goal: go beyond storage memory. Build a tiny "CML-lite" provider that adds the one thing verbatim / episodic / extracted / hybrid all lack — constraint awareness: surfacing a stored rule ("I'm allergic to shellfish", "depth 16 → OOM") even when the query shares zero words with it. Then diagnose it with the same 3-probe / LoCoMo harness from Phase 2.

Inspired by the Cognitive Memory Layer (avinash-mall/CognitiveMemoryLayer), a neuro-inspired system (Docker + Postgres + Neo4j + Redis + ~25 GB of models) we cannot run in a Colab slot — so we capture its cognitive core in ~90 lines of pure Python, fully offline.

The 5 cognitive constraint types

Phase 4 notebook (1)

Results (honest, tutorial-scale)

LoCoMo — where the 4 architectures are distinct

Memory footprints differ 10x (419 / 1282 / 19 / 438) — genuinely different architectures. No single strategy dominates.

Autoresearch trace (real L4)

3-arm utilization lab results

Memory helps by +10pp accuracy (provider vs no-memory). The oracle ceiling (0.60) shows room for improvement — better retrieval/representation could close the gap.

CLI Package

pip install -e source # offline deps
pip install -e "source[api]" # + openai/tiktoken
diagnostic-framework diagnose --strategies verbatim,extracted_facts,episodic,hybrid --top_k 5
benchmark-cli compare --providers verbatim,extracted_facts,episodic,hybrid --sample_size 20
benchmark-cli serve --port 8080

LLM-as-Judge Lab

Notebook 04_llm_as_judge_lab.ipynb runs a live LLM judge and compares it to the offline token-overlap proxy. Where they disagree: semantic relevance the proxy misses. This is the proposal's headline evaluation protocol, demonstrated live.

Memory Diagnostic (Phase 3D)

5 memory conditions x K=5 repeated probes, frozen pre-decision snapshots, real LLM agent:

4-Phase Schedule (180 min)

Timing & Pacing

For a 180-min session: 158 min spare for lecture, discussion, Q&A.

FAQ

Glossary

Authors & Credits

Software & Data Attribution

• Phase 1 techniques adapted from Agent Memory Techniques by Nir Diamant (GitHub, Apache-2.0); see licenses/APACHE-2.0.txt. The Phase 1 technique notebooks are self-contained adaptations.
• Memory providers (verbatim, extracted_facts, episodic, hybrid) — original implementations inspired by techniques 06/10/09/13. MIT-licensed.
• Phase 4 cognitive-constraint provider ("CML-lite") — original pure-Python re-implementation of the key idea of the Cognitive Memory Layer (avinash-mall/CognitiveMemoryLayer). MIT-licensed; the reference system is neuro-inspired and is not vendored.
• LoCoMo — snap-research/locomo (research use).
• LongMemEval — xiaowu0162/longmemeval-cleaned (research use).
• MemoryArena — ZexueHe/memoryarena (research use).
• Autoresearch — Karpathy's nanochat-derived pretraining loop (our fork).
• CLI package (memory-diagnostic-toolkit) — original, MIT.
• All other code is original to the tutorial authors.