KDD 2026 Tutorial · GitHub Pages Submission Site

Systematic Diagnosis and Benchmarking of Memory Systems in Autonomous AI Research Agents: A Low-Resource Framework

This page packages the KDD 2026 tutorial materials: source code, numbered notebooks, walkthrough, Modal GPU results, semantic evaluation, figures, selected logs, and reproducibility instructions.

Start the walkthrough Open notebook 1 in Colab Download final summary

Source Code

Curated Python implementation for memory providers, diagnostic probes, semantic evaluators, Modal GPU execution, and figures.

Open source README

Notebook Tutorial

Ten numbered Jupyter/Colab notebooks following the KDD tutorial timeline from setup to external testing integrations.

Open notebook guide

Walkthrough

Step-by-step teaching guide with local, Colab, and Modal GPU commands plus interpretation of final results.

Read walkthrough

Provenance Logs

Selected clean logs for Modal GPU, autoresearch-agent reevaluation, notebook execution, and local smoke checks.

Open Modal log

Tutorial title and tutors

Systematic Diagnosis and Benchmarking of Memory Systems in Autonomous AI Research Agents: A Low-Resource Framework

This tutorial introduces a practical framework for diagnosing retrieval, context use, and evaluation failures in memory systems for autonomous research agents.

Nur Arifin Akbar

Università degli Studi di Palermo, Dipartimento Matematica e Informatica, Italy.

Researcher working on AI systems, autonomous agents, retrieval-augmented generation, and memory architectures for long-horizon research workflows.

nurarifin.akbar@unipa.it

Rahool Dembani

Singularlogic, Athens, Greece.

Researcher and AI engineer focused on production LLM systems, hybrid memory architectures, enterprise AI, scalability, reliability, and cost-effective autonomous systems.

rdembani@singularlogic.eu

Gregorius Airlangga

Atma Jaya Catholic University of Indonesia, Jakarta, Indonesia.

Researcher and faculty member working on retrieval-augmented generation, natural language processing, AI evaluation, reproducibility, and resource-efficient AI.

gregorius.airlangga@atmajaya.ac.id

Ripto Mukti Wibowo

Atma Jaya Catholic University of Indonesia, Jakarta, Indonesia.

Researcher working on autonomous AI systems, machine learning, computational intelligence, memory optimisation, resource-constrained deployment, and systematic AI evaluation.

ripto.wibowo@atmajaya.ac.id

Biagio Lenzitti

Università degli Studi di Palermo, Dipartimento Matematica e Informatica, Italy.

Researcher whose work spans health informatics, machine learning, cyber risk assessment, distributed data sharing, data security, and medical informatics.

biagio.lenzitti@unipa.it

Domenico Tegolo

Università degli Studi di Palermo, Dipartimento Matematica e Informatica, Italy.

Associate Professor working on computational vision, mathematical modelling, biomedical image analysis, neural networks, artificial intelligence, and image-analysis architectures.

domenico.tegolo@unipa.it

Open tutorial notebooks in Google Colab

Use these GitHub-backed Colab links when viewing the website. They open the rendered, runnable notebook instead of downloading a plain `.ipynb` file.

1. Setup

Open in Colab

2. Datasets

Open in Colab

3. Memory architectures

Open in Colab

4. Diagnostic probes

Open in Colab

5. Full benchmark

Open in Colab

6. Dashboard

Open in Colab

7. Autoresearch loop

Open in Colab

8. Timeline fit

Open in Colab

9. Colab guidance

Open in Colab

10. Integrations

Open in Colab

What participants learn

The tutorial teaches how to diagnose memory systems in autonomous research agents using reproducible, low-resource benchmarking. Participants compare memory strategies, inspect retrieval failures, test context utilization, and use results as memory for an autonomous research-agent loop.

INGESTINDEXSEARCHANSWEREVALUATEREPORT

Tutorial framing: The live path is low-resource and notebook-first. The paper-quality path uses Modal GPU to rerun the complete benchmark with full offline semantic coverage.
Datasets Memories Retrieve Evaluate Failure probesretrieval · use · cause Semantic layeroffline · Rhesis · Semantica Reportsfigures · logs · notebooks A low-resource teaching loop plus a Modal GPU path for paper-quality full evaluation.

Walkthrough for KDD 2026 tutorial delivery

Setup and timeline. Start with notebook 1 to introduce the tutorial schedule, goals, and low-resource assumptions.
Dataset registry. Use notebook 2 to explain which datasets are included, optional, or externally downloaded.
Memory architectures. Use notebook 3 to compare no-memory, verbatim, extracted facts, episodic, and hybrid strategies.
Three diagnostic probes. Use notebook 4 for retrieval relevance, context utilization, and failure root-cause analysis.
Full benchmark and dashboard. Use notebook 5 and notebook 6 to reproduce tables and figures.
Autonomous research-agent loop. Use notebook 7 to generate ideas, retrieve prior experiment memories, and debug decisions.
Fit analysis and next steps. Use notebook 8, notebook 9, and notebook 10 for organizer guidance and optional Rhesis/Semantica checks.

Notebook timeline

Part 1architectures Part 2diagnostic probes Part 3benchmarking Part 4applications 1–345–67–10 Numbered notebooks map to the accepted KDD tutorial flow while preserving a low-resource Colab path.

Final paper-quality benchmark

The final Modal GPU run evaluates LoCoMo, LongMemEval, and MemoryArena with full offline semantic coverage. No-memory is a clean failure baseline; LongMemEval provides consistent retrieval-backed results; MemoryArena strongly favors extracted facts.

DatasetStrategyQuestionsPrecisionRecallHitSem. Cov.Sem.Pass
LoCoMono_memory15400.0000.0000.0001.0000.0000.000
LoCoMoverbatim15400.0570.2490.2771.0000.4320.277
LoCoMoextracted_facts15400.0550.2420.2711.0000.4220.271
LoCoMoepisodic15400.0640.2780.3111.0000.4650.311
LoCoMohybrid15400.0580.2560.2841.0000.4350.284
LongMemEvalverbatim15000.3930.5160.5751.0000.6360.575
LongMemEvalextracted_facts15000.3930.5150.5731.0000.6350.573
LongMemEvalepisodic15000.3930.5160.5751.0000.6360.575
MemoryArenaverbatim48500.1800.3480.3561.0000.3670.356
MemoryArenaextracted_facts48500.6130.7850.7941.0000.8040.794
MemoryArenaepisodic48500.1900.3500.3591.0000.3700.359

Metrics JSON · Summary TSV

Figures and interpretation

Real retrieval diagnostics
Retrieval diagnostics. Compares evidence hit rates across datasets and memory strategies. It shows why a single memory architecture does not dominate.
Semantic scores
Semantic scores. Adds a paper-quality evaluation layer beyond raw retrieval IDs, highlighting MemoryArena extracted facts and LongMemEval consistency.
Semantic vs retrieval
Semantic-vs-retrieval alignment. Shows whether stronger retrieval recall translates into more faithful/evaluable answers.
Evaluator coverage
Evaluator coverage. Confirms complete offline semantic coverage for the final Modal GPU run.
Failure modes
Failure analysis. Summarizes diagnosed memory failures such as retrieval misses, low-signal priors, and redundant ideas.
Memory growth
Memory growth. Helps participants reason about cost, storage, and retrieval quality as memories accumulate.
Idea novelty
Idea novelty. Supports the autonomous research-agent case study by visualizing generated idea novelty.
LoCoMo retrieval
LoCoMo detail. Focuses on the long-conversation setting where episodic memory performs best but remains challenging.

Autonomous research-agent case study

The agent turns benchmark results into experiment memory, proposes new research ideas, retrieves prior outcomes, and decides whether to keep, revise, or discard each idea.

  • Ideas: 12
  • Kept: 3, revised: 3, discarded: 6
  • Memory utilization rate: 0.5833
  • Redundant idea rate: 0.5
  • Mean semantic score: 0.3047

Open case-study report

Research-agentmemory-debug loop Generate ideas Retrieve memory Keep / revise Diagnose failure

Reproduce

python -m compileall -q source
python source/run.py --mode synthetic --backend offline --episodes 5

For full paper-quality runs, provide the real datasets separately and use Modal GPU:

python source/run.py \
  --runner modal \
  --modal-gpu A10G \
  --modal-detach \
  --mode real \
  --backend offline \
  --datasets locomo longmemeval memoryarena \
  --max-conversations 999 \
  --max-questions 999 \
  --top-k 5 \
  --eval-backend offline \
  --visualize

Submission package

Final metrics

Real benchmark JSON
Autoresearch JSON

Summaries

Real benchmark TSV
Autoresearch TSV

Logs

Modal GPU log
Autoresearch log
Notebook 5 log

Colab notebooks

Benchmark notebook in Colab
Integrations notebook in Colab

Data availability

The full real datasets are not vendored in this clean submission. See data/README.md for details. API keys must be provided through environment variables only.