The Research Landscape

A dense cluster of arxiv preprints released in late May and early June 2025 collectively constitutes a snapshot of the agent engineering discipline at a moment of rapid maturation. These preprints span six dominant technical themes: skill lifecycle management and self-evolution, memory architectures and their failure modes, evaluation and failure diagnosis tooling, safety and adversarial attack research, inference efficiency, and production deployment frameworks. Individual systems report benchmark gains ranging from 7 percentage points to over 44 percentage points, suggesting that each theme is producing measurable, if uneven, progress.

Skill Management and Self-Evolution

The most active cluster of papers addresses how agents acquire, organize, and retire procedural skills. SLIM (Skill LIfecycle Management) treats the active external skill set as a dynamic optimization variable, using leave-one-skill-out validation to estimate each skill’s marginal contribution and applying three lifecycle operations: retention, retirement, and expansion. On ALFWorld and SearchQA, SLIM outperforms the best baselines by an average of 7.1 percentage points [2].

FederatedSkill takes a privacy-preserving approach to collaborative skill evolution, transmitting semantic skill diffs rather than raw trajectories. Evaluated across 20 distinct agent task families, it achieves up to a 44.4 percent increase in success rate and a 37.5 percent reduction in computational cost over self-evolving baselines [42]. SkillPyramid introduces a hierarchical skill consolidation framework with a self-evolution mechanism for composing and validating new skills during task execution; across ALFWorld, WebShop, and ScienceWorld it increases average reward by 38.0 percent and reduces execution steps by 27.7 percent [41].

LatentSkill converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork, storing skill knowledge in weight space rather than context space. On ALFWorld, LatentSkill improves success by 21.4 and 13.4 points on seen and unseen splits respectively, while using 64.1 percent fewer prefill tokens [56].

Memory Systems: Architectures and Failure Modes

Memory research in this wave divides into new architectural proposals and a sobering set of benchmark findings about where all current systems break down.

On the architectural side, delta-mem augments a frozen full-attention backbone with a compact 8x8 online state matrix updated by delta-rule learning, achieving 1.31x the backbone score on MemoryAgentBench and 1.20x on LoCoMo without full fine-tuning [5]. Mem-pi uses a dedicated model to generate context-specific guidance on demand rather than retrieving static entries, achieving over 30 percent relative improvement on web navigation tasks [16]. MRAgent represents memory as a Cue-Tag-Content graph with an active reconstruction mechanism, reporting improvements of up to 23 percent over strong baselines on LoCoMo and LongMemEval [59]. CoMem decouples memory management from the primary agent workflow via a k-step-off asynchronous pipeline, delivering 1.4x latency improvements on SWE-Bench-Verified while preserving most task performance [40].

The MEME benchmark exposes a structural failure shared by all evaluated systems. Across six tasks covering multi-entity and evolving memory, all six evaluated memory systems collapse on dependency reasoning under default configurations: Cascade accuracy averages 3 percent and Absence accuracy averages 1 percent, despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and stronger LLMs largely fail to close this gap. Only a file-based agent paired with Claude Opus 4.7 partially recovers, but at approximately 70x the baseline cost [8].

Evaluation, Diagnosis, and Benchmarking

A recurring finding across the evaluation papers is that methodology, not model capability, is the primary bottleneck in understanding agent failures.

The holistic span-level diagnosis framework pairs top-down agent-level diagnosis with bottom-up per-span assessment. On the TRAIL benchmark it achieves relative gains over the strongest prior baselines of up to 38 percent on category F1, up to 3.5x on localization accuracy, and up to 12.5x on joint localization-categorization accuracy. Notably, the same frontier model achieves several times higher localization accuracy when used inside the framework than as a monolithic judge over the full trace [14].

Causal Agent Replay (CAR) addresses attribution by modeling agent runs as structural causal models and applying do-operations to individual steps, re-executing the trajectory forward to measure shifts in outcome distribution. The approach is motivated by the observation that state-of-the-art step-level accuracy on the Who&When benchmark using LLM-judge attribution is approximately 14 percent [68].

EvalAgent automates the end-to-end evaluation pipeline by encoding evaluation domain expertise as reusable skills. It improves the Eval@1 metric from 17.5 percent to 65 percent and achieves 79.5 percent human expert preference over baseline approaches; removing evaluation skills causes Eval@1 to drop back to 30 percent [11]. The Agent Planning Benchmark (APB) provides a planning-specific diagnostic with 4,209 multimodal cases across 22 domains, revealing systematic weaknesses in long-horizon planning, tool-noise robustness, and calibrated refusal across 12 evaluated models [51]. CL-Bench introduces continual learning evaluation across six expert-validated domains, finding that dedicated memory systems do not outperform naive in-context learning on cross-instance knowledge reuse [60].

Safety, Adversarial Attacks, and Defenses

The safety papers cover both novel attack surfaces and the limits of existing defenses.

Metis reformulates jailbreaking as inference-time policy optimization within an adversarial partially observable Markov decision process. Across 10 diverse models it achieves an average Attack Success Rate of 89.2 percent, including 76.0 percent on O1 and 78.0 percent on GPT-5-chat. By replacing stochastic search with directed optimization, Metis reduces token costs by an average of 8.2x and up to 11.4x [1].

Mobius Injection weaponizes autonomous agents into zombie nodes to launch agent-based DDoS attacks by exploiting a structural vulnerability called Semantic Closure. Experiments show single-node call amplification up to 51.0x and multi-node p95 latency inflation up to 229.1x. The proposed defense, Agent Component Energy (ACE) Analysis, detects malicious recursive triggers by measuring anomalous energy in the agent’s component graph [10].

SkillSafetyBench provides 155 adversarial cases across 47 tasks and 30 safety categories, demonstrating that localized non-user attacks on skill materials can consistently induce unsafe behavior even when the user request is benign [4]. The memory poisoning study identifies four write channels and nine structural vulnerabilities, developing a taxonomy of six attack classes and showing that agents designed to write and retrieve memory more aggressively are more exploitable [49].

On the defensive side, a stateful online monitor uses real-time clustering to aggregate weak suspiciousness signals across many agent transcripts, catching distributed attacks 30 percent earlier than standard monitors and flagging cyber misuse before it reaches the most harmful stages, with negligible additional latency for approximately 99 percent of user traffic [37].

Inference Efficiency and Production Deployment

Systems-level work in this wave targets the gap between benchmark performance and deployment economics.

The stateful inference architecture converts the per-turn O(n_t) cost of conventional serving into an O(delta_t) delta-only cost by maintaining a persistent KV cache that advances by ingesting only new tokens. Against vLLM and SGLang on novel fully-generated workloads, the reference implementation is 2.1x faster per turn on a 6-turn agentic workflow and 4.2x faster on the median turn of a 35-turn one [32].

Vortex combines a Python-embedded frontend language with a page-centric tensor abstraction for expressing sparse attention algorithms. AI agents using Vortex to automatically generate and refine algorithms reach up to 3.46x higher throughput than full attention while preserving accuracy; on the MLA-based GLM-4.7-Flash, Vortex reaches up to 4.7x higher throughput [55].

CoMem’s asynchronous summarization pipeline, noted above for its memory architecture, also delivers latency gains that scale favorably with increased system throughput, offering a modular path for independent optimization of agent reasoning and memory compression [40].

Nubank’s production framework for customer support agents at 100M-user scale integrates structured context engineering, human-in-the-loop prompt iteration, and LLM judge evaluation with measured inter-rater agreement. In a card-delivery deployment, large-scale A/B testing yields a 37 percentage-point improvement in AI transactional Net Promoter Score and a 29 percentage-point gain in self-service rate over prior agent variants. The framework also reports a strong correlation between offline simulation metrics and online outcomes [66].

FAQ

Q. Do any of the memory architectures solve the dependency reasoning failure identified by MEME? As of the evaluated configurations, no system closes the gap at practical cost. All six evaluated memory systems score an average of 3 percent on Cascade and 1 percent on Absence tasks. The only partial recovery requires a file-based agent with Claude Opus 4.7 as its internal model, at approximately 70x the baseline cost [8].

Q. How does the stateful inference architecture compare to vLLM specifically? The paper benchmarks the reference implementation against both vLLM and SGLang on novel, fully-generated workloads. On a 6-turn agentic workflow the stateful system is 2.1x faster per turn, and on the median turn of a 35-turn workflow it is 4.2x faster, with the advantage attributed to stateful reuse and speculation rather than caching alone [32].

Q. What does SkillSafetyBench reveal about model-level alignment as a defense? SkillSafetyBench’s experiments with multiple CLI agents and model backends show that localized non-user attacks on skill materials can consistently induce unsafe behavior even when the user request is benign. The findings suggest that agent safety depends not only on model-level alignment but also on how agents interpret skills, trust workflow context, and act through executable environments [4].

Q. Is the Metis jailbreaking framework effective only against weaker models? Metis maintains high efficacy on resilient frontier models, reporting 76.0 percent Attack Success Rate on O1 and 78.0 percent on GPT-5-chat, where traditional baselines exhibit substantial performance degradation [1].

Q. Does the Nubank production framework show that offline evaluation metrics predict online performance? The Nubank paper reports a strong correlation between offline simulation metrics and online outcomes across five production deployments, and describes this correlation as evidence that evaluation-driven development reliably predicts production impact [66].

Key takeaways

  • All evaluated memory systems collapse on dependency reasoning tasks (Cascade: 3%, Absence: 1% average accuracy), and the only partial fix currently requires configurations that are approximately 70x more expensive than baseline [8].
  • Evaluation methodology, not model capability, is identified as the primary bottleneck in agent failure diagnosis: the same frontier model achieves several times higher localization accuracy inside a structured span-level framework than as a monolithic judge [14].
  • Skill lifecycle management approaches show consistent benchmark gains, with SkillPyramid reporting 38.0% average reward improvement and FederatedSkill reporting up to 44.4% success rate improvement over self-evolving baselines [41][42].
  • Stateful inference reduces per-turn cost from O(n_t) to O(delta_t), delivering 2.1x to 4.2x speedups over vLLM and SGLang on multi-turn agentic workloads [32].
  • Production deployment at Nubank demonstrates a 37 percentage-point improvement in AI transactional Net Promoter Score, with offline simulation metrics correlating strongly with online outcomes across five distinct deployment domains [66].