Agent evaluation / Research project

ReScopeBench

Towards Robust Long-Lived Agents: Benchmarking Experience Lifecycles in Evolving Environments

When an environment changes without warning, which parts of their experience do agents keep, and which do they revise?

Paper · Coming soon BibTeX GitHub Dataset

Three layered maps show an agent acquiring resource connections, reusing them and revising a local path after a change. — Fig. 00 The experience lifecycle Schematic

Shijue Huang^1* Hangyu Guo^1* Hengheng Zhang³ Guo An³ Zhitao He¹ Qi Tian^2,3† Yiren Feng^1†

¹The Hong Kong University of Science and Technology²Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)³Huawei Inc.

* Equal contribution · † Corresponding authors

Interactive environments: 18
Tasks across streams: 352
Silent change events: 26
Models evaluated: 20

01 / Benchmark

What is ReScopeBench?

An agent can find a file after it moves and finish the task while its notes still point to the old location. ReScopeBench tracks both what agents deliver and what they save for the next task.

Benchmark overview. Interactive environments are built around discoverable experience. Tasks share dependencies, and local changes invalidate selected parts of that experience.

↗ Enlarge

01.1

Discover

Agents explore an unfamiliar environment. Structural experience records how resources connect. Rule experience captures principles and constraints that guide actions.

01.2

Reuse

Later tasks draw on the same knowledge. Paired runs begin from the same state. One retains memory, skills, and retrievable history between tasks. The other clears them.

01.3

Revise

Local changes invalidate selected facts while others remain true. With a limited exploration budget, the agent must decide what to recheck.

Read the paper abstract

As agents become more capable, they are moving beyond one-off tasks toward sustained assistance across successive user goals. Such assistance requires agents to discover how unfamiliar environments work, reuse what they learn across tasks, and keep that experience current as environments evolve. Existing benchmarks increasingly study experience reuse and adaptation, but provide limited insight into which parts of accumulated experience agents revise, retain, or lose. To address this gap, we introduce ReScopeBench, a benchmark for the experience lifecycle across 18 interactive environments. It distinguishes structural experience about how an environment is organized and accessed from rule experience about reusable principles and constraints that guide decisions across tasks. To study how experience is acquired, reused, and revised, we propose an experience-first construction approach. It builds task streams around shared experience and introduces silent environmental changes. We evaluate task performance and track changes in saved experience throughout each stream. Across 20 models, we find that retained experience improves task performance, but agents struggle to keep it current as environments change. Even when the required experience is correctly recorded, agents do not always fulfill requests successfully. These findings identify selective maintenance and effective use as complementary challenges for long-lived agents.

02 / Evaluation

Main results

All 20 models score higher on average when they retain experience. Yet even the best model records only 37.63% of update targets after the first task affected by a change.

Table 2 / Model comparison

Table 2. Task performance and saved experience across 18 environments. Bold scores are the highest in each column, including ties.
Model	Performance			Experience				Avg.
Model	BP	EP	DP	SC	RC	UC	RT	Avg.
Compact Open-Weight Models
Qwen3.5-9B Alibaba	0.51	0.89	0.74	15.14	18.43	1.67	2.95	5.76
Qwen3.5-35B-A3B Alibaba	0.82	2.75	0.89	18.19	18.21	0.40	4.18	6.49
Qwen3.8-27B Alibaba	20.07	62.48	38.21	60.90	55.18	29.05	59.17	46.44
Commercial Models
GPT-5.4 OpenAI	0.09	14.96	12.38	47.75	33.15	12.91	22.85	20.58
MiniMax-M3 MiniMax	5.61	34.50	26.18	45.59	36.63	11.70	35.57	27.97
GPT-6 Luna OpenAI	20.83	54.69	30.06	56.54	51.09	17.50	49.82	40.08
GPT-5.5 OpenAI	20.55	66.23	30.29	60.34	53.84	17.50	44.88	41.95
Doubao-Seed-2.1-Pro ByteDance	18.20	64.58	39.88	62.87	59.79	29.75	64.00	48.44
Kimi-K3 Moonshot	22.23	74.08	39.35	65.50	59.72	28.75	64.50	50.59
DeepSeek-V4-Pro-0813 DeepSeek	19.67	70.59	44.75	64.21	63.00	32.44	63.41	51.15
GLM-5.3-Flash Z.ai	22.35	73.39	42.42	65.76	57.83	32.07	65.62	51.35
GPT-6.1 Sol OpenAI	24.84	73.97	39.10	71.63	60.66	24.73	64.62	51.36
Gemini-3.8-Flash Google	25.70	72.47	40.51	65.49	60.56	33.53	65.14	51.91
MiMo-V2.6-Pro Xiaomi	24.60	73.94	44.73	65.86	61.79	27.93	67.25	52.30
DeepSeek-V4.1-Flash DeepSeek	19.98	74.73	47.50	68.63	65.18	33.41	65.56	53.57
Qwen3.8-Max Alibaba	24.09	73.44	46.98	67.56	66.25	31.10	68.64	54.01
GLM-5.3 Z.ai	23.78	75.02	48.81	67.38	66.32	33.00	69.88	54.89
GPT-6 Astra OpenAI	25.86	77.50	42.11	75.68	66.17	29.37	74.12	55.83
Claude Opus 5.5 Anthropic	21.94	74.98	46.75	71.70	73.22	33.27	72.89	56.39
Gemini-4-Argon Google	25.86	81.12	50.37	76.83	70.81	37.63	78.72	60.19

Scores range from 0 to 100. Each environment has equal weight. Bold marks the best score in each column across all 20 models.

How to read the seven metrics

Task performance

BP and EP measure mean task scores without and with retained experience. DP measures the score with experience on the first task affected by each change.

Saved experience

SC and RC measure correct coverage of the current structural and rule reference inventories. UC measures how much changed or new information is recorded after the first affected task. RT rewards earlier joint completion of all update targets and previously correct, still-valid information within each affected experience unit.

03 / Findings

Inside the task streams

We examine how experience relates to task performance, how agents adapt to environmental change, and how they allocate tool use.

Analysis 01 · Figure 3

How do experience and task performance evolve?

We track task scores and saved structural and rule experience across all 20 models and 18 environments. Streams are grouped by length, with the same environments equally weighted at every position.

Task performance above and structural and rule experience coverage below, for 16-, 24-, and 32-task streams. Dark-gray curves show the mean of all 20 models, alongside four representative models.

↗ Enlarge

Dark-gray curves average all 20 models. Markers distinguish GPT-5.4, GPT-5.5, GPT-6.1 Sol, and GPT-6 Astra. Bottom ticks show change positions; vertical lines mark changes shared by every environment in a group.

Gains accumulate across tasks

Performance with experience rises across all three groups, while performance without experience remains low. Shared change onsets interrupt these gains before recovery.

Different experience becomes outdated

In 24-task streams, structural coverage falls by 18.92 points at T09, while rule coverage stays comparatively stable. At T17, rule coverage falls by 15.97 points while structural coverage changes little.

Coverage is not task success

GPT-5.4 gains structural coverage across all groups but remains far below the other displayed models in task performance. Recording reference knowledge does not ensure its effective use for a specific request.

Analysis 02 · Figure 4

How do agents adapt to different changes?

We compare change timing, count, and mechanism across 26 events, 18 environments, and all 20 models. A separate content analysis follows previously learned relations to distinguish revision from first acquisition.

Adaptation by change timing and total change count, performance advantage by six change mechanisms, and separate revision and retention curves for previously correct experience.

↗ Enlarge

Performance advantage is the score with experience minus the score without experience. Offset 0 is the first affected task. The content analysis covers 487 eligible cases in eight environments; bands show the interquartile range of model means.

Later events have higher scores

In two-change streams, the first-four-task mean rises from 55.94 after the first event to 62.15 after the second. Three-change streams score 68.62, 70.50, and 72.53. Different tasks and change targets prevent interpreting this as faster recovery.

Similar drops, different recovery

Structure and tool changes initially reduce the performance advantage by 35.98 and 36.63 points. By the fourth affected task, the gaps from their pre-change levels are 0.16 and 8.69 points. These compare different environment groups.

Revision lags retention

Across the first four affected tasks, replacement coverage rises from 21.47% to 64.63%. Coverage of previously correct relations that remain valid stays between 88.07% and 91.07%.

Analysis 03 · Figure 5

How do agents allocate tool use?

We analyze 151,569 foreground tool calls from 7,040 tasks with retained experience, covering all 20 models and 18 environments. Environment tools query or act on current state, history tools retrieve prior records, and experience tools read or modify memory and skills.

Tool-call allocation, interaction rounds, and experience write attempts around environmental changes. Four representative models are shown with the mean of all 20 models.

↗ Enlarge

Counts include failed calls and exclude automatic memory loading and background review. A tool round is one assistant turn containing calls and their responses. Before averages the three preceding tasks; At change denotes the first affected task.

History use varies most

The four representative models make 13.07–13.81 environment calls per task. History calls vary from 0.14 for GPT-5.4 to 6.41 for GPT-5.5, accounting for much of the difference in their total activity.

Interaction peaks near change

The all-model mean rises from 10.72 rounds before change to 12.70 at onset, then falls to 11.21 by the fourth affected task. All displayed models except GPT-5.4 show a similar increase.

More interaction is not more writing

GPT-5.5 uses more rounds at onset, yet its experience write attempts fall from 1.08 to 0.97 per task. GPT-6.1 Sol and GPT-6 Astra increase their writes. These counts measure activity, not correct revision.

04 / A closer look

One directory changed. Five fields still worked.

In the Filesystem environment, GPT-5.5 had learned where to deliver the Riverlight Ledger index. A silent change moved the destination directory for Grove’s amber lane.

T04

A working record

The agent saved the manifest, destination, routing token, and overwrite policy. The amber lane pointed to the original directory.

T07

The old route fails

A delivery using the saved directory returned DELIVERY_POLICY_MISMATCH. The agent recorded a warning to recheck the lane.

T09

A local repair

Two tool reads revealed a new directory handle and an unchanged policy. The repaired record supported five later deliveries without another environment query for the manifest.

Inspect the original case record ↗

05 / Reference

Cite this work

BibTeX

@misc{huang2026rescopebench,
  title  = {Towards Robust Long-Lived Agents: Benchmarking Experience Lifecycles in Evolving Environments},
  author = {Huang, Shijue and Guo, Hangyu and Zhang, Hengheng and An, Guo and He, Zhitao and Tian, Qi and Feng, Yiren},
  year   = {2026},
  note   = {Preprint},
  url    = {https://ReScopeBench.github.io/}
}

06 / Funding

Acknowledgements

This research was financially supported by the Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), under Grant No. GML-KF-26-06.