Failing to Explore: Language Models on Interactive Tasks

A suite of controllable tasks that expose under-exploration in language models and highlight simple interventions that help.

Abstract

We evaluate language models on their ability to explore interactive environments under a limited interaction budget. We introduce three parametric tasks with controllable exploration difficulty, spanning continuous and discrete environments. Across state-of-the-art models, we find systematic under-exploration and suboptimal solutions, with performance often significantly worse than simple explore–exploit heuristic baselines and scaling weakly as the budget increases. Finally, we study two lightweight interventions: splitting a fixed budget into \(p\) parallel executions, which surprisingly improves performance despite a no-gain theoretical result for our tasks, and periodically summarizing the interaction history, which preserves key discoveries and further improves exploration.

Github arXiv
Figure 1 from the paper

Interactive Tasks

Each task is a parametric environment designed to make exploration important.

HillSearch task preview

HillSearch

Query a hidden function on [0, 10] to find a high value point.

Play the demo →
TreeSearch task preview

TreeSearch

Reveal rewards on a hidden tree while staying connected to the root. Try to maximize the maximum reward.

Play the demo →
MaxSatSearch task preview

MaxSatSearch

Propose Boolean assignments and and see how many clauses they satisfy. Try to unlocks high reward.

Play the demo →

Highlights

Key findings and design principles from the paper.

Mitigation Techniques

Simple interventions that consistently improve exploration.

BibTeX

Loading...