What if the key to making AI agents better at scientific discovery and medical diagnosis was hiding in a classic board game? That's exactly what researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and Harvard University discovered when they taught AI models to play a collaborative version of Battleship.
The Problem: AI Struggles with Strategic Questioning
While AI agents excel at well-defined tasks like customer service and software development, they hit a wall when it comes to fields requiring strategic inquiry—like medical diagnosis and scientific research. The core issue? Most language models simply aren't good at asking the right questions to efficiently gather information in uncertain environments.
Enter "Collaborative Battleship"
The MIT and Harvard researchers created an ingenious twist on the classic guessing game. Instead of simply calling out coordinates, their "Collaborative Battleship" requires players to ask and answer natural language questions. One player acts as the "captain" who asks questions about hidden ship locations, while their teammate serves as the "spotter" providing real-time yes-or-no answers.
First, they had over 40 humans play the game to create the "BattleshipQA" dataset. This human baseline revealed something surprising: top language models like GPT-5 could already beat humans at the game, but smaller models performed poorly—and crucially, most models struggled with asking truly useful questions.
The Monte Carlo Solution
To address the questioning problem, the researchers implemented a Monte Carlo inference strategy. This approach helps models reason about potential guesses like weighted particles—options that seem more valid get heavier weighting with each new answer, while less likely possibilities fade away.
The results were remarkable. Llama 4 Scout, a relatively small language model, jumped from winning just 8% of games against humans to an impressive 82% win rate. Even more striking: this enhanced smaller model could outpace the much larger and more expensive GPT-5 while operating at around 1% of its cost.
Making AI Better Spotters Too
The team also tackled the challenge of helping AI models give better answers. Their solution was elegantly simple: convert natural language questions into Python code that explicitly tells the model how to verify answers. For example, a question like "Is there a ship in column one that spans two rows?" becomes coded instructions for the AI to search the specific area and assess the ship's dimensions.
This "auto-formalization" approach boosted accuracy across the board. The lightweight GPT-4o-mini saw nearly a 30% performance improvement, while even the large Claude 4 Opus gained about 8 percentage points in accuracy.
Beyond Battleship: Testing on Guess Who?
To validate their approach, the researchers tested their enhanced models on "Guess Who?"—a game requiring players to efficiently narrow down 100 character options. The results were equally impressive:
- Llama 4 Scout improved from 30% to over 72% success rate
- GPT-4o jumped from 62% to 90% accuracy
Practical Implications for Prompt Engineering
This research offers several key insights for prompt engineers and AI practitioners:
Strategic Question Design: The study shows that helping AI models think about information gathering as a strategic process—rather than just asking random questions—can dramatically improve outcomes.
Code-Assisted Verification: Converting complex queries into executable code can significantly improve AI accuracy, especially for smaller models that need more structured guidance.
World Model Integration: As lead author Gabriel Grand notes, "asking informative questions depends on the ability to predict and simulate the world." Giving AI agents access to structured world models enhances their questioning capabilities.
Future Directions
While the results are promising, the researchers acknowledge that Collaborative Battleship is still a relatively simple test bed. They're working toward more complex scenarios and exploring human-AI collaboration to see if mixed teams perform even better.
As Stanford's Robert Hawkins (who wasn't involved in the study) points out, "the real bottleneck for AI agents isn't just the calculation of optimal questions, but the pragmatic reasoning needed to make the most of their answers."
This research suggests that the future of AI agents lies not just in having more powerful models, but in teaching them the art of asking better questions—a skill that could revolutionize everything from scientific research to customer service.
Source: MIT News by Alex Shipps