TutorMoments: Do AI tutors know when to help and when to hold back?

📄 Tech Report: https://tutormoments.allen.ai/static/paper/tutormoments-preview.pdf | 📊 Data: https://huggingface.co/datasets/allenai/tutormoments-preview | 💻 Code: https://github.com/allenai/tutormoments

Today we're introducing a preview of TutorMoments , a framework to measure whether cutting-edge LLMs can balance one of the hardest trade-offs in education: when to step in and help a student and when to hold back and let the student do more of the work.

TutorMoments is a replay-based evaluation built off real one-on-one math tutoring sessions. Experienced math teachers go through transcripts collected from a U.S. tutoring program and flag the moments where a tutor had to choose between making a problem easier to get started on and pushing the student to do more of the reasoning themselves. TutorMoments then takes the transcript up to that decision point, hands it to a language model, and has the model take over as the tutor in a simulated session – with the student played by another language model – to see what the LLM tutor does.

Told only to "tutor well," we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off (when to help versus when to hold back) in the tutor's prompt improves performance, but it doesn't close the gap to human tutoring that consistently fits the moment, and LLMs still differ widely in how reliably they make that call.

As part of our commitment to open research, we're releasing a dataset of de-identified tutoring transcripts , the code for running our replay pipeline , and the model tutor replays of the key moments we evaluated in those transcripts for reproducibility. We hope TutorMoments gives educators, researchers, and the teams building AI tutors a sharper way to ask how a model handles the pedagogical decisions that matter most—and helps the field build tutors that adapt to each student instead of doing the work for them.

Ask a good math tutor for help and you'll likely get a question back like, "What do you know about what the problem is asking?" That isn't unhelpfulness–part of strong teaching is diagnosing what students do know and providing the right support for them in the moment. Immediately volunteering support would rob a student of the intellectual work that helps them learn. Sometimes support is needed; other times what's most effective is a push to solidify understanding by explaining a correct answer.

Language models, though, are trained to be helpful, and a helpful assistant tends to do the hard part for you—explaining the concept, laying out the steps, and guiding you to the answer. In a tutoring session, that can cut short the productive struggle—the effortful, sometimes frustrating problem-solving that learning research has long tied to stronger understanding.