Online Mistake Detection in Instruction-Conditioned Procedural Video
Abstract
An assistant monitoring a person performing a procedure is most useful when it can detect mistakes while there is still time to recover. This requires judging actions against task instructions as the video unfolds, identifying the attempted step, locating it in time, and assessing its execution. Existing online detectors do not use the instruction text, while instruction-conditioned methods operate offline. In this paper, we study online, instruction-conditioned mistake detection in egocentric videos. Using only observed video, task instructions, and prior predictions, our model identifies and localizes completed action attempts and classifies them as correct, modified, slip, corrections, or additions; omitted steps are determined after the video ends. We compare a modular pipeline, with separate step identification, temporal localization, and execution verification, to a joint model that predicts all outputs in a single step. To support supervised learning from mistakes, we repartition EgoPER to include erroneous videos in all splits. The approaches achieve similar step-segmentation F1@0.5 (93.6% and 92.1%), while the joint model achieves higher binary mistake-detection F1@0.5 (84.1% vs. 80.7%), with its largest gain on slips (56.4% vs. 37.5%). Component-level analysis suggests that error propagation between stages contributes to the modular approach's lower detection score. On test videos containing mistakes, both designs outperform released AEM and ESTANet checkpoints under our evaluation protocol.
Community
TL;DR: We tackle online, instruction-conditioned mistake detection in egocentric procedural videos—catching errors as they happen rather than offline after the task is done.
Key takeaways:
- Joint vs. Modular: Predicting step ID, temporal boundaries, and execution quality (correct, modified, slip, correction, addition) in a single joint step beats a modular pipeline on binary mistake detection (84.1% vs. 80.7% [email protected]), largely by avoiding error propagation between stages.
- Catching subtle errors: The joint model shows a massive jump on detecting execution slips (56.4% vs. 37.5%).
- Benchmark fix: We repartitioned EgoPER so erroneous videos are included across all splits, enabling proper supervised learning from mistakes.
Happy to answer any questions about the setup, error taxonomy, or EgoPER split!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Through a PRISM, Clearly: Step-Sequence Reasoning for Mistake Recognition in Procedural Videos (2026)
- EgoActionBench: Benchmarking Dense Action Captioning for Egocentric Human Videos (2026)
- Hierarchical Step Decomposition for Training-free Online Video Step Grounding (2026)
- OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction (2026)
- ASSEMBLE: Atomic Skills for Evidence-Grounded Video Reasoning (2026)
- VideoJev: System One Model for Video Understanding and Interaction (2026)
- Egocentric Action Recognition with Objects and Vision-Language Affordances (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.25567 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper