In a joint blog post published October 22, 2026 on Hugging Face, Microsoft and Hugging Face described ThinkingBox-Training, now available on GitHub. The work builds on an earlier benchmark, ThinkingBox-Bench, which grades AI agents by comparing the final state of a database against the expected outcome rather than scoring the tool calls an agent makes. According to the post, the team reused that same executable check as a reward signal to fine-tune an agent on tasks outside the benchmark. Using a method they call 'offline policy-relative difficulty' to select training tasks, they fine-tuned the Qwen3.8-27B model with GRPO for 50 steps. The post reports average success (pass@1) rose from 51.70% to 60.78%, and the number of tasks completed correctly on all twenty attempts rose from 38 to 107, with a small decline in at-least-once coverage. In one example, a corporate hotel-booking task requiring manager approval, the base model changed the booking without approval in 15 of 20 attempts; after fine-tuning, the model preserved the booking and payment state in all 20. The post is co-authored by researchers from Microsoft, Hugging Face and a former University of Pittsburgh intern. These figures come from the vendors' own announcement.
- Qwen3.8-27B average success (pass@1) rose from 51.70% to 60.78% after 50 GRPO steps
- Tasks passing all 20 attempts rose from 38 to 107, with a small drop in at-least-once coverage
- ThinkingBox-Bench grades 507 tasks by final database state, not by the tool calls made
- Results and training code published October 22, 2026; ThinkingBox-Training released on GitHub
What it means for you
This is a research result about how to make AI agents more consistent, not a product you can buy or switch on. The useful idea underneath it, though, is one any operations leader should notice: these researchers judged the agent by what actually changed in the records at the end, not by whether the agent said it was done. An agent that closes a ticket or changes a booking while leaving the underlying data wrong is a real operational failure mode, and headline accuracy hides it.
Try this
Pick one process where you already let a tool or automation take an action on your behalf — closing tickets, updating bookings, issuing refunds. Spot-check the actual records it touched over the last week against what should have happened, rather than trusting the 'completed' status. That check is the same logic this research uses.
Who should care
Technical teams building or fine-tuning their own agents, and operations owners of any workflow where an automated step writes to a system of record (orders, refunds, bookings, ticket status).
Skip this if
You aren't training your own models and your automations are simple rule-based steps rather than AI agents acting with discretion — then this is background reading, not a signal to act.
Sources: Hugging Face — read the original