Microsoft announced human-in-the-loop evaluation quality in Microsoft Foundry public preview at Ignite on November 17, 2026. The workflow connects agent traces, reviewer feedback, evaluation datasets, and reusable test-case criteria. Microsoft announcement.
My proposed pilot would begin with a small set of approved scenarios and a domain expert who can describe the expected behavior. Record the evaluation purpose, source of each example, and the people responsible for approving the dataset.
Capture Useful Feedback
Microsoft describes ratings, tags, comments, and expected outputs on traces. Selected examples can be added to an evaluation dataset. Feedback workflow.
I would include successful interactions as well as failure cases. For multi-turn scenarios, have the reviewer check that the context needed to judge the final response is present and that the expected outcome is specific enough to test.
Inspect Generated Criteria
The preview can generate test-case rubrics when a model deployment is selected. Microsoft says the expected outputs remain as written and the generated criteria can be reviewed before evaluation. Rubric generation.
Ask a second reviewer to compare each criterion with the original feedback and business requirement. Record disagreements and revise the benchmark before using scores to assess a later agent version.
Manage Dataset Access
Microsoft lists tracing, a connected Application Insights resource, appropriate permissions, and a supported region among the prerequisites. Production traces can contain sensitive information. Preview prerequisites.
My review would define the dataset owner, permitted audience, retention approach, and approved handling of source conversations. Use synthetic or appropriately approved examples for an initial rehearsal.
Practical Cloud Engineer Takeaway
Select representative cases with a named domain reviewer.
Preserve context and explicit expected outcomes.
Inspect generated rubrics before using them to score responses.
Review privacy and access requirements for trace-derived datasets.
Who Should Care?
Foundry developers, evaluation teams, and business owners responsible for agent quality.
Bottom Line
The preview offers a way to turn human feedback into reusable evaluation material. The quality of the benchmark still depends on careful case selection and review of the criteria.
No Foundry project, trace, dataset, model deployment, or Azure resource was changed for this article.
Sources
Stay radical, stay curious, and keep pushing the boundaries of what is possible in the cloud.
Chriz
Beyond Cloud with Chriz
Comments