Pairing Expert Critique with LLM-Driven Refinement: An Iterative Approach for Curating Quality AI Evaluation Datasets
4.40T1 sourcearXiv cs.HC
Source record
Published by arXiv cs.HC (T1 source). The original is at https://arxiv.org/abs/2611.05749.
Pipeline notes
The summary and note below are generated by the signal pipeline — they are Beyond Desk’s reading, not quotations from the source.
SummarySME-C combines lightweight subject-matter-expert critiques with LLM-driven iterative refinement to build domain-specific evaluation datasets and rubrics. In a financial-brief study, 411 professionals produced 3,853 critiques for 520 queries; blind validation with 100 more professionals found improvements over top-ranked baseline responses.
Why it mattersDocuments a repeatable expert-in-the-loop evaluation workflow with substantial field data and distinct results for critique-guided and preference-vote refinement.
Cited by
No citations on record.