Cairnscale working example
Which enquiries need a person?
Six invented enquiries test a small decision before anyone relies on it.
Compare the same six requests
A baseline that always chose website matched 2 of six labels. A local language model with a fixed prompt matched 4. It sent a product-page problem and a reporting discrepancy for human review. That adds two reviews an operator could have avoided.
A reviewer owns the ambiguous cases. An invalid response also goes to review. An enquiry is never sent or acted on by this exercise. A production pilot needs a larger, representative set, agreed error limits and evidence from the actual model version.
Read all six inputs, labels, prompt and results (JSON) · Download the evaluation runner
Discuss a similar job
Tell us the outcome, who needs it and one way they would check it.
We work entirely online. A description is enough to start. Keep records, credentials and attachments for the transfer method we agree.
Start an email enquiryEnquiry privacy · hello@cairnscale.uk