An Instruction Is Not an Invariant
The prompt said scenarios must be distinct. It said so the entire time the model was returning five overlapping ones. And the control we shipped for multi-step cases turned out never to reach the model at all.
2 posts
The prompt said scenarios must be distinct. It said so the entire time the model was returning five overlapping ones. And the control we shipped for multi-step cases turned out never to reach the model at all.
Ask most QA assistants whether a release is ready and you get the general opinion of the internet. The useful answer depends on what your team wrote down — which means the retrieval matters more than the model.