Consulting / Field guide
Write AI pilot acceptance criteria before the first demo
A polished demo shows what can work. Acceptance criteria tell the team whether the pilot is ready for real responsibility.
2XD Studio · · 2 min read
Name the unit of work
For an order intake pilot, the unit is one reviewed order ready to enter the destination system. Count completed units rather than documents opened or model calls made. State which suppliers and formats are in scope. Otherwise the team can improve a score by quietly excluding inconvenient samples.
Choose checks with different consequences
Treat the vendor, currency, total, and destination account as critical fields. An incorrect description may be easy to fix; sending a valid amount to the wrong account is not. Establish which mistakes block rollout and which can enter a review queue. Include a manual fallback and identify who owns it.
Write the decision in advance
Prepare a short decision sheet with baseline time, observed correction rate, queue capacity, and unresolved incidents. Agree who can authorize expansion. If the pilot saves time on clean documents but doubles review time for revisions, narrow its scope or change the design before calling it a success.
Handle an inconclusive pilot
Ten clean orders reveal little about monthly revisions. If the test period contains none of the difficult cases, replay historical revisions before expanding. Label replay results separately from live outcomes, since staff behavior and data arrival times may differ. The right next decision can be to collect better evidence rather than renew or cancel immediately.
Reader questions
Do we need a single pass percentage?
A combined percentage can hide serious failures. Use separate thresholds for critical errors, coverage, and handling time.
What if there is no historical baseline?
Time the current process on a representative sample before switching it off. Keep the original examples for a side-by-side comparison.