Key takeaways
- A pilot exists to support a single decision: whether to invest at scale.
- Measure a baseline with agreed definitions and a single source of truth before the pilot starts.
- Write technical, operational and adoption criteria in advance and have the owner and finance sign them.
- Run offline validation and shadow mode before users act on outputs, and set a fixed date for the go or no-go call.
What a pilot is for
A pilot exists to answer one question: should we invest in running this at scale? It is not a demonstration that AI can work in general, and it is not a free trial that drifts on until the budget runs out. Pilots that end without a clear decision usually had no written success criteria, no baseline and no date by which the decision would be made.
Treat the pilot as a structured experiment. You state what you expect to happen, measure what actually happens, and compare the two against criteria everyone agreed to before the work began.
Scope it tightly
Choose one line, one asset class, one inspection station or one product family. Narrow scope keeps data work manageable and makes it easier to attribute any change to the pilot rather than to some other improvement happening at the same time.
Write down the scope in a short charter that covers:
- The specific problem and the decision the AI will support
- The physical boundary: which equipment, which shifts, which products
- The data sources and who will provide access
- Who will use the output and how they will act on it
- What is explicitly out of scope
- The duration, usually long enough to see the events you care about several times
Duration is often underestimated. If the failures you want to predict happen once a month, a six-week pilot will see one or two of them, which tells you very little. Either extend the pilot, widen the asset population, or choose a leading indicator you can observe more often.
Measure the baseline first
Before the pilot starts, measure the current state using the same definitions you will use afterward. If the goal is fewer unplanned stops, pull the stop history by cause for a representative period. If the goal is better defect detection, measure how many defects the current inspection catches and misses, which may require a short audit with a second inspection.
Agree on the source of truth. If maintenance counts downtime from the CMMS and production counts it from the MES, the numbers will differ, and the pilot will end in an argument about data instead of a decision about value.
Define success criteria in advance
Set criteria in three groups, and get the owner, finance and the people who will use the system to sign them.
| Group | Example criteria |
|---|---|
| Technical performance | Detection rate, false alarm rate, prediction lead time, accuracy against held-out data |
| Operational performance | Change in downtime, scrap, inspection labor or schedule adherence against baseline |
| Adoption and fit | Share of alerts reviewed, user feedback, time to act, whether the workflow survived contact with real shifts |
Set a threshold for each and decide which are mandatory. A model with excellent accuracy that operators ignore has failed. So has a well-used tool whose alerts are mostly wrong.
Example: a hypothetical vision pilot might require that the system catch at least as many confirmed defects as the current manual check on a blind audit set, with no more than a stated number of false rejects per shift, and that operators review every flagged part within the cycle time. The numbers are yours to set; the point is that they are written down before the results arrive.
Staff the pilot properly
Pilots are often run on borrowed time, by people who already have full-time jobs. Name the roles explicitly and allocate hours to them. A typical small pilot needs an owner from operations or maintenance who is accountable for the decision, a process expert who knows the equipment and can explain what the data means, a data or controls engineer who can get data out of plant systems, users from each affected shift, and someone from IT or OT security who approves connections. If a vendor is involved, name their counterpart for each role as well.
Agree how much time each person will spend per week and protect it. When the process expert is pulled into a breakdown for two weeks, the pilot loses two weeks. Building that risk into the schedule is better than pretending it will not happen.
Run it in phases
- Data and setup. Connect sources, validate data quality, install hardware if needed. Expect this to take longer than planned.
- Offline validation. Train or configure the model on historical data and test it on a period it has not seen. This tells you whether the approach can work at all.
- Shadow mode. Run the model live but do not act on its output. Compare its predictions with what actually happens. This catches problems with live data feeds and drift that offline testing misses.
- Assisted operation. Users see and act on the output under a defined procedure. Log every alert, every action and every outcome.
- Review. Compare results with the baseline and the success criteria.
Hold a short weekly review with the owner, the users and whoever runs the model. Track issues in a simple log. Small problems such as a dropped data feed or a confusing alert message, left alone for weeks, can quietly ruin a pilot.
Making the go or no-go call
At the end, there are four honest outcomes:
- Go. Criteria met. Plan the scale-up, including the production-grade data pipeline, support model, training and budget.
- Go with changes. Value is evident but something specific needs fixing, such as alert thresholds or a workflow step. Define the fix and a short follow-up period.
- Pause. The approach may work but a prerequisite is missing, usually data. Fund the prerequisite and revisit.
- Stop. The criteria were not met and there is no clear path to meeting them. Document what you learned and move on.
A stop decision is a successful pilot if it was reached quickly and cheaply. The failure mode to avoid is the pilot that never ends, consuming attention while nobody is willing to call it.
Plan for scale before you start
Ask during scoping what it would take to run the solution on ten lines instead of one. Pilots often rely on workarounds, such as manual data exports, a vendor engineer watching the model, or a single enthusiastic supervisor, that will not survive scale. Identify those dependencies early and estimate the cost of replacing them. That estimate belongs in the business case alongside the pilot results.
Related use cases
Maintenance & reliability
Predictive maintenance
Use sensor and maintenance history to spot developing equipment failures early and schedule repairs before they stop the line.
Quality
Visual quality inspection
Cameras and computer vision models inspect parts on the line to catch surface, assembly and dimensional defects consistently.
Quality
Quality prediction & virtual metrology
Predict product quality from live process data so problems are caught before parts are measured, tested or shipped.
Operations & production
Process & yield optimization
Models recommend or automatically adjust process setpoints to keep quality on target while improving yield, throughput and energy use.