Key takeaways
- Write and share your own requirements before demos so vendors respond to the same brief.
- Run a bounded proof of value with a held-back test set, written criteria and independent scoring.
- Settle data ownership, training use, export and deletion explicitly in the contract.
- Negotiate pricing basis, service levels, security obligations, change control and exit terms before signing.
Define your requirements before the first demo
Vendor demos are designed to impress, and without your own requirements on paper it is easy to evaluate the presentation rather than the product. Before contacting vendors, write a short requirements document covering the problem, the data sources and systems involved, the users and how they will act on the output, deployment constraints such as on premises, cloud or edge, security requirements, integration points, and what success looks like in measurable terms. Share it with every vendor so you compare responses to the same brief.
The evaluation checklist
| Area | What to check |
|---|---|
| Problem fit | Has the vendor solved this specific problem on similar equipment, processes or products? Ask for references you can call, in your industry if possible. |
| Data requirements | What data does the product need, at what rate and history length? What happens when data is missing or noisy? |
| Integration | Which protocols and systems does it connect to natively, such as OPC UA, MQTT, historians, MES, CMMS and ERP? What has to be custom built? |
| Deployment | Can it run where you need it: on premises, private cloud, public cloud, or at the edge? What are the hardware requirements? |
| Security | How does it fit into your IEC 62443 zones? Does it require inbound connections? How is remote support access controlled? What certifications and audit reports can it provide? |
| Explainability | Can users see why the system produced an alert or recommendation? |
| Model lifecycle | How are models monitored, retrained and updated? Who does that work, and how are changes validated and rolled back? |
| Usability | Can operators, technicians and engineers use it without a data scientist present? |
| Scalability | What changes, in effort and cost, when you go from one line to ten, or one site to five? |
| Company | Financial stability, size of the support team, roadmap, and what happens to your system if the vendor is acquired or exits the market. |
Structure a proof of value
A proof of value is a short, bounded test using your data and your conditions. It differs from a pilot in that it is often done before contract, with limited scope, to decide which vendor, if any, proceeds. Structure it so results are comparable and the outcome is clear.
- Define the dataset or test set. For a historical test, provide the same data to each vendor, hold back a period they do not see, and score them on it. For a live test, agree the equipment and period.
- Agree criteria in writing. Detection rate, false alarm rate, lead time, defect escapes, or whatever matters for the use case, with thresholds.
- Fix the duration and effort. State how much of your team's time the vendor can expect and how long the test runs.
- Clarify commercial terms. Who pays, what happens to any hardware installed, and whether a successful result converts to a contract at pre-agreed terms.
- Score independently. Have your own engineers verify results against ground truth rather than accepting the vendor's report.
Watch for how the vendor handles your messy data. A team that asks good questions about tag meanings, operating modes and failure records is usually a better partner than one that produces impressive charts without asking.
Make reference calls count
Vendors choose their references, so the calls tend to be positive. You can still learn a great deal by asking specific questions. Ask to speak with someone who uses the product daily, not only the executive sponsor. Then ask:
- How long did it take from contract to the first useful result, and what took longest?
- How much of your own staff time did implementation and ongoing support require?
- What did the system get wrong in the first months, and how did the vendor respond?
- How has pricing changed since you signed, particularly as you added assets or sites?
- Are you running it at scale, or is it still on the original pilot line?
- Knowing what you know now, what would you negotiate differently?
If possible, find a user the vendor did not suggest, for example through an industry association or a peer group. Visiting a running installation, even briefly, often tells you more than any presentation.
Data ownership and use
Data terms are often the most consequential part of the contract and the least read. Settle these points explicitly:
- You own your raw data, derived data and labels, and can export them in a usable, documented format at any time and at the end of the contract.
- Whether the vendor may use your data to train models used for other customers, and if so, whether it is anonymized and aggregated, and whether you can opt out.
- Who owns models trained specifically on your data, and whether you can keep using them if you leave.
- Where data is stored and processed, including subcontractors and cloud regions.
- How data is deleted at the end of the relationship, with written confirmation.
- Confidentiality of process parameters, recipes and customer-specific information.
Contract terms to negotiate
- Pricing basis and escalation. Understand the unit you pay on, such as assets, users, sites, cameras or data volume, and how price changes at renewal and as you scale.
- Service levels. Availability, support response times and escalation paths, with remedies when they are missed. For edge systems, clarify what keeps working if the vendor's cloud is unavailable.
- Performance commitments. Where possible, tie part of the fee or the renewal to agreed outcomes from the proof of value.
- Security obligations. Vulnerability disclosure, patch timelines, breach notification and the right to audit or receive audit reports.
- Change control. Notice before changes to models, software or infrastructure that affect production, and the right to defer updates during critical periods.
- Exit and transition. Data export, transition assistance, and what happens to edge hardware and licenses if you leave.
- Liability. How responsibility is allocated if the system's output contributes to a quality escape or equipment damage. Expect vendors to limit this; know what you are accepting.
Red flags
- Claims of accuracy or savings with no explanation of how they were measured.
- Reluctance to test on your data or to let your engineers verify results.
- Requirements for inbound connections into the control network.
- Vague answers about who owns your data and models.
- No clear plan for model monitoring and retraining after go-live.
- References only from pilots, with no customers running at scale.
Related use cases
Maintenance & reliability
Predictive maintenance
Use sensor and maintenance history to spot developing equipment failures early and schedule repairs before they stop the line.
Quality
Visual quality inspection
Cameras and computer vision models inspect parts on the line to catch surface, assembly and dimensional defects consistently.
Maintenance & reliability
GenAI knowledge assistants & copilots
Chat-based assistants answer maintenance and frontline questions from manuals, work orders and procedures, with sources cited.
Operations & production
Machine monitoring & OEE analytics
Automatically capture machine states, downtime reasons and output to measure OEE accurately and target the biggest losses.