Checklist ยท Practical AI education

Evaluating AI vendors: a checklist for manufacturers

What to ask, how to structure a proof of value, and which contract terms and data ownership clauses to settle before you sign.

Key takeaways

  • Write and share your own requirements before demos so vendors respond to the same brief.
  • Run a bounded proof of value with a held-back test set, written criteria and independent scoring.
  • Settle data ownership, training use, export and deletion explicitly in the contract.
  • Negotiate pricing basis, service levels, security obligations, change control and exit terms before signing.

Define your requirements before the first demo

Vendor demos are designed to impress, and without your own requirements on paper it is easy to evaluate the presentation rather than the product. Before contacting vendors, write a short requirements document covering the problem, the data sources and systems involved, the users and how they will act on the output, deployment constraints such as on premises, cloud or edge, security requirements, integration points, and what success looks like in measurable terms. Share it with every vendor so you compare responses to the same brief.

The evaluation checklist

AreaWhat to check
Problem fitHas the vendor solved this specific problem on similar equipment, processes or products? Ask for references you can call, in your industry if possible.
Data requirementsWhat data does the product need, at what rate and history length? What happens when data is missing or noisy?
IntegrationWhich protocols and systems does it connect to natively, such as OPC UA, MQTT, historians, MES, CMMS and ERP? What has to be custom built?
DeploymentCan it run where you need it: on premises, private cloud, public cloud, or at the edge? What are the hardware requirements?
SecurityHow does it fit into your IEC 62443 zones? Does it require inbound connections? How is remote support access controlled? What certifications and audit reports can it provide?
ExplainabilityCan users see why the system produced an alert or recommendation?
Model lifecycleHow are models monitored, retrained and updated? Who does that work, and how are changes validated and rolled back?
UsabilityCan operators, technicians and engineers use it without a data scientist present?
ScalabilityWhat changes, in effort and cost, when you go from one line to ten, or one site to five?
CompanyFinancial stability, size of the support team, roadmap, and what happens to your system if the vendor is acquired or exits the market.

Structure a proof of value

A proof of value is a short, bounded test using your data and your conditions. It differs from a pilot in that it is often done before contract, with limited scope, to decide which vendor, if any, proceeds. Structure it so results are comparable and the outcome is clear.

  1. Define the dataset or test set. For a historical test, provide the same data to each vendor, hold back a period they do not see, and score them on it. For a live test, agree the equipment and period.
  2. Agree criteria in writing. Detection rate, false alarm rate, lead time, defect escapes, or whatever matters for the use case, with thresholds.
  3. Fix the duration and effort. State how much of your team's time the vendor can expect and how long the test runs.
  4. Clarify commercial terms. Who pays, what happens to any hardware installed, and whether a successful result converts to a contract at pre-agreed terms.
  5. Score independently. Have your own engineers verify results against ground truth rather than accepting the vendor's report.

Watch for how the vendor handles your messy data. A team that asks good questions about tag meanings, operating modes and failure records is usually a better partner than one that produces impressive charts without asking.

Make reference calls count

Vendors choose their references, so the calls tend to be positive. You can still learn a great deal by asking specific questions. Ask to speak with someone who uses the product daily, not only the executive sponsor. Then ask:

  • How long did it take from contract to the first useful result, and what took longest?
  • How much of your own staff time did implementation and ongoing support require?
  • What did the system get wrong in the first months, and how did the vendor respond?
  • How has pricing changed since you signed, particularly as you added assets or sites?
  • Are you running it at scale, or is it still on the original pilot line?
  • Knowing what you know now, what would you negotiate differently?

If possible, find a user the vendor did not suggest, for example through an industry association or a peer group. Visiting a running installation, even briefly, often tells you more than any presentation.

Data ownership and use

Data terms are often the most consequential part of the contract and the least read. Settle these points explicitly:

  • You own your raw data, derived data and labels, and can export them in a usable, documented format at any time and at the end of the contract.
  • Whether the vendor may use your data to train models used for other customers, and if so, whether it is anonymized and aggregated, and whether you can opt out.
  • Who owns models trained specifically on your data, and whether you can keep using them if you leave.
  • Where data is stored and processed, including subcontractors and cloud regions.
  • How data is deleted at the end of the relationship, with written confirmation.
  • Confidentiality of process parameters, recipes and customer-specific information.

Contract terms to negotiate

  • Pricing basis and escalation. Understand the unit you pay on, such as assets, users, sites, cameras or data volume, and how price changes at renewal and as you scale.
  • Service levels. Availability, support response times and escalation paths, with remedies when they are missed. For edge systems, clarify what keeps working if the vendor's cloud is unavailable.
  • Performance commitments. Where possible, tie part of the fee or the renewal to agreed outcomes from the proof of value.
  • Security obligations. Vulnerability disclosure, patch timelines, breach notification and the right to audit or receive audit reports.
  • Change control. Notice before changes to models, software or infrastructure that affect production, and the right to defer updates during critical periods.
  • Exit and transition. Data export, transition assistance, and what happens to edge hardware and licenses if you leave.
  • Liability. How responsibility is allocated if the system's output contributes to a quality escape or equipment damage. Expect vendors to limit this; know what you are accepting.

Red flags

  • Claims of accuracy or savings with no explanation of how they were measured.
  • Reluctance to test on your data or to let your engineers verify results.
  • Requirements for inbound connections into the control network.
  • Vague answers about who owns your data and models.
  • No clear plan for model monitoring and retraining after go-live.
  • References only from pilots, with no customers running at scale.

Related use cases

Maintenance & reliability

Predictive maintenance

Use sensor and maintenance history to spot developing equipment failures early and schedule repairs before they stop the line.

Machine learningIndustrial IoT & edge
Established7 software companies

Quality

Visual quality inspection

Cameras and computer vision models inspect parts on the line to catch surface, assembly and dimensional defects consistently.

Computer visionMachine learning
Proven, scaling8 software companies

Maintenance & reliability

GenAI knowledge assistants & copilots

Chat-based assistants answer maintenance and frontline questions from manuals, work orders and procedures, with sources cited.

Generative AI & LLMs
Emerging9 software companies

Operations & production

Machine monitoring & OEE analytics

Automatically capture machine states, downtime reasons and output to measure OEE accurately and target the biggest losses.

Industrial IoT & edgeMachine learning
Established8 software companies

Put AI to work

Find the AI opportunities hiding in your plant.

Genvo, our implementation partner, works with manufacturers to pick the right first project, connect the systems you already run and put AI agents and tools into daily use, with people approving what goes out.

Assessments and consultations are provided by Genvo. How our partnership works.