Procure an AI-enabled service by use case and consequence
AI procurement begins with a bounded use case, the people affected and the consequence of a wrong, unsafe or unavailable output. Buying “AI capability” without that boundary makes evidence impossible to evaluate because the same model can carry very different risk in drafting, search, recommendation, classification or automated decision support.
Describe the task, input, output, user, reviewer, downstream action and prohibited use. State whether the system advises, drafts, ranks, detects or acts. Human oversight is meaningful only when the reviewer has time, information, authority and a practical way to reject or correct the output.
Treat supplier claims as hypotheses until evidence matches the proposed configuration and context. A benchmark, model card or policy can inform due diligence, but it does not prove performance with your data, language, workflow, access controls or tolerance for error.
| Area | Question | Evidence |
|---|---|---|
| Purpose | What task and decision does it support? | Approved use-case statement |
| Performance | How does it behave on representative cases? | Evaluation set and results |
| People | Who can review, contest and stop it? | Oversight workflow |
| Data | What is collected, retained and reused? | Data-flow and contract evidence |
| Operation | How are change, incidents and exit handled? | Service and control model |
Classify impact before choosing evidence depth
Assess the materiality of error, scale, reversibility, affected population, sensitivity, autonomy and reliance. A tool that proposes internal meeting summaries does not need the same controls as one that influences access, employment, credit, health, safety or legal rights.
Identify applicable law, regulation, policy and contractual duties with qualified reviewers. Avoid publishing a universal risk classification from a generic checklist. Jurisdiction, sector, role and technical design can materially change the assessment.
Use the classification to set gates, testing, approval, monitoring and fallback. High impact should lead to stronger evidence and tighter operating controls, not merely a longer questionnaire completed by the same supplier sales team.
Understand the complete system and supplier chain
Map foundation model, application layer, retrieval sources, prompts, filters, hosting, logging, monitoring, human workflow and external tools. Record which party controls each component and which changes can occur without customer approval. The product name alone does not define the system being procured.
Ask whether inputs or outputs are retained, used for training, reviewed by people or transferred to subprocessors. Confirm tenant separation, administrative access, deletion, encryption, location and incident handling for the proposed plan and region.
Track dependencies and substitution options. A vendor may change a model version or provider while keeping the interface stable. Decide what notice, re-evaluation and acceptance are required when that change can alter behaviour, cost, data flow or risk.
Request evidence that explains limitations
Ask for model and system documentation, intended and excluded uses, evaluation method, known limitations, data governance, safety testing, security practice and post-release monitoring. Prefer evidence with scope, date and method over broad statements about responsibility or accuracy.
The NIST AI Risk Management Framework organises voluntary risk work around governance, mapping, measurement and management. NIST notes that AI RMF 1.0 is being revised, so record the version used and treat it as a framework for questions rather than a certificate.
For generative systems, the NIST Generative AI Profile identifies risks and suggested actions across the lifecycle. Adapt it to the use case and current evidence; a supplier’s statement of alignment does not establish that every action is implemented.
Build a representative evaluation set
Create cases from real task patterns without exposing unnecessary live data. Cover common inputs, difficult edge cases, different languages or user groups, prohibited requests and situations where the correct response is uncertainty or refusal. Preserve expected outcomes and reviewer rationale.
Choose measures that fit the task. Accuracy may be meaningful for a bounded classification, while drafting quality needs structured human criteria and factual verification. Include harmful failure, omission, unsupported confidence, consistency, latency and the effort required to review.
Keep supplier tuning and test exposure visible. A demonstration on familiar examples does not estimate live performance. Hold back cases, repeat tests after material changes and document temperature, retrieval, system prompts and other configuration that influences output.
Design human oversight as an operating control
State who reviews outputs, what they must check and which decisions remain human. Provide source context and uncertainty where it helps review. Do not claim “human in the loop” when a high-volume queue leaves reviewers only seconds or when the interface encourages automatic acceptance.
Create escalation and contest routes for users and affected people. Record how corrections influence the individual case, underlying data, prompt, model configuration and monitoring. A correction that disappears into a support ticket is not a complete control.
Measure reviewer workload, disagreement, override and missed error during the bounded software pilot. Oversight can fail through fatigue, unclear authority or poor evidence even when the workflow contains an approval button.
Secure tools, data and generated actions
Apply least privilege to users, administrators, integrations and any tools the model can call. Treat retrieved content and external inputs as potentially untrusted. Separate a model’s ability to propose an action from authority to execute it.
Test prompt injection, data leakage, unsafe tool use, excessive agency, insecure output handling and abuse relevant to the architecture. Ask how the supplier discovers, communicates and remediates vulnerabilities and how customers can disable affected capability.
Log enough to investigate decisions and incidents without creating an uncontrolled store of sensitive prompts and outputs. Define access, retention, redaction and export. Monitoring data should follow the same evidence and privacy discipline as the service it observes.
Put change, incident and exit terms into the service model
Require notice for material model, provider, data-use, safety-control and subprocessor changes. Define when the organisation can retest, reject, disable or terminate affected capability. Continuous provider improvement does not remove the buyer’s need to understand behavioural change.
Prepare incident contacts, evidence preservation, containment, customer communication and recovery. Include AI-specific events such as systematic harmful output, leaked information, compromised retrieval or an unsafe tool action in the tabletop programme.
Use the SaaS exit plan to cover data, prompts, configuration, evaluation artefacts, logs and dependent workflows. An AI feature may be easy to disable technically while leaving a process unable to operate without it.
Make the approval conditional and reviewable
Present the use case, classification, supplier chain, evidence, evaluation, oversight, security, data flow, operating model, cost and open risks in one decision pack. Separate what the supplier claims, what the organisation tested and what remains unknown.
Approve a defined configuration, population, task and period with named owners and stop conditions. Expansion to new decisions, data or users should trigger review rather than inheriting approval because the same product is involved.
Responsible procurement does not promise risk-free AI. It establishes a specific purpose, evidence proportionate to consequence and an operating ability to detect, challenge, change and stop the system. That is a stronger basis for value than novelty or a generic assurance badge.
Control value claims and operating cost
Ask which measurable task outcome justifies using AI rather than a simpler rule, search or workflow. Compare review effort, error handling, monitoring, model consumption, retrieval maintenance and incident response with the current process. A fast draft can create more total work when every factual statement needs repair.
Separate supplier benchmark performance from local value evidence. During the pilot, measure successful completion, review time, escalation, override and harmful failure for the approved use case. Do not turn adoption or token volume into a benefit without showing the operating consequence.
Set a review point where the owner can narrow, redesign or retire the feature. AI capability should not become permanent because teams have already integrated it. Preserve the evidence and fallback that allow the organisation to choose a simpler service when it remains the better fit.
