"Add AI" is not a brief. "Build a chatbot that knows our docs" is barely better — it usually produces a demo that impresses a board slide and fails the first real workflow. If you are hiring an agency for an AI feature, brief the outcome, the evaluation harness, and the human gates — not the model brand. Teams that ship durable AI products treat evaluations as the scope: a versioned set of realistic tasks, graders, pass thresholds, and regression gates on every change ( OpenAI evaluation best practices ). That framing transfers cleanly to agency contracts.
Replace the demo request with a workflow brief
Write one paragraph that names: the user role, the job they do today, the artifact or decision the AI should improve, and what "done" looks like without AI poetry. Example: "CS lead triages inbound tickets; AI drafts a reply using only tickets + approved KB; agent edits and sends; target is draft accepted with ≤2 edits on 70% of tier-1 tickets in a held-out week." Attach today’s baseline (manual time, error modes, tools). If you cannot measure baseline, your success metric is fiction — fund a one-week measurement spike before model work. Bring three anonymized examples of excellent human output and three failure modes; those become the seed of your eval set.
What must be in the agency brief
Workflow ownership. Who is accountable when the model is wrong — the end user, a reviewer role, or an automated rollback? Name the system of record the AI may read and write. Data boundaries. Which tenants’ data may enter prompts or fine-tunes? PII/PHI rules, retention, logging redaction, and whether third-party model APIs are allowed. Put "no training on customer data" (or the opposite, explicitly) in writing. Human-in-the-loop (HITL). Which actions need approval before side effects (send email, change CRM fields, create tickets)? Escalation triggers, sampling rate for review, and approval SLA. Silent writes are how AI projects lose trust. Evaluation set. Start with 30–100 real tasks (anonymized), including edge cases and known failure modes. Define automated graders where possible and human rubrics where judgment matters. Require the same cases to run in CI against the full product harness — prompts, tools, routing — not only raw model playgrounds ( harness-aware evals ). Success metrics. Outcome metrics (task success, edit distance, time-to-resolution) plus cost and latency budgets. A feature that is "clever" but twelve seconds and expensive per call may be unusable in your UI even if the demo sparkles.
Contract patterns that prevent chatbot tourism
Prefer a paid, time-boxed pilot with a kill clause: deliver eval harness + baseline scores + one workflow path, then decide on the main build. Tie acceptance to passing thresholds on the held-out set, not to a live demo day. Require traces (inputs, tool calls, outputs) for failed cases so production bugs become new eval items. Ban scope language like "integrate GPT/Claude with our app" without naming tools, write permissions, and HITL. Allow model swaps behind an interface so vendor choice is an implementation detail, not the product. Ask for a cost model per thousand tasks at expected volume before you scale a prompt that looked cheap in a playground. Put change control in the SOW: new tools, new write permissions, and new data sources are change orders, not "small prompt tweaks." That single rule stops scope from expanding every time a stakeholder sees a viral agent demo.
What to explicitly leave out of v1
Open-ended "company brain" chat with write access to everything. Unbounded tool use. Fine-tuning before you have failing eval clusters that justify it. Multi-agent orchestration theater when a single retrieval + draft + approve loop would do. Those belong in a later phase once the first workflow clears its gates. Also postpone broad "AI everywhere" UI: one entry point with clear expectations beats a sparkle icon on every screen that nobody trusts. Resist vendor-driven roadmaps that start with a chat widget before the workflow brief exists.
How founders should run the first agency week
Day 1–2: lock workflow, data boundaries, and HITL matrix. Day 3–5: stand up eval set and scoring. Only then authorize feature UI. Review a scored report, not a screenshare of happy-path chat. If the agency cannot show failing cases and a plan to fix them, you bought a demo. Publish a one-page "AI operating rules" doc for your team: allowed data classes, forbidden actions, escalation contacts, and how customers are told a human reviewed the output. Agencies execute faster when policy is written once instead of rediscovered in every ticket. Schedule a monthly eval review the same way you schedule billing reviews — production failures should graduate into new test cases, not tribal knowledge in Slack. Keep legal and security in the loop early for data processing addenda and subprocessors, but do not let questionnaire theater replace the eval harness. CodeFlamme briefs AI work the same way we brief SaaS MVPs: dated owners, measurable acceptance, and reversible architecture — so founders buy a workflow improvement, not a chatbot souvenir.
Related reading: before you sign off on any AI scope, compare what founders should demand from production RAG with how to design approval gates for agentic products . If you are still deciding whether the feature should exist at all, see when to add AI to your product and when it is expensive theatre , and when you price the pilot, our guide to fixed-price vs milestone billing shows how to tie payment to passing evals. Want a second pair of eyes on your AI brief? Talk to the CodeFlamme team .
