It takes an afternoon to build an impressive LLM demo — and much longer to build a feature customers can rely on. Use this LLM integration checklist to review any AI feature before launch. It applies whether you use OpenAI, Azure OpenAI, Anthropic Claude or another provider.
Scope and value
1. A specific, measurable use case
"Add AI" is not a use case. "Draft a first reply to support tickets in under ten seconds, accepted without edits most of the time" is. Define the task, the user and what success looks like.
2. Real examples collected
Gather real inputs with the correct outputs before writing prompts. They become your evaluation set (see item 9).
3. The right tool for each part
Use normal code for rules, calculations and lookups; use the model for language and judgment. Many failures come from asking the model to do arithmetic or enforce business rules.
Data and security
4. Minimum necessary data
Send only what the task needs. Remove or mask personal data where possible and check your provider's data usage and retention terms.
5. Keys on the server
Call models only from your backend; store keys in a vault. See integrating an LLM API in ASP.NET Core.
6. Permission-aware retrieval
If you use RAG, filter documents by the user's access rights before they reach the model.
7. Prompt injection defenses
Treat user input and retrieved documents as untrusted. Keep system instructions separate, restrict tools, and require approval for sensitive actions. Review the OWASP Top 10 for LLM Applications.
8. Output handling
Encode model output before displaying it, validate structured output (JSON) against a schema, and never execute generated code or queries without checks.
Quality
9. An evaluation set in CI
Run your examples automatically whenever prompts, models or retrieval settings change, and track the score over time.
10. Grounding and honesty
Instruct the model to answer from provided sources, to cite them and to say "I don't know" when the answer is not there.
11. Human review where it matters
Keep a person in the loop for content that is customer-facing, legal, financial or medical — at least until evaluation proves reliability.
Cost and performance
12. Cost controls
- Choose the smallest model that meets the quality bar for each task.
- Limit input and output tokens; trim retrieved context to what is relevant.
- Cache repeated answers and embeddings.
- Set rate limits and budgets per user and per tenant.
13. Latency and user experience
Stream responses where possible, show progress, set timeouts and move long tasks to background jobs with notifications.
Operations
14. Monitoring and logging
Log requests, token usage, latency, errors and user feedback (thumbs up/down) per feature — without logging secrets or unnecessary personal data. Alert on cost spikes and error rates.
15. Fallbacks and provider flexibility
Handle provider outages and rate limits gracefully, keep the rest of the app working, and put model calls behind an abstraction so you can switch or combine providers — see Azure OpenAI vs OpenAI.
Printable summary
| Area | Checks |
|---|---|
| Scope | Measurable use case · real examples · right tool for each part |
| Security | Minimum data · server-side keys · permission-aware retrieval · injection defenses · safe output handling |
| Quality | Evaluation in CI · grounding and citations · human review |
| Cost and UX | Model choice · token limits · caching · budgets · streaming and timeouts |
| Operations | Monitoring · alerts · fallbacks · provider abstraction |
Key takeaway: most LLM features fail on evaluation, security and cost — not on the model. Treat AI like any other production dependency.
Want a second pair of eyes before launch? Our AI integration team reviews and builds LLM features for .NET and Node.js products, and our architecture reviews cover AI systems too.
