Building Custom AI Integrations and Embedding Prompts in Products

Using ChatGPT directly is fine for one-off tasks. Embedding AI into your product requires integrations, API calls, and prompt management at scale. I've integrated AI into three products and documented the patterns. The tricky part is reliability and cost at volume. I'm documenting the integration patterns that work.

Prompt Management and API Integration Architecture

Architecture: your app calls an AI endpoint (OpenAI API, Anthropic API, self-hosted). You pass: [SYSTEM_PROMPT] + [USER_INPUT]. The API returns: [OUTPUT]. Management problem: system prompts are hardcoded or scattered. Solution: prompt repository. Store prompts in a database or version control. Your app fetches the latest prompt before calling the API. Structure: prompt_id, version, system_prompt_text, updated_at. Example: 'summarize_email' v3 fetches from database, passes to API, gets summary. If you update the prompt (v4), next call uses the new version. Benefit: you can update prompts without deploying code. Testing an improvement: deploy v4 to 5% of users, measure, compare to v3. If better, deploy v4 to 100%. This is A/B testing for prompts at product scale. I implemented this on three products. Cost per call: $0.02-0.05. Volume: 100k calls/month. Cost management: set routing logic (route easy requests to cheaper/faster models, complex to GPT-4). This saved 40% on API costs.

Caching is valuable. If 20% of requests repeat (same user, same question), cache the response. No API call needed. This saves cost and improves latency.

• System prompts in database, not hardcoded in code

• Versioning: v1, v2, v3 stored separately, easy rollback

• A/B testing: route subsets to different versions, measure performance

• Cost control: route by complexity; cheaper models for simple requests

• Caching: for repeated requests, serve cached response

• Monitoring: track latency, cost, error rate per prompt version