F 14 Anthropic Ai

Status: 🟢 ready to test Owner: Adilet Last updated: 2026-05-08 Last verified on production: —

Goal

An admin stores the Anthropic API key in the database (no env var required), tests it from the admin UI, and three fire-and-forget AI agents (intake / run summary / cycle close) light up across the platform. Errors are categorised into 8 friendly modes with retry buttons.

Actors

• Primary — Admin (configures + retries)
• Secondary — Anthropic API; downstream agents that auto-run

Preconditions

• Admin is signed in.
• A working Anthropic API key (starts with sk-ant-).

Trigger

Admin opens /app/admin/integrations and clicks the "Configure" CTA on the Anthropic card.

Happy path — configure key

Happy path — three downstream agents fire

Once configured, three agents auto-run on respective triggers:

All three call Anthropic with claude-sonnet-4-20250514. Vision uses { type: 'image', source: { type: 'url', url } } directly with public DO Spaces URLs (no base64). Capped at 3 images per intake call.

Happy path — retry intake

(Retry is admin/lead only — testers can't burn budget.)

Failure categorisation

src/lib/ai/intake.ts → categoriseError() maps Anthropic errors to:

Acceptance criteria

• AC-1 — Given the admin saves an Anthropic key, when the next issue is filed, then within 60s issues.ai_suggestions is populated AND the ai_runs row's status='done' with cost_usd_cents set.
• AC-2 — Given a bad key, when "Test" is clicked, then the response surfaces "auth_failed" friendly copy and Save is blocked (or save anyway depending on UX — verify on prod).
• AC-3 — Given the key is removed (DELETE /api/admin/integrations/anthropic), when the next issue is filed, then isAiEnabled() returns false, no Anthropic call is made, and the issue still saves cleanly.
• AC-4 — Given a vision-bearing issue with 5 images, when intake runs, then only the first 3 images are sent to the model (cap).
• AC-5 — Given a model fails to call the submit_intake_analysis tool, when the response is parsed, then ai_runs.failure_reason='parse_error' and the AiSuggestionsCard shows "Re-running may help" with a Retry button (admin/lead only).
• AC-6 — Given a non-admin views the AiSuggestionsCard's failure state, then there is no Retry button visible.
• AC-7 — Given a cycle is set to status='closed', when the cycle-close agent finishes, then a new cycle_documents row exists with kind='report' and the body is the AI's summary.
• AC-8 — Given the SDK client cache is in-memory and keyed on the resolved API key, when an admin rotates the key, then subsequent calls use the new key without a server restart.

Test data / fixtures

• Baseline seed
• A working Anthropic API key with at least $0.10 of credit
• An active cycle with an open run + a tester ready to file an issue

Negative paths

Manual QA checklist

• As admin, open /app/admin/integrations
• Confirm Anthropic card shows "Not configured"
• Click Configure → paste valid key → click Test → green
• Click Save → card flips to "Active" with masked key
• As tester, file a new issue with 1 screenshot (F-08)
• Wait ~30s, refresh issue detail → AiSuggestionsCard shows suggested title / severity / bug_type
• Verify ai_runs table has a row with kind='intake_issue', status='done', cost_usd_cents > 0
• Edit the Anthropic config → paste an invalid key → save → next issue shows the failure card with auth_failed friendly copy
• Restore valid key
• As admin, click Retry on a failed AiSuggestionsCard → new ai_runs row, suggestions populate
• As tester, sign in and view a failed AiSuggestionsCard → confirm Retry button is not visible (admin/lead only)
• Close a cycle (status='closed') → wait ~60s → new cycle_documents row appears with kind='report'
• Disconnect the integration → confirm subsequent issues file with no AI call (no ai_runs row created)

Automated test outline

test.describe("F-14 anthropic ai", () => {
  test("admin configure → test → save", async ({ page }) => { … });
  test("intake populates suggestions", async ({ request }) => { … });
  test("retry intake (admin only)", async ({ request }) => { … });
  test("disconnect short-circuits agents", async ({ request }) => { … });
  test("8 failure modes categorised", async ({ request }) => { /* one test per mock */ });
});

Code references

• UI: src/components/admin/integrations/AnthropicIntegrationCard.tsx, src/components/test-cycles/AiSuggestionsCard.tsx
• Pages: src/app/app/admin/integrations/page.tsx
• API:
  • src/app/api/admin/integrations/anthropic/route.ts (GET/PUT/DELETE)
  • src/app/api/admin/integrations/anthropic/test/route.ts
  • src/app/api/test-cycles/[id]/issues/[issueId]/intake/route.ts (retry)
• Lib:
  • src/lib/ai/client.ts (getAnthropic, isAiEnabled, loadAnthropicConfig, invalidateAnthropicCache, readAnthropicIntegrationStatus)
  • src/lib/ai/intake.ts (runIntakeForIssue, categoriseError, AI_FAILURE_REASONS)
  • src/lib/ai/run-summary.ts
  • src/lib/ai/cycle-close.ts
• Schema: organization_integrations (provider_slug='anthropic'), ai_runs, AI_RUN_KINDS, AI_RUN_STATUSES, issues.ai_suggestions + issues.ai_intake_run_id, test_runs.ai_summary

Events emitted (proposed)

• integration.configured — { provider: 'anthropic', configured_by }
• integration.tested — { provider, success, error_code }
• integration.disconnected — { provider, disconnected_by }
• ai.intake_completed — { issue_id, latency_ms, cost_usd_cents, suggested_severity, confidence, image_count }
• ai.intake_failed — { issue_id, failure_reason }
• ai.intake_retried — { issue_id, retried_by }
• ai.run_summary_completed / _failed
• ai.cycle_report_completed / _failed

Open questions / known gaps

• Cost cap per cycle: ai_runs.cost_usd_cents is logged but not enforced (no hard ceiling). Flagged in feature-matrix.md §5.2.
• Run-summary has no retry button (only intake does).
• No "regenerate cycle report" button — only fires once on close.
• Vision is image-only — no video frame extraction.
• Encrypted credentials store deferred; key is in plain organization_integrations.config.api_key jsonb.