AI Error Explanations

1. What it is

AI Error Explanations is a Claude-powered layer that sits on top of Metro2's deterministic validator. Whenever the validator emits a finding — for example DOFD_REQUIRED_FOR_COLLECTION, DATE_TOO_FUTURE, or SSN_INVALID_CHECKDIGIT — you can click “Explain & Fix” and get back:

• A 1-3 sentence plain-English explanation of what went wrong and why the bureau cares.
• A user-facing severity bucket: blocking, warning, or informational.
• A specific suggested fix that targets a single field, when the model is confident enough to recommend one.
• Citations linking back to the CRRG, our in-house field reference, or the error-code database.
• Related findings that are likely caused by the same root issue.

The explanation is rendered in an ExplanationCard component that lives next to the finding. When confidence is high enough, an Apply button performs the fix in one click — with a built-in safety net described in Section 6.

2. When it helps

The feature is built for three concrete moments where Metro 2's terse, code-driven validator gets in the way:

Onboarding new ops teams

New analysts spend their first week staring at codes like ECOA_J1_CONFLICT wondering what to do. AI explanations collapse the “look it up in the CRRG, find the field, figure out the correct value, edit the row” loop into a paragraph and a button.

Batch-fixing import errors

When a freshly imported file produces hundreds of findings, the batch view groups them by error code and confidence. You can review a single representative explanation, then apply all high-confidence suggestions for that code across every affected record.

Quickly understanding obscure CRRG rules

Even seasoned compliance staff occasionally hit a rule they haven't seen in years. The explanation includes citations so you can verify the model's reasoning against the actual reference material instead of taking it on faith.

3. Privacy & safety

We make three hard guarantees before any record data leaves your environment.

PII is redacted before the model sees it

• SSN → only the last 4 digits are sent. Everything before is replaced with XXX-XX-.
• Date of birth → reduced to a year (YYYY). The model can still reason about age but can't identify the individual.
• Account numbers → truncated to the last 4 digits with the rest masked.
• Free-text address lines, names, and phone numbers → replaced with structural placeholders (<CONSUMER_NAME>, <ADDRESS_LINE>).

Grammar-guaranteed JSON output

Claude is called in strict tool-use mode against our emit_explanation JSON schema. That means the model cannot return free-form prose, can't skip fields, and can't invent unexpected ones. There is no “maybe Claude hallucinated a different shape this time” failure mode — the token-level grammar prevents it.

One-click apply re-validates and reverts on failure

Every apply call runs the full Metro 2 validator against the mutated record before persisting. If the original finding is still present, or if any new finding appears that wasn't there before, the mutation is not written to the database. The apply log records the outcome (reverted_validation_failed or reverted_new_finding) and the UI surfaces the error.

Protected fields

A short allowlist of internal fields is hard-blocked from AI mutation regardless of confidence. These include identifiers (id, company_id, user_id, file_upload_id), timestamps (created_at, updated_at), storage pointers (file_path, raw_record), validator state (status, validation_errors, validation_warnings), and anything prefixed internal_ or codex_.

4. Where you'll see it

The feature shows up in two places in the dashboard.

Per-finding: “Explain & Fix” on the badge

Anywhere you see the ValidationStatusBadge — the record drawer, the records list, the file detail page — each individual finding now has an Explain & Fix control. Clicking it opens an inline ExplanationCard for that single finding.

Batch view: per-file explain page

Each file has a dedicated batch explanation page at:

/dashboard/files/[id]/explain

The batch view groups findings by error code, ranks them by total impact (record count × severity), and lets you:

• Review a representative explanation for each code instead of reading the same explanation 200 times.
• See the suggested-fix distribution: how many records would receive an auto-applicable fix vs. how many need user input.
• Click “Apply all high-confidence” for a given code — this applies the suggested fix to every record in the group whose explanation has confidence ≥ 0.85 and which survives the re-validate check.

5. Reading an explanation card

An ExplanationCard renders five elements, top to bottom.

6. Applying a fix

Clicking Apply triggers a six-step pipeline. Steps 4-6 are the re-validate-and-revert safety net.

1. Confidence gate. If suggested_fix.confidence is below the configured threshold (default 0.85), the call is rejected with outcome rejected_low_confidence. The mutation never runs.
2. User-input gate. If requires_user_input: true or the suggested field / value is null, the call is rejected with rejected_user_input_required.
3. Protected-field gate. If the suggested field is on the protected list, the call is rejected with rejected_protected_field.
4. Type coercion. The suggested value (always a string from the model) is coerced to the field's real type — date fields are normalized to YYYY-MM-DD, amount fields are parsed to numbers, everything else stays string. Coercion failure produces outcome error with no mutation.
5. Re-validate. The full Metro 2 validator runs against the mutated record in memory. We compute the finding sets before and after and compare.
6. Apply or revert.
   • If the original finding is gone and no new finding appeared, the mutation is written to metro2_records, the explanation row is marked applied, and the apply log records outcome: applied.
   • If the original finding is still present after the proposed edit, no mutation is written and the apply log records outcome: reverted_validation_failed.
   • If any new finding appears that wasn't there before, no mutation is written and the apply log records outcome: reverted_new_finding along with the list of new findings.

Every applied mutation is mirrored into the standard record_audit_log so it shows up in the record history alongside manual edits, with the same user + timestamp + before/after trail.

7. Confidence threshold

The default apply threshold is 0.85. It can be overridden per environment via the AI_APPLY_CONFIDENCE_THRESHOLD env var (any value in 0.0 - 1.0).

Why 0.85?

We tuned this against an internal eval set during the rollout. At 0.85 the precision of suggested fixes (fraction that pass re-validate without introducing new findings) is high enough that the safety net almost never has to revert. Below 0.85 the false-fix rate climbed sharply — even with the revert net, surfacing low quality suggestions wastes operator attention.

Confidence semantics

The gate is a hard server-side check, not just UI state — even a manually-crafted POST to /apply with a low-confidence explanation will be rejected.

8. Cost cap & model

Per-company monthly cap

Every company has a monthly USD cap on AI spend stored on companies.ai_monthly_cap_usd. The default is $25/month. When usage approaches the cap, new Explain calls return AiCapExceededError and the UI surfaces a friendly “monthly cap reached” state instead of silently failing. The counter resets at the start of each calendar month UTC.

Default model

By default we use Claude Sonnet 4.6 with prompt caching. The system prompt — which includes the CRRG-derived corpus, the field reference, and the strict tool definition — is pinned with a 1-hour TTL cache breakpoint. The per-record payload (the redacted finding) is the only uncached part of each call.

Per-call economics

9. Enabling / disabling per company

The feature is controlled by two company-level settings.

Changes to these columns take effect on the next request — there is no cache to bust on the application side. You don't need to redeploy.

Disabling for a single user

The feature is per-company, not per-user. If you need to keep a specific analyst out of the AI flow, control it at the role/permission level rather than toggling the company flag.

10. Feedback

Every explanation card has a small thumbs-up / thumbs-down control. Hitting either button writes a row to the feedback table along with the user id, the explanation id, and an optional free-text comment.

• Thumbs-up — the explanation was accurate and useful, regardless of whether you applied the fix.
• Thumbs-down — the explanation was wrong, misleading, or unhelpful. We treat these as high-priority signals.

Feedback feeds the future eval harness. As we accumulate flagged cases we'll re-prompt them against new model versions and prompt-corpus updates to confirm regressions don't slip through. Nothing about your feedback is sent back to Anthropic — it stays in our database.

11. Common questions

Can it auto-apply everything?

No, not in v1. Every apply is an explicit click. We'll evaluate background auto-apply once we have enough feedback data to be confident in per-code precision — until then, a human stays in the loop for each mutation.

Why is the apply button disabled?

It will be disabled for one of three reasons:

• Low confidence: the model's confidence is below the threshold (default 0.85).
• Requires user input: the model returned requires_user_input: true — usually because the correct value isn't present anywhere in the record (e.g., a missing DOFD that can't be derived).
• Protected field: the suggested mutation targets a field on the protected list (identifiers, timestamps, storage pointers, validator state, anything internal_ or codex_ prefixed).

Does it edit my data without permission?

No. The explain endpoint is read-only — it generates an explanation and writes it to the explanations table, but never touches metro2_records. The only path that mutates a record is the /explain/apply endpoint, which only fires when you click the Apply button (or the “apply all high-confidence” batch action). There is no background process that applies AI suggestions.

Will it ever pull from CRRG PDFs?

Pending CDIA license review. Today the prompt corpus is built from our in-house error-code database and field reference — both written from scratch by Switch Labs. We deliberately do not embed CRRG PDF text in the corpus. Once we have CDIA's blessing we'll add deep-linked citations that point directly into the licensed PDF. Citations currently labeled crrg in the citation chips refer to our paraphrased, publicly-citeable summary — not verbatim CRRG content.

What models does it use?

Sonnet 4.6 by default. Opus is available per-company via ai_model_preference. Both use strict tool-use against the same emit_explanation schema, so the contract is identical — you're trading cost for reasoning depth, not changing output shape.

What happens if I'm at the monthly cap?

New explain requests return a 429-style error and the UI surfaces a “monthly cap reached” message with a link to billing. Previously generated explanations remain readable, and applying them still works — apply doesn't consume AI budget. The cap resets at 00:00 UTC on the 1st of each month.

12. API reference

All endpoints accept either session auth (when called from the dashboard) or an API key via Authorization: Bearer <key>. All responses are JSON.

Explain a single record

POST /api/metro2-records/{id}/explain

Generates (or returns the cached version of) an explanation for one finding on one record. Body specifies which finding to explain when the record has multiple.

curl -X POST https://metro2.switchlabs.dev/api/metro2-records/REC_123/explain \
  -H "Authorization: Bearer $METRO2_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "error_code": "DOFD_REQUIRED_FOR_COLLECTION",
    "field_name": "fcra_date_of_first_delinquency"
  }'

Explain every finding in a file (batch)

POST /api/metro2-files/{id}/explain

Kicks off background chunking via /api/background/explain-chunk. Returns immediately with a job id you can poll, or with status: "already_running" if a batch is in flight for this file.

curl -X POST https://metro2.switchlabs.dev/api/metro2-files/FILE_abc/explain \
  -H "Authorization: Bearer $METRO2_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "chunk_size": 50,
    "skip_existing": true
  }'

Apply a suggested fix

POST /api/metro2-records/{id}/explain/apply

Applies a previously-generated explanation's suggested fix through the full confidence + protected-field + re-validate-and-revert pipeline. Returns the outcome and the before/after validation snapshots.

curl -X POST https://metro2.switchlabs.dev/api/metro2-records/REC_123/explain/apply \
  -H "Authorization: Bearer $METRO2_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "explanation_id": "EXP_456"
  }'

Example successful response:

{
  "outcome": "applied",
  "old_value": "20251301",
  "new_value": "2025-01-13",
  "validation_before": { "errors": [...], "warnings": [] },
  "validation_after":  { "errors": [],    "warnings": [] }
}

Example reverted response:

{
  "outcome": "reverted_new_finding",
  "old_value": "0",
  "new_value": "1000",
  "error": "Applying would introduce new finding(s): AMOUNT_PAST_DUE_EXCEEDS_BALANCE|amount_past_due|...",
  "validation_before": { "errors": [...], "warnings": [] },
  "validation_after":  { "errors": [...new...], "warnings": [] }
}

Submit feedback on an explanation

POST /api/metro2-explanations/{id}/feedback

Records a thumbs-up or thumbs-down with an optional comment. Idempotent on (explanation_id, user_id) — resubmitting overwrites the prior rating.

curl -X POST https://metro2.switchlabs.dev/api/metro2-explanations/EXP_456/feedback \
  -H "Authorization: Bearer $METRO2_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rating": "down",
    "comment": "Suggested fix cleared the finding but used the wrong date format upstream of the bureau spec."
  }'

13. Troubleshooting

The explanation seems wrong

Click thumbs-down on the card — this flags it for the eval harness. If you already applied the fix and the explanation was misleading, you can revert the field manually (the audit log shows the AI-applied value as the previous value, so rolling back is just a regular edit). Drop a one-line comment with the feedback so the eval reviewer has context.

Apply failed with “Applying would introduce new finding(s)”

This is the safety net working as designed — reverted_new_finding. The proposed edit cleared the original error but would have created a different one. No mutation was written. Open the apply log entry (linked from the card) to see the exact new findings; they often point at a more fundamental issue with the record that the model didn't have enough context to spot.

Apply failed with “Original finding still present”

Outcome reverted_validation_failed. The model's suggested value, when run through coercion and the validator, did not actually resolve the finding it was supposed to fix. No mutation was written. Thumbs-down the explanation so we can re-train against the case.

The Apply button is disabled even though confidence looks high

Check whether the suggested-fix block has requires_user_input: true, or whether the target field is on the protected list. The button is disabled in either case regardless of confidence.

I got an “AI explanations disabled for this company” error

ai_explanations_enabled is false on your company row. An admin can flip it via the company settings page.

I got an “AI monthly cap exceeded” error

You've hit ai_monthly_cap_usd for the month. Either wait until the 1st of next month (UTC) or have an admin raise the cap. Applying previously-generated explanations does not count against the cap — only new explain calls do.

Batch explain seems stuck

The batch view kicks off background chunks via /api/background/explain-chunk. Each chunk processes ~50 findings, then enqueues the next. If progress stalls for more than ~2 minutes, refresh the page — the chunker is idempotent and will skip explanations that already exist. Re-running the batch is safe.

14. Related guides

• Payment History Grid — how to read and edit the 24-month payment grid, where a lot of AI explanations land.
• Analytics Suite — where AI-applied fixes show up in error-rate trend charts and per-code dashboards.