VerdifaX

Troubleshooting

Common issues when integrating with Verdifax, with the resolution steps that close them. Organized by error symptom so you can find the right page from a stack trace or a curl response.

Authentication

{"ok": false, "error": "missing X-Verdifax-Key header"}

You called an authenticated endpoint without an API key. Three fixes:

  1. Set the header, Verdifax uses X-Verdifax-Key, not the common Authorization: Bearer ... pattern.

    curl -H "X-Verdifax-Key: $VERDIFAX_ADMIN_KEY" \
  https://api.verdifax.com/runs

  2. Get a key if you don't have one. Keys for the hosted API at https://api.verdifax.com are issued after an access request at verdifax.com/request; POST /admin/keys there requires an admin key and returns 401 without one. On a self-hosted deployment (which needs access to the private image) where no key has ever been created (open mode), the first POST /admin/keys is accepted without auth and returns the secret once. Mint that first key with admin scope; a key minted without it is customer scope and gets 403 on /admin/*:

    curl -X POST http://localhost:9090/admin/keys \
  -H "Content-Type: application/json" \
  -d '{"name": "ops-admin", "scope": "admin", "confirm_admin_scope": true}'

    Once any key exists, revoked or not, POST /admin/keys requires an admin key. If every admin key is lost or revoked, run the break-glass verdifax-api --mint-admin-key <name> inside the machine that hosts the API server.

  3. Persist the secret, POST /admin/keys returns the secret exactly once. After that, it's hashed in the orchestrator's database and not retrievable. If you lost it, revoke and create a new one.

401 Unauthorized or 403 admin scope required on /admin/erase or /admin/keys/{id}

Admin endpoints, including /admin/erase and /admin/keys, require an admin-scope API key. A valid customer-grade key gets 403 admin scope required. If you got a 401, the key you sent is either revoked, malformed, or for a different deployment. Re-check via GET /admin/keys with an admin key (it lists every key without secrets).

Manifest verification

verified: false with non-empty recomputed_hash

The manifest hash recomputed from the sealed fields doesn't match what was stored. Two possible causes:

  1. The run was sealed under an older formula era. Verify the formula_version field in the response. If it says v1.6 (Phase 16, Kernel Artifacts) but your verifier expects v1.7 (Phase 17, CRES), the run pre-dates CRES (the current era is v1.9 (execution record)). All eras verify under the same orchestrator binary; the dispatch picks the right formula based on which JSON keys are present in the sealed manifest. If your local verifier hard-codes an era, update it to use the dispatch table.

  2. Tampering or storage corruption. The hash is recomputed from the parsed values of the sealed manifest fields, so JSON formatting (whitespace, key order) does not affect it. If any sealed field's value differs from what Verdifax sealed, even by one character, the hash mismatches. This is the intended invariant; treat it as evidence the artifact has been modified.

verified: true but the audit PDF still complains

The PDF and the JSON manifest are independent surfaces. If the PDF shows a section as (not declared) but the JSON has the field populated, the issue is in the PDF rendering layer, not the sealed data. Re-render the PDF via GET /runs/{id}/report.pdf?format=comprehensive and the section should populate from the live JSON.

Audit PDFs

404 Not Found on /runs/{id}/report.pdf

The endpoint is /runs/{id}/report.pdf, not /runs/{id}/audit.pdf or /runs/{id}/audit. Older Verdifax deployments used /audit; that path was renamed to /report for clarity.

PDF renders but section X is empty

Audit PDFs degrade gracefully, if the run didn't reach a stage, the section shows (not declared) and the manifest hash still seals. This is the intended behavior; absence is informational, not an error.

If a stage should have populated a section but didn't, check the run's status field via GET /runs/{id}:

  • pepg_deny, PEPG denied the request before downstream stages ran. Check the DenyReceipt at /runs/{id}/deny-receipt.
  • ccv_halt, per-run budget breached at the post-DKEC check. Check the CCVHaltReceipt at /runs/{id}/ccv-halt-receipt.
  • macc_halt, the budget bucket's cumulative budget breached. Check the MACCHaltReceipt at /runs/{id}/macc-halt-receipt.
  • stage_error, a stage hit an error. Check the run's error_stage and error_message fields.

CRES (cryptographic record erasure)

POST /admin/erase is admin-only. It destroys a per-field data-encryption key and writes a sealed deletion receipt. The encryption path is not yet wired into /execute, so the destroyed key encrypts no stored field and the run's sealed bundle is unchanged. A deletion receipt therefore records that a key was destroyed; it is not evidence that any readable content was removed.

POST /admin/erase returns 400 ciphertext_hash must be 64-char SHA-256 hex

You passed a malformed ciphertext_hash. Three checks:

  1. The string is exactly 64 characters.
  2. It contains only 0-9 and a-f (lowercase only).
  3. It's not the plaintext, the orchestrator wants the SHA-256 of the ciphertext, which the caller should have recorded at encryption time.

POST /admin/erase returns 404 run not found on a valid run_id

The endpoint is admin-only, and it also looks the run up scoped to the API key that's calling it. If you're using an admin key that didn't create the run, you'll get 404 instead of 403 (defense-in-depth: you can't enumerate runs you don't own). A customer-grade key gets 403 admin scope required before the lookup.

Receipt verifies offline but the orchestrator says dek_still_present

Two surfaces report different things:

  • Recomputing the receipt hash from its canonical preimage (the fields and joining rule the audit PDF lists) confirms only that the receipt is self-consistent. It can't see the orchestrator's keystore. The open-source verifier does not check deletion receipts.
  • The orchestrator's Verify() method also checks that the DEK is gone from the keystore. If it reports dek_still_present but the receipt is otherwise valid, the shred operation either failed or was reverted by a backup-restore. Re-issue the shred.

Sigstore Rekor anchoring

LedgerProofHash is empty in the manifest

You're running with VERDIFAX_LEDGER_MODE=mock (the default for local dev + tests). To get real Rekor anchoring, set VERDIFAX_LEDGER_MODE=rekor on the orchestrator and restart.

Rekor's search UI takes ~30 seconds to surface a newly anchored entry. A run whose PDF shows a Rekor link was anchored: a failed Rekor submission does not produce an unanchored OK run, it fails the run at stage LEDGER, which is stored with status stage_error. The orchestrator logs such a failure as "pipeline rejected" with stage LEDGER and an error of the form orchestrator: stage LEDGER failed: rekor: .... Most of those failures are transient, and the next run anchors successfully.

Performance

/execute is slow when VERDIFAX_AIVP_MODE=t4

Real-mode AIVP invokes the live Anthropic Claude API per request. Per-call latency is dominated by Claude's response time (~50ms-500ms p50). Switch to VERDIFAX_AIVP_MODE=mock for tests and local development; reserve t4 for production governance.

Orchestrator OOM at 256MB

Default Fly machine is shared-cpu-1x with 256MB. If you're seeing OOM kills under load, bump to shared-cpu-1x 512MB by editing the [[vm]] block in fly.toml:

[[vm]]
  size = "shared-cpu-1x"
  memory = "512mb"

Per-run memory is dominated by the AIVP subprocess (~10MB resident) plus PDF rendering (peaks at ~50MB during comprehensive format generation). 256MB is enough for steady-state ~50 req/sec; above that, bump.

Where to file unresolved issues

If your problem isn't covered here: