BlameTrail

Live demo — this is what BlameTrail looks like on the Pro plan.

Get Started Free
Back to Incidents

Payment processing failures — 500 error spike

resolved
critical
availability
Incident Timeline

Deploy Suspected (#1)

at start

v2.14.0 · a1b2c3d · by alex.chen

Incident Opened

+12m

critical availability incident detected

AI Summary Requested

+13m

Sentry Issue Linked

+13m

payment-api · PROJ-4821

Incident Acknowledged

+14m

AI Summary Generated

+14m

Root cause analysis completed

Jira Ticket Linked

+17m

PAY-1247 — Payment processing failures — 500 error spike

Observability Summary

+19m

Metrics and logs analysis completed

Incident Resolved

+39m

AI Summary

Updated with details from the most likely recent deploy, including recent code changes when available.

The Payment API began returning HTTP 500 errors on POST /v1/payments/charge at 08:17 PM. The error rate jumped from a baseline of ~0.5% to 42% within 2 minutes. **Root cause:** Deploy v2.14.0 (commit a1b2c3d) introduced a null-reference bug in `src/services/payment/validator.ts`. The new Stripe metadata validation accesses `metadata.customer_tier` without a null check. Roughly 40% of payment requests lack this field, causing `TypeError: Cannot read properties of null (reading 'customer_tier')`. **Impact:** ~1,240 failed payment attempts over 27 minutes. No duplicate charges detected — failures occurred before the charge was submitted to Stripe. **Resolution:** Rollback deploy v2.14.1 (commit f4e5d6a) added an optional-chaining guard. Error rate returned to baseline within 3 minutes of deployment.

CompletedUpdated with deploy detailsUpdated 5/4/2026, 8:19:19 PM

Why This Deploy Is Suspected

Structured evidence behind the current top suspect deploy.

#1Score: 94%Confidence: 94%a1b2c3dv2.14.0feature/tiered-pricing

Suspicion Signal

Deployed 12 minutes before incident. Changed payment validation logic in the exact endpoint that started failing.

Commit

feat: validate Stripe metadata.customer_tier for tiered pricing

Pull Request

Add tiered pricing validation to payment flow

Deployed: 5/4/2026, 8:05:19 PMBy: alex.chenEnv: productionSource: github-actions

Most Relevant Changed Files

src/services/payment/validator.ts
modified+34 -8relevance 0.95

Directly modifies payment validation logic — the exact code path producing 500 errors.

src/services/payment/types.ts
modified+12 -2relevance 0.6

Adds CustomerTier type definition used by the new validation.

src/services/payment/__tests__/validator.test.ts
modified+45relevance 0.3

Test file — tests pass but don't cover the null metadata case.

Fix Actions

Generate a revert or AI-powered fix and open a pull request.

AI Fix
PR Created
92% confidencelow risk

Add optional chaining guard for metadata.customer_tier in payment validator

PR #849
Merged
Revert
PR Created

Revert commit a1b2c3d: feat: validate Stripe metadata.customer_tier for tiered pricing

PR #848
Closed
Trace Exploration
Tempo

24 traces queried · 5 with errors

payment-api:POST /v1/payments/charge (3)
payment-api:validateMetadata (2)
ScoreServiceOperationDurationSpansErrors
78
payment-apiPOST /v1/payments/charge1240ms82
62
payment-apiPOST /v1/payments/charge890ms61
35
payment-apiGET /v1/payments/history320ms40

Latency Regressions

moderate
1 operation regressed
payment-api/POST /v1/payments/charge
severe
p50
180ms→310ms
+72.2%
p95
340ms→890ms
+161.8%
p99
520ms→1240ms
+138.5%
Showing data from May 4, 08:17:19 PM to May 4, 08:44:19 PM
resolved

AI Observability Summary

AI-generated analysis of metrics, logs, and deploy context for this incident.

Correlated a sharp error rate spike (0.5% → 42%) with deploy v2.14.0, which landed 12 minutes before incident onset. Logs confirm TypeError in payment/validator.ts when metadata.customer_tier is null. The spike resolves after rollback deploy v2.14.1.

Likely Causes

Null-reference in Stripe metadata validation (validator.ts:47)

high

Logs show TypeError: Cannot read properties of null (reading 'customer_tier') on 40% of requests. Introduced in commit a1b2c3d.

Missing null-check for optional Stripe metadata field

high

metadata.customer_tier is only present on enterprise accounts (~60% of traffic). The new code assumes it always exists.

Suspicious Commits

a1b2c3dAI Flagged

feat: validate Stripe metadata.customer_tier for tiered pricing

Introduced metadata.customer_tier access without null guard in payment validation hot path.

By: Alex ChenMay 4, 08:05 PM

Observed Anomalies

  • Error rate jumped from 0.5% to 42% within 2 minutes of deploy v2.14.0
  • p99 latency dropped from 340ms to 12ms (fast failures — requests error before reaching Stripe)
  • No corresponding increase in Stripe API errors — failures occur pre-charge

Recommended Actions

  • Add optional chaining: metadata?.customer_tier
  • Add null-metadata test case to validator.test.ts
  • Consider adding a pre-deploy canary that replays recent production payloads
Completedhigh confidenceGenerated 56d ago
What Changed Before This Incident?
a1b2c3d
AI Flagged
v2.14.0
feature/tiered-pricing

feat: validate Stripe metadata.customer_tier for tiered pricing

by alex.chenMay 4, 08:05:19 PM
Error Rate (%)
Prometheus

Max

42.3%

Avg

14.7%

Min

0.2%

Current

0.4%

Recent Correlated Logs
TimestampSeverityMessage
May 4, 08:18:19 PMerrorTypeError: Cannot read properties of null (reading 'customer_tier') at validatePayment (validator.ts:47)
May 4, 08:19:19 PMerrorPayment charge failed: internal validation error for customer cus_9Qx4mR1kLp — request aborted before Stripe call
May 4, 08:20:19 PMwarningError rate threshold exceeded: 15.2% (threshold: 5%) on POST /v1/payments/charge
May 4, 08:21:19 PMerrorTypeError: Cannot read properties of null (reading 'customer_tier') at validatePayment (validator.ts:47)
May 4, 08:24:19 PMerrorBatch failure: 47/120 payment requests returned HTTP 500 in the last 60 seconds
May 4, 08:28:19 PMerrorTypeError: Cannot read properties of null (reading 'customer_tier') at validatePayment (validator.ts:47)
May 4, 08:34:19 PMinfoDeploy v2.14.1 initiated by ops-bot — rollback of v2.14.0 (commit f4e5d6a)
May 4, 08:40:19 PMinfoError rate recovering: 2.1% on POST /v1/payments/charge (was 42%)
May 4, 08:43:19 PMinfoError rate returned to baseline: 0.4% on POST /v1/payments/charge — incident auto-resolved