Production Engineering - Best Practices

Create a free account to save your progress

Earn XP, track streaks, and sync your dashboard across devices.

Lesson

You've integrated Stripe, Clerk, and Resend. Now the question is: what happens when they go down? What happens when you hit rate limits at 3am? What happens when your APIWhat is api?A set of rules that lets one program talk to another, usually over the internet, by sending requests and getting responses. key leaks? These are the operational realities that separate a production app from a demo.

Robust error handling

The first thing to accept is that external APIs will fail. Networks glitch, services have outages, keys expire, rate limits get hit. Your code must have an opinion about what to do in each case.

async function callExternalAPI<T>(
  fn: () => Promise<T>,
  options: { service: string; fallback?: () => T }
): Promise<T> {
  try {
    return await fn();
  } catch (error: any) {
    // Rate limited - slow down and retry
    if (error.status === 429 || error.code === 'rate_limit') {
      console.warn(`Rate limited by ${options.service}. Back off and retry.`);
      throw new RateLimitError(options.service);
    }

    // Auth failure - alert immediately, this needs human attention
    if (error.status === 401 || error.code === 'invalid_api_key') {
      await alertOncall(`Invalid API key for ${options.service}`);
      throw new AuthError(options.service);
    }

    // Service unavailable - use fallback if available
    if (error.status >= 500) {
      console.error(`${options.service} is down:`, error.message);
      if (options.fallback) {
        return options.fallback();
      }
    }

    throw error;
  }
}

Retry with exponential backoffWhat is exponential backoff?A retry strategy where each attempt waits twice as long as the previous one, giving an overloaded server progressively more time to recover.

When a request fails due to a transient error (network hiccup, brief rate limit), retrying immediately usually makes things worse. Exponential backoff spaces out your retries so the service has time to recover.

async function withRetry<T>(
  fn: () => Promise<T>,
  maxAttempts = 3,
  baseDelayMs = 1000
): Promise<T> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error: any) {
      const isLastAttempt = attempt === maxAttempts;
      const isRetryable = error.status === 429 || error.status >= 500;

      if (isLastAttempt || !isRetryable) {
        throw error;
      }

      // Exponential backoff with jitter: 1s, 2s, 4s (+ random offset)
      const delay = baseDelayMs * Math.pow(2, attempt - 1) + Math.random() * 500;
      console.log(`Attempt ${attempt} failed. Retrying in ${Math.round(delay)}ms...`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }

  throw new Error('Should not reach here');
}

// Usage
const customer = await withRetry(() =>
  stripe.customers.create({ email: user.email })
);

The random jitter in the delay is intentional. If 1000 clients all hit a rate limit at the same time and retry at exactly the same interval, they'll all hit the limit again simultaneously. Jitter spreads them out.

Circuit breakerWhat is circuit breaker?A pattern that stops sending requests to a failing service after repeated errors, giving it time to recover before trying again. pattern

A circuit breaker monitors failures to an external service. When failures exceed a threshold, it "opens" the circuit and stops sending requests for a period. This prevents a slow or failing service from backing up your request queue and slowing down your whole app.

type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

class CircuitBreaker {
  private failures = 0;
  private state: CircuitState = 'CLOSED';
  private lastFailureTime = 0;
  private readonly threshold: number;
  private readonly resetTimeoutMs: number;

  constructor(threshold = 5, resetTimeoutMs = 60_000) {
    this.threshold = threshold;
    this.resetTimeoutMs = resetTimeoutMs;
  }

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'OPEN') {
      const elapsed = Date.now() - this.lastFailureTime;

      if (elapsed < this.resetTimeoutMs) {
        throw new Error('Circuit breaker is OPEN - service temporarily unavailable');
      }

      // Try one request to see if service has recovered
      this.state = 'HALF_OPEN';
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess() {
    this.failures = 0;
    this.state = 'CLOSED';
  }

  private onFailure() {
    this.failures++;
    this.lastFailureTime = Date.now();

    if (this.failures >= this.threshold) {
      this.state = 'OPEN';
      console.error(`Circuit breaker OPENED after ${this.failures} failures`);
    }
  }
}

// Use one circuit breaker per external service
const stripeBreaker = new CircuitBreaker(5, 30_000);

const customer = await stripeBreaker.call(() =>
  stripe.customers.create({ email: user.email })
);

Secrets management

APIWhat is api?A set of rules that lets one program talk to another, usually over the internet, by sending requests and getting responses. keys are credentials. Treat them with the same care as passwords.

Practice	Status
Store keys in `.env` file	Good for local dev
Add `.env` to `.gitignore`	Required
Hardcode keys in source code	Never
Commit `.env` to git	Never
Use environment variables in production	Required
Use a secrets manager (Doppler, AWS Secrets Manager)	Best for teams

# .env (local development - never commit this file)
STRIPE_SECRET_KEY=sk_test_...
RESEND_API_KEY=re_...
CLERK_SECRET_KEY=sk_test_...

# In code
const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

If you accidentally commitWhat is commit?A permanent snapshot of your staged changes saved in Git's history, identified by a unique hash and accompanied by a message describing what changed. a secret to git, rotate the key immediately, don't try to rewrite git history and hope no one noticed. Keys in git history are compromised.

Avoiding vendor lock-inWhat is vendor lock-in?When your code depends on features unique to one platform, making it expensive or difficult to switch to a different provider.

When you write Resend-specific code throughout your app, switching to SendGrid requires touching dozens of files. The adapter pattern wraps the providerWhat is provider?A wrapper component that makes data available to all components nested inside it without passing props manually. behind a common interface, swap the implementation without touching anything else.

// Define a common interface
interface EmailProvider {
  sendEmail(options: {
    to: string;
    subject: string;
    html: string;
    from?: string;
  }): Promise<{ success: boolean; id?: string }>;
}

// Resend implementation
class ResendEmailProvider implements EmailProvider {
  private client = new Resend(process.env.RESEND_API_KEY!);

  async sendEmail({ to, subject, html, from }) {
    const { data, error } = await this.client.emails.send({
      to,
      subject,
      html,
      from: from ?? 'app@yourdomain.com'
    });

    return { success: !error, id: data?.id };
  }
}

// SendGrid implementation (same interface)
class SendGridEmailProvider implements EmailProvider {
  async sendEmail({ to, subject, html, from }) {
    await sgMail.send({ to, subject, html, from: from ?? 'app@yourdomain.com' });
    return { success: true };
  }
}

// Pick provider from config - swap with one env var change
const emailProvider: EmailProvider =
  process.env.EMAIL_PROVIDER === 'sendgrid'
    ? new SendGridEmailProvider()
    : new ResendEmailProvider();

// Your app code never mentions Resend or SendGrid
await emailProvider.sendEmail({
  to: user.email,
  subject: 'Welcome!',
  html: '<h1>Hello</h1>'
});

Quick reference

Pattern	What it solves	When to use
Try/catch with typed errors	Distinguishes rate limits from auth failures from outages	Always
Exponential backoff	Avoids hammering a rate-limited service	Retryable errors (429, 5xx)
Circuit breaker	Stops a down service from cascading	High-traffic production apps
Adapter pattern	Decouples your code from specific providers	When switching providers is likely
Secrets in env vars	Prevents credential leaks	Always, no exceptions

javascript

// Robust API wrapper with retry and fallback
class RobustAPIClient {
  constructor(config) {
    this.config = config;
    this.retries = 3;
    this.backoff = 1000;
  }

  async call(endpoint, options) {
    for (let i = 0; i < this.retries; i++) {
      try {
        const response = await fetch(
          `${this.config.baseURL}${endpoint}`,
          {
            ...options,
            headers: {
              'Authorization': `Bearer ${this.config.apiKey}`,
              'Content-Type': 'application/json',
              ...options.headers
            }
          }
        );

        if (response.status === 429) {
          // Rate limit, retry with backoff
          const delay = this.backoff * Math.pow(2, i);
          await sleep(delay);
          continue;
        }

        if (!response.ok) {
          throw new Error(`HTTP ${response.status}`);
        }

        return await response.json();

      } catch (error) {
        if (i === this.retries, 1) {
          // Last retry failed
          if (this.config.fallback) {
            return this.config.fallback();
          }
          throw error;
        }
      }
    }
  }
}

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

Create a free account to save your progress

Essential to know

Robust error handling

Retry with exponential backoffWhat is exponential backoff?A retry strategy where each attempt waits twice as long as the previous one, giving an overloaded server progressively more time to recover. Ask AI for more

Circuit breakerWhat is circuit breaker?A pattern that stops sending requests to a failing service after repeated errors, giving it time to recover before trying again. Ask AI for more pattern

Secrets management

Avoiding vendor lock-inWhat is vendor lock-in?When your code depends on features unique to one platform, making it expensive or difficult to switch to a different provider. Ask AI for more

Quick reference

Retry with exponential backoffWhat is exponential backoff?A retry strategy where each attempt waits twice as long as the previous one, giving an overloaded server progressively more time to recover.

Circuit breakerWhat is circuit breaker?A pattern that stops sending requests to a failing service after repeated errors, giving it time to recover before trying again. pattern

Avoiding vendor lock-inWhat is vendor lock-in?When your code depends on features unique to one platform, making it expensive or difficult to switch to a different provider.