Error Reference

The CrawlKit API uses standard HTTP status codes and returns structured JSON error responses. This guide covers the error format, all status codes, error codes, and how to handle errors in your code.

Error Response Format

All error responses follow a consistent JSON structure:

{
  "error": "validation_error",
  "message": "The 'url' field must be a valid URL",
  "details": {
    "field": "url",
    "value": "not-a-url",
    "constraint": "Must start with http:// or https://"
  },
  "request_id": "req_a1b2c3d4e5f6"
}

HTTP Status Codes

Error Codes

The error field in the response body provides a machine-readable error code. Use these for programmatic error handling.

Handling Errors in Code

JavaScript / TypeScript

class CrawlKitError extends Error {
  constructor(
    public status: number,
    public code: string,
    message: string,
    public requestId: string,
    public details?: Record<string, unknown>
  ) {
    super(message);
    this.name = "CrawlKitError";
  }

  get isRetryable(): boolean {
    return [429, 500, 503].includes(this.status);
  }
}

async function crawlkitFetch(path: string, options: RequestInit = {}) {
  const response = await fetch(`https://api.crawlkit.app/api/v1${path}`, {
    ...options,
    headers: {
      "Authorization": `Bearer ${process.env.CRAWLKIT_API_KEY}`,
      "Content-Type": "application/json",
      ...options.headers,
    },
  });

  if (!response.ok) {
    const body = await response.json();
    throw new CrawlKitError(
      response.status,
      body.error,
      body.message,
      body.request_id,
      body.details
    );
  }

  return response.json();
}

// Usage
try {
  const result = await crawlkitFetch("/check-url", {
    method: "POST",
    body: JSON.stringify({ url: "https://crawlkit.app" }),
  });
  console.log(result);
} catch (err) {
  if (err instanceof CrawlKitError) {
    if (err.code === "insufficient_tier") {
      console.error("Upgrade required:", err.message);
    } else if (err.isRetryable) {
      console.error("Transient error, will retry:", err.message);
    } else {
      console.error(`API error [${err.code}]: ${err.message}`);
    }
  }
}

Python

import requests

class CrawlKitError(Exception):
    def __init__(self, status: int, code: str, message: str,
                 request_id: str, details: dict = None):
        super().__init__(message)
        self.status = status
        self.code = code
        self.request_id = request_id
        self.details = details

    @property
    def is_retryable(self) -> bool:
        return self.status in (429, 500, 503)

def crawlkit_fetch(path: str, method: str = "GET", json_data: dict = None):
    url = f"https://api.crawlkit.app/api/v1{path}"
    headers = {
        "Authorization": f"Bearer {os.environ['CRAWLKIT_API_KEY']}",
        "Content-Type": "application/json",
    }

    response = requests.request(method, url, headers=headers, json=json_data)

    if not response.ok:
        body = response.json()
        raise CrawlKitError(
            status=response.status_code,
            code=body.get("error", "unknown"),
            message=body.get("message", "Unknown error"),
            request_id=body.get("request_id", ""),
            details=body.get("details"),
        )

    return response.json()

# Usage
try:
    result = crawlkit_fetch("/check-url", "POST", {"url": "https://crawlkit.app"})
    print(result)
except CrawlKitError as err:
    if err.code == "insufficient_tier":
        print(f"Upgrade required: {err}")
    elif err.is_retryable:
        print(f"Transient error, will retry: {err}")
    else:
        print(f"API error [{err.code}]: {err}")

Getting Help

If you encounter persistent errors or unexpected behavior:

• Include the request_id from the error response when contacting support.
• Check the status page for any ongoing incidents.
• Review the API Reference for the specific endpoint's expected request format.
• Email support@crawlkit.app with your request ID, the endpoint, and the full error response.