Rate Limits

CrawlKit enforces per-account rate limits based on your subscription tier. Rate limits protect the platform and ensure fair usage across all customers.

Limits by Tier

Rate limits apply per account, not per API key. If you have multiple keys on the same account, they share the same rate limit budget.

Rate Limit Headers

Every API response includes rate limit headers so you can track your usage in real time:

Example Response Headers

HTTP/2 200
content-type: application/json
x-ratelimit-limit: 300
x-ratelimit-remaining: 287
x-ratelimit-reset: 1710345600

When You Hit the Limit

When you exceed your rate limit, the API returns a 429 Too Many Requests response with a Retry-After header indicating how many seconds to wait:

HTTP/2 429
content-type: application/json
retry-after: 23
x-ratelimit-limit: 60
x-ratelimit-remaining: 0
x-ratelimit-reset: 1710345623

{
  "error": "rate_limit_exceeded",
  "message": "Rate limit exceeded. Try again in 23 seconds.",
  "retry_after": 23
}

Retry Strategies

Implement exponential backoff with jitter to handle rate limits gracefully. This avoids thundering herd problems where multiple clients retry simultaneously.

JavaScript / TypeScript

async function fetchWithRetry(url: string, options: RequestInit, maxRetries = 3) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const response = await fetch(url, options);

    if (response.status !== 429) {
      return response;
    }

    if (attempt === maxRetries) {
      throw new Error("Rate limit exceeded after maximum retries");
    }

    // Use Retry-After header, or calculate exponential backoff
    const retryAfter = response.headers.get("Retry-After");
    const baseDelay = retryAfter
      ? parseInt(retryAfter, 10) * 1000
      : Math.pow(2, attempt) * 1000;

    // Add jitter: random delay between 0-1000ms
    const jitter = Math.random() * 1000;
    const delay = baseDelay + jitter;

    console.log(`Rate limited. Retrying in ${Math.round(delay)}ms...`);
    await new Promise(resolve => setTimeout(resolve, delay));
  }
}

Python

import time
import random
import requests

def fetch_with_retry(url, headers, json_data=None, max_retries=3):
    for attempt in range(max_retries + 1):
        response = requests.post(url, headers=headers, json=json_data)

        if response.status_code != 429:
            return response

        if attempt == max_retries:
            raise Exception("Rate limit exceeded after maximum retries")

        # Use Retry-After header, or calculate exponential backoff
        retry_after = response.headers.get("Retry-After")
        if retry_after:
            base_delay = int(retry_after)
        else:
            base_delay = 2 ** attempt

        # Add jitter: random delay between 0-1 seconds
        jitter = random.uniform(0, 1)
        delay = base_delay + jitter

        print(f"Rate limited. Retrying in {delay:.1f}s...")
        time.sleep(delay)

Best Practices

• Respect headers. Always check X-RateLimit-Remaining before making bursts of requests. Proactively slow down when remaining count is low.
• Use exponential backoff. Never retry immediately after a 429. Wait at least the Retry-After duration, then add jitter.
• Batch operations. Use batch endpoints like /gsc/inspect/batch (up to 50 URLs) instead of making individual requests.
• Cache responses. SEO data does not change every second. Cache check results for at least 5 minutes to reduce unnecessary API calls.
• Queue long-running jobs. For crawl jobs and enrichment, submit the job and poll for status rather than blocking on the response.
• Monitor usage. Use GET /api/v1/auth/usage to track your consumption and set up alerts before hitting limits.

Upgrading Your Tier

If you consistently hit rate limits, consider upgrading your subscription:

1. Go to crawlkit.app/settings/billing.
2. Select the tier that matches your usage pattern.
3. Rate limit increases take effect immediately — no restart or key rotation needed.

For enterprise requirements beyond the Agency tier (custom rate limits, dedicated infrastructure, SLA guarantees), contact sales@crawlkit.app.