Skip to main content
Fetches a URL through our proxy network and returns the complete HTTP response including body, headers, and timing information. Cost: 1 credit

​
Endpoint

POST /api/v1/crawl/raw

​
Request

​
Headers

HeaderRequiredDescription
AuthorizationYesApiKey ck_... or Bearer <token>
Content-TypeYesapplication/json

​
Body

{
  "url": "https://example.com",
  "options": {
    "timeout": 30000,
    "headers": {},
    "followRedirects": true,
    "maxRedirects": 5
  }
}

​
Parameters

ParameterTypeRequiredDefaultDescription
urlstringYes-The URL to fetch (must include protocol)
options.timeoutnumberNo30000Request timeout in milliseconds (max: 60000)
options.headersobjectNoCustom HTTP headers to send
options.followRedirectsbooleanNotrueWhether to follow HTTP redirects
options.maxRedirectsnumberNo5Maximum redirects to follow (max: 10)

​
Response

​
Success (200)

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "finalUrl": "https://example.com/",
    "statusCode": 200,
    "headers": {
      "content-type": "text/html; charset=UTF-8",
      "content-length": "1256"
    },
    "body": "<!doctype html><html>...</html>",
    "contentLength": 1256,
    "timing": {
      "total": 342
    },
    "creditsUsed": 1,
    "creditsRemaining": 999
  }
}

​
Response Fields

FieldTypeDescription
urlstringThe original URL you requested
finalUrlstringThe final URL after redirects
statusCodenumberHTTP status code (200, 404, etc.)
headersobjectResponse headers from the target
bodystringThe response body (HTML, JSON, etc.)
contentLengthnumberSize of the response in bytes
timing.totalnumberTotal request time in milliseconds
creditsUsednumberCredits charged (always 1)
creditsRemainingnumberYour remaining credit balance

​
Examples

​
Basic Request

curl -X POST https://api.crawlkit.com/api/v1/crawl/raw \
  -H "Authorization: ApiKey ck_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

​
With Custom Headers

curl -X POST https://api.crawlkit.com/api/v1/crawl/raw \
  -H "Authorization: ApiKey ck_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://api.example.com/data",
    "options": {
      "headers": {
        "Accept": "application/json",
        "User-Agent": "MyBot/1.0"
      }
    }
  }'

​
With Increased Timeout

curl -X POST https://api.crawlkit.com/api/v1/crawl/raw \
  -H "Authorization: ApiKey ck_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://slow-website.com",
    "options": {
      "timeout": 60000
    }
  }'

​
Error Responses

​
Invalid URL (400)

{
  "success": false,
  "error": {
    "code": "INVALID_URL",
    "message": "URL must include protocol (http:// or https://)"
  }
}

​
Unauthorized (401)

{
  "success": false,
  "error": {
    "code": "UNAUTHORIZED",
    "message": "Invalid or missing API key"
  }
}

​
Insufficient Credits (402)

{
  "success": false,
  "error": {
    "code": "INSUFFICIENT_CREDITS",
    "message": "You have 0 credits remaining"
  }
}

​
Timeout (408)

{
  "success": false,
  "error": {
    "code": "TIMEOUT",
    "message": "Request timed out after 30000ms"
  }
}

​
Crawl Failed (502)

{
  "success": false,
  "error": {
    "code": "CRAWL_FAILED",
    "message": "Failed to connect to target server"
  }
}

​
Use Cases

Web Scraping

Extract data from any website

API Proxying

Access APIs through our proxy network

Content Monitoring

Track changes on web pages

Data Collection

Gather information at scale