Start crawling and discover pages for training

POST 
/knowledge-bases/crawler

Starts a website crawl that discovers pages for later AI training. WHEN TO USE: use this when you need to scan a site before training; use trainDiscoveredUrls to ingest selected pages; use getCrawlingStatusForLatestOperation to poll progress. RETURNS: the discovery operation id, current status, and the URL being crawled.

Request

Version stringrequired

API Version

Available optionsv3

application/json

application/json

Body

Bodyrequired

locationIdstringrequired

Location ID as string

urlstringrequired

Website URL as string

optionstringrequired
Mode as string
Available optionsExactPathDomain
knowledgeBaseIdstringrequired

knowledge base ID as string

preSelectedUrlsstring[]

Pre-selected URLs from sitemap preview — when provided, only these URLs are crawled

autoTrainboolean

Automatically train the discovered URLs after crawling

Example (auto)

{
  "locationId": "tDtDnQdgm2LXpyiqYvZ6",
  "url": "https://kubernetes.io/tDtDnQdgm2LXpyiqYvZ6",
  "option": "Exact",
  "knowledgeBaseId": "tDtDnQdgm2LXpyiqYvZ6",
  "preSelectedUrls": [
    "https://example.com/page-1",
    "https://example.com/page-2"
  ],
  "autoTrain": false
}

application/json

Crawling and discovery started successfully

application/json

Schema

Schema

operationIdstringrequired

Operation ID for tracking the discovery process

statusstringrequired
Current status of the website discovery operation
Available optionsPendingProcessingSuccessfulFailedExistingRestrictedCancelledAbortedTraining
urlstringrequired

The URL being discovered/crawled

Example (auto)

{
  "operationId": "op_abc123xyz",
  "status": "Processing",
  "url": "https://example.com"
}