Web Crawler

Documentation for Knowledge Base API

📄️Preview Sitemap URLs

Returns a paginated, searchable list of sitemap URLs without starting a crawl.

📄️Start crawling and discover pages for training

Starts a website crawl that discovers pages for later AI training.

📄️Get all trained page links by knowledge base

Lists discovered and trained website pages for a knowledge base, with pagination and optional URL filter.

📄️Delete trained pages

Removes trained website pages from a knowledge base so the AI no longer references them.

📄️Get crawling status for the latest operation

Fetches crawl/train progress for a specific operation, or the latest operation when operationId is omitted.

📄️Train discovered website pages and ingest into the knowledge base

Queues selected discovered pages so the AI ingests their content into the knowledge base.