XCrawl is a web scraping and crawling API for developers who need page content in formats they can use in applications or research workflows. It can fetch an individual URL, crawl a collection of pages, map a site's URLs, or search across supported engines. Instead of handling every page as an undifferentiated HTML document, a request can ask for Markdown, extracted links, a screenshot, or structured JSON. Teams choosing it should consider whether they need a single-page response, a controlled site crawl, or an asynchronous job with a callback.
What can XCrawl do?
For individual pages, XCrawl supports synchronous requests that return results in the response and asynchronous requests that can send webhook callbacks. Output choices include Markdown, cleaned HTML, raw HTML, links, summaries, screenshots, and structured JSON. Prompted JSON extraction can use an optional JSON Schema when the intended fields need a defined structure. JavaScript rendering is enabled by default, with controls for when rendering waits and for the browser viewport. The main-content setting is on by default, but it can be changed when material outside the principal content area matters.
For site-wide collection, crawling supports a page limit, depth, URL inclusion and exclusion patterns, and sitemap use. A separate mapping capability lists URLs within a site, while search works across supported engines. Proxy country and sticky-session settings are available for scrape requests. These controls let a developer specify which pages to collect, what representation to return, and whether to receive the result in the original request or later through a callback.
Who is XCrawl best for?
XCrawl fits developers building workflows that need web pages as text, HTML, links, screenshots, or extracted fields. A team collecting content from a known set of URLs can use single-page scraping; one exploring a site can set boundaries for a crawl or map its URLs first. It also suits projects that need browser rendering or asynchronous retrieval rather than a simple fetch. The API requires a key, so it is a choice for people willing to integrate requests and handle returned data.
XCrawl pricing in brief
XCrawl offers a free trial with a one-time credit allowance, followed by paid tiers with recurring credit allocations. The paid tiers differ in their included credits, concurrent-request caps, and per-credit excess fees. Choosing a tier therefore depends on the volume of requests and the number that need to run at once. Credit usage deserves checking against the intended operations rather than assuming that every operation has the same cost.
Limitations to know about XCrawl
The API requires a key, and the free allowance is a time-limited trial rather than an ongoing monthly allocation. Paid tiers have concurrent-request caps and charge excess fees per credit. Crawl requests start with a default page ceiling and depth limit unless you configure them, so a broad site crawl needs deliberate settings. Scrape requests return main content by default; Markdown output should not be mistaken for a guaranteed copy of every visible element on a page. The available facts do not establish extraction success rates or a complete credit-cost table for every operation.
Alternatives to XCrawl
If the goal is to answer research questions rather than collect page outputs through an API, Perplexity AI searches the web and returns cited answers. For developers assembling their own retrieval workflows, LangChain provides document-loading and retrieval components. XCrawl returns page data, while these options focus on cited answers or components for building a retrieval application. The choice depends on which of those outputs the project needs.
