x-crawl

Flexible Node.js AI-assisted crawler library

Making crawler work more efficient, intelligent and convenient

🤖 AI Assistance

Powerful AI assistance function makes crawler work more efficient, intelligent and convenient.

A single crawling API is suitable for multiple configurations, and each configuration method has its own advantages.

Support crawling dynamic pages, static pages, interface data and file data.

Crawling dynamic pages supports automated operations, keyboard input, event operations, etc.

Zero configuration or custom configuration to avoid fingerprint recognition and tracking us from different locations.

Asynchronous or synchronous crawling modes without switching the crawling API.

No interval, fixed interval and random interval, determine whether to crawl with high concurrency.

Customize the number of retries to avoid crawling failures due to transient problems.

With failure retries, custom error times and HTTP status code automatic proxy rotation.

Depending on the priority of a single crawl target, it can be crawled ahead of other targets.

Controllable crawling information, colored string information will be output in the terminal.

Has a type and implements a complete type through generics.