lapwinglabs/x-ray-crawler

Friendly web crawler for x-ray

JavaScript

No license

x-ray-crawler is a flexible web crawling library designed to work with x-ray, offering features like rate limiting, concurrency control, and request hooks for efficient data extraction. It’s ideal for developers building web scrapers or automated tools to gather and process data from websites. The library supports customizable drivers, delayed requests, and total request limits, making it suitable for both simple and complex crawling tasks.

Total donated

Undistributed

Share with your subscribers:

Recipients

How the donated funds are distributed

Support the dependencies of lapwinglabs/x-ray-crawler

cheeriojs/cheerio

The fast, flexible & elegant library for parsing and manipulating HTML and XML.

debug-js/debug

Lightweight debugging utility for Node.js and the browser

sindresorhus/component-emitter

Event emitter

MatthewMueller/enqueue

queue up function calls

lapwinglabs/http-context

High-level request & response object. Inspired by Koa's context

wilmoore/selectn.js

Curried property accessor function that resolves deeply-nested object properties via dot/bracket-notation string path while mitigating TypeErrors via friendly and composable API.

aheckmann/sliced

A faster Node.js alternative to Array.prototype.slice.call(arguments)

ladjs/superagent

elegant & feature rich browser / node HTTP with a fluent API

lapwinglabs/x-ray-parse

Selector parser

MatthewMueller/yieldly

Turn the function into a thunk if the last argument isn't a function

jestjs/jest

Delightful JavaScript Testing.

Support the dependencies

Support the repos that depend on lapwinglabs/x-ray-crawler

Support the repos that depend on this repository

Top contributors

18 contributions

7 contributions

3 contributions

2 contributions

1 contributions

1 contributions

1 contributions

Recent events

Kivach works on the Obyte network, and therefore you can track all donations.

No events yet

Support the dependencies of lapwinglabs/x-ray-crawler

cheeriojs/cheerio

The fast, flexible & elegant library for parsing and manipulating HTML and XML.

debug-js/debug

Lightweight debugging utility for Node.js and the browser

sindresorhus/component-emitter

Event emitter

MatthewMueller/enqueue

queue up function calls

lapwinglabs/http-context

High-level request & response object. Inspired by Koa's context

wilmoore/selectn.js

Curried property accessor function that resolves deeply-nested object properties via dot/bracket-notation string path while mitigating TypeErrors via friendly and composable API.

aheckmann/sliced

A faster Node.js alternative to Array.prototype.slice.call(arguments)

ladjs/superagent

elegant & feature rich browser / node HTTP with a fluent API

lapwinglabs/x-ray-parse

Selector parser

MatthewMueller/yieldly

Turn the function into a thunk if the last argument isn't a function

jestjs/jest

Delightful JavaScript Testing.