bda-research/node-webcrawler

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

JavaScript

MIT License

Total donated

Undistributed

Share with your subscribers:

Recipients

How the donated funds are distributed

Support the dependencies of bda-research/node-webcrawler

cheeriojs/cheerio

The fast, flexible & elegant library for parsing and manipulating HTML and XML.

ashtuchkin/iconv-lite

Convert character encodings in pure javascript.

request/request

Simplified HTTP request client.

chaijs/chai

BDD/TDD assertion library for node.js and the browser. Test framework agnostic.

mochajs/mocha

simple, flexible, fun test framework

sinonjs/sinon

JavaScript test spies, stubs and mocks.

jsdom/jsdom

A JavaScript implementation of many web standards

frankkanis/charset-parser

Parse charset string from http header and hmtl meta

bnoordhuis/node-iconv

Text recoding in JavaScript for fun and profit!

mike442144/seenreq

A library to test if a url(request) is crawled, usually used in a web crawler. Compatible with `request` and `node-crawler`

SGrondin/bottleneck

Distributed task scheduler and rate limiter

Support the dependencies

Support the repos that depend on bda-research/node-webcrawler

wbio/reviews-collector-android

Collect reviews for Android apps from the Google Play Store

Musixmatchdev/musixmatch-node-sdk

Musixmatch SDK for Node.js

aeolingamenfel/lighthouse-crawler

Simple CLI to crawl a website/webapp and perform Lighthouse tests.

wbio/reviews-collector-ios

Collect reviews for iOS apps from the Apple App Store

Support the repos that depend on this repository

Top contributors

paulvalla

135 contributions

63 contributions

44 contributions

8 contributions

5 contributions

5 contributions

3 contributions

2 contributions

2 contributions

2 contributions

Recent events

Kivach works on the Obyte network, and therefore you can track all donations.

No events yet

Support the dependencies of bda-research/node-webcrawler

cheeriojs/cheerio

The fast, flexible & elegant library for parsing and manipulating HTML and XML.

ashtuchkin/iconv-lite

Convert character encodings in pure javascript.

request/request

Simplified HTTP request client.

chaijs/chai

BDD/TDD assertion library for node.js and the browser. Test framework agnostic.

mochajs/mocha

simple, flexible, fun test framework

sinonjs/sinon

JavaScript test spies, stubs and mocks.

jsdom/jsdom

A JavaScript implementation of many web standards

frankkanis/charset-parser

Parse charset string from http header and hmtl meta

bnoordhuis/node-iconv

Text recoding in JavaScript for fun and profit!

mike442144/seenreq

A library to test if a url(request) is crawled, usually used in a web crawler. Compatible with `request` and `node-crawler`

SGrondin/bottleneck

Distributed task scheduler and rate limiter