TRAWL4 is a lightweight, command‑line Node.js web crawler that recursively follows links across a site, respects robots.txt, and stores each page’s URL and content in a MySQL database for later analysis. It’s built for developers or data‑engineers who need an inexpensive, low‑memory (≈240 MB) solution for site‑wide scraping or spidering without using heavyweight frameworks.
Total donated
Undistributed
Share with your subscribers:
Recipients
How the donated funds are distributed
Support the dependencies of tudorilisoi/trawl4
babel module loader for webpack
The fast, flexible & elegant library for parsing and manipulating HTML and XML.
Lightweight debugging utility for Node.js and the browser
Parse, validate, manipulate, and display dates
The fastest http(s) client (and much more) for Node.js - Node.js bindings for libcurl
NodeJS Object-relational mapping
A library for promises (CommonJS/Promises/A,B,D)
Parser for robots.txt
Add sleep() and usleep() to nodejs
Packs ECMAScript/CommonJs/AMD modules for the browser. Allows you to split your codebase into multiple bundles, which can be loaded on demand. Supports loaders to preprocess files, i.e. json, jsx, es7
yargs the modern, pirate-themed, successor to optimist.
Support the repos that depend on this repository
Top contributors
tudorilisoi
173 contributions
Recent events
Kivach works on the Obyte network, and therefore you can track all donations.
No events yet