NodeJS robots.txt parser with support for wildcard (*) matching.
JavaScript
156
19
MIT License
Total donated
Undistributed
Share with your subscribers:
Recipients
How the donated funds are distributed
Support the dependencies of samclarke/robots-parser
BDD/TDD assertion library for node.js and the browser. Test framework agnostic.
simple, flexible, fun test framework
the Istanbul command line interface
Support the repos that depend on samclarke/robots-parser
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
A set of shared utilities that can be used by crawlers
Distributed web crawler powered by Headless Chrome
Node.js shared library for Franklin bulk operations
Distributed web crawler powered by Headless Chrome
Distributed web crawler powered by Headless Chrome
Check the favicon of a website
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
robots.txt agent with cache.
Distributed web crawler powered by Headless Chrome
A web crawler written in TypeScript.
Site report generator
Fast asynchronous NodeJS module for crawling/scraping a web through worker_threads.
Automated auditing, performance metrics, and best practices for the web. / Lighthouse 7.3.0
Distributed web crawler powered by Headless Chrome
Automated auditing, performance metrics, and best practices for the web.
Structural analysis tools for complex web sites
Simple sitemap generator for remote resources
A node.js library to search, parse and fetch covers from https://www.thecoverproject.net/
A simple web crawler
fork from headless-chrome-crawler and update puppeteer to the latest version
MCP servers exposed as a CLI
Comprehensive website validation
A Model Context Protocol server that provides web content fetching capabilities
Rules related to checking for any SEO issues on the page given
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
Automated auditing, performance metrics, and best practices for the web.
Top contributors
samclarke
77 contributions
![dependabot[bot]'s profile](https://avatars.githubusercontent.com/dependabot[bot])
dependabot[bot]
5 contributions
brendankenny
1 contributions
brendonboshell
1 contributions
danhab99
1 contributions
kdzwinel
1 contributions
SimonC-Audigent
1 contributions
Recent events
Kivach works on the Obyte network, and therefore you can track all donations.
No events yet