NodeJS robots.txt parser with support for wildcard (*) matching.
JavaScript
159
19
MIT License
Total donated
Undistributed
Share with your subscribers:
Recipients
How the donated funds are distributed
Support the dependencies of samclarke/robots-parser
BDD/TDD assertion library for node.js and the browser. Test framework agnostic.
simple, flexible, fun test framework
the Istanbul command line interface
Support the repos that depend on samclarke/robots-parser
🍢 Skewer web data perfectly - Smart Indonesian web crawler library
A Model Context Protocol server that provides web content fetching capabilities
Fast, token-efficient web content extraction - fetch web pages and convert to clean Markdown
Web crawler MCP server for extracting text content from web pages
Web crawler MCP server for extracting text content from web pages
CLI tool to convert websites into RAG-ready local markdown knowledge bases with AI-powered content improvement.
A Model Context Protocol server that provides web content fetching capabilities
MCP server for fetch deepwiki.com and turn content into LLM readable markdown
MCP servers exposed as a CLI
A Model Context Protocol server that provides web content fetching capabilities
A Model Context Protocol server that provides web content fetching capabilities
A Model Context Protocol server that provides web content fetching capabilities with automatic image saving and optional AI display
Automated auditing, performance metrics, and best practices for the web.
Check the favicon of a website
A node.js library to search, parse and fetch covers from https://www.thecoverproject.net/
Comprehensive website validation
Fast asynchronous NodeJS module for crawling/scraping a web through worker_threads.
Node.js shared library for Franklin bulk operations
Site report generator
Web crawling tool
A small Web scraper that publishes its data into data-fair datasets.
Structural analysis tools for complex web sites
Top contributors
samclarke
77 contributions
![dependabot[bot]'s profile](https://avatars.githubusercontent.com/dependabot[bot])
dependabot[bot]
5 contributions
brendankenny
1 contributions
brendonboshell
1 contributions
danhab99
1 contributions
kdzwinel
1 contributions
SimonC-Audigent
1 contributions
Recent events
Kivach works on the Obyte network, and therefore you can track all donations.
No events yet