This is a Node.js library for parsing robots.txt files according to RFC 9309, enabling developers to check if web crawling is allowed for specific URLs and user agents. It's designed for web crawlers, search engines, and any application that needs to respect website crawling rules.
Total donated
Undistributed
Share with your subscribers:
Recipients
How the donated funds are distributed
Support the dependencies of samclarke/robots-parser
BDD/TDD assertion library for node.js and the browser. Test framework agnostic.
simple, flexible, fun test framework
the Istanbul command line interface
Support the repos that depend on this repository
Top contributors
samclarke
77 contributions
![dependabot[bot]'s profile](https://avatars.githubusercontent.com/dependabot[bot])
dependabot[bot]
8 contributions
brendankenny
1 contributions
brendonboshell
1 contributions
danhab99
1 contributions
kdzwinel
1 contributions
SimonC-Audigent
1 contributions
Recent events
Kivach works on the Obyte network, and therefore you can track all donations.
No events yet