cgiffard/node-simplecrawler

Higher-order functions and common patterns for asynchronous code

Convert character encodings in pure javascript.

A specification compliant robots.txt parser with wildcard (*) matching support.

URI.js is a Javascript library for working with URLs.

BDD/TDD assertion library for node.js and the browser. Test framework agnostic.

Generates markdown API documentation from jsdoc annotated source code

simple, flexible, fun test framework

Easily create XML sitemaps for your website.

Ingest utilities

sitespeed.io is an open-source tool for comprehensive web performance analysis, enabling you to test, monitor, and optimize your website’s speed using real browsers in various environments.

Behaviour Assertion Sheets: CSS-like declarative syntax for client-side integration testing and quality assurance.

A library for checking basic SEO signals of a website

Implements https://github.com/simplecrawler to get all html links within the domain

Specify the sitemap index and download the xml information locally or on a node.js server.

Ingest utilities

Easily create XML sitemaps for your website.

Follows and collects breadcrumbs accross the web

Website crawler and differencer

@vokus/crawler

Analyze the web performance of your site

Spider test suite application for crawling and validating your website

SQLite FetchQueue Implementation for Simplecrawler

A library for checking basic SEO signals of a website

Easily create XML sitemaps for your website.

Automate full website screenshots and PDF generation with multiple viewports

Simple sitemap generator for remote resources

A CLI tool for validating pages while crawling a website.

Analyze the web performance of your site

Crawl a site to generate a backstopjs config

Creates a snapshot of a url with HTML suffix removed in favour of directories

Easily create XML sitemaps for your website with url-parse 1.5.10 based on sitemap generator

Crawls a given site and checks for W3C validity.

A command line tool that takes a URL or a file, then uses html-validator (a wrapper for https://validator.w3.org/nu/) to validate each page.

Crawl 100% JS single page apps with phantomjs and node.

Link manipulation tool

Specify the sitemap index and download the xml information locally or on a node.js server.