🌜
🌞

Best JavaScript Crawler Libraries

@nodelib/fs.walk

@nodelib/fs.walk

v1.2.8

A library for efficiently walking a directory recursively

Last Commit
Jun 11, 2022
GitHub Stargazers
48
License
MIT
Contributors
3
isbot

isbot

v3.5.0

🤖 detect bots/crawlers/spiders via the user agent.

Last Commit
Sep 21, 2022
GitHub Stargazers
514
License
Unlicense
Contributors
22
apify

apify

v3.0.2

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Last Commit
Sep 23, 2022
GitHub Stargazers
6,591
License
Apache-2.0
Contributors
33
cei-crawler

cei-crawler

v3.0.0

Crawler para pegar dados do Canal Eletronico do Investidor

Last Commit
Nov 22, 2021
GitHub Stargazers
275
License
MIT
Contributors
6
fdir

fdir

v5.2.0

The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s

Last Commit
Jun 29, 2022
GitHub Stargazers
1,081
License
MIT
Contributors
7
@opd/crawler

@opd/crawler

v1.6.1

web crawler based on Puppeteer

Last Commit
-
GitHub Stargazers
11
License
MIT
Contributors
1
twitter-crawler

twitter-crawler

v1.0.4

NodeJS Crawler for Twitter

Last Commit
-
GitHub Stargazers
9
License
Apache-2.0
Contributors
5
node-html-crawler

node-html-crawler

v1.2.3

Crawler (spider) of site web pages by domain name

Last Commit
Aug 4, 2021
GitHub Stargazers
30
License
MIT
Contributors
3
nest-crawler

nest-crawler

v1.9.0

An easiest crawling and scraping module for NestJS

Last Commit
-
GitHub Stargazers
30
License
MIT
Contributors
2
es6-crawler-detect

es6-crawler-detect

v3.3.0

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

Last Commit
Jul 23, 2022
GitHub Stargazers
80
License
MIT
Contributors
8
@algolia/netlify-plugin-crawler

@algolia/netlify-plugin-crawler

v1.0.15

This plugin links your Netlify site with Algolia's Crawler. It will trigger a crawl on each successful build.

Last Commit
Jun 27, 2022
GitHub Stargazers
250
License
MIT
Contributors
6
puppeteer-extra-plugin-stealth

puppeteer-extra-plugin-stealth

v2.11.1

Stealth mode: Applies various techniques to make detection of headless puppeteer harder.

Last Commit
Aug 9, 2022
GitHub Stargazers
4,327
License
MIT
Contributors
32
x-ray-crawler

x-ray-crawler

v2.0.5

x-ray's crawler

Last Commit
Apr 9, 2021
GitHub Stargazers
41
License
MIT
Contributors
6