simplecrawler/simplecrawler issues and pull requests

#498 - docs: fix discoverResources signature in example

Pull Request - State: open - Opened by bard about 4 years ago

#497 - addDownloadCondition example?

Issue - State: closed - Opened by xeroxstar about 4 years ago - 1 comment

#496 - Adding to the queue on "complete" callback doesn't work

Issue - State: open - Opened by Ivanca about 4 years ago

#495 - crawler.supportedMimeTypes not moving after first page

Issue - State: open - Opened by ankurarora over 4 years ago

#494 - addFetchCondition to get only text/html content type?

Issue - State: closed - Opened by msudol over 4 years ago - 2 comments

#493 - Proxy for each request in the queue?

Issue - State: open - Opened by alex-w0 over 4 years ago - 2 comments

#492 - Generic error missing

Issue - State: open - Opened by Ivanca over 4 years ago - 1 comment

#491 - Bug fix: Remove fragment (to avoid bad 404s)

Pull Request - State: open - Opened by Ivanca over 4 years ago

#490 - Introduced fetch

Pull Request - State: open - Opened by pcdeshmukh over 4 years ago

#489 - Update jsdoc-to-markdown to the latest version 🚀

Pull Request - State: open - Opened by greenkeeper[bot] over 4 years ago
Labels: greenkeeper

#488 - Fail to decode application/x-gzip

Issue - State: open - Opened by FelixRe0 almost 5 years ago

#487 - An in-range update of mocha is breaking the build 🚨

Issue - State: open - Opened by greenkeeper[bot] almost 5 years ago - 2 comments
Labels: greenkeeper

#486 - Fix format in FS cache backend error constructor

Pull Request - State: closed - Opened by kbychkov almost 5 years ago

#485 - Fix format in FS cache backend error constructor (Take 3)

Pull Request - State: closed - Opened by Mr0grog almost 5 years ago - 2 comments

#484 - Fix format in FS cache backend error constructor (take 2)

Pull Request - State: closed - Opened by Mr0grog almost 5 years ago - 1 comment

#483 - Fix string substitution in FS cache backend error constructor

Pull Request - State: closed - Opened by Mr0grog almost 5 years ago - 8 comments

#482 - Update mocha to the latest version 🚀

Pull Request - State: closed - Opened by greenkeeper[bot] about 5 years ago - 2 comments
Labels: greenkeeper

#481 - docs: does this spider support "waiting for all the JS to finish" because actually crawling the resulting page?

Issue - State: open - Opened by Pomax about 5 years ago - 2 comments

#480 - How to await "fetchcomplete"?

Issue - State: open - Opened by Pradyumna-medarametla about 5 years ago - 1 comment

#479 - Which method to use avoid crawling URL's that end with .js /.css /.png/.jpg

Issue - State: closed - Opened by Pradyumna-medarametla about 5 years ago - 1 comment
Labels: Usage Help

#478 - SQLite FetchQueue Implementation for Simplecrawler

Issue - State: open - Opened by LeMoussel over 5 years ago

#477 - An in-range update of eslint is breaking the build 🚨

Issue - State: closed - Opened by greenkeeper[bot] over 5 years ago - 2 comments
Labels: greenkeeper

#476 - Crawler stuck on url 'Exceeded maximum number of redirects'

Issue - State: open - Opened by stijn-lcp over 5 years ago - 1 comment

#475 - Adding fetchcondition to check broken links

Issue - State: closed - Opened by stijn-lcp over 5 years ago - 2 comments

#474 - Request path contains unescaped characters

Issue - State: open - Opened by guidodizi over 5 years ago

#473 - Update iconv-lite to the latest version 🚀

Pull Request - State: closed - Opened by greenkeeper[bot] over 5 years ago
Labels: greenkeeper

#472 - Update eslint to the latest version 🚀

Pull Request - State: closed - Opened by greenkeeper[bot] over 5 years ago - 1 comment
Labels: greenkeeper

#471 - Cannot find site page

Issue - State: closed - Opened by kr-ilya over 5 years ago - 5 comments

#470 - cache: send headers If-None-Match and If-Modified-Since

Pull Request - State: closed - Opened by deton over 5 years ago - 1 comment

#469 - Update async to the latest version 🚀

Pull Request - State: closed - Opened by greenkeeper[bot] over 5 years ago
Labels: greenkeeper

#468 - Update jsdoc-to-markdown to the latest version 🚀

Pull Request - State: closed - Opened by greenkeeper[bot] almost 6 years ago
Labels: greenkeeper

#467 - feature: validate outgoing request headers before sending them

Issue - State: open - Opened by upugo-dev almost 6 years ago

#466 - Greenkeeper/mocha 6.1.1

Pull Request - State: closed - Opened by kbychkov almost 6 years ago

#465 - An in-range update of mocha is breaking the build 🚨

Issue - State: closed - Opened by greenkeeper[bot] almost 6 years ago - 2 comments
Labels: greenkeeper

#464 - chore(package): update lockfile package-lock.json

Pull Request - State: closed - Opened by kbychkov almost 6 years ago

#463 - chore(package): update eslint to version 5.16.0

Pull Request - State: closed - Opened by kbychkov almost 6 years ago

#462 - chore: update travis installation script

Pull Request - State: closed - Opened by kbychkov almost 6 years ago

#461 - test: fix cookie expires

Pull Request - State: closed - Opened by kbychkov almost 6 years ago

#460 - An in-range update of eslint is breaking the build 🚨

Issue - State: closed - Opened by greenkeeper[bot] almost 6 years ago - 2 comments
Labels: greenkeeper

#459 - fix jsdoc2md generated links

Pull Request - State: closed - Opened by kbychkov almost 6 years ago

#458 - Fix docs of events `crawlstart` and `discoverycomplete`

Pull Request - State: closed - Opened by siwinski almost 6 years ago

#457 - [README.md] Re-run `npm run docs`

Pull Request - State: closed - Opened by siwinski almost 6 years ago - 3 comments

#456 - Update dependencies to enable Greenkeeper 🌴

Pull Request - State: closed - Opened by greenkeeper[bot] almost 6 years ago
Labels: greenkeeper

#455 - Update dependencies

Pull Request - State: closed - Opened by kbychkov almost 6 years ago

#454 - Rewrite deprecated use of `Buffer`

Pull Request - State: closed - Opened by kbychkov almost 6 years ago - 2 comments

#453 - chore: update badges

Pull Request - State: closed - Opened by kbychkov almost 6 years ago - 1 comment

#452 - Can't compile app because of simplecrawler

Issue - State: open - Opened by bruceandroid almost 6 years ago - 1 comment

#451 - Fix: handle unwanted path char in chaching file path

Pull Request - State: open - Opened by FranziskaP almost 6 years ago

#450 - fix: get rid of multiple `fetchQueueItem` execution for the same items

Pull Request - State: closed - Opened by kbychkov about 6 years ago

#449 - FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Issue - State: open - Opened by motiejuss about 6 years ago - 2 comments

#448 - How to pause/resume the Crawler?

Issue - State: closed - Opened by bushev about 6 years ago - 1 comment

#447 - Fix TypeError ERR_INVALID_CALLBACK on fs.writeFile for node.js v10

Pull Request - State: closed - Opened by deton about 6 years ago - 6 comments

#446 - Crawler.stop(true) not working as expected

Issue - State: open - Opened by braj1999 about 6 years ago - 2 comments

#445 - Update dependencies

Pull Request - State: closed - Opened by kbychkov about 6 years ago

#444 - Make tests to run independently

Pull Request - State: closed - Opened by kbychkov about 6 years ago - 7 comments

#443 - Oldest unfetched item duplicates

Issue - State: closed - Opened by kbychkov over 6 years ago

#442 - Minor improvements to default discoverRegex

Pull Request - State: closed - Opened by fredrikekelund over 6 years ago

#441 - Pattern for CSS url() syntax matches wrongly also some JS url() function calls

Issue - State: closed - Opened by florianheinze over 6 years ago

#440 - fix repo links

Pull Request - State: closed - Opened by lgraubner over 6 years ago - 1 comment

#439 - Links getting skipped due to escape sequence in href

Issue - State: closed - Opened by braj1999 over 6 years ago - 4 comments

#438 - Crawling aint starting.

Issue - State: open - Opened by BotGenius over 6 years ago - 1 comment

#437 - Does simplecrawler support distributed queuing?

Issue - State: closed - Opened by myrtleTree33 over 6 years ago - 1 comment

#436 - Provision to Validate against Self signed certificate

Issue - State: open - Opened by braj1999 over 6 years ago - 1 comment

#435 - savetodisk: fix require error

Pull Request - State: open - Opened by dsteinel over 6 years ago

#434 - Feature request to support NTLM and Kerberos authentication

Issue - State: open - Opened by braj1999 over 6 years ago - 1 comment

#433 - Uncaught TypeError - invalid input

Issue - State: closed - Opened by selmi-karim over 6 years ago - 7 comments

#432 - Add a replacement in cleanURL

Pull Request - State: closed - Opened by baptistejamin over 6 years ago - 3 comments

#431 - Using Simplecrwaler for Tor pages

Issue - State: open - Opened by zabihimayvan over 6 years ago - 1 comment

#430 - Emit fetchstart event before the request has been initiated

Pull Request - State: closed - Opened by gombosg almost 7 years ago - 2 comments

#429 - fetchStart is unable to modify request

Issue - State: open - Opened by gombosg almost 7 years ago - 3 comments

#428 - HSTS/307 detected as 301

Issue - State: open - Opened by rvizcaino80 almost 7 years ago - 6 comments

#427 - Multiple hosts

Issue - State: closed - Opened by mdalmazzi almost 7 years ago - 5 comments

#426 - Please help to Answer Question on Stackoverflow

Issue - State: closed - Opened by coommark almost 7 years ago - 2 comments

#425 - Checking cookie domain can be problematic

Issue - State: open - Opened by ollieh-m almost 7 years ago

#424 - Add local outgoing IP

Pull Request - State: open - Opened by raunsbaekdk almost 7 years ago

#423 - Update crawler.js

Pull Request - State: open - Opened by Piemontez almost 7 years ago

#422 - Update C:\Users\gsthiuwa\Documents\GitHub\simplecrawler\lib\crawler.j…

Pull Request - State: open - Opened by billyhiuwali almost 7 years ago - 1 comment

#421 - Does this work on relative paths out of the box?

Issue - State: closed - Opened by mhluska almost 7 years ago - 1 comment

#420 - Always fails when proxy=true ?

Issue - State: closed - Opened by ghost almost 7 years ago

#419 - srcset source termination

Issue - State: closed - Opened by PRGfx almost 7 years ago - 1 comment

#418 - Using crawler.addDownloadCondition causes the scraper to complete without downloading anything.

Issue - State: open - Opened by g-a-v-i-n about 7 years ago - 4 comments

#412 - how to use simplecrawler to crawl multiple URLs?

Issue - State: open - Opened by ashamia over 7 years ago - 2 comments

#410 - Update dependencies to enable Greenkeeper 🌴

Pull Request - State: closed - Opened by greenkeeper[bot] over 7 years ago - 3 comments
Labels: greenkeeper

#409 - New documentation approach!

Pull Request - State: closed - Opened by fredrikekelund over 7 years ago

#408 - How to extract canonical URL from HTML Source?

Issue - State: closed - Opened by LeMoussel over 7 years ago - 8 comments

#406 - How to catch/fire network errors?

Issue - State: open - Opened by LeMoussel over 7 years ago - 8 comments

#402 - Basic and X509 Auth, Proxy fixes and Puppeteer support for client side generated content

Pull Request - State: closed - Opened by konstantinblaesi over 7 years ago - 8 comments

#399 - The header content contains invalid characters

Issue - State: open - Opened by nacimgoura over 7 years ago - 11 comments

#394 - Write full documentation with sphinx-js

Issue - State: closed - Opened by fredrikekelund over 7 years ago

#383 - Prettier formatting

Pull Request - State: closed - Opened by fredrikekelund over 7 years ago - 3 comments

#375 - Cache is useless

Issue - State: closed - Opened by Vanuan over 7 years ago

#361 - Look into proxy logic

Issue - State: open - Opened by fredrikekelund almost 8 years ago - 14 comments

#350 - The cacheindex.json are always empty after program exit.

Issue - State: open - Opened by visig9 about 8 years ago - 4 comments

#345 - Async and addFetchCondition / queueadd

Issue - State: closed - Opened by maxcorbeau about 8 years ago - 10 comments
Labels: New Feature

#327 - Request path contains unescaped characters

Issue - State: closed - Opened by ahansson89 over 8 years ago - 3 comments
Labels: Bug, Can't Reproduce

#315 - Crawler#respectRobotsTxt now also looks for nofollow robots meta tags

Pull Request - State: closed - Opened by fredrikekelund over 8 years ago - 8 comments
Labels: New Feature

#292 - Crawler: Enabled stop to forcibly terminate in-flight requests

Pull Request - State: closed - Opened by cgiffard over 8 years ago - 11 comments

#244 - Using simplecrawler with PhantomJS (not integrating!)

Issue - State: open - Opened by moshewe almost 9 years ago - 9 comments
Labels: New Feature

#222 - WIP: Added basic CLI as well as some default output formatters (reporters.)

Pull Request - State: closed - Opened by cgiffard about 9 years ago - 2 comments
Labels: Underway

#183 - Added decodeResponses option

Pull Request - State: closed - Opened by fredrikekelund over 9 years ago - 8 comments

GitHub / simplecrawler/simplecrawler issues and pull requests