buren/wayback_archiver issues and pull requests

#72 - Ruby 3 fixes and local robots

Pull Request - State: closed - Opened by buren 3 months ago

#71 - Update robots requirement from < 0.10.1 to < 0.10.2

Pull Request - State: open - Opened by dependabot[bot] almost 2 years ago
Labels: dependencies

#70 - Update spidr requirement from ~> 0.6.1 to >= 0.6.1, < 0.8.0

Pull Request - State: open - Opened by dependabot[bot] almost 2 years ago
Labels: dependencies

#69 - fix 429, limit to 5 requests per minute

Pull Request - State: open - Opened by bartman081523 over 2 years ago

#68 - Road to v2

Issue - State: open - Opened by buren over 2 years ago - 7 comments

#67 - Road to v2

Issue - State: closed - Opened by buren over 2 years ago

#66 - Uncaught timeout exception in http.rb

Issue - State: open - Opened by danshearer over 2 years ago - 3 comments

#65 - Update rake requirement from ~> 12.3 to ~> 13.0

Pull Request - State: open - Opened by dependabot[bot] almost 3 years ago
Labels: dependencies

#64 - Update wayback_archiver.gemspec

Pull Request - State: closed - Opened by bartman081523 almost 3 years ago

#51 - cannot load such file -- robots (LoadError) with ruby verison 2.5

Issue - State: closed - Opened by bartman081523 about 3 years ago - 10 comments

#50 - byebug depedency version

Pull Request - State: closed - Opened by bartman081523 over 3 years ago

#49 - WARNING: open-ended dependency on byebug

Issue - State: closed - Opened by bartman081523 over 3 years ago

#48 - rexml dependency version

Pull Request - State: closed - Opened by bartman081523 over 3 years ago - 1 comment

#47 - Invalid gemspec in [wayback_archiver.gemspec]

Issue - State: closed - Opened by bartman081523 over 3 years ago - 1 comment

#46 - Error in running the script

Issue - State: closed - Opened by xplosionmind over 3 years ago - 8 comments

#45 - Added Dependency: rexml

Pull Request - State: closed - Opened by bartman081523 over 3 years ago

#44 - cannot load such file -- rexml/document

Issue - State: closed - Opened by shoeper almost 4 years ago - 2 comments

#43 - Update simplecov requirement from ~> 0.14.1 to ~> 0.16.1

Pull Request - State: open - Opened by dependabot[bot] almost 4 years ago
Labels: dependencies

#42 - Upgrade to GitHub-native Dependabot

Pull Request - State: closed - Opened by dependabot-preview[bot] almost 4 years ago
Labels: dependencies

#41 - Don't respect robots.txt file by default

Pull Request - State: closed - Opened by buren almost 4 years ago

#40 - Add WaybackArchiver::respect_robots_txt

Pull Request - State: closed - Opened by buren almost 4 years ago

#39 - Jekyll integration

Issue - State: closed - Opened by xplosionmind about 4 years ago - 3 comments

#38 - Update Spidr gem constraint

Pull Request - State: closed - Opened by jgarber623 almost 5 years ago - 3 comments

#37 - Consider not respecting robots.txt by default

Issue - State: closed - Opened by buren about 5 years ago - 1 comment

#36 - TravisCI fix and bundler v2.0 upgrade

Pull Request - State: closed - Opened by buren about 5 years ago

#35 - Set default concurrency to 1, due to harsher rate limits

Pull Request - State: closed - Opened by buren about 5 years ago

#34 - Update bundler requirement from ~> 1.3 to ~> 2.1

Pull Request - State: closed - Opened by dependabot-preview[bot] about 5 years ago - 1 comment
Labels: dependencies

#33 - avoid 429, ignore robots

Pull Request - State: closed - Opened by bartman081523 over 5 years ago - 3 comments

#32 - Rate limiting – HTTP 429, Too Many Requests

Issue - State: open - Opened by buren over 5 years ago - 11 comments

#31 - Update rake requirement from ~> 10.3 to ~> 12.3

Pull Request - State: closed - Opened by dependabot-preview[bot] almost 6 years ago
Labels: dependencies

#30 - Update bundler requirement from ~> 1.3 to ~> 2.0

Pull Request - State: closed - Opened by dependabot-preview[bot] almost 6 years ago - 1 comment
Labels: dependencies

#29 - Update simplecov requirement from ~> 0.14.1 to ~> 0.16.1

Pull Request - State: closed - Opened by dependabot-preview[bot] almost 6 years ago - 1 comment
Labels: dependencies

#28 - Fix hosts argument regression

Issue - State: closed - Opened by buren almost 6 years ago
Labels: bug

#27 - Add support for multiple hosts

Pull Request - State: closed - Opened by buren almost 6 years ago

#26 - Collect only OK,(200,302) pages

Pull Request - State: closed - Opened by bartman081523 almost 6 years ago - 2 comments

#25 - `check_path': path conflicts with opaque (URI::InvalidURIError)

Issue - State: closed - Opened by bartman081523 about 6 years ago - 4 comments

#24 - Fix #16 : Pull: Archive every found file

Pull Request - State: closed - Opened by bartman081523 about 6 years ago - 3 comments

#23 - Refactor Request#perform_request error handling

Pull Request - State: closed - Opened by buren over 7 years ago

#22 - Consider retrying on certain HTTP response codes, i.e 429, 502, 503

Issue - State: open - Opened by buren over 7 years ago

#21 - Protect sitemap index duplicates

Pull Request - State: closed - Opened by buren over 7 years ago

#20 - Brakes when crawling hemnet.se sitemap

Issue - State: closed - Opened by buren over 7 years ago

#19 - Adapters

Pull Request - State: closed - Opened by buren over 7 years ago

#18 - Update the Travis CI config to run on Ruby 2.0

Issue - State: closed - Opened by buren over 7 years ago

#17 - Replace usage of String#match? in HttpCode

Issue - State: closed - Opened by buren over 7 years ago

#16 - Configure crawler to send more page types than just html

Issue - State: closed - Opened by buren over 7 years ago

#15 - Figure out what to do when a domain redirects to a subdomain when starting to crawl

Issue - State: closed - Opened by buren over 7 years ago - 1 comment

#14 - Rewrite - CLI, Proper concurrency, added test suite, proper Sitemap

Pull Request - State: closed - Opened by buren over 7 years ago

#13 - Logger

Pull Request - State: closed - Opened by buren over 7 years ago

#12 - Concurrent

Pull Request - State: closed - Opened by buren over 7 years ago

#11 - Use the concurrent-ruby gem

Issue - State: closed - Opened by buren over 7 years ago

#10 - most URLs sent to Wayback have 502 Bad Gateway status

Issue - State: closed - Opened by milliken about 8 years ago - 3 comments
Labels: bug

#9 - Add simple process queue

Pull Request - State: closed - Opened by buren about 8 years ago

#8 - Replace site_mapper with spidr

Pull Request - State: closed - Opened by buren about 8 years ago

#7 - error while collecting URLs during crawl

Issue - State: closed - Opened by milliken about 8 years ago - 6 comments

#6 - Allow URL with path as the start URL for the crawler

Issue - State: closed - Opened by buren over 8 years ago - 1 comment
Labels: enhancement

#5 - Use spidr as the crawler

Issue - State: closed - Opened by buren over 8 years ago - 1 comment
Labels: enhancement

#4 - Add --report flag and functionality

Issue - State: open - Opened by buren over 8 years ago
Labels: enhancement

#3 - Make crawler threaded

Issue - State: closed - Opened by buren over 10 years ago - 1 comment
Labels: feature

#2 - Crawler

Pull Request - State: closed - Opened by buren over 10 years ago

#1 - Follow redirects

Issue - State: closed - Opened by buren over 10 years ago
Labels: enhancement

GitHub / buren/wayback_archiver issues and pull requests