benjaminestes/crawl issues and pull requests

#34 - [README] Add binary installation instructions

Pull Request - State: open - Opened by cstrouse over 4 years ago

#33 - Changed to a persistent queue from a per-level queue, added MaxPages configuration option and tweaked final output

Pull Request - State: open - Opened by williamjulianvicary almost 5 years ago

#32 - Change body hashing method from SHA512 to Simhash

Issue - State: open - Opened by cstrouse about 5 years ago - 6 comments
Labels: enhancement

#31 - First request of spider doesn't use configured User-Agent

Issue - State: closed - Opened by cstrouse about 5 years ago - 3 comments

#30 - Make IdleConnTimeout configurable

Pull Request - State: closed - Opened by cstrouse about 5 years ago - 1 comment

#29 - Add version printing command. Closes #18

Pull Request - State: closed - Opened by cstrouse about 5 years ago

#28 - Add schema import instructions using BigQuery CLI

Pull Request - State: closed - Opened by cstrouse about 5 years ago

#27 - Fix error in inlinks sql example's groupby clause

Pull Request - State: closed - Opened by cstrouse about 5 years ago

#26 - Unable to use generated schema in BigQuery due to type problem

Issue - State: closed - Opened by cstrouse about 5 years ago - 4 comments

#25 - Add example config generation command

Issue - State: open - Opened by benjaminestes over 5 years ago
Labels: enhancement

#24 - Sitemap command should download sitemap, not crawl

Issue - State: closed - Opened by benjaminestes over 5 years ago
Labels: bug

#23 - Sitemap crawling should respect robots.txt

Issue - State: open - Opened by benjaminestes over 5 years ago
Labels: bug

#22 - SQL for hreflang report should ensure hreflang attribute has a value

Issue - State: open - Opened by benjaminestes over 5 years ago

#21 - Add basic authentication

Issue - State: closed - Opened by benjaminestes almost 6 years ago - 1 comment

#20 - Update README with clearer installation instructions

Issue - State: open - Opened by benjaminestes almost 6 years ago

#19 - Call out and expand example analysis files

Issue - State: open - Opened by benjaminestes about 6 years ago
Labels: enhancement

#18 - Add version printing command

Issue - State: closed - Opened by benjaminestes about 6 years ago
Labels: enhancement

#17 - Ensure # connections > 0 when initializing a Crawler.

Issue - State: closed - Opened by benjaminestes about 6 years ago
Labels: bug

#16 - Crawl package should be responsible for parsing config files

Issue - State: closed - Opened by benjaminestes about 6 years ago
Labels: enhancement

#15 - Handle Config errors with great messages

Issue - State: open - Opened by benjaminestes about 6 years ago - 3 comments
Labels: enhancement

#14 - Does robots.txt exclusion code maintain different records for http:// and https:// protocols?

Issue - State: closed - Opened by benjaminestes about 6 years ago - 1 comment

#13 - Nicen code for CLI.

Issue - State: closed - Opened by benjaminestes about 6 years ago
Labels: enhancement

#12 - Add BigQuery SQL files to repo

Issue - State: closed - Opened by benjaminestes about 6 years ago
Labels: enhancement

#11 - Update installation instructions

Issue - State: closed - Opened by benjaminestes about 6 years ago
Labels: bug

#10 - Add tests

Issue - State: closed - Opened by benjaminestes about 6 years ago
Labels: bug

#9 - Add semantic versioning

Issue - State: closed - Opened by benjaminestes about 6 years ago
Labels: enhancement