nietaki/crawlie issues and pull requests

#40 - Misc doc changes

Pull Request - State: open - Opened by kianmeng over 2 years ago - 1 comment

#39 - Rate Limiting

Issue - State: closed - Opened by mikhailbot almost 5 years ago

#38 - Modernize libraries and code to match current versions

Pull Request - State: open - Opened by axelson over 5 years ago - 1 comment

#37 - Not compatible with the latest GenStage

Issue - State: open - Opened by axelson over 5 years ago

#36 - minor edits

Pull Request - State: open - Opened by RichMorin over 5 years ago - 1 comment

#35 - Making crawlie work with GenStage and Flow 0.12.0

Pull Request - State: closed - Opened by nietaki about 7 years ago - 1 comment

#34 - Update README.md

Pull Request - State: open - Opened by loongmxbt about 7 years ago - 1 comment

#33 - Create some sort of CONTRIBUTING.md file

Issue - State: open - Opened by nietaki about 7 years ago

#32 - Adding content-type(s) to Response struct. Closes #28.

Pull Request - State: closed - Opened by nietaki about 7 years ago - 1 comment

#31 - Make it possible to pass some information from the parent page

Issue - State: open - Opened by nietaki about 7 years ago
Labels: enhancement

#30 - Stats tracking

Pull Request - State: closed - Opened by nietaki about 7 years ago - 4 comments

#29 - Allowing skipping pages in ParserLogic.parse

Pull Request - State: closed - Opened by nietaki about 7 years ago - 1 comment

#28 - Add content_type and content_type_simple to the Response struct

Issue - State: closed - Opened by nietaki about 7 years ago
Labels: enhancement

#27 - Allow for ParserLogic.parse to skip a page instead of just parsing

Issue - State: closed - Opened by nietaki about 7 years ago - 1 comment
Labels: enhancement, blocker

#26 - Provide the option of tracking crawling statistics

Issue - State: closed - Opened by nietaki about 7 years ago - 2 comments
Labels: enhancement

#25 - remove duplicate "crawling finished" debug messages

Issue - State: closed - Opened by nietaki about 7 years ago - 1 comment
Labels: bug, blocked

#24 - moving the visited check to when pages are added. Closes #22

Pull Request - State: closed - Opened by nietaki about 7 years ago - 1 comment

#23 - Remove `initial` from the UrlManager State

Issue - State: closed - Opened by nietaki about 7 years ago

#22 - Do not add duplicate uris to the UrlManager State

Issue - State: closed - Opened by nietaki about 7 years ago

#21 - Moving to using URI.t in both Page and the HTTP Client

Pull Request - State: closed - Opened by nietaki about 7 years ago - 1 comment

#20 - Rename `extract_links` to `extract_uris`

Issue - State: closed - Opened by nietaki about 7 years ago

#19 - Move to using URI.t instead of strings for urls

Issue - State: closed - Opened by nietaki about 7 years ago

#18 - Updating GenStage and Flow to 0.11.x. Closes #17

Pull Request - State: closed - Opened by nietaki about 7 years ago - 1 comment

#17 - Update GenStage to 0.11

Issue - State: closed - Opened by nietaki about 7 years ago

#16 - Adding the Crawlie.Response struct

Pull Request - State: closed - Opened by nietaki about 7 years ago - 2 comments

#15 - Make Elixir Syntax Highlighting work

Pull Request - State: closed - Opened by tazsingh about 7 years ago - 2 comments

#14 - Add a simple usage example to the README

Issue - State: open - Opened by nietaki over 7 years ago
Labels: enhancement

#13 - Moving from heap to a priority queue for storing discovered pages.

Pull Request - State: closed - Opened by nietaki over 7 years ago - 1 comment

#12 - Have crawlie operate in the library's supervision tree instead of the caller's

Issue - State: closed - Opened by nietaki over 7 years ago - 2 comments

#11 - Tracking in-flight urls in UrlManager instead of relying on timeouts

Pull Request - State: closed - Opened by nietaki over 7 years ago - 1 comment

#10 - Links with depth over "max_depth" don't get sent to the Manager anymore.

Pull Request - State: closed - Opened by nietaki over 7 years ago - 1 comment

#9 - Replace the heap with a priority queue

Issue - State: closed - Opened by nietaki over 7 years ago
Labels: enhancement

#8 - Tune the Flow parameters

Issue - State: closed - Opened by nietaki over 7 years ago - 1 comment
Labels: enhancement

#7 - Signal completion of the fetches to the UrlManager instead of relying on timeouts to wrap it up.

Issue - State: closed - Opened by nietaki over 7 years ago
Labels: enhancement

#6 - Pass more response data to the parser logic

Issue - State: closed - Opened by nietaki over 7 years ago
Labels: enhancement

#5 - limiting urls to a domain

Issue - State: closed - Opened by nietaki over 7 years ago - 2 comments
Labels: enhancement

#4 - Don't send links that are too deep back to the `UrlManager`

Issue - State: closed - Opened by nietaki over 7 years ago - 3 comments
Labels: enhancement

#3 - Elliminating duplicate urls

Issue - State: closed - Opened by nietaki over 7 years ago
Labels: enhancement

#2 - Fix the `:url_manager_timeout` logic

Issue - State: closed - Opened by nietaki over 7 years ago - 2 comments
Labels: bug

#1 - Merge the option with defaults inside Crawlie.crawl

Issue - State: closed - Opened by nietaki over 7 years ago

GitHub / nietaki/crawlie issues and pull requests