crwlrsoft/crawler issues and pull requests

#62 - Improve the Sitemap::getUrlsFromSitemap() step

Pull Request - State: closed - Opened by otsch almost 2 years ago

#61 - Change urlPathMatches filter rule

Pull Request - State: closed - Opened by otsch almost 2 years ago

#60 - Add Filter to filter URL paths by regex

Pull Request - State: closed - Opened by otsch almost 2 years ago

#59 - Improve response cache

Pull Request - State: closed - Opened by otsch almost 2 years ago

#58 - Add default timeouts for the default guzzle client

Pull Request - State: closed - Opened by otsch almost 2 years ago

#57 - Fix retrieving compressed cache items

Pull Request - State: closed - Opened by otsch almost 2 years ago

#56 - Add option to compress cached data to FileCache

Pull Request - State: closed - Opened by otsch almost 2 years ago

#55 - Fail silently when `robots.txt` can't be parsed

Pull Request - State: closed - Opened by otsch almost 2 years ago

#54 - Add test for cookies set via javascript

Pull Request - State: closed - Opened by otsch almost 2 years ago

#53 - New functionality to paginate

Pull Request - State: closed - Opened by otsch almost 2 years ago

#52 - Add JsonFileStore feature

Pull Request - State: closed - Opened by Cyberschorsch almost 2 years ago - 2 comments

#51 - Controll what to do in case of error responses

Pull Request - State: closed - Opened by otsch almost 2 years ago

#50 - Handling 429 - Too Many Requests

Pull Request - State: closed - Opened by otsch almost 2 years ago

#49 - Add feature list to readme and v0.6 changelog

Pull Request - State: closed - Opened by otsch about 2 years ago

#48 - Increase default throttling

Pull Request - State: closed - Opened by otsch about 2 years ago

#47 - New schema.org step

Pull Request - State: closed - Opened by otsch about 2 years ago

#46 - New step to get Metadata from HTML

Pull Request - State: closed - Opened by otsch about 2 years ago

#45 - Check HTML <base> tag when resolving URLs

Pull Request - State: closed - Opened by otsch about 2 years ago

#44 - Improve Politeness features

Pull Request - State: closed - Opened by otsch about 2 years ago

#43 - Add new HTTP::crawl() step

Pull Request - State: closed - Opened by otsch about 2 years ago

#42 - Improve SimpleCsvFileStore for nested results

Pull Request - State: closed - Opened by otsch about 2 years ago

#41 - Add filter methods to DomQuery class

Pull Request - State: closed - Opened by otsch about 2 years ago

#40 - Last v0.5 preparations

Pull Request - State: closed - Opened by otsch about 2 years ago

#39 - Add option to use headless browser with HttpLoader

Pull Request - State: closed - Opened by otsch about 2 years ago

#38 - Improve Composer scripts

Pull Request - State: closed - Opened by szepeviktor about 2 years ago - 1 comment

#37 - Question

Issue - State: closed - Opened by michael-rubel about 2 years ago - 2 comments

#36 - Get absolute links when extracting data

Pull Request - State: closed - Opened by otsch about 2 years ago

#35 - Improve default results from last step outputs

Pull Request - State: closed - Opened by otsch about 2 years ago

#34 - Improve behavior of group's combineToSingleOutput

Pull Request - State: closed - Opened by otsch about 2 years ago

#33 - Fix wrong cookie expires date format issue

Pull Request - State: closed - Opened by otsch over 2 years ago

#32 - Add uniqueInputs functionality

Pull Request - State: closed - Opened by otsch over 2 years ago

#31 - Reduce log messages

Pull Request - State: closed - Opened by otsch over 2 years ago

#30 - Remove second argument from Loop::withInput()

Pull Request - State: closed - Opened by otsch over 2 years ago

#29 - Make Dom::extract() work with single selector and fix behavior when selectors don't match

Pull Request - State: closed - Opened by otsch over 2 years ago

#28 - Remove null from Loader::hooks

Pull Request - State: closed - Opened by szepeviktor over 2 years ago - 1 comment

#27 - Clean up logic of Io's constructor

Pull Request - State: closed - Opened by szepeviktor over 2 years ago - 4 comments

#26 - Fix typehint in GetLinks

Pull Request - State: closed - Opened by szepeviktor over 2 years ago - 1 comment

#25 - Switch order of happy path and unhappy path in GetLink

Pull Request - State: closed - Opened by szepeviktor over 2 years ago - 1 comment

#24 - Replace for loops with array_filter in GetLink

Pull Request - State: closed - Opened by szepeviktor over 2 years ago - 1 comment

#23 - Remove return true/false from booleans

Pull Request - State: closed - Opened by szepeviktor over 2 years ago

#22 - Improve PHPStan config

Pull Request - State: closed - Opened by szepeviktor over 2 years ago

#21 - Change getLink(s) argument to optional

Pull Request - State: closed - Opened by otsch over 2 years ago

#20 - Crawler output hooks

Pull Request - State: closed - Opened by otsch over 2 years ago

#19 - Change all private methods to protected

Pull Request - State: closed - Opened by otsch over 2 years ago

#18 - Limit outputs a step will yield at max

Pull Request - State: closed - Opened by otsch over 2 years ago

#17 - Fix Json step not accepting Http response input

Pull Request - State: closed - Opened by otsch over 2 years ago

#16 - Update logo again

Pull Request - State: closed - Opened by otsch over 2 years ago

#15 - Readme logo

Pull Request - State: closed - Opened by otsch over 2 years ago

#14 - Add logger to the store

Pull Request - State: closed - Opened by otsch over 2 years ago

#13 - Add url step filters

Pull Request - State: closed - Opened by otsch over 2 years ago

#12 - Get links steps domain and host constraints

Pull Request - State: closed - Opened by otsch over 2 years ago

#11 - CSV auto mapping with column headlines

Pull Request - State: closed - Opened by otsch over 2 years ago

#10 - Rename filter() to where() and add orWhere()

Pull Request - State: closed - Opened by otsch over 2 years ago

#9 - Implemented Step filters

Pull Request - State: closed - Opened by otsch over 2 years ago

#8 - Add filter functionality to Csv step

Pull Request - State: closed - Opened by otsch over 2 years ago

#7 - Log memory usage

Pull Request - State: closed - Opened by otsch over 2 years ago

#6 - Fix Generators usage

Pull Request - State: closed - Opened by otsch over 2 years ago

#5 - Change version number in readme link

Pull Request - State: closed - Opened by otsch over 2 years ago

#4 - Implement Group setResultKey and addKeysToResult

Pull Request - State: closed - Opened by otsch over 2 years ago

#3 - Add runAndTraverse method to Crawler

Pull Request - State: closed - Opened by otsch over 2 years ago

#2 - Unique step outputs

Pull Request - State: closed - Opened by otsch over 2 years ago

#1 - Final changes for v0.1 launch

Pull Request - State: closed - Opened by otsch over 2 years ago

GitHub / crwlrsoft/crawler issues and pull requests