crwlrsoft/crawler issues and pull requests

#162 - Make maxOutputs() work with `Group` steps

Pull Request - State: closed - Opened by otsch about 1 month ago

#161 - Cache ttl prolongation

Pull Request - State: closed - Opened by otsch about 1 month ago

#160 - Remove deprecated HttpLoader methods

Pull Request - State: closed - Opened by otsch about 2 months ago

#159 - Remove/change deprecated paginator stuff

Pull Request - State: closed - Opened by otsch about 2 months ago

#158 - Remove addToResult() and multiple loaders

Pull Request - State: closed - Opened by otsch about 2 months ago

#157 - v2.0

Pull Request - State: open - Opened by otsch 2 months ago

#156 - Restrict retrying cached error responses

Pull Request - State: closed - Opened by otsch 2 months ago

#155 - Fix typos

Pull Request - State: closed - Opened by szepeviktor 2 months ago

#154 - New paginator stop rules

Pull Request - State: closed - Opened by otsch 2 months ago

#153 - Add URL refiners

Pull Request - State: closed - Opened by otsch 2 months ago

#152 - Mozilla 5.0 compatible user agent

Pull Request - State: closed - Opened by otsch 2 months ago

#151 - Centralize gzip compression

Pull Request - State: closed - Opened by otsch 2 months ago

#150 - Detect gzip encoding in Http

Pull Request - State: closed - Opened by szepeviktor 2 months ago - 2 comments

#149 - Prevent PHP warnings

Pull Request - State: closed - Opened by otsch 2 months ago

#148 - Apply cache filter also for usage of cached responses

Pull Request - State: closed - Opened by otsch 2 months ago

#147 - Replace browserHelper() with browser()

Pull Request - State: closed - Opened by otsch 3 months ago

#146 - Fix issue with setting chrome executable

Pull Request - State: closed - Opened by otsch 4 months ago

#145 - Also add a getTimeout() function

Pull Request - State: closed - Opened by otsch 4 months ago

#144 - Allow further headless chrome configuration

Pull Request - State: closed - Opened by otsch 4 months ago

#143 - Question regarding "Failed to load % cURL error 60: SSL: no alternative certificate subject name matches target host name"

Issue - State: closed - Opened by severfire 5 months ago - 1 comment

#142 - keep() instead of addToResult() and sub crawlers

Pull Request - State: closed - Opened by otsch 6 months ago

#141 - Fail soft when input key is missing

Pull Request - State: closed - Opened by otsch 7 months ago

#140 - Fix issue in Http::crawl() step

Pull Request - State: closed - Opened by otsch 7 months ago

#139 - JSON step improvement

Pull Request - State: closed - Opened by otsch 7 months ago

#138 - Fix issue with cache filters and redirects

Pull Request - State: closed - Opened by otsch 7 months ago

#137 - Enable updating cached responses via the Loader

Pull Request - State: closed - Opened by otsch 8 months ago

#136 - Enable adding to result from nested output

Pull Request - State: closed - Opened by otsch 8 months ago

#135 - Merge HttpBaseLoader back to HttpLoader

Pull Request - State: closed - Opened by otsch 8 months ago

#134 - Fix issue in Sitemap::getUrlsFromSitemap()

Pull Request - State: closed - Opened by otsch 8 months ago

#133 - Process URLs from sitemap in chunks

Issue - State: closed - Opened by derjochenmeyer 8 months ago - 9 comments

#132 - HttpLoader Separation

Pull Request - State: closed - Opened by otsch 8 months ago

#131 - Documentation request

Issue - State: closed - Opened by derjochenmeyer 8 months ago - 2 comments

#130 - New DomQuery::formattedText() method

Pull Request - State: closed - Opened by otsch 8 months ago

#129 - Enable manipulating nested query params

Pull Request - State: closed - Opened by otsch 9 months ago

#128 - Fix reading uncompressed files

Pull Request - State: closed - Opened by otsch 10 months ago

#127 - Fix paginating with multiple initial inputs

Pull Request - State: closed - Opened by otsch 10 months ago

#126 - Add forgotten getter method

Pull Request - State: closed - Opened by otsch 10 months ago

#125 - Validate CSS Selectors and XPath Queries earlier

Pull Request - State: closed - Opened by otsch 10 months ago

#124 - Symfony 7 support

Pull Request - State: closed - Opened by otsch 10 months ago

#123 - Run tests als in PHP 8.3 in CI

Pull Request - State: closed - Opened by otsch 11 months ago

#122 - Query params paginator and paginator improvements

Pull Request - State: closed - Opened by otsch 11 months ago

#121 - Flexible Auto-Retries for any kind of error responses (4xx, 5xx)

Issue - State: open - Opened by otsch about 1 year ago - 2 comments

#120 - Enable the use of Proxies

Pull Request - State: closed - Opened by otsch about 1 year ago

#119 - Fixes in Http steps

Pull Request - State: closed - Opened by otsch about 1 year ago

#118 - Fix tracking request end for redirected requests

Pull Request - State: closed - Opened by otsch about 1 year ago

#117 - New onCacheHit Loader hook

Pull Request - State: closed - Opened by otsch about 1 year ago

#116 - Move Microseconds class to utils package

Pull Request - State: closed - Opened by otsch about 1 year ago

#115 - Fix throttling when using the headless browser

Pull Request - State: closed - Opened by otsch about 1 year ago

#114 - Improve message composition

Pull Request - State: closed - Opened by szepeviktor about 1 year ago - 10 comments

#113 - Improve variable names

Issue - State: closed - Opened by szepeviktor about 1 year ago - 1 comment

#112 - Ignore special non HTTP links

Pull Request - State: closed - Opened by otsch about 1 year ago

#111 - Fix reading input sitemap in HTTP crawl step

Pull Request - State: closed - Opened by otsch about 1 year ago

#110 - "Microseconds" not found

Issue - State: closed - Opened by flanderboy about 1 year ago - 1 comment

#109 - Fatal error: is not a valid URL

Issue - State: closed - Opened by flanderboy over 1 year ago - 5 comments

#108 - Limit Pagination Crawler

Issue - State: closed - Opened by flanderboy over 1 year ago - 2 comments

#107 - Fix typo in HttpTest

Pull Request - State: closed - Opened by szepeviktor over 1 year ago

#106 - Improve Json step

Pull Request - State: closed - Opened by otsch over 1 year ago

#105 - Fix getting only certain HTML meta data properties

Pull Request - State: closed - Opened by otsch over 1 year ago

#104 - Fix issue with array of arrays in result

Pull Request - State: closed - Opened by otsch over 1 year ago

#103 - sub steps

Issue - State: closed - Opened by TheCrealm over 1 year ago - 4 comments

#102 - Add version 1.1 in CHANGELOG.md

Pull Request - State: closed - Opened by otsch over 1 year ago

#101 - New method HttpLoader::cacheOnlyWhereUrl()

Pull Request - State: closed - Opened by otsch over 1 year ago

#100 - Use new crwlr/utils Json class

Pull Request - State: closed - Opened by otsch over 1 year ago

#99 - Use proxy

Issue - State: closed - Opened by vladsvd over 1 year ago - 7 comments

#98 - Filter by output key aliases

Pull Request - State: closed - Opened by otsch over 1 year ago

#97 - Negate filter functionality

Pull Request - State: closed - Opened by otsch over 1 year ago

#96 - Crawl step can optionally use canonical links

Pull Request - State: closed - Opened by otsch over 1 year ago

#95 - Get HTTP request parameters from input data

Pull Request - State: closed - Opened by otsch over 1 year ago

#94 - Http::crawl() step respect maxOutputs limit

Pull Request - State: closed - Opened by otsch over 1 year ago

#93 - Links/URLs without fragment part

Pull Request - State: closed - Opened by otsch over 1 year ago

#92 - Remove UTF-8 BOM from beginning of strings

Pull Request - State: closed - Opened by otsch over 1 year ago

#91 - Question about submitting form data

Issue - State: closed - Opened by DMGPage over 1 year ago - 6 comments

#90 - Fix duplicate list point with broken link

Pull Request - State: closed - Opened by otsch over 1 year ago

#89 - Paginating links with JavaScript href

Issue - State: closed - Opened by pjdevries over 1 year ago - 2 comments

#88 - Upgrade to pest v2

Pull Request - State: closed - Opened by otsch over 1 year ago

#87 - Add Crawler::runAndDump()

Pull Request - State: closed - Opened by otsch over 1 year ago

#86 - Output key aliases for addToResult()

Pull Request - State: closed - Opened by otsch over 1 year ago

#85 - Fix JSON keys with empty string value

Pull Request - State: closed - Opened by otsch over 1 year ago

#84 - Improve CI in four ways

Pull Request - State: closed - Opened by szepeviktor over 1 year ago

#83 - Remove PHP version displaying steps

Pull Request - State: closed - Opened by szepeviktor over 1 year ago

#82 - Upgrade actions/checkout action

Pull Request - State: closed - Opened by szepeviktor over 1 year ago

#81 - Remove deprecated --no-suggest from CI

Pull Request - State: closed - Opened by szepeviktor over 1 year ago

#80 - Speed up CI

Pull Request - State: closed - Opened by szepeviktor over 1 year ago

#79 - Improve fixing JSON having keys without quotes

Pull Request - State: closed - Opened by otsch over 1 year ago

#78 - Extracting <script> tags

Issue - State: closed - Opened by dimitardimitrov2 over 1 year ago - 2 comments

#77 - PHP 8.0 support

Issue - State: closed - Opened by MathiasReker over 1 year ago - 1 comment

#76 - v1.0

Pull Request - State: closed - Opened by otsch over 1 year ago

#75 - How to process extracted values?

Issue - State: closed - Opened by HelgeSverre over 1 year ago - 6 comments

#74 - Finish v0.7

Pull Request - State: closed - Opened by otsch over 1 year ago

#73 - Keeping input data uses original input

Pull Request - State: closed - Opened by otsch over 1 year ago

#72 - Base Http step also works with array of URLs

Pull Request - State: closed - Opened by otsch over 1 year ago

#71 - Deprecate the loop feature

Pull Request - State: closed - Opened by otsch over 1 year ago

#70 - Fix JSON object keys without quotes

Pull Request - State: closed - Opened by otsch over 1 year ago

#69 - Fix losing result data to add later

Pull Request - State: closed - Opened by otsch over 1 year ago

#68 - Add option to provide chrome executable name

Pull Request - State: closed - Opened by otsch over 1 year ago

#67 - Minor validation methods improvement

Pull Request - State: closed - Opened by otsch almost 2 years ago

#66 - Html/Xml data extraction multiple layers

Pull Request - State: closed - Opened by otsch almost 2 years ago

#65 - Improve adding data to final Result objects

Pull Request - State: closed - Opened by otsch almost 2 years ago

#64 - New outputKey() and keepInputData() step methods

Pull Request - State: closed - Opened by otsch almost 2 years ago

#63 - Add max redirects limit

Pull Request - State: closed - Opened by otsch almost 2 years ago

GitHub / crwlrsoft/crawler issues and pull requests