internetarchive/warcprox issues and pull requests

#196 - AttributeError: 'Backend' object has no attribute 'activate_builtin_random'

Issue - State: open - Opened by bjorn3 7 months ago - 4 comments

#195 - v.2.5.1 - Watching Youtube over warcprox not working

Issue - State: closed - Opened by yacylover 8 months ago - 4 comments

#194 - Add SOCKS proxy options

Pull Request - State: closed - Opened by vbanos 9 months ago

#193 - draft: update warcprox benchmarks

Pull Request - State: open - Opened by galgeek 9 months ago

#192 - updates for Python 3.11

Pull Request - State: closed - Opened by galgeek 10 months ago - 8 comments

#191 - Add custom WARC headers

Issue - State: open - Opened by TheTechRobo 10 months ago

#190 - Bump aiohttp from 2.0.7 to 3.8.5 in /benchmarks

Pull Request - State: open - Opened by dependabot[bot] 12 months ago
Labels: dependencies

#189 - Drop idna==2.10 version lock

Pull Request - State: closed - Opened by vbanos 12 months ago

#188 - draft: limit revisits mixin

Pull Request - State: closed - Opened by galgeek about 1 year ago - 2 comments

#187 - Limit dependency version cryptography>=2.3,<=39.0.0

Pull Request - State: closed - Opened by vbanos about 1 year ago

#186 - GitHub Action to run pytest on the qa branch

Pull Request - State: closed - Opened by cclauss about 1 year ago

#185 - qa

Pull Request - State: closed - Opened by galgeek about 1 year ago

#184 - draft: skip duplicate revisits, per ait-job-id

Pull Request - State: closed - Opened by galgeek about 1 year ago - 4 comments

#183 - skip revisits

Pull Request - State: closed - Opened by galgeek about 1 year ago

#182 - GitHub Action to run pytest

Pull Request - State: closed - Opened by cclauss about 1 year ago

#181 - GitHub Action to replace Travis CI

Pull Request - State: closed - Opened by cclauss about 1 year ago - 1 comment

#180 - Delete .travis.yml

Pull Request - State: closed - Opened by cclauss about 1 year ago - 1 comment

#179 - Seperate WARC file for each request

Issue - State: open - Opened by Yakabuff over 1 year ago - 1 comment

#178 - Seperate WARC file for each request

Issue - State: closed - Opened by Yakabuff over 1 year ago

#177 - zlib decompression for warcprox-meta blocks

Pull Request - State: closed - Opened by galgeek almost 2 years ago

#176 - Warning of deprecated Python API

Issue - State: open - Opened by anjackson almost 2 years ago

#175 - Randomize TLS fingerprint

Pull Request - State: closed - Opened by vbanos about 2 years ago - 4 comments

#174 - GitHub Action to lint Python

Pull Request - State: closed - Opened by cclauss about 2 years ago

#173 - tune MIN_BATCH_SEC, MAX_BATCH_SEC

Pull Request - State: closed - Opened by galgeek about 2 years ago - 2 comments

#172 - Adding url canonicalization tests and handling of edge cases to reduc…

Pull Request - State: closed - Opened by adam-miller about 2 years ago

#171 - Adds hop path logging

Pull Request - State: closed - Opened by adam-miller about 2 years ago

#170 - Migrate to GitHub Apps, Since June 15th, 2021, the building on travis-ci.org is ceased

Issue - State: open - Opened by MaxPeal over 2 years ago

#169 - support for a upstream proxy, to make it work behind a Corporate Proxy

Issue - State: open - Opened by MaxPeal over 2 years ago

#168 - Adds hop path logging

Pull Request - State: closed - Opened by adam-miller over 2 years ago

#167 - fix logging buglet iii

Pull Request - State: closed - Opened by galgeek over 2 years ago

#166 - fix logging buglet

Pull Request - State: closed - Opened by galgeek over 2 years ago

#165 - in-batch dedup

Pull Request - State: closed - Opened by galgeek over 2 years ago - 2 comments

#164 - `fcntl` module does not exist on Windows

Issue - State: open - Opened by tari over 2 years ago

#163 - idna==2.10

Pull Request - State: closed - Opened by galgeek about 3 years ago - 1 comment

#162 - Fixes malformed crawl log lines

Pull Request - State: closed - Opened by adam-miller over 3 years ago

#161 - Checking for content type header consiting of only empty spaces and r…

Pull Request - State: closed - Opened by adam-miller over 3 years ago

#160 - Bump aiohttp from 2.0.7 to 3.7.4 in /benchmarks

Pull Request - State: closed - Opened by dependabot[bot] over 3 years ago - 1 comment
Labels: dependencies

#159 - no luck installing by following the instructions (solved)

Issue - State: open - Opened by loveencounterflow over 3 years ago

#158 - set failed_url.timestamp

Pull Request - State: closed - Opened by galgeek almost 4 years ago

#157 - buglet fix for PR 145

Pull Request - State: closed - Opened by galgeek almost 4 years ago - 1 comment

#156 - Avoid `UnicodeDecodeError` in Python 3.5, 3.6

Pull Request - State: closed - Opened by ldko almost 4 years ago - 2 comments

#155 - Expanding logging to handle DNS failures, print error message to craw…

Pull Request - State: closed - Opened by adam-miller almost 4 years ago

#154 - bump version

Pull Request - State: closed - Opened by galgeek almost 4 years ago

#153 - DedupableMixin.should_dedup() improvement

Pull Request - State: closed - Opened by vbanos almost 4 years ago - 1 comment

#152 - Travis CI: Add Python 3.8 to testing

Pull Request - State: closed - Opened by cclauss almost 4 years ago

#151 - Fix runtime error

Pull Request - State: closed - Opened by vbanos almost 4 years ago

#150 - Handle RuntimeError

Pull Request - State: open - Opened by vbanos almost 4 years ago - 1 comment

#149 - Adds logging for failed connections

Pull Request - State: closed - Opened by adam-miller about 4 years ago - 1 comment

#148 - TypeError("connection_from_host() got an unexpected keyword argument 'pool_kwargs'"

Issue - State: closed - Opened by qome over 4 years ago - 2 comments

#147 - tests need trough

Pull Request - State: closed - Opened by nlevitt over 4 years ago

#146 - Add port to custom WARC filename vars

Pull Request - State: closed - Opened by vbanos over 4 years ago

#145 - Adds logging for failed connections

Pull Request - State: closed - Opened by adam-miller over 4 years ago - 2 comments

#144 - change trough dedup `date` type to varchar

Pull Request - State: closed - Opened by nlevitt over 4 years ago

#143 - use trough.client instead of warcprox.trough

Pull Request - State: closed - Opened by nlevitt over 4 years ago

#142 - Another exception when trying to close a WARC file

Pull Request - State: closed - Opened by vbanos almost 5 years ago - 2 comments

#141 - try to fix test failing due to url-encoding

Pull Request - State: closed - Opened by nlevitt almost 5 years ago

#140 - Handle ValueError when trying to close WARC file

Pull Request - State: closed - Opened by vbanos almost 5 years ago - 7 comments

#139 - Skip cdx dedup for volatile URLs with session params

Pull Request - State: closed - Opened by vbanos almost 5 years ago - 1 comment

#138 - Increase remote_connection_pool maxsize

Pull Request - State: closed - Opened by vbanos almost 5 years ago - 5 comments

#137 - avoid clobbering existing warc

Issue - State: open - Opened by traverseda almost 5 years ago - 11 comments

#136 - Optimise WarcWriter.maybe_size_rollover()

Pull Request - State: closed - Opened by vbanos about 5 years ago - 2 comments

#135 - Check if connection is still open when trying to close

Pull Request - State: closed - Opened by vbanos about 5 years ago - 1 comment

#134 - Catch BadStatusLine exception

Pull Request - State: closed - Opened by vbanos about 5 years ago - 2 comments

#133 - handle multiple dedup-buckets, rw or ro (and dedup brozzler test crawls against collection seed)

Pull Request - State: closed - Opened by galgeek about 5 years ago

#132 - Increase IO buffer size to improve WarcWriter performance

Pull Request - State: closed - Opened by vbanos about 5 years ago - 2 comments

#131 - Cache bad target hostname:port to avoid reconnection attempts

Pull Request - State: closed - Opened by vbanos about 5 years ago - 4 comments

#130 - Improve target url validation

Pull Request - State: closed - Opened by vbanos about 5 years ago - 1 comment

#129 - Increase urllib parse cache size

Pull Request - State: closed - Opened by vbanos about 5 years ago - 1 comment

#128 - Compile RecordedUrl regex to improve performance

Pull Request - State: closed - Opened by vbanos about 5 years ago

#127 - Cache digest_str result in memory to improve performance

Pull Request - State: closed - Opened by vbanos about 5 years ago - 2 comments

#126 - Some pages for sites (like scp-wiki.net) have their "From" capture date be before their "To" capture date

Issue - State: closed - Opened by ollie-iterators about 5 years ago

#125 - Too many redirects Error

Issue - State: closed - Opened by ollie-iterators about 5 years ago - 1 comment

#124 - IncompleteRead fix with test

Pull Request - State: closed - Opened by nlevitt about 5 years ago

#123 - Continue request when http.client.IncompleteRead is raised

Pull Request - State: closed - Opened by vbanos about 5 years ago - 1 comment

#122 - avoid exception sending error to client

Pull Request - State: closed - Opened by nlevitt about 5 years ago - 3 comments

#121 - Avoid exception when trying to send error to client

Pull Request - State: closed - Opened by vbanos about 5 years ago - 4 comments

#120 - fixing travis build

Pull Request - State: closed - Opened by nlevitt about 5 years ago

#119 - Increase the MAXHEADERS limit of http client

Pull Request - State: closed - Opened by vbanos about 5 years ago - 3 comments

#118 - account for surt fix in urlcanon 0.3.0

Pull Request - State: closed - Opened by nlevitt over 5 years ago

#117 - travis-ci python 3.7

Pull Request - State: closed - Opened by nlevitt over 5 years ago

#116 - Add option to load logging conf from YAML file

Pull Request - State: closed - Opened by vbanos over 5 years ago - 7 comments

#115 - Any known issues using warcprox with SSLv3?

Issue - State: open - Opened by anjackson over 5 years ago - 5 comments

#114 - Use in-memory LRU cache in CDX Server dedup

Pull Request - State: closed - Opened by vbanos over 5 years ago

#113 - Configurable max threads in CdxServerDedupLoader

Pull Request - State: closed - Opened by vbanos over 5 years ago

#112 - Shows valid fake TLS certificate when the actual site has invalid certificates

Issue - State: open - Opened by kliu128 over 5 years ago - 2 comments

#111 - Thoughts on a custom browser solution for local research

Issue - State: open - Opened by hanoii over 5 years ago - 7 comments

#110 - Help getting started?

Issue - State: open - Opened by hanoii over 5 years ago - 2 comments

#109 - Warc close api

Pull Request - State: closed - Opened by nlevitt over 5 years ago

#108 - T995 dependencies 2.1

Pull Request - State: closed - Opened by kerchner over 5 years ago - 3 comments

#102 - arch diagram

Issue - State: closed - Opened by nlevitt almost 6 years ago - 3 comments

#91 - Max request duration option

Pull Request - State: closed - Opened by vbanos about 6 years ago - 2 comments

#89 - Max request duration limit

Pull Request - State: closed - Opened by vbanos about 6 years ago - 7 comments

#88 - Dedup only urls with Warcprox-Meta and warc-prefix

Pull Request - State: closed - Opened by vbanos about 6 years ago - 5 comments

#79 - Skip CDX dedup check for captures with empty payload

Pull Request - State: closed - Opened by vbanos over 6 years ago

#78 - Increase proxy_client socket send buffer and use TCP_NODELAY

Pull Request - State: closed - Opened by vbanos over 6 years ago

#76 - Fix ListenerPostfetchProcessor typo

Pull Request - State: closed - Opened by vbanos over 6 years ago

#75 - Configurable SpooledTemporaryFile max memory size

Pull Request - State: closed - Opened by vbanos over 6 years ago

#73 - Improve incoming request handling by accepting all requests

Pull Request - State: closed - Opened by vbanos over 6 years ago - 1 comment

#70 - Support client connections using HTTP/1.1

Pull Request - State: closed - Opened by vbanos over 6 years ago - 3 comments

#67 - Use updated list of SSL ciphers

Pull Request - State: closed - Opened by vbanos over 6 years ago - 2 comments

GitHub / internetarchive/warcprox issues and pull requests