TeamHG-Memex/scrapy-rotating-proxies issues and pull requests

#73 - feat: Authorized-Proxy support + PROXY_LIST_PATH URL support

Pull Request - State: closed - Opened by ruvn-1fgas 24 days ago

#71 - Proxies are only differentiated by domain:port

Issue - State: open - Opened by DJousto 2 months ago

#70 - Use Queue to selec the next proxy

Issue - State: open - Opened by devfox-se 7 months ago

#69 - how to use proxy authentication username and password

Issue - State: open - Opened by zcafetoday almost 2 years ago - 1 comment

#68 - Python gt 3.6

Pull Request - State: closed - Opened by matiskay over 2 years ago

#67 - How to Update the proxy_list.

Issue - State: open - Opened by Kamranbarlas almost 3 years ago

#66 - it keeps checking my proxy servers Dead or not than actually doing its job of scraping data

Issue - State: open - Opened by vijaykurhade almost 3 years ago - 1 comment

#65 - how to ban the proxy with timeout?

Issue - State: open - Opened by leolle about 3 years ago - 1 comment

#64 - Proxies working for longer time slows scraping down.

Issue - State: open - Opened by Kamranbarlas about 3 years ago - 2 comments

#63 - Where is the settings.py? Should I create a new one for me?

Issue - State: open - Opened by alimertkoc about 3 years ago - 1 comment

#62 - Read proxy list from an URL

Pull Request - State: open - Opened by datawookie about 3 years ago - 4 comments

#61 - Is there a way to bypass the proxy check entirely? Everything Slowed

Issue - State: open - Opened by doverradio over 3 years ago

#60 - Please add ability to not assign proxy that is in-use

Issue - State: open - Opened by iamumairayub over 3 years ago - 1 comment

#59 - Proxy-Authorization added

Pull Request - State: open - Opened by dtalkachou over 3 years ago - 8 comments

#58 - AttributeError: Response content isn't text in "response_is_ban"

Issue - State: open - Opened by iamumairayub almost 4 years ago - 1 comment

#57 - Make typing dependency conditional < python 3.5

Pull Request - State: open - Opened by yadalik almost 4 years ago

#56 - RETRY_HTTP_CODES is not taken into account

Issue - State: open - Opened by caffeinatedMike almost 4 years ago

#55 - scrapy with splash and rotating proxies

Issue - State: open - Opened by mianwaqas94 almost 4 years ago - 1 comment

#54 - Crawlera → Zyte Smart Proxy Manager

Pull Request - State: open - Opened by Gallaecio almost 4 years ago - 1 comment

#53 - Update proxy list once a day

Issue - State: open - Opened by moritz-fenske about 4 years ago - 1 comment

#52 - Dependency `typing` breaks `pip install`

Issue - State: open - Opened by StasDeep about 4 years ago - 3 comments

#51 - Shouldn't an IgnoreRequest being raised if max_proxies_to_try is reached

Issue - State: open - Opened by psychonaute about 4 years ago

#50 - This middleware breaks default throttling

Issue - State: open - Opened by 3hhh about 4 years ago

#49 - Using get_random() to select a proxy is not optimal

Issue - State: open - Opened by fredd-427 about 4 years ago

#48 - Banned responses get to the engine in the end

Issue - State: open - Opened by 3hhh about 4 years ago

#47 - Insensible middleware priority recommendations

Issue - State: open - Opened by 3hhh about 4 years ago - 2 comments

#46 - Only rotate proxy when banned or dead

Issue - State: open - Opened by Marow about 4 years ago

#45 - None of the Proxies are checked - leading to perpetual process where scraping never starts

Issue - State: open - Opened by caffeinatedMike over 4 years ago

#44 - stop rotating proxy

Issue - State: open - Opened by MaziarTalebi over 4 years ago

#43 - integrate middleware with standalone Scrapy script

Issue - State: closed - Opened by beanben over 4 years ago - 1 comment

#42 - Clean proxy in logs

Pull Request - State: open - Opened by StasDeep over 4 years ago - 2 comments

#41 - Log proxy stats only if ROTATING_PROXY_LOGSTATS_INTERVAL is non-zero

Pull Request - State: closed - Opened by StasDeep over 4 years ago - 3 comments

#40 - Refresh the list of proxies during scraping

Issue - State: open - Opened by dibodin over 4 years ago - 11 comments

#39 - Possibility to use proxies from DB?

Issue - State: open - Opened by fparareda over 4 years ago - 1 comment

#38 - Add dynamic signaling

Pull Request - State: open - Opened by n1kn4x over 4 years ago - 2 comments

#37 - raise error.ReactorNotRestartable() twisted.internet.error.ReactorNotRestartable

Issue - State: closed - Opened by Arregator over 4 years ago - 1 comment

#36 - Proxies Info stucked so much times

Issue - State: open - Opened by Marow almost 5 years ago

#35 - Add Support for Priority Proxies

Pull Request - State: open - Opened by iamashks almost 5 years ago - 1 comment

#34 - All proxies unchecked except one

Issue - State: open - Opened by milicamilivojevic almost 5 years ago - 4 comments

#33 - Fix the setting of `Proxy-Authorization` header

Issue - State: open - Opened by tianhuil almost 5 years ago

#32 - retrying request after ban does not respect DOWNLOAD_DELAY and autothrottle

Issue - State: open - Opened by kakarukeys about 5 years ago

#31 - What about getting proxy list from url?

Issue - State: open - Opened by aidiss about 5 years ago - 2 comments

#30 - Strip away password and/or username from proxy url before sending them to logs

Issue - State: open - Opened by umrashrf over 5 years ago

#29 - about proxy without scheme(protocol)

Issue - State: open - Opened by searchlyf over 5 years ago - 1 comment

#28 - Problem using proxy with scrapy

Issue - State: open - Opened by Nizarazo over 5 years ago - 1 comment

#27 - Retry Request

Issue - State: open - Opened by kadimon over 5 years ago - 1 comment

#26 - WIP: add BanAwareCachePolicy for http caching

Pull Request - State: open - Opened by Granitosaurus over 5 years ago - 1 comment

#25 - Always return float value for mean_backoff_time

Pull Request - State: closed - Opened by tlinhart over 5 years ago - 2 comments

#24 - fix bug with good proxy stats

Pull Request - State: closed - Opened by Granitosaurus over 5 years ago - 3 comments

#23 - DOWNLOAD_DELAY will not work if use proxy.

Issue - State: open - Opened by GuoYu-Peng almost 6 years ago - 1 comment

#22 - Does it works with https or socks proxy?

Issue - State: closed - Opened by ghostku almost 6 years ago - 1 comment

#21 - add stats tracking to proxy middleware

Pull Request - State: closed - Opened by Granitosaurus almost 6 years ago - 5 comments

#20 - Do not cache failed requests

Issue - State: open - Opened by tlinhart about 6 years ago - 4 comments

#19 - add signal to periodically update scrapy stats with proxy statuses

Pull Request - State: closed - Opened by Granitosaurus over 6 years ago - 2 comments

#18 - Disable interval logging for DefaultSpider

Issue - State: open - Opened by Granitosaurus almost 7 years ago

#17 - Scrapy stuck when page not response

Issue - State: open - Opened by herbert-h almost 7 years ago - 5 comments

#16 - In distributed crawling environment

Issue - State: open - Opened by vijay-axis almost 7 years ago

#15 - Cookiejar handling while crawling the same page after a ban

Issue - State: closed - Opened by mouchh about 7 years ago - 1 comment

#14 - Default middleware priority 610 is too close to downloader for compressed responses.

Issue - State: open - Opened by Granitosaurus about 7 years ago

#13 - Provide load_proxies() hook

Issue - State: open - Opened by Suor about 7 years ago

#12 - Proxies Stuck in unchecked state

Issue - State: open - Opened by john-parton over 7 years ago - 11 comments

#11 - Auth Proxy

Issue - State: open - Opened by bezkos over 7 years ago - 5 comments

#10 - Make backoff cap configurable

Pull Request - State: closed - Opened by petermoore14 over 7 years ago - 2 comments

#9 - Make backoff cap configurable

Issue - State: closed - Opened by petermoore14 over 7 years ago - 2 comments

#8 - Changes needed to correct authentication issue

Pull Request - State: closed - Opened by petermoore14 over 7 years ago - 3 comments

GitHub / TeamHG-Memex/scrapy-rotating-proxies issues and pull requests