internetarchive/Zeno issues and pull requests

#132 - Panic on /workers access

Issue - State: open - Opened by CorentinB about 1 month ago
Labels: bug, P1

#131 - Verify & test our XML extraction in the context of sitemaps

Issue - State: open - Opened by CorentinB about 1 month ago
Labels: enhancement, P3

#130 - fix: ensure there is no infinite recursion of URLs

Pull Request - State: closed - Opened by NGTmeaty about 1 month ago

#129 - Add additional tests and validate current URL behavior

Pull Request - State: closed - Opened by NGTmeaty about 1 month ago
Labels: enhancement

#128 - Fix the handover bypass when seeds are loaded from list and handover is disabled

Pull Request - State: closed - Opened by equals215 about 1 month ago
Labels: bug, P1

#127 - Add Pyroscope profiling support

Pull Request - State: closed - Opened by NGTmeaty about 1 month ago
Labels: enhancement

#126 - Add proper YouTube archiving via YT-DLP

Pull Request - State: open - Opened by CorentinB about 1 month ago
Labels: enhancement, internal-only, P3

#125 - Correct seedList enqueuing process

Pull Request - State: closed - Opened by equals215 about 1 month ago
Labels: enhancement, P2

#124 - Ingest seeds before starting workers

Pull Request - State: closed - Opened by CorentinB about 1 month ago
Labels: bug

#123 - Send on closed channel panic

Issue - State: open - Opened by CorentinB about 2 months ago - 2 comments
Labels: bug, P1

#122 - Split Zeno in smaller packages with a better structure

Pull Request - State: open - Opened by equals215 about 2 months ago - 3 comments
Labels: enhancement, internal-only, P1

#121 - Queue all items from seeds list before starting to crawl

Issue - State: closed - Opened by CorentinB about 2 months ago
Labels: enhancement, P3

#120 - Calming down queue stats

Pull Request - State: closed - Opened by yzqzss about 2 months ago

#119 - Change default queue behaviour

Pull Request - State: closed - Opened by equals215 about 2 months ago
Labels: enhancement, internal-only, P1

#118 - WAL tests fail

Issue - State: closed - Opened by CorentinB about 2 months ago - 3 comments
Labels: bug, P1

#117 - Enable `log` package to distribute a stored logger to all other packages

Pull Request - State: closed - Opened by equals215 about 2 months ago - 3 comments
Labels: bug, enhancement, P1

#116 - Lock-free AwaitWALCommitted (and smoother queue?)

Pull Request - State: closed - Opened by yzqzss about 2 months ago

#115 - Define Zeno's queuing behavior properly

Issue - State: closed - Opened by CorentinB about 2 months ago - 2 comments
Labels: enhancement, internal-only, P1

#114 - Fix idna.ToASCII fail on punycode encoded URLs with port

Pull Request - State: closed - Opened by CorentinB about 2 months ago
Labels: bug

#113 - Transform Zeno architecture to a crawling pipeline effectively making use of Go channels

Issue - State: open - Opened by equals215 about 2 months ago
Labels: enhancement, internal-only, P2

#112 - Break down Zeno in smaller packages, especially `crawl` package which has grown too big

Issue - State: open - Opened by equals215 about 2 months ago
Labels: enhancement, internal-only, P1

#111 - Reuse free space from popped items

Pull Request - State: open - Opened by equals215 about 2 months ago - 3 comments
Labels: enhancement, internal-only

#110 - fix: hanging on indexManager.Close()

Pull Request - State: closed - Opened by yzqzss about 2 months ago - 2 comments

#109 - Improve WAL concurrency performance by @yzqzss and make it optional

Pull Request - State: closed - Opened by equals215 about 2 months ago - 13 comments

#108 - SIGSEGV logging in BatchEnqueue

Issue - State: closed - Opened by CorentinB about 2 months ago - 3 comments
Labels: bug, P1

#107 - Using group commit to improve WAL concurrency performance

Pull Request - State: closed - Opened by yzqzss about 2 months ago - 1 comment
Labels: enhancement

#106 - Queue handover v2

Pull Request - State: closed - Opened by equals215 about 2 months ago
Labels: enhancement, internal-only

#105 - Remove `runtime.Gosched()` in polling

Pull Request - State: closed - Opened by yzqzss about 2 months ago - 1 comment
Labels: enhancement

#104 - Optimize `get list` loading performance

Pull Request - State: closed - Opened by yzqzss about 2 months ago
Labels: enhancement

#103 - --exclude-host not found

Issue - State: closed - Opened by CorentinB about 2 months ago
Labels: bug

#102 - Fix readItemsFromQueue() CPU 100%, fix various data races, fix hang when all items are deduplicated

Pull Request - State: closed - Opened by yzqzss about 2 months ago - 5 comments
Labels: bug

#101 - Have `queue.Enqueue()` handover items to idle workers and optimize workers routines

Pull Request - State: closed - Opened by equals215 about 2 months ago - 5 comments
Labels: enhancement, internal-only

#99 - Implement host rotation and Enqueue/Dequeue access regulation via atomic booleans

Pull Request - State: closed - Opened by equals215 about 2 months ago - 2 comments
Labels: enhancement, internal-only

#98 - Add dequeue enqueue stats

Pull Request - State: open - Opened by CorentinB about 2 months ago - 2 comments

#97 - create url_string_test.go

Pull Request - State: open - Opened by willmhowes about 2 months ago - 2 comments
Labels: enhancement

#96 - Restore HQ flags

Issue - State: closed - Opened by CorentinB about 2 months ago
Labels: bug

#95 - Implement linkheader parsing

Pull Request - State: closed - Opened by HarshNarayanJha about 2 months ago - 5 comments
Labels: enhancement

#94 - Queue and Index should reuse free space

Issue - State: open - Opened by equals215 about 2 months ago
Labels: enhancement, internal-only, P2

#93 - Persist & load queue stats

Pull Request - State: closed - Opened by CorentinB about 2 months ago
Labels: enhancement, internal-only

#92 - Add logging capabilities for queue (index too) using custom `log` package

Issue - State: closed - Opened by equals215 about 2 months ago - 1 comment
Labels: enhancement, internal-only, P1

#91 - Fix commit hash in User Agent can't be calculated when not present

Pull Request - State: closed - Opened by CorentinB about 2 months ago
Labels: bug

#90 - Instantiate a `CODE_OF_CONDUCT.md` as the repo drags some traction

Issue - State: open - Opened by equals215 about 2 months ago
Labels: documentation, internal-only, P3

#89 - Panic when starting Zeno with go run

Issue - State: closed - Opened by CorentinB about 2 months ago
Labels: bug

#88 - Change log location

Pull Request - State: closed - Opened by nick2432 2 months ago - 7 comments
Labels: bug

#87 - Extract URLs from ebook formats (EPUB, MOBI..)

Issue - State: open - Opened by CorentinB 2 months ago
Labels: enhancement, P4

#86 - Extract URLs from images

Issue - State: open - Opened by CorentinB 2 months ago - 2 comments
Labels: enhancement, P4

#85 - Replace github.com/tomnomnom/linkheader with stdlib

Issue - State: closed - Opened by CorentinB 2 months ago - 10 comments
Labels: enhancement, good first issue

#84 - Replace github.com/clbanning/mxj/v2 with stdlib

Issue - State: open - Opened by CorentinB 2 months ago
Labels: enhancement

#83 - Revamp index mechanism with a WAL

Pull Request - State: closed - Opened by equals215 2 months ago - 4 comments
Labels: enhancement, internal-only

#82 - pprof API expose silently fail when port is used

Issue - State: open - Opened by CorentinB 2 months ago
Labels: bug, good first issue

#81 - Replicate the active workers count for `--live-stats` on the new version of workers

Issue - State: closed - Opened by equals215 2 months ago - 1 comment
Labels: bug, enhancement, internal-only

#80 - Fix workers not stopping properly and temp fix for workers hanging unexpectedly

Pull Request - State: closed - Opened by equals215 2 months ago - 1 comment
Labels: bug

#79 - Fix crash when --api is not set

Pull Request - State: closed - Opened by NGTmeaty 2 months ago - 1 comment
Labels: bug

#78 - Rewriting the queue

Pull Request - State: closed - Opened by CorentinB 2 months ago - 11 comments
Labels: enhancement, internal-only

#77 - Fix live-stats

Pull Request - State: closed - Opened by CorentinB 2 months ago

#76 - Add deduping stats

Pull Request - State: closed - Opened by CorentinB 2 months ago
Labels: enhancement

#75 - Remove Gin dependency

Pull Request - State: closed - Opened by CorentinB 2 months ago - 1 comment
Labels: enhancement

#74 - Add basic UI to manage Zeno

Issue - State: open - Opened by CorentinB 2 months ago
Labels: enhancement

#73 - Logs are written to the wrong location

Issue - State: closed - Opened by willmhowes 2 months ago - 4 comments
Labels: bug

#72 - live-stats flag is broken

Issue - State: closed - Opened by willmhowes 2 months ago - 1 comment
Labels: bug

#71 - fix: logfile declaration in log.go:New()

Pull Request - State: closed - Opened by equals215 3 months ago - 2 comments

#70 - Log rotation cause panic / SIGSEGV

Issue - State: closed - Opened by CorentinB 3 months ago - 2 comments
Labels: bug

#69 - Disk space pause is not working

Issue - State: open - Opened by NGTmeaty 3 months ago - 3 comments
Labels: bug

#68 - Add site-specific code for Facebook

Pull Request - State: closed - Opened by CorentinB 3 months ago - 1 comment

#67 - Rewrite of cmd+config packages using `spf13` `cobra` and `viper`

Pull Request - State: closed - Opened by equals215 3 months ago - 2 comments
Labels: enhancement

#66 - Give Zeno the logging it deserves

Pull Request - State: closed - Opened by equals215 3 months ago - 3 comments
Labels: enhancement

#65 - Add state management for workers

Pull Request - State: closed - Opened by equals215 3 months ago
Labels: enhancement

#64 - Add site-specific support for Streamable

Pull Request - State: closed - Opened by CorentinB 3 months ago

#63 - Add state management for workers

Pull Request - State: closed - Opened by equals215 3 months ago - 1 comment
Labels: enhancement

#62 - Allow crawl space threshold to be set on CLI, report space avail

Pull Request - State: closed - Opened by machawk1 4 months ago - 2 comments

#61 - Allow free space threshold to be customizable

Issue - State: closed - Opened by machawk1 4 months ago

#60 - Enhance JSON extraction & add XML extraction

Pull Request - State: closed - Opened by CorentinB 6 months ago

#59 - Add support to send a cookie with CDX requests

Pull Request - State: closed - Opened by NGTmeaty 8 months ago
Labels: enhancement

#58 - Small fixes to fix memory usage

Pull Request - State: closed - Opened by CorentinB 8 months ago

#57 - Properly save sync.map version of Frontier

Pull Request - State: closed - Opened by NGTmeaty 11 months ago
Labels: bug

#56 - Add: Telegram support

Pull Request - State: closed - Opened by CorentinB 12 months ago

#55 - Add headless / headfull capabilities to Zeno

Pull Request - State: open - Opened by CorentinB about 1 year ago
Labels: enhancement

#54 - Fix improperly escaped URLs

Pull Request - State: closed - Opened by NGTmeaty about 1 year ago
Labels: enhancement

#53 - Fix potential blocking

Pull Request - State: closed - Opened by CorentinB about 1 year ago

#52 - Fix ElasticSearch

Pull Request - State: closed - Opened by NGTmeaty about 1 year ago
Labels: bug

#51 - Limit total current requests per domain

Pull Request - State: closed - Opened by NGTmeaty about 1 year ago

#50 - Add --exclude-string & daily ES indexes

Pull Request - State: closed - Opened by CorentinB over 1 year ago

#49 - Improve logging

Pull Request - State: closed - Opened by NGTmeaty over 1 year ago

#48 - Add ElasticSearch logging

Pull Request - State: closed - Opened by CorentinB over 1 year ago

#47 - WARC update

Pull Request - State: closed - Opened by NGTmeaty over 1 year ago

#46 - Various bug fixes

Pull Request - State: closed - Opened by NGTmeaty over 1 year ago

#45 - User-Agent improvements

Pull Request - State: closed - Opened by NGTmeaty over 1 year ago

#44 - Add cloudflarestream.com support as a plugin

Pull Request - State: closed - Opened by CorentinB over 1 year ago

#43 - AWS and mismatch for User-Agent Zeno

Issue - State: closed - Opened by tvb almost 2 years ago - 2 comments

#42 - No such file or directory panic

Issue - State: closed - Opened by CorentinB almost 2 years ago

#41 - Make WARC temp dir configurable via --warc-temp-dir

Pull Request - State: closed - Opened by CorentinB about 2 years ago

#40 - Update WARC and asset extraction improvements

Pull Request - State: closed - Opened by NGTmeaty about 2 years ago
Labels: enhancement

#39 - Code cleanup and asset exclusion based on previous settings

Pull Request - State: closed - Opened by NGTmeaty about 2 years ago
Labels: enhancement

#38 - Add: seencheck when executing redirects

Pull Request - State: closed - Opened by CorentinB about 2 years ago

#37 - feat: update warc to 0.8.21

Pull Request - State: closed - Opened by NGTmeaty about 2 years ago

#36 - Various fixes

Pull Request - State: closed - Opened by NGTmeaty about 2 years ago - 1 comment
Labels: enhancement

#35 - Investigate mailto: links

Issue - State: closed - Opened by NGTmeaty about 2 years ago - 1 comment

#34 - More efficient handling of text/ payloads if we aren't trying to scrape them

Pull Request - State: closed - Opened by NGTmeaty about 2 years ago

#33 - Add: crawl rate limiter

Pull Request - State: closed - Opened by CorentinB about 2 years ago

#32 - Add setting for warc pool size and update to warc 0.8.20

Pull Request - State: closed - Opened by NGTmeaty about 2 years ago
Labels: enhancement

GitHub / internetarchive/Zeno issues and pull requests