Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / guillaume-be/rust-tokenizers issues and pull requests

#102 - Port of 'rust-tokenizer' to C#/.NET

Issue - State: open - Opened by vermorel 8 months ago

#101 - Add lock files to .gitignore

Pull Request - State: closed - Opened by guillaume-be about 1 year ago

#100 - Bump webpki from 0.22.0 to 0.22.2 in /main

Pull Request - State: closed - Opened by dependabot[bot] about 1 year ago - 1 comment
Labels: dependencies

#99 - Bump openssl from 0.10.48 to 0.10.57 in /main

Pull Request - State: closed - Opened by dependabot[bot] about 1 year ago - 1 comment
Labels: dependencies

#98 - Fix RoBERTa segment ids

Pull Request - State: closed - Opened by guillaume-be about 1 year ago

#97 - Upgrade readme to reflect new tokenizers API

Pull Request - State: closed - Opened by ToluClassics about 1 year ago

#96 - Slight Error in Readme?

Issue - State: open - Opened by ToluClassics about 1 year ago - 1 comment

#96 - Slight Error in Readme?

Issue - State: closed - Opened by ToluClassics about 1 year ago - 2 comments

#95 - change target branch for CI

Pull Request - State: closed - Opened by guillaume-be over 1 year ago

#94 - [perf] Avoid unnecessary collect.

Pull Request - State: closed - Opened by ttsugriy over 1 year ago - 1 comment

#93 - [perf] Remove unnecessary vec extend.

Pull Request - State: closed - Opened by ttsugriy over 1 year ago

#92 - Reexport TLS options from `cached-path`

Pull Request - State: closed - Opened by mweber15 over 1 year ago

#91 - Bump openssl from 0.10.48 to 0.10.55 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago - 2 comments
Labels: dependencies

#90 - Start work on wav2vec2 tokenizer

Pull Request - State: open - Opened by j4qfrost over 1 year ago

#89 - sentencepiece is not the same

Issue - State: open - Opened by igor-yusupov over 1 year ago

#88 - Allow addition of tokens in vocab/tokenizer

Pull Request - State: closed - Opened by guillaume-be over 1 year ago

#87 - Bump h2 from 0.3.15 to 0.3.17 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago
Labels: dependencies

#86 - Fix Clippy warnings

Pull Request - State: closed - Opened by guillaume-be over 1 year ago

#85 - Bump openssl from 0.10.45 to 0.10.48 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago - 1 comment
Labels: dependencies

#84 - 8.0.0 release

Pull Request - State: closed - Opened by guillaume-be almost 2 years ago

#83 - CI Fix

Pull Request - State: closed - Opened by guillaume-be almost 2 years ago

#82 - Bump bumpalo from 3.3.0 to 3.12.0 in /main

Pull Request - State: closed - Opened by dependabot[bot] almost 2 years ago - 1 comment
Labels: dependencies

#81 - NLLB Tokenizer support.

Pull Request - State: closed - Opened by npatsakula about 2 years ago - 4 comments

#80 - Use AsRef<Path> instead of &str.

Pull Request - State: closed - Opened by npatsakula about 2 years ago - 4 comments

#79 - Structural errors.

Pull Request - State: open - Opened by npatsakula about 2 years ago - 1 comment

#78 - Special token map extension

Pull Request - State: closed - Opened by guillaume-be about 2 years ago - 4 comments

#78 - Special token map extension

Pull Request - State: closed - Opened by guillaume-be about 2 years ago - 4 comments

#77 - Respect special_tokens_map.json.

Pull Request - State: closed - Opened by yk-goblin about 2 years ago - 5 comments

#76 - DRAFT: NLLB tokenizer support.

Pull Request - State: closed - Opened by npatsakula over 2 years ago - 2 comments

#75 - Bump smallvec from 1.4.0 to 1.8.0 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#75 - Bump smallvec from 1.4.0 to 1.8.0 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#74 - Bump smallvec from 1.4.2 to 1.8.0 in /python-bindings

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#74 - Bump smallvec from 1.4.2 to 1.8.0 in /python-bindings

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#73 - Bump futures-task from 0.3.5 to 0.3.21 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#73 - Bump futures-task from 0.3.5 to 0.3.21 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#72 - Bump futures-util from 0.3.5 to 0.3.21 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#71 - Bump crossbeam-utils from 0.8.3 to 0.8.8 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#71 - Bump crossbeam-utils from 0.8.3 to 0.8.8 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#70 - Bump crossbeam-utils from 0.8.3 to 0.8.8 in /python-bindings

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#69 - Bump lock_api from 0.4.1 to 0.4.6 in /python-bindings

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#68 - Bump tokio from 1.3.0 to 1.19.2 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#68 - Bump tokio from 1.3.0 to 1.19.2 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#67 - Bump crossbeam-deque from 0.8.0 to 0.8.1 in /python-bindings

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#67 - Bump crossbeam-deque from 0.8.0 to 0.8.1 in /python-bindings

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#66 - Bump crossbeam-deque from 0.8.0 to 0.8.1 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#66 - Bump crossbeam-deque from 0.8.0 to 0.8.1 in /main

Pull Request - State: closed - Opened by dependabot[bot] over 2 years ago
Labels: dependencies

#65 - Can't create `XLMRobertaTokenizer` from xlm-roberta dataset

Issue - State: closed - Opened by mlodato517 over 2 years ago - 2 comments

#65 - Can't create `XLMRobertaTokenizer` from xlm-roberta dataset

Issue - State: closed - Opened by mlodato517 over 2 years ago - 2 comments

#64 - 7_0_2 Release

Pull Request - State: closed - Opened by guillaume-be over 2 years ago

#64 - 7_0_2 Release

Pull Request - State: closed - Opened by guillaume-be over 2 years ago

#63 - DeBERTa v2 tokenizer

Pull Request - State: closed - Opened by guillaume-be almost 3 years ago

#62 - Fixed use of AlbertVocab in FNetTokenizer

Pull Request - State: closed - Opened by guillaume-be almost 3 years ago

#62 - Fixed use of AlbertVocab in FNetTokenizer

Pull Request - State: closed - Opened by guillaume-be almost 3 years ago

#61 - Deberta tokenizer

Pull Request - State: closed - Opened by guillaume-be almost 3 years ago

#61 - Deberta tokenizer

Pull Request - State: closed - Opened by guillaume-be almost 3 years ago

#60 - Fnet tokenizer

Pull Request - State: closed - Opened by guillaume-be about 3 years ago

#60 - Fnet tokenizer

Pull Request - State: closed - Opened by guillaume-be about 3 years ago

#59 - Make generic bounds less generic.

Pull Request - State: closed - Opened by sftse about 3 years ago - 1 comment

#58 - prepare for 6.2.5 release

Pull Request - State: closed - Opened by guillaume-be about 3 years ago

#58 - prepare for 6.2.5 release

Pull Request - State: closed - Opened by guillaume-be about 3 years ago

#57 - Fix panic with unicode chars that are expanded at the end of sentences

Pull Request - State: closed - Opened by sftse about 3 years ago - 1 comment

#57 - Fix panic with unicode chars that are expanded at the end of sentences

Pull Request - State: closed - Opened by sftse about 3 years ago - 1 comment

#56 - Updated sentencepiece bpe

Pull Request - State: closed - Opened by guillaume-be about 3 years ago

#55 - Updated MBart tokenizer to use >>ISO639<< format

Pull Request - State: closed - Opened by guillaume-be over 3 years ago

#54 - BPE SentencePiece tokenizers

Pull Request - State: closed - Opened by guillaume-be over 3 years ago

#53 - Optimize leading_byte lookup for language code splitting

Pull Request - State: closed - Opened by guillaume-be over 3 years ago

#53 - Optimize leading_byte lookup for language code splitting

Pull Request - State: closed - Opened by guillaume-be over 3 years ago

#52 - Mbart implementation

Pull Request - State: closed - Opened by guillaume-be over 3 years ago

#52 - Mbart implementation

Pull Request - State: closed - Opened by guillaume-be over 3 years ago

#51 - Reading SentencePieceVocab from text file

Issue - State: open - Opened by MikaelCall over 3 years ago - 5 comments

#51 - Reading SentencePieceVocab from text file

Issue - State: open - Opened by MikaelCall over 3 years ago - 5 comments

#50 - Pegasus tokenizer

Pull Request - State: closed - Opened by guillaume-be over 3 years ago

#50 - Pegasus tokenizer

Pull Request - State: closed - Opened by guillaume-be over 3 years ago

#49 - [Question] How is this library compared to huggingface's tokenizer?

Issue - State: closed - Opened by liebkne over 3 years ago - 2 comments

#49 - [Question] How is this library compared to huggingface's tokenizer?

Issue - State: closed - Opened by liebkne over 3 years ago - 2 comments

#48 - RoBERTa tokenizer patch

Pull Request - State: closed - Opened by guillaume-be almost 4 years ago

#48 - RoBERTa tokenizer patch

Pull Request - State: closed - Opened by guillaume-be almost 4 years ago

#47 - ProphetNet tokenizer implementation

Pull Request - State: closed - Opened by guillaume-be almost 4 years ago

#47 - ProphetNet tokenizer implementation

Pull Request - State: closed - Opened by guillaume-be almost 4 years ago

#46 - Update README.md

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#46 - Update README.md

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#45 - Actions migration

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#45 - Actions migration

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#44 - Reformer tokenizer

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#44 - Reformer tokenizer

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#43 - Updated dependencies, re-generated sentencepiece proto

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#42 - Remove unnecessary arc

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#42 - Remove unnecessary arc

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#41 - Multithreaded bpe

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#41 - Multithreaded bpe

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#40 - Documentation

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#40 - Documentation

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#39 - Using tokenizers within threads is a little cumbersome

Issue - State: closed - Opened by epwalsh about 4 years ago - 10 comments

#39 - Using tokenizers within threads is a little cumbersome

Issue - State: closed - Opened by epwalsh about 4 years ago - 10 comments

#38 - Xlnet tokenizer

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#38 - Xlnet tokenizer

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#37 - Optional add prefix space

Pull Request - State: closed - Opened by guillaume-be about 4 years ago

#36 - Improved error handling

Pull Request - State: closed - Opened by guillaume-be over 4 years ago

#36 - Improved error handling

Pull Request - State: closed - Opened by guillaume-be over 4 years ago