VKCOM/YouTokenToMe issues and pull requests

#115 - Fix Cython dependency

Pull Request - State: closed - Opened by dumitruboaghi 11 months ago

#114 - Fix compile errors with gcc13 and cython dep

Pull Request - State: open - Opened by adrianlzt about 1 year ago

#113 - ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found

Issue - State: open - Opened by timotheecour about 1 year ago

#112 - Visual studio c++ 14.0 error while installing.

Issue - State: open - Opened by manjunath7472 over 1 year ago

#111 - Special tokens accessibility

Issue - State: open - Opened by TheFaheem over 1 year ago

#110 - Pre-tokenization viable?

Issue - State: closed - Opened by akontra over 1 year ago

#109 - libc++abi Runtime error

Issue - State: closed - Opened by jjdelvalle almost 2 years ago - 1 comment

#108 - Fix install Cython errors on pip & poetry

Pull Request - State: open - Opened by gburlet almost 2 years ago - 9 comments

#107 - ModuleNotFoundError: No module named 'Cython' when installing with pip version 23.1

Issue - State: closed - Opened by cosminc98 almost 2 years ago - 5 comments

#106 - + dependabot

Pull Request - State: open - Opened by borisshapa almost 2 years ago

#105 - [WIP] fast wordpiece tokenization

Pull Request - State: open - Opened by gleb-kov almost 2 years ago

#104 - Poetry support

Pull Request - State: closed - Opened by SokolovYaroslav about 2 years ago - 1 comment

#103 - Add minimum requirements for package installation

Pull Request - State: open - Opened by gbeckenkamp about 2 years ago - 4 comments

#102 - Update LICENSE

Pull Request - State: open - Opened by victor30608 over 2 years ago

#101 - up-to-date benchmarks

Pull Request - State: closed - Opened by gleb-kov over 2 years ago

#100 - remove deadcode, use std namespace explicitly

Pull Request - State: closed - Opened by gleb-kov over 2 years ago

#99 - run stress test with sanitizers

Pull Request - State: closed - Opened by gleb-kov over 2 years ago

#98 - Code refactoring

Pull Request - State: closed - Opened by gleb-kov over 2 years ago

#97 - Support custom tokens

Pull Request - State: open - Opened by 9173860 over 2 years ago

#96 - Youtokentome cannot be installed on Windows if VisualStudio not installed

Issue - State: open - Opened by yilmazay74 over 2 years ago - 5 comments

#95 - Windows Support

Issue - State: closed - Opened by harubaru over 2 years ago - 1 comment

#94 - Can't pip install on a new env (without Cython)

Issue - State: open - Opened by zachmoshe almost 3 years ago - 3 comments

#93 - compatibility of youtokentome.BPE outputs with the official implementation

Issue - State: open - Opened by l-k-11235 about 3 years ago

#92 - How to optimize the usage of videomemory?

Issue - State: closed - Opened by HetagKoroev over 3 years ago

#91 - Installing error C1083

Issue - State: open - Opened by YFrite over 3 years ago - 2 comments

#90 - Installation error using pip

Issue - State: closed - Opened by azayz over 3 years ago - 3 comments

#89 - error: Microsoft Visual C++ 14.0 or greater is required.

Issue - State: open - Opened by fourat-bs over 3 years ago - 1 comment

#88 - Decode() got an unexpected keyword argument 'ignore_ids'

Issue - State: open - Opened by raphkhan over 3 years ago

#87 - No word tokenizer under the hood?

Issue - State: open - Opened by slowwavesleep almost 4 years ago

#86 - Error during installation

Issue - State: open - Opened by ruruu127 almost 4 years ago

#85 - How does YouTokenToMe's speed compare to subword-nmt?

Issue - State: open - Opened by gowtham1997 almost 4 years ago

#84 - Using YouTokenToMe with pre-defined vocab and embeddings

Issue - State: open - Opened by alexbalandi about 4 years ago - 2 comments

#83 - type annotation for BPE.decode fixed

Pull Request - State: closed - Opened by kefirski about 4 years ago

#82 - [WIP] Fix CI. Deprecate python3.5

Pull Request - State: closed - Opened by yutkin about 4 years ago

#81 - Added Pickle support

Pull Request - State: closed - Opened by kefirski about 4 years ago - 1 comment

#80 - Tokenizing large corpus

Issue - State: open - Opened by quetz over 4 years ago - 2 comments

#79 - Decoding without IDs

Issue - State: closed - Opened by JOHW85 over 4 years ago - 2 comments

#78 - Fixing return type annotation in decode method

Pull Request - State: closed - Opened by SantaDiver over 4 years ago

#77 - Decode method returns list of strings

Issue - State: closed - Opened by SantaDiver over 4 years ago - 1 comment

#76 - Set random seed for bpe dropout

Pull Request - State: closed - Opened by yutkin over 4 years ago

#75 - Is it possible to unset random seed for BPE-dropout?

Issue - State: open - Opened by skurzhanskyi over 4 years ago - 2 comments
Labels: enhancement

#74 - [Question] How to learn joint bpe and vocabulary

Issue - State: closed - Opened by cidrugHug8 over 4 years ago - 2 comments

#73 - Controlling word tokenization

Issue - State: open - Opened by MexicanMan over 4 years ago

#72 - How to train with multiple corpus files?

Issue - State: open - Opened by hccho2 over 4 years ago

#71 - Process killed?

Issue - State: open - Opened by miguelvictor over 4 years ago - 4 comments

#70 - Add Huggingface tokenizer converter

Issue - State: closed - Opened by alxmamaev almost 5 years ago - 1 comment

#69 - how to get vocab

Issue - State: open - Opened by wqfengnlpr almost 5 years ago - 2 comments

#68 - Vocabulary contains underscore multiple times?

Issue - State: open - Opened by RuABraun almost 5 years ago

#67 - "▁" character can be separated when using BPE-dropout

Issue - State: open - Opened by TIXFeniks almost 5 years ago - 11 comments

#66 - How to generate vocab.json and merges.txt for YTTM tokenizer?

Issue - State: open - Opened by nikhilno1 almost 5 years ago - 1 comment

#65 - Special Tokens

Issue - State: open - Opened by martingajek almost 5 years ago - 1 comment

#64 - Ruby Library

Issue - State: closed - Opened by ankane almost 5 years ago

#63 - Adding new terms into pre-trained model vocab | Issue in tokenizing OOV keywords

Issue - State: open - Opened by spate141 almost 5 years ago

#62 - FIx CI on MacOS

Pull Request - State: closed - Opened by yutkin about 5 years ago

#61 - No module named 'Cython'

Issue - State: closed - Opened by Not-White about 5 years ago - 2 comments

#60 - Improve memory usage

Pull Request - State: closed - Opened by xbelonogov about 5 years ago

#59 - Doesn't consider combining characters.

Issue - State: open - Opened by IDDT about 5 years ago

#58 - Aborted (core dumped) error

Issue - State: closed - Opened by glample about 5 years ago - 4 comments

#57 - Hugging Face benchmark

Pull Request - State: closed - Opened by xbelonogov about 5 years ago - 1 comment

#56 - [Feature] Add ability to manual edit vocabulary (add/remove subwords)

Issue - State: open - Opened by zabzug-pfpt about 5 years ago - 7 comments

#55 - Replace ska namespace with vkcom namespace

Pull Request - State: closed - Opened by Oktai15 about 5 years ago

#54 - С++ Interface needs namespace `yttm`

Issue - State: closed - Opened by Oktai15 about 5 years ago - 1 comment

#53 - Is it possible to wrap the calls in pyspark UDFs?

Issue - State: open - Opened by AlexGidiotis about 5 years ago - 1 comment

#52 - [Feature] Add support for training on directories (not only on a single file)

Issue - State: open - Opened by shkarupa-alex about 5 years ago - 2 comments

#51 - Fix typo in README & update travis for Mac

Pull Request - State: closed - Opened by Oktai15 about 5 years ago

#50 - compilation issues

Issue - State: open - Opened by jwijffels about 5 years ago - 6 comments
Labels: help wanted

#49 - Add Python 3.8 classifier to setup.py

Pull Request - State: closed - Opened by yutkin about 5 years ago

#48 - Added yttm.pyx to manifest.in

Pull Request - State: closed - Opened by xbelonogov about 5 years ago

#47 - Update docs

Pull Request - State: closed - Opened by kalaidin about 5 years ago

#46 - Add Cython to Travis

Pull Request - State: closed - Opened by yutkin about 5 years ago

#45 - Add dropout

Pull Request - State: closed - Opened by xbelonogov over 5 years ago

#44 - Add an option to predefine special tokens

Issue - State: open - Opened by Kyeongpil over 5 years ago - 4 comments
Labels: enhancement

#43 - Added feature to ignore some indices during the decoding

Pull Request - State: closed - Opened by kefirski over 5 years ago

#42 - type checks assertions doesn't seem to be safe

Issue - State: closed - Opened by kefirski over 5 years ago - 1 comment
Labels: enhancement

#41 - Make Black formatting is required in Travis

Pull Request - State: closed - Opened by yutkin over 5 years ago
Labels: enhancement

#40 - Do not raise unhandled exception on unknown --output_type

Pull Request - State: closed - Opened by asivokon over 5 years ago - 2 comments
Labels: enhancement

#39 - Update README.md

Pull Request - State: closed - Opened by kefirski over 5 years ago

#38 - Update README.md

Pull Request - State: closed - Opened by kefirski over 5 years ago