Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / VKCOM/YouTokenToMe issues and pull requests

#115 - Fix Cython dependency

Pull Request - State: closed - Opened by dumitruboaghi 6 months ago

#114 - Fix compile errors with gcc13 and cython dep

Pull Request - State: open - Opened by adrianlzt 9 months ago

#112 - Visual studio c++ 14.0 error while installing.

Issue - State: open - Opened by manjunath7472 12 months ago

#111 - Special tokens accessibility

Issue - State: open - Opened by TheFaheem about 1 year ago

#110 - Pre-tokenization viable?

Issue - State: closed - Opened by akontra over 1 year ago

#109 - libc++abi Runtime error

Issue - State: closed - Opened by jjdelvalle over 1 year ago - 1 comment

#108 - Fix install Cython errors on pip & poetry

Pull Request - State: open - Opened by gburlet over 1 year ago - 9 comments

#106 - + dependabot

Pull Request - State: open - Opened by borisshapa over 1 year ago

#105 - [WIP] fast wordpiece tokenization

Pull Request - State: open - Opened by gleb-kov over 1 year ago

#104 - Poetry support

Pull Request - State: closed - Opened by SokolovYaroslav over 1 year ago - 1 comment

#103 - Add minimum requirements for package installation

Pull Request - State: open - Opened by gbeckenkamp almost 2 years ago - 4 comments

#102 - Update LICENSE

Pull Request - State: open - Opened by victor30608 almost 2 years ago

#101 - up-to-date benchmarks

Pull Request - State: closed - Opened by gleb-kov almost 2 years ago

#100 - remove deadcode, use std namespace explicitly

Pull Request - State: closed - Opened by gleb-kov almost 2 years ago

#99 - run stress test with sanitizers

Pull Request - State: closed - Opened by gleb-kov almost 2 years ago

#98 - Code refactoring

Pull Request - State: closed - Opened by gleb-kov almost 2 years ago

#97 - Support custom tokens

Pull Request - State: open - Opened by 9173860 about 2 years ago

#96 - Youtokentome cannot be installed on Windows if VisualStudio not installed

Issue - State: open - Opened by yilmazay74 about 2 years ago - 5 comments

#95 - Windows Support

Issue - State: closed - Opened by harubaru about 2 years ago - 1 comment

#94 - Can't pip install on a new env (without Cython)

Issue - State: open - Opened by zachmoshe over 2 years ago - 3 comments

#92 - How to optimize the usage of videomemory?

Issue - State: closed - Opened by HetagKoroev almost 3 years ago

#91 - Installing error C1083

Issue - State: open - Opened by YFrite about 3 years ago - 2 comments

#90 - Installation error using pip

Issue - State: closed - Opened by azayz about 3 years ago - 3 comments

#89 - error: Microsoft Visual C++ 14.0 or greater is required.

Issue - State: open - Opened by fourat-bs about 3 years ago - 1 comment

#88 - Decode() got an unexpected keyword argument 'ignore_ids'

Issue - State: open - Opened by raphkhan about 3 years ago

#87 - No word tokenizer under the hood?

Issue - State: open - Opened by slowwavesleep over 3 years ago

#86 - Error during installation

Issue - State: open - Opened by ruruu127 over 3 years ago

#85 - How does YouTokenToMe's speed compare to subword-nmt?

Issue - State: open - Opened by gowtham1997 over 3 years ago

#84 - Using YouTokenToMe with pre-defined vocab and embeddings

Issue - State: open - Opened by alexbalandi over 3 years ago - 2 comments

#83 - type annotation for BPE.decode fixed

Pull Request - State: closed - Opened by kefirski over 3 years ago

#82 - [WIP] Fix CI. Deprecate python3.5

Pull Request - State: closed - Opened by yutkin over 3 years ago

#81 - Added Pickle support

Pull Request - State: closed - Opened by kefirski over 3 years ago - 1 comment

#80 - Tokenizing large corpus

Issue - State: open - Opened by quetz almost 4 years ago - 2 comments

#79 - Decoding without IDs

Issue - State: closed - Opened by JOHW85 almost 4 years ago - 2 comments

#78 - Fixing return type annotation in decode method

Pull Request - State: closed - Opened by SantaDiver almost 4 years ago

#77 - Decode method returns list of strings

Issue - State: closed - Opened by SantaDiver almost 4 years ago - 1 comment

#76 - Set random seed for bpe dropout

Pull Request - State: closed - Opened by yutkin about 4 years ago

#75 - Is it possible to unset random seed for BPE-dropout?

Issue - State: open - Opened by skurzhanskyi about 4 years ago - 2 comments
Labels: enhancement

#74 - [Question] How to learn joint bpe and vocabulary

Issue - State: closed - Opened by cidrugHug8 about 4 years ago - 2 comments

#73 - Controlling word tokenization

Issue - State: open - Opened by MexicanMan about 4 years ago

#72 - How to train with multiple corpus files?

Issue - State: open - Opened by hccho2 about 4 years ago

#71 - Process killed?

Issue - State: open - Opened by miguelvictor about 4 years ago - 4 comments

#70 - Add Huggingface tokenizer converter

Issue - State: closed - Opened by alxmamaev over 4 years ago - 1 comment

#69 - how to get vocab

Issue - State: open - Opened by wqfengnlpr over 4 years ago - 2 comments

#68 - Vocabulary contains underscore multiple times?

Issue - State: open - Opened by RuABraun over 4 years ago

#67 - "▁" character can be separated when using BPE-dropout

Issue - State: open - Opened by TIXFeniks over 4 years ago - 11 comments

#66 - How to generate vocab.json and merges.txt for YTTM tokenizer?

Issue - State: open - Opened by nikhilno1 over 4 years ago - 1 comment

#65 - Special Tokens

Issue - State: open - Opened by martingajek over 4 years ago - 1 comment

#64 - Ruby Library

Issue - State: closed - Opened by ankane over 4 years ago

#62 - FIx CI on MacOS

Pull Request - State: closed - Opened by yutkin over 4 years ago

#61 - No module named 'Cython'

Issue - State: closed - Opened by Not-White over 4 years ago - 2 comments

#60 - Improve memory usage

Pull Request - State: closed - Opened by xbelonogov over 4 years ago

#59 - Doesn't consider combining characters.

Issue - State: open - Opened by IDDT over 4 years ago

#58 - Aborted (core dumped) error

Issue - State: closed - Opened by glample over 4 years ago - 4 comments

#57 - Hugging Face benchmark

Pull Request - State: closed - Opened by xbelonogov over 4 years ago - 1 comment

#56 - [Feature] Add ability to manual edit vocabulary (add/remove subwords)

Issue - State: open - Opened by zabzug-pfpt over 4 years ago - 7 comments

#55 - Replace ska namespace with vkcom namespace

Pull Request - State: closed - Opened by Oktai15 almost 5 years ago

#54 - С++ Interface needs namespace `yttm`

Issue - State: closed - Opened by Oktai15 almost 5 years ago - 1 comment

#53 - Is it possible to wrap the calls in pyspark UDFs?

Issue - State: open - Opened by AlexGidiotis almost 5 years ago - 1 comment

#51 - Fix typo in README & update travis for Mac

Pull Request - State: closed - Opened by Oktai15 almost 5 years ago

#50 - compilation issues

Issue - State: open - Opened by jwijffels almost 5 years ago - 6 comments
Labels: help wanted

#49 - Add Python 3.8 classifier to setup.py

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#48 - Added yttm.pyx to manifest.in

Pull Request - State: closed - Opened by xbelonogov almost 5 years ago

#47 - Update docs

Pull Request - State: closed - Opened by kalaidin almost 5 years ago

#46 - Add Cython to Travis

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#45 - Add dropout

Pull Request - State: closed - Opened by xbelonogov almost 5 years ago

#44 - Add an option to predefine special tokens

Issue - State: open - Opened by Kyeongpil almost 5 years ago - 4 comments
Labels: enhancement

#43 - Added feature to ignore some indices during the decoding

Pull Request - State: closed - Opened by kefirski almost 5 years ago

#42 - type checks assertions doesn't seem to be safe

Issue - State: closed - Opened by kefirski almost 5 years ago - 1 comment
Labels: enhancement

#41 - Make Black formatting is required in Travis

Pull Request - State: closed - Opened by yutkin almost 5 years ago
Labels: enhancement

#40 - Do not raise unhandled exception on unknown --output_type

Pull Request - State: closed - Opened by asivokon almost 5 years ago - 2 comments
Labels: enhancement

#39 - Update README.md

Pull Request - State: closed - Opened by kefirski almost 5 years ago

#38 - Update README.md

Pull Request - State: closed - Opened by kefirski almost 5 years ago

#37 - BPE-Dropout support

Issue - State: closed - Opened by kalaidin almost 5 years ago - 1 comment
Labels: enhancement

#36 - Bump version to v1.0.3

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#35 - Replace size_t on uint64_t

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#34 - cc

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#33 - Import somehow fails

Issue - State: closed - Opened by arrrrrmin almost 5 years ago - 4 comments
Labels: bug

#32 - Support pickling

Issue - State: closed - Opened by shmpanski almost 5 years ago - 4 comments
Labels: enhancement

#31 - Remove only on master CI build

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#30 - Remove only on master CI build

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#29 - Remove only on master CI build

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#28 - Remove redundant check in CI

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#27 - Add Travis-CI

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#26 - Add Travis-CI

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#25 - WIP: Add travis CI

Pull Request - State: closed - Opened by yutkin almost 5 years ago

#24 - Handling line breaks

Issue - State: closed - Opened by mgrankin almost 5 years ago - 3 comments
Labels: enhancement

#23 - Added return codes and exceptions for handling incorrect input data.

Pull Request - State: closed - Opened by xbelonogov almost 5 years ago

#22 - Loading a BPE model with a wrong path kills the Jupyter kernel

Issue - State: closed - Opened by warenlg almost 5 years ago - 2 comments

#21 - [WIP] File objects

Pull Request - State: open - Opened by vmarkovtsev almost 5 years ago - 1 comment

#20 - `youtokentome/cpp/yttm.pyx` is missing

Issue - State: closed - Opened by vmarkovtsev almost 5 years ago - 3 comments

#19 - Change the model format to binary

Pull Request - State: open - Opened by vmarkovtsev almost 5 years ago - 3 comments
Labels: enhancement

#18 - Extension for our production usage

Issue - State: closed - Opened by vmarkovtsev almost 5 years ago - 7 comments
Labels: enhancement, help wanted

#17 - Way to predefine vocab size for continious text data

Issue - State: closed - Opened by serhio7 almost 5 years ago - 1 comment
Labels: question

#16 - [Feature] Add text normalisation as SentencePiece do

Issue - State: open - Opened by keotic about 5 years ago - 2 comments
Labels: enhancement