Data

Tools

Indexes

Applications

Experiments

Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / mlc-ai/tokenizers-cpp issues and pull requests

#81 - [Rust] Bump huggingface tokenizer to 0.21.2

Pull Request - State: closed - Opened by MasterJH5574 about 1 month ago

#80 - [BUILD FIX] Use ahash to support v21.0.2

Pull Request - State: closed - Opened by AgrawalAmey about 1 month ago

#79 - Fix type binding error for Tokenizer library

Pull Request - State: closed - Opened by joeldushouyu about 1 month ago - 1 comment

#78 - Rust E3080 mismatched types for tokenizer binding

Issue - State: closed - Opened by joeldushouyu about 1 month ago

#77 - Avoid loading `added_tokens` when this variable is empty

Pull Request - State: closed - Opened by hebangwen about 1 month ago

#76 - Add support for generating SHARED library with -DBUILD_SHARED_LIBS=ON flag

Pull Request - State: open - Opened by PARTHSQL about 2 months ago - 2 comments

#75 - Test

Pull Request - State: closed - Opened by DonghakPark about 2 months ago

#74 - Add support for generating dynamic library

Issue - State: open - Opened by PARTHSQL about 2 months ago

#73 - [Rust] Bump huggingface tokenizer to 0.21.1

Pull Request - State: closed - Opened by MasterJH5574 2 months ago

#72 - rustc E0308 error when compiling version v.0.1.0

Issue - State: open - Opened by gaokao123 2 months ago

#71 - Does it support Wordpiece tokenizer?

Issue - State: open - Opened by yangxudong 2 months ago

#70 - Add build option for mt/md (#1)

Pull Request - State: closed - Opened by jhlee525 3 months ago - 1 comment

#69 - feat : windows build ps1

Pull Request - State: open - Opened by rinechran 3 months ago

#68 - Request to create GitHub Releases or Tags for each release

Issue - State: closed - Opened by rinechran 3 months ago - 2 comments

#67 - [Web] Bump web-tokenizer to 0.1.6

Pull Request - State: closed - Opened by CharlieFRuan 3 months ago - 1 comment

#66 - Fixed compilation for aarch64

Pull Request - State: closed - Opened by NaveenMittal0 4 months ago - 1 comment

#65 - Fix CMake Configuration for MacOS Support

Pull Request - State: closed - Opened by Darce-One 4 months ago - 4 comments

#64 - CMake Configuration Issues on MacOS

Issue - State: open - Opened by Darce-One 4 months ago - 3 comments

#62 - updating tokenizer version from 0.20.0 to 0.21.0

Pull Request - State: closed - Opened by NaveenMittal0 5 months ago

#61 - Problem using #include "tokenizers_c.h"

Issue - State: open - Opened by doogunwo 6 months ago - 1 comment

#60 - Add method to return attn-mask for HF Tokenizer.

Pull Request - State: open - Opened by cptspacemanspiff 6 months ago

#59 - Bump CMake minimum to v3.19

Pull Request - State: closed - Opened by afuller-TT 6 months ago

#58 - Sentence piece llama2 its not printing space

Issue - State: open - Opened by naveenmittal04 6 months ago - 1 comment

#57 - Add hf tokenizer class definition header

Pull Request - State: open - Opened by cptspacemanspiff 6 months ago

#56 - How to build in android platform?

Issue - State: open - Opened by jixiedaima 7 months ago

#55 - Fix for loading tokenizers using non-utf8 strings

Pull Request - State: closed - Opened by ThomasProg 7 months ago - 1 comment

#54 - link error in tokenizers_c.lib

Issue - State: open - Opened by Caio-lima-santos 7 months ago

#53 - example.cc running error

Issue - State: closed - Opened by zhaoxuejun1234 7 months ago - 7 comments

#52 - Add a way to force the macos target in case it is set

Pull Request - State: closed - Opened by ykhrustalev 8 months ago - 1 comment

#51 - Building for size

Issue - State: open - Opened by stellaraccident 8 months ago - 1 comment

#50 - Abort is not a great error handling strategy

Issue - State: open - Opened by stellaraccident 8 months ago

#49 - [Rust] Bump huggingface tokenizer to 0.20.0

Pull Request - State: closed - Opened by MasterJH5574 9 months ago

#47 - [CMake] Enable SentencePiece tokenizer by default

Pull Request - State: closed - Opened by MasterJH5574 9 months ago

#45 - Why MLC_ENABLE_SENTENCEPIECE_TOKENIZER OFF by default?

Issue - State: open - Opened by korciuch 10 months ago - 6 comments

#44 - How could I get the PIC library ?

Issue - State: closed - Opened by EeyoreLee 10 months ago - 1 comment

#43 - Building the example fails

Issue - State: closed - Opened by kriscao-cohere 11 months ago - 1 comment

#42 - [Web] Set TOKENIZERS_PARALLELISM to false for HFTokenizer

Pull Request - State: closed - Opened by CharlieFRuan 12 months ago

#41 - Add web binding `Tokenizer.tokenToId()`

Pull Request - State: closed - Opened by grf53 12 months ago - 2 comments

#40 - does it support multi - thread decode ?

Issue - State: open - Opened by Vincent-syr about 1 year ago - 3 comments

#38 - [CMake] Support disable SentencePiece tokenizer

Pull Request - State: closed - Opened by MasterJH5574 about 1 year ago

#37 - How to compile in Ubuntu

Issue - State: closed - Opened by gaokao123 about 1 year ago - 2 comments

#36 - [Rust] Bump HuggingFace tokenizer version

Pull Request - State: closed - Opened by MasterJH5574 about 1 year ago

#35 - tokenizer for triton inference server

Issue - State: open - Opened by geraldstanje about 1 year ago

#34 - Remove incorrect final for non-virtual func

Pull Request - State: closed - Opened by vinx13 about 1 year ago

#33 - Added EncodeBatch interface

Pull Request - State: closed - Opened by vinx13 about 1 year ago

#32 - Fail to import in Node.js

Issue - State: open - Opened by zcbenz about 1 year ago - 1 comment

#30 - conflict with torch

Issue - State: closed - Opened by zhangyuhanjc over 1 year ago - 1 comment

#29 - compile failed

Issue - State: closed - Opened by chenpei2 over 1 year ago - 1 comment

#28 - Set value of TOKENIZERS_CPP_CARGO_TARGET based on ANDROID_ABI

Pull Request - State: closed - Opened by yuxuanchiadm over 1 year ago - 1 comment

#27 - Allow parameters related to special tokens for HFTokenizer

Pull Request - State: closed - Opened by Abhishek8394 over 1 year ago

#26 - Allow hugginface tokenizer to pass arguments for add/skip special tokens

Issue - State: closed - Opened by Abhishek8394 over 1 year ago - 1 comment

#24 - Can't support cross build on Linux

Issue - State: closed - Opened by SakuragiJump over 1 year ago - 5 comments

#23 - is there any plan to support Tiktoken?

Issue - State: closed - Opened by Jasonsey over 1 year ago - 3 comments

#22 - Add support for querying vocabulary from tokenizer

Pull Request - State: closed - Opened by Ubospica over 1 year ago - 5 comments

#21 - Error when compiling tokenizers for AMD with HIP C++ compiler (hipcc)

Issue - State: closed - Opened by goliaro over 1 year ago - 1 comment

#20 - Update build.sh to make web tokenizer work with chrome extension

Pull Request - State: closed - Opened by manuongithub over 1 year ago - 2 comments

#19 - fix rwkv world tokenzier

Pull Request - State: closed - Opened by BBuf over 1 year ago

#18 - Update to reduce memory realloc

Pull Request - State: closed - Opened by tqchen almost 2 years ago

#17 - How to free encode ids without destructing tokenizer?

Issue - State: closed - Opened by ShukantPal almost 2 years ago - 8 comments

#16 - [FIX] Remove boost dependency from submodule msgpack

Pull Request - State: closed - Opened by Hzfengsy almost 2 years ago

#15 - msgpack as 3rdparty library instead of fetch_context

Pull Request - State: closed - Opened by BBuf almost 2 years ago - 1 comment

#14 - add rwkv world tokenizer

Pull Request - State: closed - Opened by BBuf almost 2 years ago - 6 comments

#13 - If the model hasn't tokenizer.json file, what should I do?

Issue - State: closed - Opened by wolf-li almost 2 years ago - 2 comments

#11 - Update rust to support latest tokenizers

Pull Request - State: closed - Opened by tqchen almost 2 years ago

#10 - Compiler error (Rust)

Issue - State: closed - Opened by jklaise almost 2 years ago - 3 comments

#9 - undefined symbol: open64 when run build.sh in web dir

Issue - State: closed - Opened by helloburke almost 2 years ago - 1 comment

#8 - An error occurred in the compilation

Issue - State: closed - Opened by Liu-xiandong about 2 years ago - 1 comment

#7 - SentencePiece Build Error - ld: error: undefined symbol: __android_log_write

Issue - State: closed - Opened by zjc664656505 about 2 years ago - 6 comments

#6 - Update CMakeLists.txt

Pull Request - State: closed - Opened by luiyen about 2 years ago - 1 comment

#5 - Fix windows build by linking against ntdll

Pull Request - State: closed - Opened by junrushao about 2 years ago

#4 - Fix readme typo

Pull Request - State: closed - Opened by erjanmx about 2 years ago - 1 comment

#3 - Add npm package

Pull Request - State: closed - Opened by tqchen about 2 years ago

#2 - Update CMakeLists.txt

Pull Request - State: closed - Opened by songkq about 2 years ago - 1 comment

#1 - add_library INTERFACE library requires no source arguments.

Issue - State: closed - Opened by songkq about 2 years ago - 12 comments