google/sentencepiece issues and pull requests

#1088 - fix crash in unigram model training

Pull Request - State: open - Opened by pilleye 4 days ago - 1 comment

#1087 - Fix ci qemu

Pull Request - State: closed - Opened by dbowring 5 days ago

#1086 - Bump cryptography from 43.0.0 to 44.0.1 in /.github/workflows/requirements in the pip group

Pull Request - State: open - Opened by dependabot[bot] 6 days ago
Labels: dependencies, python

#1085 - Fix CI build issues

Pull Request - State: open - Opened by dbowring 9 days ago - 1 comment

#1084 - Upgrade cibuildwheel to v2.20.0 (build for python3.13)

Pull Request - State: open - Opened by dbowring 12 days ago - 2 comments

#1083 - Unable to install sentencepiece on Python 3.13

Issue - State: open - Opened by glxay 13 days ago - 5 comments

#1082 - Unable to install sentencepiece on Python 3.13

Issue - State: closed - Opened by glxay 13 days ago

#1081 - Bump the github-actions group across 1 directory with 4 updates

Pull Request - State: open - Opened by dependabot[bot] 17 days ago
Labels: dependencies, github_actions

#1080 - Bump the build-time-deps group across 1 directory with 6 updates

Pull Request - State: open - Opened by dependabot[bot] 17 days ago
Labels: dependencies, python

#1079 - Vocabulary size too high

Issue - State: closed - Opened by jixiedaima 27 days ago - 1 comment

#1078 - model issue:

Issue - State: closed - Opened by jixiedaima about 1 month ago - 2 comments

#1077 - `split_by_number` only splits Western Arabic Numerals and Full-Width Digits

Issue - State: closed - Opened by Numeri about 1 month ago - 1 comment

#1076 - [Question] Could not install via pip or build from source code as a model of python

Issue - State: open - Opened by Mikachu2333 about 1 month ago - 2 comments

#1075 - is possible convert hugging face tokenizers in sentence piece .model?

Issue - State: closed - Opened by Caio-lima-santos about 1 month ago

#1074 - Update python readme prerequisites

Pull Request - State: open - Opened by Chad-007 about 2 months ago

#1073 - Add dummy prefix from args

Pull Request - State: closed - Opened by yiyangh-ps 3 months ago - 1 comment

#1072 - Bump the build-time-deps group across 1 directory with 6 updates

Pull Request - State: closed - Opened by dependabot[bot] 3 months ago - 1 comment
Labels: dependencies, python

#1071 - BPE Dropout tokenizer generates unk at the beginning of sequence

Issue - State: open - Opened by AnnaLebedeva 3 months ago

#1070 - The pip command to install the SentencePiece Python module fails.

Issue - State: open - Opened by tprrt 3 months ago - 4 comments

#1069 - Doesn't seem to work with Python 3.13

Issue - State: open - Opened by tprrt 3 months ago - 9 comments

#1068 - Initialized number of seed sentencepieces too low

Issue - State: open - Opened by DmitriiP20 3 months ago - 1 comment

#1067 - subprocess-exited-with-error

Issue - State: closed - Opened by Nana-kwame-junior 3 months ago - 2 comments

#1066 - Update artifact actions from v3 to v4

Pull Request - State: open - Opened by kasinadhsarma 3 months ago - 1 comment

#1065 - Asan detects memory leak in sentencepiece/_sentencepiece.cpython-312-x86_64-linux-gnu.so+0x6f7f4

Issue - State: open - Opened by renxida 4 months ago

#1064 - Bump the build-time-deps group across 1 directory with 4 updates

Pull Request - State: closed - Opened by dependabot[bot] 4 months ago - 1 comment
Labels: dependencies, python

#1063 - Bump the github-actions group across 1 directory with 3 updates

Pull Request - State: closed - Opened by dependabot[bot] 4 months ago - 1 comment
Labels: dependencies, github_actions

#1062 - Distributed implementation of the unsupervised unigram word segmentizer possible?

Issue - State: closed - Opened by y-he2 4 months ago

#1061 - Enhancements to CI Workflows and Python Module Initialization with Minor Fixes

Pull Request - State: open - Opened by kasinadhsarma 4 months ago - 1 comment

#1060 - Compatibility Issue when using v0.2.0 with transformers and tensorflow

Issue - State: open - Opened by aws-tianquaw 5 months ago - 3 comments

#1059 - "space must not be included in normalized string" when training with a sentence iterator

Issue - State: closed - Opened by bauwenst 5 months ago - 1 comment

#1058 - Bump the github-actions group across 1 directory with 3 updates

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 1 comment
Labels: dependencies, github_actions

#1057 - Bump the build-time-deps group across 1 directory with 3 updates

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 1 comment
Labels: dependencies, python

#1056 - Use libsentencepiece.0.dylib in macos. When load model, model_factory.cc(43) LOG(ERROR) Unknown model_type: 16

Issue - State: closed - Opened by codeAndxv 5 months ago - 2 comments

#1055 - Manually modify the vocabulary list

Issue - State: open - Opened by lileica 5 months ago - 1 comment

#1054 - Can't load Llama3 tokenizer.model

Issue - State: closed - Opened by fabriceyhc 5 months ago - 1 comment

#1053 - Is it possible to add normalization rules into a trained sentence piece model?

Issue - State: closed - Opened by lost-libra 5 months ago - 2 comments

#1052 - Training with a custom base vocabulary and handling reserved tokens

Issue - State: closed - Opened by rteehas 5 months ago - 1 comment

#1051 - Crashes on out of range inputs depending on other inputs

Issue - State: open - Opened by colehaus 5 months ago - 1 comment
Labels: bug

#1050 - logprobs in the vocabulary file do not match the values computed from the tokenized training document

Issue - State: closed - Opened by pnugues 5 months ago - 2 comments

#1049 - Bump the build-time-deps group in /.github/workflows/requirements with 2 updates

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 1 comment
Labels: dependencies, python

#1048 - Bump the github-actions group with 2 updates

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 1 comment
Labels: dependencies, github_actions

#1047 - With unigram algorithm, constant piece at end of each sentences does not become a token

Issue - State: open - Opened by jogardi 6 months ago
Labels: bug

#1046 - Error Attribute Error: type object 'SentencePieceTrainer' has no attribute 'train'. Did you mean: 'Train'?

Issue - State: open - Opened by bop578530 6 months ago - 1 comment

#1045 - builds for android devices

Issue - State: closed - Opened by RaoufiTech 6 months ago

#1044 - decode token one by one

Issue - State: closed - Opened by nigelzzz 6 months ago - 1 comment

#1043 - decode one by one can't show space

Issue - State: closed - Opened by nigelzzz 6 months ago - 3 comments

#1042 - Why is the Hugging Face encoding 1 greater compared to the Google SentencePiece encoding when using the XLM-RoBERTa SentencePiece tokenizer?

Issue - State: closed - Opened by RaoufiTech 6 months ago - 1 comment

#1041 - Bump the build-time-deps group across 1 directory with 3 updates

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago
Labels: dependencies, python

#1040 - Bump the github-actions group with 3 updates

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago
Labels: dependencies, github_actions

#1039 - multi-thread batch encode seems slower than list comprehension

Issue - State: closed - Opened by Mr-Grin 7 months ago - 1 comment

#1038 - Update setup.py

Pull Request - State: closed - Opened by raushanksec 7 months ago - 1 comment

#1037 - Add support for windows arm64

Pull Request - State: closed - Opened by Nagico2 7 months ago - 1 comment

#1036 - Bump the build-time-deps group across 1 directory with 4 updates

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 1 comment
Labels: dependencies, python

#1035 - Bump the pip group in /.github/workflows/requirements with 2 updates

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago
Labels: dependencies, python

#1034 - Bump certifi from 2023.11.17 to 2024.7.4 in /.github/workflows/requirements in the pip group

Pull Request - State: closed - Opened by dependabot[bot] 8 months ago
Labels: dependencies, python

#1033 - Bump the github-actions group across 1 directory with 6 updates

Pull Request - State: closed - Opened by dependabot[bot] 8 months ago
Labels: dependencies, github_actions

#1032 - Bump the build-time-deps group in /.github/workflows/requirements with 4 updates

Pull Request - State: closed - Opened by dependabot[bot] 8 months ago
Labels: dependencies, python

#1031 - Zero Width Joiner issue for Sinhala Language

Issue - State: open - Opened by Nadil-K 8 months ago

#1030 - No typings in Python package

Issue - State: open - Opened by marcospgp 8 months ago - 1 comment
Labels: enhancement

#1029 - When I set SPM_PROTOBUF_PROVIDER to "package" in CMakeLists.txt, the compilation fails.

Issue - State: open - Opened by hhxdestiny 8 months ago

#1028 - trainer_interface.cc: Integer value -1 is outside the valid range of values [0, 255] for the enumeration type 'ScriptType'

Issue - State: open - Opened by kcoul 8 months ago - 1 comment
Labels: enhancement

GitHub / google/sentencepiece issues and pull requests