Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / openai/tiktoken issues and pull requests

#276 - Encode an empty string gives empty tokens

Issue - State: closed - Opened by flexwang 7 months ago - 2 comments

#275 - K

Issue - State: closed - Opened by marseko 8 months ago

#275 - K

Issue - State: closed - Opened by marseko 8 months ago

#272 - minor fix

Pull Request - State: closed - Opened by igeni 8 months ago

#272 - minor fix

Pull Request - State: closed - Opened by igeni 8 months ago

#271 - Remove dependency on requests

Pull Request - State: open - Opened by tristan-jl 8 months ago

#271 - Remove dependency on requests

Pull Request - State: open - Opened by tristan-jl 8 months ago

#270 - Description of repository has a typo

Issue - State: closed - Opened by markusheimerl 8 months ago - 1 comment

#270 - Description of repository has a typo

Issue - State: closed - Opened by markusheimerl 8 months ago - 1 comment

#268 - Create SECURITY.md

Pull Request - State: closed - Opened by Monkei49 8 months ago

#268 - Create SECURITY.md

Pull Request - State: closed - Opened by Monkei49 8 months ago

#266 - Create devcontainer.json

Pull Request - State: closed - Opened by 79075 8 months ago

#266 - Create devcontainer.json

Pull Request - State: closed - Opened by 79075 8 months ago

#265 - Incorrect tokenization of "Elaborate"

Issue - State: closed - Opened by Sternbach-Software 8 months ago - 1 comment

#265 - Incorrect tokenization of "Elaborate"

Issue - State: closed - Opened by Sternbach-Software 8 months ago - 1 comment

#264 - Can't install tiktoken==0.4.0 or tiktoken==0.5.1in Python 3.12

Issue - State: closed - Opened by JackObid 8 months ago - 2 comments

#264 - Can't install tiktoken==0.4.0 or tiktoken==0.5.1in Python 3.12

Issue - State: closed - Opened by JackObid 8 months ago - 2 comments

#263 - TikTok-Unlock-master.zip

Pull Request - State: closed - Opened by 79075 8 months ago

#263 - TikTok-Unlock-master.zip

Pull Request - State: closed - Opened by 79075 8 months ago

#262 - Update README.md

Pull Request - State: closed - Opened by tic-top 9 months ago

#262 - Update README.md

Pull Request - State: closed - Opened by tic-top 9 months ago

#261 - <|endoftext|>，Why can't ChatGPT recognize it?

Issue - State: closed - Opened by Kayce001 9 months ago - 1 comment

#261 - <|endoftext|>，Why can't ChatGPT recognize it?

Issue - State: closed - Opened by Kayce001 9 months ago - 1 comment

#260 - In-code documentation update

Pull Request - State: open - Opened by louisbrulenaudet 9 months ago

#260 - In-code documentation update

Pull Request - State: open - Opened by louisbrulenaudet 9 months ago

#259 - Unable to route GET requests through proxy

Issue - State: open - Opened by aevilevitch 9 months ago - 2 comments

#259 - Unable to route GET requests through proxy

Issue - State: open - Opened by aevilevitch 9 months ago - 2 comments

#258 - Add possessive quantifiers to avoid catastrophic backtracking

Pull Request - State: closed - Opened by l0rinc 9 months ago - 3 comments

#258 - Add possessive quantifiers to avoid catastrophic backtracking

Pull Request - State: closed - Opened by l0rinc 9 months ago - 3 comments

#256 - Add turbo 16k model for Azure

Pull Request - State: closed - Opened by kartikagrawal2503 9 months ago - 1 comment

#256 - Add turbo 16k model for Azure

Pull Request - State: closed - Opened by kartikagrawal2503 9 months ago - 1 comment

#255 - Simplify byte_pair_merge

Pull Request - State: closed - Opened by hauntsaninja 9 months ago - 1 comment

#255 - Simplify byte_pair_merge

Pull Request - State: closed - Opened by hauntsaninja 9 months ago - 1 comment

#253 - Inline custom mapping function in _byte_pair_merge

Pull Request - State: closed - Opened by hauntsaninja 9 months ago

#253 - Inline custom mapping function in _byte_pair_merge

Pull Request - State: closed - Opened by hauntsaninja 9 months ago

#252 - Avoid calling byte_pair_encode for existing tokens

Pull Request - State: closed - Opened by hauntsaninja 9 months ago

#252 - Avoid calling byte_pair_encode for existing tokens

Pull Request - State: closed - Opened by hauntsaninja 9 months ago

#250 - Enhancement: Add convenience token-counting functions to this package

Issue - State: open - Opened by pamelafox 9 months ago - 4 comments

#250 - Enhancement: Add convenience token-counting functions to this package

Issue - State: open - Opened by pamelafox 9 months ago - 4 comments

#245 - Panic (stack overflow) when encoding a certain string

Issue - State: open - Opened by Crazytieguy 10 months ago - 7 comments

#239 - Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)

Pull Request - State: open - Opened by l0rinc 10 months ago - 4 comments

#238 - Support enterprise network support for self-hosted encodings (UPDATED v0.6.0)

Pull Request - State: open - Opened by blaney83 10 months ago - 1 comment

#238 - Support enterprise network support for self-hosted encodings (UPDATED v0.6.0)

Pull Request - State: open - Opened by blaney83 10 months ago - 1 comment

#234 - 1) Optimize regular expressions used for splitting by ~20%

Pull Request - State: closed - Opened by l0rinc 11 months ago

#234 - 1) Optimize regular expressions used for splitting by ~20%

Pull Request - State: closed - Opened by l0rinc 11 months ago

#231 - Pickling tokenizer fails due to builtins.CoreBPE

Issue - State: closed - Opened by jerheff 11 months ago - 4 comments

#231 - Pickling tokenizer fails due to builtins.CoreBPE

Issue - State: closed - Opened by jerheff 11 months ago - 4 comments

#218 - Plugins found: ['tiktoken_ext.openai_public']

Issue - State: closed - Opened by voghoei 12 months ago - 15 comments

#218 - Plugins found: ['tiktoken_ext.openai_public']

Issue - State: open - Opened by voghoei 12 months ago - 14 comments

#205 - Can't install tiktoken in Python 3.12

Issue - State: closed - Opened by pamelafox about 1 year ago - 7 comments

#205 - Can't install tiktoken in Python 3.12

Issue - State: closed - Opened by pamelafox about 1 year ago - 7 comments

#198 - tiktoken error on AWS lambda

Issue - State: open - Opened by vicky141998 about 1 year ago - 5 comments

#198 - tiktoken error on AWS lambda

Issue - State: open - Opened by vicky141998 about 1 year ago - 5 comments

#190 - Adding a __version__ attribute

Issue - State: closed - Opened by rasbt about 1 year ago - 5 comments

#186 - Replace `<|endoftext|>` with `ENDOFTEXT`

Pull Request - State: closed - Opened by alvarobartt about 1 year ago - 1 comment

#185 - Improve `tiktoken.encoding_for_model` with better option

Pull Request - State: closed - Opened by xhluca about 1 year ago - 1 comment

#185 - Improve `tiktoken.encoding_for_model` with better option

Pull Request - State: closed - Opened by xhluca about 1 year ago - 1 comment

#181 - Make `Encoding` serializable

Pull Request - State: closed - Opened by mariosasko about 1 year ago - 3 comments

#181 - Make `Encoding` serializable

Pull Request - State: closed - Opened by mariosasko about 1 year ago - 3 comments

#179 - Please commit your Cargo.lock file in the repository

Issue - State: open - Opened by charles-dyfis-net about 1 year ago - 3 comments

#179 - Please commit your Cargo.lock file in the repository

Issue - State: open - Opened by charles-dyfis-net about 1 year ago - 3 comments

#169 - tiktoken_ext.__pycache__ Issue On Amazone Lambda

Issue - State: open - Opened by FarzamHejaziK over 1 year ago - 1 comment

#169 - tiktoken_ext.__pycache__ Issue On Amazone Lambda

Issue - State: open - Opened by FarzamHejaziK over 1 year ago - 1 comment

#152 - fix pycache folder being added instead of encoding info files

Pull Request - State: open - Opened by dheerajiiitv over 1 year ago - 9 comments

#152 - fix pycache folder being added instead of encoding info files

Pull Request - State: open - Opened by dheerajiiitv over 1 year ago - 9 comments

#143 - from __future__ import annotations to use dict for type annotation

Pull Request - State: closed - Opened by youkaichao over 1 year ago

#137 - Any introduction about api `encode_with_unstable`?

Issue - State: open - Opened by fseasy over 1 year ago - 2 comments

#137 - Any introduction about api `encode_with_unstable`?

Issue - State: open - Opened by fseasy over 1 year ago - 2 comments

#97 - Unofficial bindings / ports in other languages

Issue - State: open - Opened by hauntsaninja over 1 year ago - 11 comments

#94 - Add support for Pyodide

Issue - State: closed - Opened by Emasoft over 1 year ago - 6 comments

#94 - Add support for Pyodide

Issue - State: closed - Opened by Emasoft over 1 year ago - 6 comments

#93 - fix error message

Pull Request - State: closed - Opened by jonathanagustin over 1 year ago - 1 comment

#93 - fix error message

Pull Request - State: closed - Opened by jonathanagustin over 1 year ago - 1 comment

#80 - ValueError: Unknown encoding cl100k_base

Issue - State: closed - Opened by rembo1992 over 1 year ago - 5 comments

#61 - Unable to find ENCODING_CONSTRUCTORS

Issue - State: closed - Opened by dheerajiiitv over 1 year ago - 6 comments

#61 - Unable to find ENCODING_CONSTRUCTORS

Issue - State: closed - Opened by dheerajiiitv over 1 year ago - 6 comments

#60 - Exporting BPE file to vocab/merges files used by Huggingface tokenizers

Issue - State: closed - Opened by colinclement over 1 year ago - 15 comments

#50 - Add WASM/JS bindings

Pull Request - State: open - Opened by dqbd over 1 year ago - 2 comments

#44 - adds gpt-3.5-turbo and prefix matching to encoding_for_model()

Pull Request - State: closed - Opened by ted-at-openai over 1 year ago - 2 comments

#44 - adds gpt-3.5-turbo and prefix matching to encoding_for_model()

Pull Request - State: closed - Opened by ted-at-openai over 1 year ago - 2 comments

#25 - Could I train this tokenizer using my own data from scratch?

Issue - State: closed - Opened by Zephyr1022 almost 2 years ago - 3 comments