Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / niieani/gpt-tokenizer issues and pull requests

#66 - expose vocabulary size

Issue - State: closed - Opened by pelikhan 13 days ago - 3 comments
Labels: released

#65 - [2.6.0] TS2835: Relative import paths need explicit file extensions

Issue - State: closed - Opened by CryZo 14 days ago - 3 comments
Labels: released

#64 - Add lite token counter

Pull Request - State: closed - Opened by shism2 15 days ago - 2 comments

#63 - fix: initialize encodings array in parts

Pull Request - State: closed - Opened by niieani 19 days ago - 1 comment
Labels: released

#62 - RangeError: Maximum call stack size exceeded on iOS Browsers (Safari & WebKit-based)

Issue - State: closed - Opened by denizcdemirci 22 days ago - 4 comments
Labels: released

#61 - chore: Configure Renovate

Pull Request - State: open - Opened by renovate[bot] about 1 month ago

#60 - chore(deps): update dependency typescript to v5.6.3 - autoclosed

Pull Request - State: closed - Opened by renovate[bot] about 1 month ago - 1 comment

#59 - fix(deps): update dependency js-tiktoken to ^1.0.15

Pull Request - State: closed - Opened by renovate[bot] about 1 month ago - 1 comment
Labels: released

#57 - fix(deps): update dependency gpt-tokenizer to ^2.4.0

Pull Request - State: closed - Opened by renovate[bot] about 2 months ago - 1 comment
Labels: released

#56 - feat: added o1-preview and o1-mini chat completion models

Pull Request - State: closed - Opened by mbukeRepo about 2 months ago - 1 comment
Labels: released

#55 - chore(deps): update github/codeql-action action to v3 - autoclosed

Pull Request - State: closed - Opened by renovate[bot] about 2 months ago

#54 - chore(deps): update actions/setup-node action to v4 - autoclosed

Pull Request - State: closed - Opened by renovate[bot] about 2 months ago

#53 - chore(deps): update dependency @niieani/scaffold to ^1.7.35 - autoclosed

Pull Request - State: closed - Opened by renovate[bot] about 2 months ago - 1 comment

#52 - feat: performance optimizations

Pull Request - State: closed - Opened by niieani 2 months ago - 1 comment
Labels: released

#51 - chore(deps): lock file maintenance

Pull Request - State: open - Opened by renovate[bot] 2 months ago - 1 comment

#50 - feat: significantly improve performance, memory usage & initialization time

Pull Request - State: closed - Opened by niieani 2 months ago - 1 comment
Labels: released

#49 - [Bug] Cannot upgrade to TS 5.6x due to build errors

Issue - State: closed - Opened by corinagum 2 months ago - 7 comments
Labels: released

#48 - empty chat will return 3 tokens

Issue - State: closed - Opened by BennyAlex 2 months ago - 1 comment

#47 - encodeChat: lineToEncode.match is not a function

Issue - State: closed - Opened by BennyAlex 2 months ago - 3 comments

#45 - Gpt 4o mini support?

Issue - State: closed - Opened by semanticist21 4 months ago - 1 comment

#44 - fix(deps): update dependency rfc4648 to ^1.5.3

Pull Request - State: closed - Opened by renovate[bot] 4 months ago - 2 comments
Labels: released

#43 - chore: setup o200k_base encoding

Pull Request - State: closed - Opened by mbukeRepo 6 months ago - 4 comments
Labels: released

#42 - Support New o200k_base

Issue - State: closed - Opened by RedwindA 6 months ago - 1 comment
Labels: released

#41 - Picture tokens

Issue - State: open - Opened by arthurwolf 10 months ago - 3 comments
Labels: help wanted

#40 - vercel/pkg Error with gpt-tokenizer

Issue - State: closed - Opened by Ranork 12 months ago - 1 comment

#39 - the `isWithinTokenLimit` may be broken

Issue - State: closed - Opened by zhouhan760503 12 months ago - 1 comment

#38 - Is the algorithm of this repo consistent with openai tiktoken?

Issue - State: closed - Opened by Zwe1 about 1 year ago - 1 comment

#36 - ChatMessage[] TypeScript warning

Issue - State: closed - Opened by thdoan about 1 year ago - 3 comments
Labels: released

#35 - Huge memory consumsation of isWithinTokenLimit

Issue - State: closed - Opened by aminsol about 1 year ago - 6 comments
Labels: released

#34 - How to set the model in encode / encode generator like encodeChat or encodeChatGenerator?

Issue - State: closed - Opened by MiniSuperDev about 1 year ago - 1 comment
Labels: released

#33 - chore(deps): update actions/checkout action to v4

Pull Request - State: closed - Opened by renovate[bot] about 1 year ago - 1 comment
Labels: released

#32 - Different calculating result between gpt-tokenizer and openai

Issue - State: closed - Opened by Penggeor about 1 year ago - 1 comment

#31 - chore(deps): update dependency tsx to ^3.12.10 - autoclosed

Pull Request - State: closed - Opened by renovate[bot] about 1 year ago - 2 comments

#28 - ReferenceError: Property 'TextDecoder' doesn't exist, js engine: hermes

Issue - State: closed - Opened by YunySun over 1 year ago - 1 comment

#27 - Error: Invariant: Method expects to have requestAsyncStorage, none available

Issue - State: closed - Opened by chenqilin70 over 1 year ago
Labels: invalid

#26 - isWithinTokenLimit fails due to lack of model being provided

Issue - State: closed - Opened by emfastic over 1 year ago - 7 comments
Labels: released

#25 - chore(deps): update dependency @niieani/scaffold to ^1.7.32 - autoclosed

Pull Request - State: closed - Opened by renovate[bot] over 1 year ago - 2 comments

#24 - chore(deps): update yarn to v3.8.5 - autoclosed

Pull Request - State: closed - Opened by renovate[bot] over 1 year ago - 2 comments

#23 - chore(deps): update yarn to v3.5.1 - autoclosed

Pull Request - State: closed - Opened by renovate[bot] over 1 year ago - 2 comments

#22 - Improving documentation about the use of api vs importing function directly

Issue - State: open - Opened by nahumzs over 1 year ago - 2 comments

#21 - Support function calls?

Issue - State: open - Opened by seoker over 1 year ago - 3 comments
Labels: help wanted

#20 - Tokens are slightly different from OpenAI Tokenizer

Issue - State: closed - Opened by nilsreichardt over 1 year ago - 1 comment

#19 - Add support for `gpt-3.5-turbo-0613` and `gpt-3.5-turbo-16k-0613`

Issue - State: closed - Opened by nilsreichardt over 1 year ago - 2 comments
Labels: released

#18 - avoid expensive initialization

Issue - State: closed - Opened by mimoo over 1 year ago - 8 comments
Labels: help wanted, released

#16 - This module is not ready for CJK characters

Issue - State: closed - Opened by mashihua over 1 year ago - 3 comments
Labels: invalid

#15 - encodeChatGenerator undefined?

Issue - State: closed - Opened by congqi09 over 1 year ago - 16 comments
Labels: released

#14 - Dependency Dashboard

Issue - State: open - Opened by renovate[bot] over 1 year ago

#13 - Add gpt-4-32k to mappings

Pull Request - State: closed - Opened by wesbos over 1 year ago - 1 comment

#12 - Loading with script tags doesn't work

Issue - State: closed - Opened by baobabKoodaa over 1 year ago - 1 comment
Labels: released

#11 - comparison to other tokenizers

Issue - State: closed - Opened by transitive-bullshit over 1 year ago - 2 comments
Labels: released

#10 - How can I get prompts_tokens?

Issue - State: closed - Opened by magelikescoke over 1 year ago - 3 comments
Labels: released

#9 - [email protected] token count is smaller than ChatGPT api response

Issue - State: closed - Opened by Webifi over 1 year ago - 1 comment

#8 - feat: complete rewrite to support different models

Pull Request - State: closed - Opened by niieani over 1 year ago - 1 comment
Labels: released

#7 - CDN version is broken

Issue - State: closed - Opened by alejandro5042 over 1 year ago - 2 comments
Labels: released

#6 - Calculated tokens much higher than actual

Issue - State: closed - Opened by Qarj over 1 year ago - 10 comments
Labels: released, released on @beta

#5 - Which encoding is this using?

Issue - State: closed - Opened by ricardomatias over 1 year ago - 3 comments
Labels: released, released on @beta

#4 - Import Issue on codebases with both module and common js files

Issue - State: closed - Opened by gbrlmtrz over 1 year ago - 4 comments
Labels: released

#3 - fix: export `encodeGenerator`

Pull Request - State: closed - Opened by justinsteven over 1 year ago - 1 comment
Labels: released

#2 - feat: add a way to count tokens without encoding the whole text

Pull Request - State: closed - Opened by niieani over 1 year ago

#1 - Configure Renovate - autoclosed

Pull Request - State: closed - Opened by renovate[bot] over 1 year ago