Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / microsoft/Tokenizer issues and pull requests

#51 - docs: update CONTRIBUTING.md

Pull Request - State: closed - Opened by eltociear 3 months ago

#50 - Bump braces from 3.0.2 to 3.0.3 in /tokenizer_ts

Pull Request - State: open - Opened by dependabot[bot] 3 months ago
Labels: dependencies

#49 - allow passing in a parsd dictionary to the tokenizer

Pull Request - State: closed - Opened by connor4312 3 months ago - 1 comment

#48 - Error: ENOENT: no such file or directory, mkdir '/var/model'

Issue - State: closed - Opened by sbernadsky 4 months ago - 1 comment

#47 - Support for text-embedding-3-large

Issue - State: closed - Opened by veedi 4 months ago - 1 comment

#46 - Add support for gpt-4o

Pull Request - State: closed - Opened by shengyfu 4 months ago

#45 - Tokenizer TS Web Browser Compatability

Issue - State: open - Opened by Wolfleader101 5 months ago - 1 comment

#44 - bump version to 1.0.6

Pull Request - State: closed - Opened by sbatten 6 months ago

#43 - TS Optimization: see if there's a minimum length for LRU entries

Issue - State: closed - Opened by connor4312 6 months ago - 1 comment

#42 - perf: avoid spread argument when pushing tokens

Pull Request - State: closed - Opened by connor4312 6 months ago - 1 comment

#41 - perf: replace lru-cache module with a simpler map

Pull Request - State: closed - Opened by connor4312 6 months ago

#40 - perf: reduce allocations when encoding text

Pull Request - State: closed - Opened by connor4312 6 months ago

#38 - TS Optimization: avoid allocations when encoding text

Issue - State: closed - Opened by connor4312 6 months ago

#36 - start on adding start/end to BinaryMap get

Pull Request - State: closed - Opened by andreamah 6 months ago - 1 comment

#35 - Add a performance notebook and improve TS performance

Pull Request - State: closed - Opened by connor4312 6 months ago - 3 comments

#34 - Don't ship `.map` files

Pull Request - State: closed - Opened by mjbvz 6 months ago - 3 comments

#33 - Small optimizations

Pull Request - State: closed - Opened by mjbvz 6 months ago - 1 comment

#32 - Speed up `uint8ArrayToString`

Pull Request - State: closed - Opened by mjbvz 6 months ago - 1 comment

#31 - ts: initial perf improvements

Pull Request - State: closed - Opened by connor4312 6 months ago

#30 - Use `exec` for regex matching

Pull Request - State: closed - Opened by mjbvz 6 months ago

#29 - Fix Important blockquote

Pull Request - State: closed - Opened by ericstj 6 months ago

#28 - Point to Microsoft.ML.Tokenizers

Pull Request - State: closed - Opened by ericstj 6 months ago - 1 comment

#27 - Adding new APIs to avoid passing in allowed special tokens

Pull Request - State: closed - Opened by shengyfu 9 months ago

#26 - Fix `encodeTrim*` on special strings with repeat tokens

Pull Request - State: closed - Opened by lramos15 10 months ago

#23 - Fix caching for other APIs as well

Pull Request - State: closed - Opened by shengyfu 12 months ago

#22 - store cache miss

Pull Request - State: closed - Opened by sbatten 12 months ago

#21 - remove node-fetch

Pull Request - State: closed - Opened by sbatten 12 months ago

#20 - Token count accuracy questions with GPT3.5

Issue - State: closed - Opened by jsypkens about 1 year ago - 2 comments

#19 - Fix link in README

Pull Request - State: closed - Opened by zamoshchin about 1 year ago

#18 - Example of how to pre-download the BPE rank file

Issue - State: closed - Opened by KyleMit about 1 year ago - 2 comments

#17 - docs: fix syntax error in readme sample code

Pull Request - State: closed - Opened by KyleMit about 1 year ago

#16 - Bump word-wrap from 1.2.3 to 1.2.4 in /tokenizer_ts

Pull Request - State: closed - Opened by dependabot[bot] about 1 year ago
Labels: dependencies

#15 - Can I use the SentencePiece Model file?

Issue - State: closed - Opened by tylike over 1 year ago - 1 comment

#14 - Update version to 1.3.2

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#13 - Update readme.md

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#12 - Update readme files

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#11 - Update to manual trigger for release workflow

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#10 - Adding lock to allow concurrent access to the cache

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#9 - Create github actions for build/publish tokenizer-ts

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#8 - Adding Typescript implementation of the tiktoken algorithm.

Pull Request - State: closed - Opened by shengyfu over 1 year ago - 1 comment

#7 - Fix default commandline args for model name

Pull Request - State: closed - Opened by nt-7 over 1 year ago

#6 - Refactor: Migrate from deprecated WebClient to async HttpClient

Pull Request - State: closed - Opened by nt-7 over 1 year ago - 3 comments

#4 - Add license file to nuget package

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#3 - Add perf benchmark and update nuget package metadata

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#2 - Update nuget feed config to enable azure devops build

Pull Request - State: closed - Opened by shengyfu over 1 year ago

#1 - Initial check in for open source

Pull Request - State: closed - Opened by shengyfu over 1 year ago