Ecosyste.ms: Issues
An open API service for providing issue and pull request metadata for open source projects.
GitHub / microsoft/Tokenizer issues and pull requests
#51 - docs: update CONTRIBUTING.md
Pull Request -
State: closed - Opened by eltociear 3 months ago
#50 - Bump braces from 3.0.2 to 3.0.3 in /tokenizer_ts
Pull Request -
State: open - Opened by dependabot[bot] 3 months ago
Labels: dependencies
#49 - allow passing in a parsd dictionary to the tokenizer
Pull Request -
State: closed - Opened by connor4312 3 months ago
- 1 comment
#48 - Error: ENOENT: no such file or directory, mkdir '/var/model'
Issue -
State: closed - Opened by sbernadsky 4 months ago
- 1 comment
#47 - Support for text-embedding-3-large
Issue -
State: closed - Opened by veedi 4 months ago
- 1 comment
#46 - Add support for gpt-4o
Pull Request -
State: closed - Opened by shengyfu 4 months ago
#45 - Tokenizer TS Web Browser Compatability
Issue -
State: open - Opened by Wolfleader101 5 months ago
- 1 comment
#44 - bump version to 1.0.6
Pull Request -
State: closed - Opened by sbatten 6 months ago
#43 - TS Optimization: see if there's a minimum length for LRU entries
Issue -
State: closed - Opened by connor4312 6 months ago
- 1 comment
#42 - perf: avoid spread argument when pushing tokens
Pull Request -
State: closed - Opened by connor4312 6 months ago
- 1 comment
#41 - perf: replace lru-cache module with a simpler map
Pull Request -
State: closed - Opened by connor4312 6 months ago
#40 - perf: reduce allocations when encoding text
Pull Request -
State: closed - Opened by connor4312 6 months ago
#39 - TS Optimization: avoid allocations and array slicing in byte pair encoding
Issue -
State: closed - Opened by connor4312 6 months ago
#38 - TS Optimization: avoid allocations when encoding text
Issue -
State: closed - Opened by connor4312 6 months ago
#37 - TS Optimization: make the LRU cache pluggable and faster
Issue -
State: closed - Opened by connor4312 6 months ago
#36 - start on adding start/end to BinaryMap get
Pull Request -
State: closed - Opened by andreamah 6 months ago
- 1 comment
#35 - Add a performance notebook and improve TS performance
Pull Request -
State: closed - Opened by connor4312 6 months ago
- 3 comments
#34 - Don't ship `.map` files
Pull Request -
State: closed - Opened by mjbvz 6 months ago
- 3 comments
#33 - Small optimizations
Pull Request -
State: closed - Opened by mjbvz 6 months ago
- 1 comment
#32 - Speed up `uint8ArrayToString`
Pull Request -
State: closed - Opened by mjbvz 6 months ago
- 1 comment
#31 - ts: initial perf improvements
Pull Request -
State: closed - Opened by connor4312 6 months ago
#30 - Use `exec` for regex matching
Pull Request -
State: closed - Opened by mjbvz 6 months ago
#29 - Fix Important blockquote
Pull Request -
State: closed - Opened by ericstj 6 months ago
#28 - Point to Microsoft.ML.Tokenizers
Pull Request -
State: closed - Opened by ericstj 6 months ago
- 1 comment
#27 - Adding new APIs to avoid passing in allowed special tokens
Pull Request -
State: closed - Opened by shengyfu 9 months ago
#26 - Fix `encodeTrim*` on special strings with repeat tokens
Pull Request -
State: closed - Opened by lramos15 10 months ago
#25 - `encodeTrim*` functions behave weirdly with some (special?) tokens
Issue -
State: closed - Opened by nicocanali 10 months ago
#23 - Fix caching for other APIs as well
Pull Request -
State: closed - Opened by shengyfu 12 months ago
#22 - store cache miss
Pull Request -
State: closed - Opened by sbatten 12 months ago
#21 - remove node-fetch
Pull Request -
State: closed - Opened by sbatten 12 months ago
#20 - Token count accuracy questions with GPT3.5
Issue -
State: closed - Opened by jsypkens about 1 year ago
- 2 comments
#19 - Fix link in README
Pull Request -
State: closed - Opened by zamoshchin about 1 year ago
#18 - Example of how to pre-download the BPE rank file
Issue -
State: closed - Opened by KyleMit about 1 year ago
- 2 comments
#17 - docs: fix syntax error in readme sample code
Pull Request -
State: closed - Opened by KyleMit about 1 year ago
#16 - Bump word-wrap from 1.2.3 to 1.2.4 in /tokenizer_ts
Pull Request -
State: closed - Opened by dependabot[bot] about 1 year ago
Labels: dependencies
#15 - Can I use the SentencePiece Model file?
Issue -
State: closed - Opened by tylike over 1 year ago
- 1 comment
#14 - Update version to 1.3.2
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#13 - Update readme.md
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#12 - Update readme files
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#11 - Update to manual trigger for release workflow
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#10 - Adding lock to allow concurrent access to the cache
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#9 - Create github actions for build/publish tokenizer-ts
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#8 - Adding Typescript implementation of the tiktoken algorithm.
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
- 1 comment
#7 - Fix default commandline args for model name
Pull Request -
State: closed - Opened by nt-7 over 1 year ago
#6 - Refactor: Migrate from deprecated WebClient to async HttpClient
Pull Request -
State: closed - Opened by nt-7 over 1 year ago
- 3 comments
#4 - Add license file to nuget package
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#3 - Add perf benchmark and update nuget package metadata
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#2 - Update nuget feed config to enable azure devops build
Pull Request -
State: closed - Opened by shengyfu over 1 year ago
#1 - Initial check in for open source
Pull Request -
State: closed - Opened by shengyfu over 1 year ago