opencsgs/llm-inference issues and pull requests

#100 - The usage introduction of `llm-serve` is not correct in quick_start.md

Issue - State: closed - Opened by depenglee1707 7 months ago
Labels: good first issue

#100 - The usage introduction of `llm-serve` is not correct in quick_start.md

Issue - State: closed - Opened by depenglee1707 7 months ago
Labels: good first issue

#99 - Requested tokens (817) exceed context window of 512

Issue - State: open - Opened by SeanHH86 7 months ago - 3 comments
Labels: bug

#99 - Requested tokens (817) exceed context window of 512

Issue - State: open - Opened by SeanHH86 7 months ago - 3 comments
Labels: bug

#98 - Model inference cross multi-nodes

Issue - State: open - Opened by SeanHH86 7 months ago

#98 - Model inference cross multi-nodes

Issue - State: open - Opened by SeanHH86 7 months ago

#97 - API server startup slow

Issue - State: closed - Opened by SeanHH86 7 months ago - 1 comment
Labels: bug

#97 - API server startup slow

Issue - State: closed - Opened by SeanHH86 7 months ago - 1 comment
Labels: bug

#96 - fix llm-serve list

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#96 - fix llm-serve list

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#95 - refine cli, make cli self-explanatory

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#95 - refine cli, make cli self-explanatory

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#94 - support revision to aviod download latest version of mode

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#94 - support revision to aviod download latest version of mode

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#93 - remove deprecated params: stream

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#93 - remove deprecated params: stream

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#92 - support "revision" in yaml defination

Issue - State: closed - Opened by depenglee1707 7 months ago - 2 comments

#92 - support "revision" in yaml defination

Issue - State: closed - Opened by depenglee1707 7 months ago - 2 comments

#91 - Support streaming in vllm integration

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#91 - Support streaming in vllm integration

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#90 - UI not support static batch

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#90 - UI not support static batch

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#89 - fix issue: loading from local folder

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#89 - fix issue: loading from local folder

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#88 - fix issue: vllm cannot address runtime_env

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#88 - fix issue: vllm cannot address runtime_env

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#87 - vllm cannot address "runtime_env"

Issue - State: closed - Opened by depenglee1707 7 months ago - 1 comment

#87 - vllm cannot address "runtime_env"

Issue - State: closed - Opened by depenglee1707 7 months ago - 1 comment

#86 - Refine description of repo

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#86 - Refine description of repo

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#85 - adopt streaming for ui with text-generation downstream task

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#85 - adopt streaming for ui with text-generation downstream task

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#84 - fix issue: non-support streaming pipeline cannot work when call it as stream

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#84 - fix issue: non-support streaming pipeline cannot work when call it as stream

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#83 - enhance llamacpp integration to share soma logic between streaming and predict

Pull Request - State: closed - Opened by depenglee1707 7 months ago - 1 comment

#83 - enhance llamacpp integration to share soma logic between streaming and predict

Pull Request - State: closed - Opened by depenglee1707 7 months ago - 1 comment

#82 - Refactor streaming

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#82 - Refactor streaming

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#81 - Fix prompt is not string bug

Pull Request - State: closed - Opened by SeanHH86 7 months ago - 1 comment

#81 - Fix prompt is not string bug

Pull Request - State: closed - Opened by SeanHH86 7 months ago - 1 comment

#80 - fix issue: stream generation is slow

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#80 - fix issue: stream generation is slow

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#79 - enhance name of router for comparation scenario

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#79 - enhance name of router for comparation scenario

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#78 - Fix path params issue, make interface consistent

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#78 - Fix path params issue, make interface consistent

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#77 - update log

Pull Request - State: closed - Opened by SeanHH86 7 months ago

#77 - update log

Pull Request - State: closed - Opened by SeanHH86 7 months ago

#76 - Updata logs

Pull Request - State: closed - Opened by SeanHH86 7 months ago

#76 - Updata logs

Pull Request - State: closed - Opened by SeanHH86 7 months ago

#75 - Fix stream without prompt format

Pull Request - State: closed - Opened by SeanHH86 7 months ago

#75 - Fix stream without prompt format

Pull Request - State: closed - Opened by SeanHH86 7 months ago

#74 - fix generate bug for stream api of llamacpp

Pull Request - State: closed - Opened by SeanHH86 7 months ago

#74 - fix generate bug for stream api of llamacpp

Pull Request - State: closed - Opened by SeanHH86 7 months ago

#73 - correct vllm version

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#73 - correct vllm version

Pull Request - State: closed - Opened by depenglee1707 7 months ago

#72 - Failed to load qwen1_5-72b-chat-q5_k_m.gguf

Issue - State: closed - Opened by SeanHH86 7 months ago - 3 comments

#72 - Failed to load qwen1_5-72b-chat-q5_k_m.gguf

Issue - State: closed - Opened by SeanHH86 7 months ago - 3 comments

#71 - add Qwen1.5-72B-GGUF yaml and fix load json input error

Pull Request - State: closed - Opened by SeanHH86 7 months ago - 1 comment

#71 - add Qwen1.5-72B-GGUF yaml and fix load json input error

Pull Request - State: closed - Opened by SeanHH86 7 months ago - 1 comment

#70 - Make scale out policy consistent between deployments

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#70 - Make scale out policy consistent between deployments

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#69 - keep removing deprecated stuff

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#69 - keep removing deprecated stuff

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#68 - Support load Qwen1.5-72B-Chat-GPTQ-Int4 by auto_gptq

Issue - State: open - Opened by SeanHH86 8 months ago - 1 comment
Labels: enhancement

#68 - Support load Qwen1.5-72B-Chat-GPTQ-Int4 by auto_gptq

Issue - State: open - Opened by SeanHH86 8 months ago - 1 comment
Labels: enhancement

#67 - Model streaming API enhancement

Issue - State: closed - Opened by SeanHH86 8 months ago - 2 comments

#67 - Model streaming API enhancement

Issue - State: closed - Opened by SeanHH86 8 months ago - 2 comments

#66 - add streaming API support

Pull Request - State: closed - Opened by SeanHH86 8 months ago

#66 - add streaming API support

Pull Request - State: closed - Opened by SeanHH86 8 months ago

#65 - Enable chat template applied for vllm integration

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#65 - Enable chat template applied for vllm integration

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#64 - update Qwen1.5-72B yaml

Pull Request - State: closed - Opened by SeanHH86 8 months ago

#64 - update Qwen1.5-72B yaml

Pull Request - State: closed - Opened by SeanHH86 8 months ago

#63 - Fix json format issue for "transformerpipeline"

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#63 - Fix json format issue for "transformerpipeline"

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#62 - fix load json data with '\n' failed

Pull Request - State: closed - Opened by SeanHH86 8 months ago

#62 - fix load json data with '\n' failed

Pull Request - State: closed - Opened by SeanHH86 8 months ago

#61 - Remove the original implements for vllm integration

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#61 - Remove the original implements for vllm integration

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#60 - Refactor the solution of vllm integration

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#60 - Refactor the solution of vllm integration

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#59 - Install dependency llama-cpp-python failed

Issue - State: open - Opened by SeanHH86 8 months ago - 4 comments

#59 - Install dependency llama-cpp-python failed

Issue - State: open - Opened by SeanHH86 8 months ago - 4 comments

#58 - remove useless stuff

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#58 - remove useless stuff

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#57 - enable prompt template for gguf format inference

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#57 - enable prompt template for gguf format inference

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#56 - Update ray to 2.9.3

Pull Request - State: closed - Opened by SeanHH86 8 months ago

#56 - Update ray to 2.9.3

Pull Request - State: closed - Opened by SeanHH86 8 months ago

#55 - Expose model generate parameters by API server

Issue - State: open - Opened by SeanHH86 8 months ago

#55 - Expose model generate parameters by API server

Issue - State: open - Opened by SeanHH86 8 months ago

#54 - Enable chat template for huggingface transformer

Pull Request - State: closed - Opened by depenglee1707 8 months ago - 1 comment

#54 - Enable chat template for huggingface transformer

Pull Request - State: closed - Opened by depenglee1707 8 months ago - 1 comment

#53 - Generate incorrect text format when use pipeline defaulttransformers

Issue - State: closed - Opened by SeanHH86 8 months ago - 2 comments

#53 - Generate incorrect text format when use pipeline defaulttransformers

Issue - State: closed - Opened by SeanHH86 8 months ago - 2 comments

#52 - Enhance inference API to support OpenAI style

Issue - State: closed - Opened by SeanHH86 8 months ago - 3 comments
Labels: enhancement

#52 - Enhance inference API to support OpenAI style

Issue - State: closed - Opened by SeanHH86 8 months ago - 3 comments
Labels: enhancement

#51 - enable "use_bettertransformer" and "torch_compile"

Pull Request - State: closed - Opened by depenglee1707 8 months ago

#51 - enable "use_bettertransformer" and "torch_compile"

Pull Request - State: closed - Opened by depenglee1707 8 months ago

GitHub / opencsgs/llm-inference issues and pull requests