microsoft/DeepSpeed-MII issues and pull requests

#444 - server crashed for some reason, unable to proceed

Issue - State: open - Opened by Archmilio 11 months ago - 1 comment

#443 - [BUG] Issue serving Mixtral 8x7B on H100

Issue - State: open - Opened by Rogerwyf 11 months ago - 9 comments

#442 - qwen1.5 model Support?

Issue - State: open - Opened by musexiaoluo 11 months ago - 3 comments

#441 - On M3 Pro Macbook having issues with installation

Issue - State: closed - Opened by HariKunapareddy 11 months ago - 2 comments

#440 - [NEED HELP] Quantization inference

Issue - State: open - Opened by freQuensy23-coder 11 months ago - 2 comments

#439 - Quantization inference

Issue - State: open - Opened by freQuensy23-coder 11 months ago - 1 comment

#438 - What is the exact meaning of forward tokens?

Issue - State: open - Opened by frankxyy 11 months ago

#437 - Workarounds for pre-Ampere devices

Issue - State: open - Opened by jinhachung 12 months ago - 1 comment

#436 - Kernel execution error with long context length

Issue - State: open - Opened by qiangxu1996 12 months ago

#435 - Can DeepSpeed-MII inference on multi gpus with only 1 replica?

Issue - State: open - Opened by gujingit 12 months ago - 2 comments

#434 - Update version.txt after 0.2.3 release

Pull Request - State: closed - Opened by mrwyattii 12 months ago

#433 - Add quantization config option

Pull Request - State: closed - Opened by mrwyattii 12 months ago

#432 - ValueError: Unsupported model type roberta

Issue - State: open - Opened by pradeepdev-1995 12 months ago - 2 comments

#431 - MII Example shows that mii is "Slower" than Baseline!

Issue - State: open - Opened by Weigaa 12 months ago

#430 - How to use DeepSpeed-MII to deploy a LLM model from DeepSpeed/Megatron-DeepSpeed trained checkpoints?

Issue - State: open - Opened by Jye-525 12 months ago - 2 comments

#429 - Update model support

Pull Request - State: closed - Opened by mrwyattii 12 months ago

#427 - Requests.exceptions.ConnectionError:

Issue - State: closed - Opened by Weigaa 12 months ago - 4 comments

#426 - Speeding up loading in inference checkpoints

Issue - State: open - Opened by amritap-ef 12 months ago - 2 comments

#425 - Add support for Gemma models

Issue - State: open - Opened by lullabies777 12 months ago - 1 comment

#424 - When I start server, after loading model, I got an error of 'grpc.aio._call.AioRpcError'

Issue - State: closed - Opened by zzz0906 12 months ago - 5 comments

#423 - Pydantic v2 migration

Pull Request - State: closed - Opened by mrwyattii 12 months ago - 2 comments

#422 - why all-reduce takes lots of time for mixtral which is quite larger than that of vllm and tensorrt-llm

Issue - State: open - Opened by Eutenacity 12 months ago

#421 - Remove references to --extra-index-url in MII repo

Pull Request - State: closed - Opened by loadams 12 months ago

#420 - How to set trust_remote_code=True in pipeline

Issue - State: open - Opened by gujingit 12 months ago - 2 comments

#419 - Fp6 eta

Issue - State: open - Opened by nivibilla 12 months ago - 2 comments

#418 - Use of dtype in the mii fastgen

Issue - State: open - Opened by gangooteli 12 months ago - 1 comment

#417 - How does GPT2/Bert models utilize continuous batching feature in MII?

Issue - State: open - Opened by Jye-525 12 months ago - 1 comment

#416 - Is the DeepSpeed-MII will support habana (HPU) hardware?

Issue - State: open - Opened by muhammad-asn 12 months ago - 2 comments

#415 - Add `accelerate` to requirements to improve MII-legacy model load times

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#414 - Add test for loading from local dir

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#413 - Update version.txt after 0.2.2 release

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#412 - add stable diffusion CI workflow

Pull Request - State: open - Opened by mrwyattii about 1 year ago - 1 comment

#411 - Disable model check in UT

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#410 - Add support for inpainting task in DS-MII

Pull Request - State: closed - Opened by gauravrajguru about 1 year ago

#409 - fix: Fixed the issue where the mii.pipeline.pipe(stop) was ineffective

Pull Request - State: closed - Opened by kitstar about 1 year ago - 2 comments

#408 - Fix for missing EOS token

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#407 - text2img task to support negative prompts

Pull Request - State: closed - Opened by gauravrajguru about 1 year ago

#406 - How to generate multiple responses in one time?

Issue - State: open - Opened by yangzhch6 about 1 year ago - 1 comment

#405 - TypeError: expected Tensor as element 0 in argument 0, but got bool

Issue - State: closed - Opened by SiriusWy about 1 year ago - 1 comment

#404 - Update version.txt after 0.2.1 release

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#403 - Improve recovery from KV cache starvation

Pull Request - State: closed - Opened by tohtana about 1 year ago - 1 comment

#402 - The inference result is inconsistent with hf

Issue - State: open - Opened by bao-xiaoyi about 1 year ago - 1 comment

#401 - Fix generate output order

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#400 - RuntimeError: server crashed for some reason, unable to proceed

Issue - State: closed - Opened by bao-xiaoyi about 1 year ago - 2 comments

#399 - result is empty

Issue - State: closed - Opened by bao-xiaoyi about 1 year ago

#398 - ModuleNotFoundError: No module named 'mii'

Issue - State: closed - Opened by bao-xiaoyi about 1 year ago - 2 comments

#397 - Readable token streaming support

Pull Request - State: closed - Opened by greshilov about 1 year ago - 2 comments

#396 - Support for repetition penalty during inference with sampling

Issue - State: open - Opened by nischith-sarvam about 1 year ago - 1 comment

#395 - Benchmark:Performance is lower than vllm

Issue - State: open - Opened by zhaotyer about 1 year ago - 1 comment

#394 - Fix recovery from deadlock

Pull Request - State: closed - Opened by tohtana about 1 year ago

#393 - Unable to run inference on free tier Colab.

Issue - State: closed - Opened by sudhir2016 about 1 year ago - 4 comments

#392 - Update CI workflows

Pull Request - State: closed - Opened by loadams about 1 year ago

#391 - Update landing page

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#390 - did "mii.pipeline" support float16?

Issue - State: closed - Opened by wangrendong-yition about 1 year ago - 3 comments

#389 - support for mixtral family ?

Issue - State: open - Opened by S-Yacer about 1 year ago - 8 comments

#388 - How to eliminate deadlock problem?

Issue - State: open - Opened by BaiStone2017 about 1 year ago - 1 comment

#387 - import mii not working

Issue - State: open - Opened by pradeepdev-1995 about 1 year ago - 5 comments

#386 - Error on unknown generate fields

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#385 - `ValueError: channels must be divisible by 8` when new special tokens are added

Issue - State: open - Opened by s-jse about 1 year ago - 4 comments

#384 - Make the order of outputs the same as the order of inputs when using `mii.pipeline`

Pull Request - State: closed - Opened by s-jse about 1 year ago - 1 comment

#383 - RuntimeError: The server socket has failed to listen on any local network address. The server socket has failed to bind to [::]:29700 (errno: 98 - Address already in use).

Issue - State: open - Opened by Chenhzjs about 1 year ago - 2 comments

#382 - Update version.txt after 0.2.0 release

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#381 - Error: "Only able to place X replicas, but Y replicas were requested"

Issue - State: open - Opened by spring1915 about 1 year ago - 2 comments

#380 - Update required DS version

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#379 - deep speed parallel erro

Issue - State: open - Opened by king-shao about 1 year ago - 1 comment

#378 - Test

Pull Request - State: closed - Opened by deas23 about 1 year ago

#377 - Improve efficiency of scheduling and token sampiling

Pull Request - State: closed - Opened by tohtana about 1 year ago

#376 - Improve efficiency of ragged batching scheduler

Pull Request - State: closed - Opened by tohtana about 1 year ago - 1 comment

#375 - Remove inefficient loop in TopP logits processor

Pull Request - State: closed - Opened by tohtana about 1 year ago - 1 comment

#374 - Mistral 8*7B Out of memory

Issue - State: open - Opened by byerose about 1 year ago - 1 comment

#373 - Add model support unit test

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#372 - Make generate params pydantic model

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#371 - pydantic.errors.PydanticUserError

Issue - State: closed - Opened by ArlanCooper about 1 year ago - 2 comments

#370 - Restrict when legacy unit tests are run

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#369 - fix address already in use error on UT

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#368 - How to get the logit tensor of generated text?

Issue - State: open - Opened by randomx207 about 1 year ago - 4 comments

#367 - fix bug when mii_config is None

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#366 - I wonder if we can use batch inference and offload in mii pipeline ?

Issue - State: open - Opened by Kevin-shihello-world about 1 year ago - 2 comments

#365 - for loop calling Non Persistent Pipeline will cause Deadlock

Issue - State: open - Opened by CxsGhost about 1 year ago - 1 comment

#364 - Add restful_api_host into server args.

Pull Request - State: closed - Opened by sarattha about 1 year ago

#363 - one of mii.client() Options, ignore_eos doesn't work

Issue - State: closed - Opened by BaiStone2017 about 1 year ago

#362 - restful_api_host did not use in anywhere

Issue - State: open - Opened by Bhurinut about 1 year ago - 1 comment

#361 - When running mii.serv, it keeps print waiting for server to start.

Issue - State: closed - Opened by cninnovationai about 1 year ago - 6 comments

#360 - Update supported models list

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#359 - Add pipeline unit tests

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#358 - Problem while running facebook/opt-125m with MII

Issue - State: closed - Opened by wangtianxia-sjtu about 1 year ago - 2 comments

#357 - Can MII support quanted Llama2 of AWQ?

Issue - State: closed - Opened by janelu9 about 1 year ago

#356 - Reproduced readme results

Issue - State: open - Opened by Traveller2001 about 1 year ago - 8 comments

#355 - Update version.txt after 0.1.3 release

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#354 - Deployment in kubernetes

Issue - State: open - Opened by nani1149 about 1 year ago

#353 - mixtral support

Issue - State: closed - Opened by martinshkreli about 1 year ago - 2 comments
Labels: enhancement

#352 - Loosen unit test performance assert

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#351 - Can you support DeepSeek's inference acceleration? Thank you very much.

Issue - State: open - Opened by joyhhheee about 1 year ago - 3 comments

#350 - Fix for error messages in persistent deployment

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#349 - Error messages spilled from persistent deployment for every request

Issue - State: closed - Opened by weiqisun about 1 year ago - 6 comments

#348 - Add RESTful API option for host

Pull Request - State: closed - Opened by mrwyattii about 1 year ago

#347 - How to stream tokens?

Issue - State: open - Opened by mevince about 1 year ago - 1 comment
Labels: enhancement

#346 - The choice of the split size for splitAndFuse

Issue - State: open - Opened by frankxyy about 1 year ago - 4 comments

#345 - Can deepspeed-MII run on AMD GPU?

Issue - State: open - Opened by sunpian1 about 1 year ago - 2 comments
Labels: enhancement

#344 - restful api host need configuration

Issue - State: closed - Opened by cableyang about 1 year ago - 1 comment
Labels: enhancement

GitHub / microsoft/DeepSpeed-MII issues and pull requests