triton-inference-server/client issues and pull requests

#821 - How to send self.client._stream._enqueue_request(None) in python through AsyncStreamInfer in c++ client

Issue - State: closed - Opened by lywyue 3 days ago

#820 - Sticky Load Balancing during inference not hitting all the Triton Servers sitting behind the Load Balancer

Issue - State: open - Opened by animikhaich 7 days ago

#819 - [Feature Request] Support using system versions of gRPC, Protobuf, Abseil in CMake

Issue - State: open - Opened by prm-james-hill about 1 month ago

#818 - [Question] Help with Client-Side Batching for Large Requests in Triton

Issue - State: open - Opened by harsh-boloai about 1 month ago

#817 - Fix C/C++ libb64 ABI incompatibility bug

Pull Request - State: closed - Opened by matthewkotila about 1 month ago

#816 - Read back data written in cuda shared memory in python

Issue - State: open - Opened by MaximeDebarbat 2 months ago

#815 - Performance Discrepancy Between Triton Client SDK and perf_analyzer

Issue - State: open - Opened by wensimin 2 months ago - 2 comments

#814 - refactor: Making `tritonclient.http` and `tritonclient.grpc` health checks uniform

Pull Request - State: open - Opened by KrishnanPrash 3 months ago

#813 - Fix L0_https--base SSL issue

Pull Request - State: closed - Opened by yinggeh 3 months ago
Labels: bug

#812 - Update `main` branch post 24.11

Pull Request - State: closed - Opened by mc-nv 3 months ago

#811 - Lock grpcio version

Pull Request - State: closed - Opened by mc-nv 3 months ago - 2 comments

#810 - fix: Partial fix for L0_https

Pull Request - State: closed - Opened by yinggeh 3 months ago
Labels: bug

#809 - Prevent failure if cache is empty

Pull Request - State: closed - Opened by mc-nv 3 months ago

#808 - Removing maven installation as it causes side package installation

Pull Request - State: closed - Opened by mc-nv 3 months ago

#807 - Export RHEL Libs

Pull Request - State: closed - Opened by fpetrini15 3 months ago - 1 comment

#806 - InferenceServerHttpClient::Create() failed with error: std::bad_alloc

Issue - State: open - Opened by megadl 3 months ago

#805 - Enable support for Ubuntu 24.04

Pull Request - State: closed - Opened by mc-nv 3 months ago

#804 - Incorrect grpcio version Python client

Issue - State: open - Opened by tigersoldier 3 months ago

#803 - [GRPC_CLIENT] LoadModel with serialized model raise an error: "Poll failed for model directory" on native windows triton.

Issue - State: open - Opened by FilipLeja 4 months ago

#802 - Build Updates for Ubuntu24.04

Pull Request - State: closed - Opened by nv-anants 4 months ago

#801 - Fix cuda linking bug

Pull Request - State: closed - Opened by matthewkotila 4 months ago

#800 - Revert "Remove unnecessary linking of CUDA Runtime library (#798)"

Pull Request - State: closed - Opened by matthewkotila 4 months ago - 1 comment

#799 - Revert "Remove unnecessary linking of CUDA Runtime library"

Pull Request - State: closed - Opened by fpetrini15 4 months ago

#798 - Remove unnecessary linking of CUDA Runtime library

Pull Request - State: closed - Opened by matthewkotila 4 months ago

#797 - refactor: Remove C shared memory shim. Refine shared memory utils behavior. Add unit test for shared memory

Pull Request - State: closed - Opened by GuanLuo 4 months ago - 3 comments

#796 - The dependency information of the Python package needs to be updated

Issue - State: open - Opened by penguin-wwy 4 months ago

#795 - build: upgrade commons io

Pull Request - State: closed - Opened by jbkyang-nvi 4 months ago - 2 comments

#795 - build: upgrade commons io

Pull Request - State: closed - Opened by jbkyang-nvi 4 months ago - 2 comments

#794 - build: update commons io

Pull Request - State: closed - Opened by jbkyang-nvi 4 months ago

#793 - Enabling Ninja build

Pull Request - State: open - Opened by nicolasnoble 4 months ago

#792 - [Question] numpy 2 support

Issue - State: open - Opened by m3at 4 months ago - 5 comments

#791 - Potential memory leak

Issue - State: open - Opened by prasannagn16 5 months ago - 2 comments

#790 - RHEL SDK

Pull Request - State: closed - Opened by fpetrini15 5 months ago - 1 comment

#790 - RHEL SDK

Pull Request - State: closed - Opened by fpetrini15 5 months ago - 1 comment

#789 - Tritonclient CUDA shared memory set_shared_memory_region_from_dlpack fails with non-set stream for Torch tensor.

Issue - State: open - Opened by piotrm-nvidia 5 months ago

#788 - build: Bump tritonclient deps to grpcio>=1.63.0 and protobuf>=5.26.1,<6

Pull Request - State: closed - Opened by tonyay163 5 months ago - 13 comments

#787 - tritonclient protobuf / grpcio version mismatch

Issue - State: closed - Opened by tonyay163 5 months ago - 1 comment

#787 - tritonclient protobuf / grpcio version mismatch

Issue - State: closed - Opened by tonyay163 5 months ago - 1 comment

#786 - Fix typo

Pull Request - State: open - Opened by MasterYi1024 5 months ago

#785 - genai-perf exception offline-mode

Issue - State: open - Opened by sochiseung 5 months ago

#784 - 原始是byte[]的数据，没法传。会报triton.client.InferenceException: [request id: <id_unknown>] inference input data-type is 'INT8', model expects 'BYTES' for 'xxxxx'

Issue - State: open - Opened by freeleeq 5 months ago - 3 comments

#783 - Re-use client or create new one each time?

Issue - State: open - Opened by datdo-msft 5 months ago

#782 - KV-cache of Tensorrt_llm

Issue - State: open - Opened by huynguyen-bku 5 months ago

#781 - Cancel the requests in streaming case by default when closing the connection

Pull Request - State: open - Opened by Tabrizian 5 months ago - 1 comment

#780 - Use try-with-resources for InferenceServerClient to handle resource management and potential TimeoutException

Issue - State: open - Opened by 1ComputerMaster 6 months ago

#779 - tensorrtllm and vllm backend results are different using genai-perf

Issue - State: open - Opened by upskyy 6 months ago - 1 comment

#778 - Unexpected Behavior: ModelInferRequest Fields Overwritten with Incorrect Values in Triton C++ Client

Issue - State: open - Opened by fighterhit 6 months ago

#777 - Failing with Generic Error message: Failed to obtain stable measurement.

Issue - State: open - Opened by Kanupriyagoyal 6 months ago - 11 comments

#776 - Docs for GRPC Error codes (#775)

Pull Request - State: closed - Opened by indrajit96 6 months ago
Labels: cherry-pick

#775 - Docs for GRPC Error codes

Pull Request - State: closed - Opened by indrajit96 6 months ago

#774 - When using the Java Triton Client, why must INT8 type InputTensors be passed using int_contents? This causes the inference results of passing INT8 Embedding Tensors to the model to always not align with the Python client.

Issue - State: closed - Opened by kebibulaient24 6 months ago

#774 - When using the Java Triton Client, why must INT8 type InputTensors be passed using int_contents? This causes the inference results of passing INT8 Embedding Tensors to the model to always not align with the Python client.

Issue - State: closed - Opened by kebibulaient24 6 months ago

#773 - Redirect GenAI-Perf to new repo

Pull Request - State: closed - Opened by fpetrini15 7 months ago

#772 - GenAI-Perf goodput support

Pull Request - State: closed - Opened by AndyDai-nv 7 months ago - 1 comment

#772 - GenAI-Perf goodput support

Pull Request - State: closed - Opened by AndyDai-nv 7 months ago - 1 comment

#771 - Purge PA from Client Repo

Pull Request - State: closed - Opened by fpetrini15 7 months ago - 1 comment

#770 - Update CMake file configuraion

Pull Request - State: closed - Opened by mc-nv 7 months ago

#769 - Port VLM functionalities from main branch to the refactor branch

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#768 - When custom profile export provided, update CSV/JSON naming

Pull Request - State: closed - Opened by dyastremsky 7 months ago - 1 comment

#768 - When custom profile export provided, update CSV/JSON naming

Pull Request - State: closed - Opened by dyastremsky 7 months ago - 1 comment

#767 - Support input payload generation for tensorrtllm engine

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#767 - Support input payload generation for tensorrtllm engine

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#766 - Small Parsing Fixes

Pull Request - State: closed - Opened by IzzyPutterman 7 months ago

#766 - Small Parsing Fixes

Pull Request - State: closed - Opened by IzzyPutterman 7 months ago

#765 - variable number of images per prompt

Pull Request - State: closed - Opened by mwawrzos 7 months ago

#764 - Updating genAI perf version post 24.07 release

Pull Request - State: closed - Opened by pvijayakrish 7 months ago

#764 - Updating genAI perf version post 24.07 release

Pull Request - State: closed - Opened by pvijayakrish 7 months ago

#763 - Allow client to request subset of ensemble model outputs

Issue - State: open - Opened by Vincouux 7 months ago - 1 comment

#762 - Add tensorrtllm_engine service-kind

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#762 - Add tensorrtllm_engine service-kind

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#761 - Fix: PA unit test failure due to temporary object issue

Pull Request - State: closed - Opened by lkomali 7 months ago - 1 comment

#760 - Update GenAI-Perf README.md

Pull Request - State: closed - Opened by nv-hwoo 7 months ago - 1 comment

#760 - Update GenAI-Perf README.md

Pull Request - State: closed - Opened by nv-hwoo 7 months ago - 1 comment

#759 - Random image format in SyntheticImageGenerator

Pull Request - State: closed - Opened by mwawrzos 7 months ago - 1 comment

#758 - Add documentation for multi-modal support in GenAI-Perf

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#758 - Add documentation for multi-modal support in GenAI-Perf

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#757 - Fix typo in tutorial.md

Pull Request - State: closed - Opened by nv-hwoo 7 months ago - 3 comments

#756 - Support Vision Language Model in GenAI-Perf

Pull Request - State: closed - Opened by nv-hwoo 7 months ago - 3 comments

#756 - Support Vision Language Model in GenAI-Perf

Pull Request - State: closed - Opened by nv-hwoo 7 months ago - 3 comments

#755 - NVCF assets support

Pull Request - State: closed - Opened by mwawrzos 7 months ago

#754 - Support synthetic image generation in GenAI-Perf

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#753 - Add command line arguments for synthetic image generation

Pull Request - State: closed - Opened by nv-hwoo 7 months ago - 3 comments

#752 - Refactor LLM inputs into multiple files

Pull Request - State: closed - Opened by dyastremsky 7 months ago - 3 comments

#752 - Refactor LLM inputs into multiple files

Pull Request - State: closed - Opened by dyastremsky 7 months ago - 3 comments

#751 - Synthetic image generator

Pull Request - State: closed - Opened by mwawrzos 7 months ago

#750 - When JSON parsing fails, return the failed string

Pull Request - State: closed - Opened by dyastremsky 7 months ago

#749 - Support multi-modal input from file for OpenAI Chat Completions

Pull Request - State: closed - Opened by nv-hwoo 7 months ago

#748 - feat: Treat HTTP request completion as final response

Pull Request - State: closed - Opened by nnshah1 7 months ago - 1 comment

#747 - Fix typo

Pull Request - State: closed - Opened by AndyDai-nv 7 months ago - 1 comment

#747 - Fix typo

Pull Request - State: closed - Opened by AndyDai-nv 7 months ago - 1 comment

#746 - Clarify new arguments in documentation

Pull Request - State: closed - Opened by dyastremsky 7 months ago

#746 - Clarify new arguments in documentation

Pull Request - State: closed - Opened by dyastremsky 7 months ago

#745 - Move GenAI-Perf profiling to its own subcommand

Pull Request - State: closed - Opened by dyastremsky 8 months ago

#744 - Update the name of Hugging Face TEI

Pull Request - State: closed - Opened by dyastremsky 8 months ago

#744 - Update the name of Hugging Face TEI

Pull Request - State: closed - Opened by dyastremsky 8 months ago

#743 - Update GAP tutorial of vllm backend

Pull Request - State: closed - Opened by AndyDai-nv 8 months ago

#743 - Update GAP tutorial of vllm backend

Pull Request - State: closed - Opened by AndyDai-nv 8 months ago

#742 - feat: Client-side input shape/element validation

Pull Request - State: open - Opened by yinggeh 8 months ago - 2 comments
Labels: enhancement

#741 - Update version genai-perf to 0.0.4

Pull Request - State: closed - Opened by fpetrini15 8 months ago

#740 - Revert "Update version 0.0.4 (#739)"

Pull Request - State: closed - Opened by fpetrini15 8 months ago

GitHub / triton-inference-server/client issues and pull requests