Overview

Build reliable high performance LLM inference on NVIDIA GPUs with Triton and TensorRT LLM from first prototype to multi node production. Running large language models at scale is not just about picking a model. You have to fit massive checkpoints into GPU memory, keep latency predictable under load, ship updates safely, and keep costs under control while traffic patterns change. This book gives you a practical end to end path for doing that with NVIDIA Triton Inference Server and TensorRT LLM. It walks through hardware sizing, engine building, Triton configuration, Kubernetes deployment, observability, autoscaling, and real case studies so you can move from experiments to dependable production services. Understand the LLM inference stack on NVIDIA GPUs and where Triton and TensorRT LLM fit among other runtimes Select model architectures, tokenizers, and checkpoints that are compatible with TensorRT LLM and your hardware budget Build and validate TensorRT LLM engines, including decoder and encoder decoder models with accuracy checks and quantization choices Tune paged KV cache, inflight batching, and advanced parallelism strategies such as tensor, pipeline, and expert parallelism Configure Triton model repositories, backends, dynamic and sequence batching, instance groups, and multi model multi tenant layouts Deploy Triton and TensorRT LLM on Kubernetes with GPU device plugins, scheduling rules, Helm charts, and GitOps based rollouts Operate sharded models across nodes, manage startup and cache warmup, and handle failure modes and recovery patterns Design LLM APIs with streaming token responses, apply gateway level routing, and integrate Triton endpoints into application frameworks Build retrieval augmented generation pipelines on Triton, serving both embedding models and generative models behind consistent endpoints Set up GPU telemetry exporters, Triton metrics, dashboards, and a systematic tuning loop for latency, throughput, and cost Apply concrete playbooks for single node services and cluster scale sharded deployments, including cost modeling and capacity planning The book includes detailed configuration snippets, Kubernetes manifests, and working code samples for Triton clients, RAG components, telemetry exporters, and distributed TensorRT LLM builds, so you can adapt proven patterns instead of starting from scratch. If you want your LLM services on NVIDIA GPUs to be fast, observable, and production ready, grab your copy today.

Full Product Details

9798277387214

Table of Contents

Reviews

Author Information

Tab Content 6

Countries Available