Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference

Amazon SageMaker adds prefix‑aware routing to reduce LLM inference latency. The feature routes requests with the same prompt prefix to the same instance, keeping the KV cache warm. Benchmarks on Llama 3.1 70B show a 77 % drop in P50 time‑to‑first‑token and a jump in cache hit rates from 25 % to over 80 %.

Opening Kapyn…