# Challenges with Real-time Inference at Scale

**URL:** <https://discuss.huggingface.co/t/challenges-with-real-time-inference-at-scale/140367>\
**Category:** Beginners\
**Created:** [February 12, 2025, 6:34am UTC](https://discuss.huggingface.co/t/challenges-with-real-time-inference-at-scale/140367 "2025-02-12T06:34:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JamesLee2295](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@JamesLee2295](https://discuss.huggingface.co/u/JamesLee2295)\
**Post date:** [February 12, 2025, 6:34am UTC](https://discuss.huggingface.co/t/challenges-with-real-time-inference-at-scale/140367/1 "2025-02-12T06:34:39Z")

</div>

Hello! We’re implementing an AI chatbot that supports real-time customer interactions, but the inference time of our LLM becomes a bottleneck under heavy user traffic. Even with GPU-backed infrastructure, the scaling costs are climbing quickly. Has anyone optimized LLMs for high-throughput applications or found any company provides platforms/services that handle this efficiently? Would love to hear about approaches to reduce latency without sacrificing quality.