Trend Feed: Inference & Serving

Daily items the curator mapped to this module.

Tue, Sep 15, 2026

D1 · Inference & Servingintermediate · 2 min

Show HN: Self-adjusting vLLM at production scale

Surfaced from Hacker News.
Open the source to read the details.

Why it matters. Relevant to Inference & Serving. Matched Inference & Serving on: vllm. Read it through that lens.

Sun, Sep 13, 2026

D1 · Inference & Servingintermediate · 2 min

The Local LLM community feels like the golden era of the internet all over again

Lately because of the current hardware shortage, unfortunately or fortunately, we can’t just throw infinite cloud compute at our problems, but we’re forced to actually care about what’s happening under the hood.
We’re tweaking inference engines, learning quantization math, and optimizing architecture just to squeeze as much performance as possible for the lowest possible setups.
Fact: Just recently, the forked llama.cpp(s) and halogen-flash-server of Strix Halo pushed the performance through the roof, achieving double performance in decode (52tok/s), 5-6x performance in prefill (1300tok/s) for

Why it matters. Relevant to Inference & Serving. Matched Inference & Serving on: prefill, decode. Read it through that lens.

Wed, Sep 9, 2026

D1 · Inference & Servingintermediate · 2 min

Why the hell is LM Studio making LM Studio so difficult to download?

Who is the marketing genius at LM Studio that decided that going ALL IN on pushing their new Bionic Agent product meant they are going to make it a giant pain in the ass to find and download actual LM Studio.
This is the dumbest marketing decision I’ve ever seen.
I used to love LM Studio, it was the middle stepping stone in the logical progression of inference.

Why it matters. Relevant to Inference & Serving. Matched Inference & Serving on: vllm. Read it through that lens.

Thu, Aug 27, 2026

D1 · Inference & Servingintermediate · 2 min

Four reproducible vLLM parser failures that return 200 with the wrong tool call

Surfaced from Hacker News.
Open the source to read the details.

Why it matters. Relevant to Inference & Serving. Matched Inference & Serving on: vllm. Read it through that lens.

Sun, Aug 23, 2026

D1 · Inference & Servingintermediate · 2 min

I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens

What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 p…
One clean run is about $36 of GPU time.
Left warm, it is $1,363 a day.

Why it matters. Relevant to Inference & Serving. Matched Inference & Serving on: vllm, decode, ttft. Read it through that lens.

Mon, Jul 27, 2026

D1 · Inference & Servingintermediate · 2 min

Kimi K3 on vLLM: Up to 370 Tokens/sec

Surfaced from Hacker News.
Open the source to read the details.

Why it matters. Relevant to Inference Internals & Performance. Matched Inference Internals & Performance on: vllm. Read it through that lens.

Tue, Jul 14, 2026

D1 · Inference & Servingintermediate · 2 min

Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)

Surfaced from Hacker News.
Open the source to read the details.

Why it matters. Relevant to Inference Internals & Performance. Matched Inference Internals & Performance on: latency. Read it through that lens.

Wed, Jun 24, 2026

D2 · Model Efficiencyadvanced · 3 min

A faster exact-verification speculative decoder lands in a popular serving stack

A new draft-model scheme raises the token acceptance rate, so more tokens are verified per target pass.
It uses exact verification, so output stays distributionally identical to the target model.
Reported speedups are largest at low batch sizes, where decode is most bandwidth-bound.

Why it matters. Speculative decoding cuts latency without touching quality when verification is exact — but the win shrinks as batch size grows and spare FLOPs disappear. Know your serving regime before adopting it.

Quick check: Why does exact-verification speculative decoding not change output quality?

Correct: A rejection-sampling verification step provably reproduces the target model's distribution
The draft model is quantized to match the target
It only runs when the prompt is cached
It averages the draft and target logits

Exact methods accept/reject draft tokens so the final distribution equals the target's; the draft only affects speed via acceptance rate.