Problems / LLM serving, medium

An LLM judge on every answer

Every answer is graded inline by an LLM judge with 20 slots before it returns. The judge is now the slowest part.

Free to open. Runs in your browser, no sign-up.

Your goals

Success ≥ 99%, p99 ≤ 5 s.

40 requests a second for 30 seconds: 100% request (reads).

What each component does in this design, and what calls it. The simulator mimics each one from these settings.

Component	Its job here	Settings
Load balancerLoad balancer	Traffic enters here.Calls AI app.Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header.	–
AI appApp service	Load balancer forwards to it.Calls GPU serving pool, LLM judge.Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.	3 replicas × 500 slots · timeout 30 s
GPU serving poolSelf-hosted serving pool	AI app calls it.GPU servers batch requests together; throughput rises with batch size until memory runs out.	400 slots
LLM judgeLLM-as-judge	AI app calls it.An LLM judge grades outputs at scale, but it is another model call with its own latency, cost and biases.	20 slots

Try it first. Open one hint at a time if you get stuck.

Hint 1

The judge waits on its own model call.

Hint 2

Slots needed = rate × judge time.

One way to solve it

Give the judge more capacity (or grade a sample asynchronously).