Problems / LLM serving, medium

Too many fine-tunes on one GPU

One multi-LoRA server hosts 200 customer adapters on a shared base model. At 60 requests a second the shared batches overflow.

Free to open. Runs in your browser, no sign-up.

Your goals

Success ≥ 99%, p99 ≤ 5 s.

60 requests a second for 30 seconds: 100% request (reads).

What each component does in this design, and what calls it. The simulator mimics each one from these settings.

Component	Its job here	Settings
Load balancerLoad balancer	Traffic enters here.Calls AI app.Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header.	–
AI appApp service	Load balancer forwards to it.Calls Multi-LoRA server.Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.	3 replicas × 500 slots · timeout 30 s
Multi-LoRA serverMulti-LoRA server	AI app calls it.Multi-LoRA serving runs one base model for many fine-tuned adapters, batching requests for different adapters together.	32 slots

Try it first. Open one hint at a time if you get stuck.

Hint 1

All adapters share one base model’s slots.

Hint 2

Add servers and spread adapters across them.

One way to solve it

Double the multi-LoRA server’s capacity.