Problems / AI systems, medium

RAG at twice the traffic

Your support assistant doubles to 80 questions a second. The model’s token quota does not.

Free to open. Runs in your browser, no sign-up.

Your goals

Success ≥ 99%, time to first token p99 ≤ 1.5 s.

80 requests a second for 30 seconds: 100% question (reads).

What each component does in this design, and what calls it. The simulator mimics each one from these settings.

Component	Its job here	Settings
Load balancerLoad balancer	Traffic enters here.Calls AI gateway.Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header.	–
AI gatewayApp service	Load balancer forwards to it.Calls Semantic cache (checked first), Embedding model, Vector DB, LLM endpoint.Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.	3 replicas × 200 slots · timeout 30 s
Semantic cacheSemantic cache	AI gateway checks it first.A semantic cache matches questions by meaning using embeddings, not exact text.	25% hit rate
Embedding modelEmbedding model	AI gateway calls it.Embeddings turn text into vectors where similar meanings sit close together.	–
Vector DBVector DB	AI gateway calls it.Approximate indexes like HNSW trade a little recall for much faster search.	–
LLM endpointLLM endpoint	AI gateway calls it.Latency has two parts: time to the first token, then a steady stream of tokens per second.	300 streams · 6M input tokens/min

Try it first. Open one hint at a time if you get stuck.

Hint 1

Look for 429s on the model.

Hint 2

More API servers don’t raise a provider quota.

One way to solve it

Raise the semantic cache hit rate, enable prefix caching, or get more quota.