Problems / RAG and retrieval, hard
Corrective RAG grades retrieved chunks with an evaluator model before answering. The evaluator has 20 slots and grades every question.
Free to open. Runs in your browser, no sign-up.
Success ≥ 99%, p99 ≤ 5 s.
60 requests a second for 30 seconds: 100% request (reads).
What each component does in this design, and what calls it. The simulator mimics each one from these settings.
| Component | Its job here | Settings |
|---|---|---|
| Load balancerLoad balancer | Traffic enters here.Calls AI app.Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header. | – |
| AI appApp service | Load balancer forwards to it.Calls Hybrid search, Retrieval evaluator, GPU serving pool.Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s. | 3 replicas × 500 slots · timeout 30 s |
| Hybrid searchHybrid search | AI app calls it.Hybrid search combines keyword matching (BM25) with vector similarity. | 400 slots · 2 steps |
| Retrieval evaluatorLLM-as-judge | AI app calls it.An LLM judge grades outputs at scale, but it is another model call with its own latency, cost and biases. | 20 slots |
| GPU serving poolSelf-hosted serving pool | AI app calls it.GPU servers batch requests together; throughput rises with batch size until memory runs out. | 400 slots |
Try it first. Open one hint at a time if you get stuck.
The evaluator is a model call per question.
Follow one question through each retrieval stage; the stage that turns red is the one to fix first.
Give the evaluator more capacity.