Problems / AI systems, hard
40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.
Free to open. Runs in your browser, no sign-up.
Keep success ≥ 99% and time to first token p99 ≤ 2 s. Watch the shared embedding quota and the vector index size.
40 requests a second for 50 seconds: 90% ask (reads), 10% upload (writes).
What each component does in this design, and what calls it. The simulator mimics each one from these settings.
| Component | Its job here | Settings |
|---|---|---|
| Load balancerLoad balancer | Traffic enters here.Calls Q&A API.Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header. | – |
| Q&A APIApp service | Load balancer forwards to it.Calls Semantic cache (checked first), Embedding model, Vector DB, Reranker, Answer model, S3 (documents), Ingestion queue (SQS) (publishes).Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s. | 3 replicas × 300 slots · timeout 30 s |
| Semantic cacheSemantic cache | Q&A API checks it first for reads.A semantic cache matches questions by meaning using embeddings, not exact text. | 30% hit rate |
| Embedding modelEmbedding model | Q&A API calls it for reads. Ingestion workers (chunk + embed) calls it.Embeddings turn text into vectors where similar meanings sit close together. | – |
| Vector DBVector DB | Q&A API calls it for reads. Ingestion workers (chunk + embed) calls it.Approximate indexes like HNSW trade a little recall for much faster search. | – |
| RerankerReranker | Q&A API calls it for reads.A reranker reads each retrieved passage together with the question and reorders by relevance. | 48 slots |
| Answer modelLLM endpoint | Q&A API calls it for reads.Latency has two parts: time to the first token, then a steady stream of tokens per second. | 300 streams · 20M input tokens/min |
| S3 (documents)Object storage | Q&A API calls it for writes. Ingestion workers (chunk + embed) calls it.Object stores like S3 keep copies across devices and facilities for very high durability. | 5000 slots · 300,000 calls/min |
| Ingestion queue (SQS)Message queue | Q&A API publishes to it for writes.Calls Ingestion workers (chunk + embed).If a consumer does not finish within the visibility timeout, the message is delivered again. | – |
| Ingestion workers (chunk + embed)Worker | Ingestion queue (SQS) consumes from it.Calls S3 (documents), Embedding model, Vector DB.Queues deliver at least once, so a worker can see the same message twice. | 16 concurrent jobs |