1Context and scope
People upload documents and ask questions about them. An upload is stored in S3 and queued; ingestion workers read it, chunk it, embed the chunks and write them to a vector database. A question checks a semantic cache, is embedded, finds similar chunks in the vector database, reranks them and asks a model to answer from them. Both paths use the same embedding model and its quota, and the index has grown past the vector database’s memory.
2Goals and non-goals
Goals
- 40 requests a second: 90% questions, 10% uploads.
- Success at least 99%; time to first token p99 at most 2 s.
- Uploaded documents are searchable within seconds, and none are lost.
Non-goals
- Answer quality (chunking strategy and evaluation are separate work).
- Per-user permissions on documents.
- Parsing scanned images (that needs OCR first).
3Capacity estimate
Back-of-the-envelope numbers from the template. The simulation uses the same ones, so you can check them against a run.
| Quantity | Value | How |
|---|---|---|
| Questions reaching the embedding model | ≈ 25/s | 36/s × (1 − 30% semantic-cache hits) |
| Embedding tokens | ≈ 700k/min | (25 questions + 4 uploads)/s × 400 tokens; the quota is 600k/min |
| After a 55% cache hit rate | ≈ 485k/min | under the quota |
| Vector index | ≈ 74 GB | 8M × 1,536 dims × 4 B × 1.5 overhead, on 64 GB replicas |
| Model response time | ≈ 4.8 s | 500 ms to first token + 299 tokens ÷ 70 tokens/s |
| S3 request rate | ≈ 8/s | S3 handles at least 3,500 writes and 5,500 reads a second per prefix |
4The design
Each component in the template, in the order a request meets it, and why it is there.
- Q&A API
Routes questions down the answer path and uploads down the ingestion path.
- Semantic cache
Returns a stored answer for a question similar enough to one already answered: no embedding, search or model call.
- Embedding model
Turns text into vectors, for questions and for document chunks. One model, one quota, shared by both paths.
- Vector DB
Finds the chunks closest to the question (an approximate nearest-neighbour index such as HNSW). Fast only while the index fits in memory.
- Reranker
Reorders the top chunks with a more precise model so the best ones reach the prompt.
- Answer model
Writes the answer from the chunks. Its time to first token is what users feel.
- S3, ingestion queue and workers
S3 keeps the original documents; the queue lets uploads return at once while workers chunk and embed at their own pace, retrying on failure.
5Key concepts
Two paths, one quota
Ingestion bursts (a customer uploads a thousand files) eat the embedding quota that questions need. Separate quotas or keys for ingestion, or rate-limited workers, keep questions fast.
Embeddings can’t fall back
Vectors from a different embedding model don’t match the ones in your index, so a backup embedding provider would return nonsense. Protect the embedding path with caching and quota instead, and re-embed everything if you ever switch models.
Index memory
Vector indexes like HNSW are fast because they live in memory. Index size ≈ vectors × dimensions × 4 bytes × overhead. Past the memory size, queries slow down. Shard across replicas, use fewer dimensions, or quantize.
Queue-based ingestion
The upload returns once the file is in S3 and a message is queued. Workers take messages; if one fails, SQS delivers it again after the visibility timeout. Make ingestion idempotent so a redelivery doesn’t duplicate chunks.
6Alternatives considered
- A higher semantic cache hit rate saves calls but risks answering a different question with an old answer; tune the similarity threshold.
- Bigger embedding dimensions can improve recall but grow the index and its memory cost.
- A reranker improves which chunks reach the model but adds latency to every question.
- Synchronous ingestion shows results at once but makes uploads slow and fragile; queued ingestion is eventually consistent.
7Failure modes and fixes
What goes wrong in the simulation, why, and the change that fixes it. Each one shows up as a finding in the lab; ask “what broke?” to see it, and for the fix when you want it.
Questions and uploads fail with embedding 429s
WhyAbout 700k tokens a minute against a 600k quota, shared by both paths.
FixRaise the semantic cache hit rate (the suggested fix), give ingestion its own quota, or ask for more.
Vector index larger than memory
Why8M vectors need about 74 GB per replica; replicas have 64 GB.
FixReplicas with more memory, sharding, or smaller or quantized vectors.
Ingestion backlog on a bulk upload
WhyWorkers retry on 429s and the queue grows.
FixMore workers only helps if the quota allows; rate-limit ingestion and give it a separate quota.
8Try it in the lab
Open the template, then type these into the copilot one at a time. The first few are also offered as buttons when the template opens.
what broke?Embedding quota errors on both paths, and the vector index over memory.
semantic cache hit rate 55%Fewer embedding calls: the 429s stop and success passes.
set ingestion workers to 32Ingestion drains faster, but uses more of the embedding quota.
stress testShows which of the embedding quota, vector DB or model gives out first.
9Check yourself
- Why does a bulk upload slow down questions?
- Why can’t the embedding model fall back to another provider the way the answer model can?
- How big is the index for 8M vectors of 1,536 dimensions, and what are three ways to shrink it?
- What happens to an upload if a worker crashes halfway through ingesting it?