162 system design problems

Each problem is a working system with something wrong in it: too little capacity, a cache sized badly, retries that amplify an outage, an attack with nothing in front, a model quota too small. Run it, watch what breaks, and fix it until every goal is green. Every problem is tested to fail as given and pass once fixed.

40 easy, 86 medium and 36 hard. Opening a problem is free.

162 problems

Scaling 12

  1. One server on launch dayYour startup runs on a single API server behind a load balancer. Launch-day traffic is 1,500 requests a second.EasySolve
  2. Black Friday spikeA shop handles 1,500 req/s comfortably. At 10 s, Black Friday traffic triples for 15 seconds.EasySolve
  3. CPU-bound APIThe API renders pages on the server: 6 ms of CPU per request on 2 vCPUs per replica, at 1,500 req/s.EasySolve
  4. Database out of CPUEvery read and write goes to one Postgres with 2 vCPUs, at 2,000 req/s. There is no cache.MediumSolve
  5. Write-heavy ledgerA ledger service is 60% writes at 1,500 req/s on a 2-vCPU database. Read replicas won’t help writes.MediumSolve
  6. The slow rampTraffic grows from 800 to 2,400 req/s over the run. The API is sized for today.EasySolve
  7. Bursty trafficAverage traffic is 700 req/s, but every 8 seconds a 2-second burst brings 4× that. The API queue is short.MediumSolve
  8. Autoscaling hits its ceilingThe API runs on a container platform that autoscales from 2 to 3 instances. Traffic needs more.MediumSolve
  9. Losing a replica at peakTwo API replicas run at 70% each. At 10 s one of them dies (a bad host), and the other must take everything.MediumSolve
  10. Search falls overEvery page calls a search index with 32 query slots and 4 ms of CPU per query on 4 vCPUs, at 1,200 req/s.MediumSolve
  11. Every request checks the sessionEach API request validates its session with an auth service that has 40 slots. At 1,500 req/s it can’t keep up.EasySolve
  12. Resizing images on uploadEvery upload is resized synchronously by an image service: 40 ms of CPU each on 4 vCPUs, at 300 uploads a second.MediumSolve

Caching 8

  1. Read-heavy product catalog2,000 req/s, 90% reads through a cache. At 20 s the cache is flushed.EasySolve
  2. Read-heavy without a cacheA product API is 95% reads at 3,000 req/s, all straight to one database.EasySolve
  3. Cold cache after a deployA 90%-hit cache shields Postgres at 2,500 req/s. At 12 s a deploy flushes the cache, and every read hits the database.MediumSolve
  4. A cache too small5 million products, 2 KB each, but the cache has 512 MB. Popular items fit; most don’t.MediumSolve
  5. TTL too shortThe cache is big enough, but entries expire after 2 seconds, before most are read again.MediumSolve
  6. The cache itself is the bottleneckA single Redis node handles at most 1,500 operations a second. Reads arrive at 2,700 a second.HardSolve
  7. Everyone reads something differentPopularity is nearly flat (Zipf 0.4) over 10 million keys, so a small cache barely helps.HardSolve
  8. Flush during a spikeTraffic doubles at 8 s, and at 12 s someone flushes the cache. Two bad things at once.HardSolve

Databases 8

  1. Slow database rescueThe database answers reads in 9 seconds. The API gives up after 5 seconds and retries twice.EasySolve
  2. URL shortener4,000 req/s, 95% redirects. Popular links are far more popular than the rest.MediumSolve
  3. Key-value store out of partitionsA session store spreads keys over 4 partitions that each handle 400 operations a second. Traffic is 2,500 a second.MediumSolve
  4. The 1.5-second queryA missing index makes every read take 1.5 seconds. The API gives up after 1 second and retries.EasySolve
  5. Connection pool exhaustedThe database allows 10 connections. Each query holds one for about 20 ms, at 1,500 queries a second.MediumSolve
  6. Stale reads from replicasRead replicas lag the primary by 2 seconds. Users edit their profile and immediately see the old one.MediumSolve
  7. Lost updates on a hot rowTwo requests read a counter, both add one, both write it back: one increment disappears. Popular rows are updated constantly.HardSolve
  8. One viral keyA URL shortener where one link went viral: its partition melts while the rest are idle.HardSolve

Resilience 5

  1. A rate limit below real trafficA rate limiter in front of the API allows 1,000 req/s. Normal peak traffic is 1,300 req/s, so real users get 429s.EasySolve
  2. Retry stormThe database slows down. The API times out after 300 ms and retries 3 times with no backoff budget, tripling the load.MediumSolve
  3. Timeout shorter than the callThe search backend takes 80–200 ms. The API gives up after 100 ms.EasySolve
  4. Payment provider outageCheckout calls an external payment API. At 10 s it goes down for 15 seconds.MediumSolve
  5. A slow dependency drags everyoneOne downstream service gets slow. API requests pile up waiting on it, and soon even requests that don’t need it fail.HardSolve

Async and streaming 6

  1. Payment webhook processor500 webhooks/s through an at-least-once queue. 0.2% of acknowledgements are lost.MediumSolve
  2. Event pipeline catching up2,000 events/s into a 6-partition stream with 30 s retention. 3 consumers.MediumSolve
  3. Duplicate chargesPayment events go through a queue that delivers at least once. Some acknowledgements are lost, so some customers are charged twice.MediumSolve
  4. Events that never arriveThe API writes the order to the database, then publishes an event. At 10 s the queue is unavailable for 5 seconds: orders are saved but their events vanish.HardSolve
  5. Consumers can’t keep upKafka keeps events for 30 seconds. One consumer processes them, slower than they arrive.EasySolve
  6. Too few partitionsThe topic has 2 partitions. Each partition is read by one consumer at a time, however many you add.HardSolve

Security 5

  1. Edge defence: survive a DDoS1,500 req/s of real users. At 15 s a 20,000 req/s flood hits for 15 s. Nothing stands in front of the load balancer.MediumSolve
  2. A 50,000 req/s floodAt 10 s a volumetric DDoS of 50,000 req/s hits a system with nothing in front of the load balancer.MediumSolve
  3. Bot trafficAt 8 s, 8,000 req/s of bots scrape and spam your API alongside real users.EasySolve
  4. Credential stuffingAttackers replay leaked passwords against your login at 5,000 req/s. Every attempt hits the database.MediumSolve
  5. The WAF blocks real usersA strict WAF rule set blocks 4% of real users as attackers.MediumSolve

AI systems 11

  1. RAG support assistant40 questions/s, 4,000-token prompts, a 6M tokens/min quota and a 25% answer-cache hit rate.MediumSolve
  2. Agent runaway15 tasks/s. The agent makes 8 model and tool calls per task; the model allows 4,000 calls a minute and 120 concurrent streams.HardSolve
  3. LLM gateway with fallback40 chats/s through your AI gateway and a prompt cache to one hosted model. At 20 s the provider has a 15 s outage.MediumSolve
  4. MCP agent platform4 tasks/s. The agent thinks (model) and acts (tools through an MCP gateway) 6 times per task, with vector memory and results in S3.HardSolve
  5. Document Q&A on S3 and a vector DB40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.HardSolve
  6. RAG at twice the trafficYour support assistant doubles to 80 questions a second. The model’s token quota does not.MediumSolve
  7. Semantic cache that never hitsThe semantic cache’s similarity threshold is so strict it hits 5% of the time. Nearly every question reaches the model.EasySolve
  8. The agent that thinks too muchAn agent takes 12 model-and-tool steps per task. Steps multiply load, cost and failures.MediumSolve
  9. Model provider outageYour chat product depends on one model provider. At 20 s it is down for 15 seconds.MediumSolve
  10. Ten steps of MCP toolsAgents call MCP tools 10 times per task. A 1% tool error rate becomes a 10% task failure rate.HardSolve
  11. Uploads starve questionsQuestions and document ingestion share one embedding quota. Uploads double and both paths start failing.HardSolve

Real world 7

  1. YouTube end to end20,000 requests/s routed by path: watch, search, comments and uploads. Metadata cache in front of Vitess MySQL, recommendations, view events into Kafka, uploads to storage and a transcoding queue.HardSolve
  2. Concert ticket dropTickets go on sale at 10 s: traffic jumps 5× for 10 seconds, mostly people refreshing the same few pages.HardSolve
  3. Breaking newsA news article goes viral: traffic ramps to 4× over the run. Pages are cacheable at the edge, but the CDN is tiny.MediumSolve
  4. Chat history on one databaseA messaging app stores every message and reads conversation history from one database: 70% reads at 2,000 req/s.MediumSolve
  5. Game leaderboardMillions of players check the leaderboard after every match; the top of the board is read constantly.MediumSolve
  6. Core banking at month endMonth-end batch traffic doubles writes at 10 s. The database runs hot and crashes under CPU pressure.HardSolve
  7. Flash-sale shopping cartsDuring a flash sale everyone updates the same few product rows (stock counts). Updates are read-then-write.HardSolve

LLM serving 20

  1. Token quota at launchA chat product sends 2,000-token prompts at 50 requests a second. The provider allows 4 million input tokens a minute.EasySolve
  2. Not enough concurrent streamsEach answer streams for about 4 seconds, and the deployment allows 100 concurrent streams. Traffic is 40 requests a second.EasySolve
  3. Provider outage at peakAt 15 s the only model provider goes down for 15 seconds, at 30 chats a second.MediumSolve
  4. Timeouts on long answersAnswers are 800 tokens at 70 tokens a second, about 12 seconds each. The app gives up after 8 seconds.EasySolve
  5. Prompt cache that rarely hitsAn exact-match prompt cache sits in front of the model but hits only 5%: timestamps in the prompt make every prompt unique. The model’s token quota is tight.MediumSolve
  6. Self-hosted GPUs at capacityYou serve an open-weights model on your own GPU pool with 32 slots. Requests take about a second; traffic is 50 a second.EasySolve
  7. GPU autoscaling ceilingThe GPU pool autoscales from 2 to 3 nodes of 16 slots each, but traffic needs more than 3 nodes.MediumSolve
  8. Prefill is the bottleneckServing is split into prefill and decode pools (disaggregated serving). Prompts got longer: the prefill pool can’t keep up, and time to first token suffers.MediumSolve
  9. Decode is the bottleneckDisaggregated serving with a small decode pool: answers got longer and decode slots fill up.MediumSolve
  10. KV cache not reusedA KV-cache tier sits in front of prefill, but only 10% of requests reuse a cached prefix: the system prompt changes per user, so prefixes never match.HardSolve
  11. Draft model can’t keep upSpeculative decoding speeds up answers, but the single draft-model GPU is now the bottleneck at 150 requests a second.MediumSolve
  12. Too many fine-tunes on one GPUOne multi-LoRA server hosts 200 customer adapters on a shared base model. At 60 requests a second the shared batches overflow.MediumSolve
  13. AI gateway key limitAll traffic goes through an AI gateway whose API key allows 1,500 requests a minute. Traffic is 40 a second.EasySolve
  14. PII redaction on the hot pathEvery prompt is scanned by a PII redactor with 2 vCPUs and 10 ms of CPU per prompt, at 400 requests a second.MediumSolve
  15. An LLM judge on every answerEvery answer is graded inline by an LLM judge with 20 slots before it returns. The judge is now the slowest part.MediumSolve
  16. Tracing drops spansEvery request writes a trace to an observability service limited to 2,000 spans a minute. At 60 requests a second, traces are rejected and requests fail with them.EasySolve
  17. Voice assistant: transcription backs upA voice assistant transcribes speech, answers, and speaks the reply. Speech-to-text has 16 slots at 30 utterances a second.MediumSolve
  18. Image generation queueAn image generator takes 4 seconds per image on 40 slots. Users request 15 images a second.EasySolve
  19. Retries into a rate limitThe app retries every failed model call 3 times immediately. When the token quota is hit, retries multiply the 429s.HardSolve
  20. Batch inference used for live trafficLive requests were routed to a batch-inference queue meant for offline jobs: it holds requests for 30+ seconds.HardSolve

RAG and retrieval 16

  1. Query rewriting on every questionAgentic RAG rewrites each question with a small model before retrieval. The rewriter has 8 slots at 80 questions a second.MediumSolve
  2. GraphRAG traversals melt the CPUMulti-hop questions traverse a knowledge graph: 10 ms of CPU each on 8 vCPUs, at 1,200 questions a second.MediumSolve
  3. Hybrid search saturatedRetrieval combines keyword and vector search. The hybrid search service has 40 slots at 1,500 queries a second.EasySolve
  4. Reranker on a single GPUA cross-encoder reranks the top 50 chunks for every question on one GPU with 16 slots, at 300 questions a second.EasySolve
  5. Embedding calls queue upEvery question is embedded first, but the embedding deployment allows 8 concurrent calls at 400 questions a second.EasySolve
  6. Semantic cache too strictA semantic cache in front of a RAG assistant hits 10%: its similarity threshold is 0.99. The model quota can’t take the rest.MediumSolve
  7. Context assembly is slowA context builder stitches retrieved chunks, history and instructions for each question. It has 40 slots at 1,000 questions a second.MediumSolve
  8. OCR before every answerA document assistant OCRs scanned pages on demand: 12 slots, 1.5 seconds per page, 15 pages a second.MediumSolve
  9. RAG answers stop during an outageRetrieval works, but the answer model’s provider is down from 15 s to 30 s.MediumSolve
  10. Too many chunks in the promptThe assistant stuffs 6,000 tokens of chunks into every prompt at 30 questions a second, against an 8-million-token-a-minute quota.MediumSolve
  11. Corrective RAG evaluator bottleneckCorrective RAG grades retrieved chunks with an evaluator model before answering. The evaluator has 20 slots and grades every question.HardSolve
  12. Retrieval traces over quotaEach RAG question writes retrieval traces to an observability API capped at 3,000 a minute, at 80 questions a second.EasySolve
  13. Multi-hop retrieval loopsAn agentic retriever makes up to 8 search hops per question, and each hop hits the search service.HardSolve
  14. Rerank results never reusedA cache in front of the reranker hits 15%, because it is keyed on the raw question text including whitespace and casing.MediumSolve
  15. GraphRAG index shared with ingestionThe GraphRAG index answers questions and absorbs entity extraction from new documents at the same time, at 1,500 operations a second.HardSolve
  16. Scrapers hammer the RAG APIAt 8 s, 3,000 req/s of scrapers start querying your RAG assistant, eating model capacity meant for users.MediumSolve

Agents and tools 22

  1. An agent that takes 12 stepsA support agent takes 12 model-and-tool steps per ticket. Tools fail 1% of the time, and every failure sinks the whole task.EasySolve
  2. MCP gateway quotaEvery tool call goes through an MCP gateway (auth, allow-lists, quotas) limited to 600 calls a minute. Agents make about 18 tool calls a second.EasySolve
  3. One slow tool serverA CRM tool server with 8 slots takes 300 ms per call. Agents call it every step.EasySolve
  4. Code sandboxes run outA data-analysis agent runs Python in sandboxes: 6 sandboxes, about 1.2 s per run, 4 runs per task at 2 tasks a second.MediumSolve
  5. Browser automation bottleneckA web agent drives headless browsers: 16 browsers, about 2.5 s per action, 3 actions per task at 3 tasks a second.MediumSolve
  6. Computer-use desktops are scarceA computer-use agent clicks through a legacy app on virtual desktops: 10 desktops, about 3 s per action, 4 actions per task.MediumSolve
  7. Approvals pile upA finance agent asks a human to approve every payment. Five reviewers take about 20 s each; tasks arrive at 0.5 a second.MediumSolve
  8. Agent memory on a small storeThe agent reads and writes long-term memory every step, on a memory store with 2 slots and 60 ms per call.EasySolve
  9. Same tool call, every stepAgents repeat the same read-only lookups, but the tool-result cache hits 10% because it is keyed on raw arguments with timestamps.MediumSolve
  10. Checkpointing every stepDurable execution saves the agent’s state after every step to a checkpoint store with 10 slots and 50 ms writes.MediumSolve
  11. Policy check on every tool callEvery tool call is authorised by a policy engine with 8 slots and 30 ms per check.EasySolve
  12. Tool search on every stepWith 400 tools, the agent searches for the right few each step. The tool-search service has 6 slots.MediumSolve
  13. Compaction calls overflowLong sessions are compacted every step by a summariser with 20 slots, about 600 ms each.MediumSolve
  14. Agents burn the token quotaEach step sends 3,000 tokens of context. Agents make 24 model calls a second against a 3-million-token-a-minute quota.MediumSolve
  15. Model outage mid-taskThe agent’s model provider goes down for 15 seconds. Every task in flight fails at its next step.MediumSolve
  16. Agent model out of streamsAgents hold a model stream for about 2 s per step. The deployment allows 30 streams; agents need about 50.EasySolve
  17. Partner agent goes offlineYour travel agent books hotels through a partner’s agent over A2A. The partner is down from 15 s to 35 s.MediumSolve
  18. Partner agent at capacityA partner agent you delegate to over A2A handles 20 tasks at once, about 2.5 s each. You send it 12 a second.MediumSolve
  19. Workflow engine overloadDeterministic steps run on a workflow engine with 6 slots (200 ms each) between the agent’s decisions.MediumSolve
  20. Loading skills every stepThe agent loads skills (instructions and scripts) from a registry with 4 slots on every step.EasySolve
  21. Agent jobs run twiceAgent jobs are queued and run by workers. When an acknowledgement is lost, the queue redelivers and the job’s side effect (sending an email) happens twice.HardSolve
  22. Tracing every agent stepEach step writes a trace span to an observability API capped at 1,200 a minute; agents make about 30 steps a second.EasySolve

Multi-agent and harness 12

  1. Subagents swamp the search toolA lead agent fans each question out to 8 parallel subagents, and each subagent searches the web. The search tool has 8 slots.MediumSolve
  2. Multi-agent token billEvery subagent sends 3,000 tokens of context. Eight subagents per question multiply token use past the provider quota.HardSolve
  3. A coding harness that never stopsA long-running coding harness takes 20 turns per task, and a 1% tool failure rate sinks a fifth of tasks.EasySolve
  4. Harness compaction backs upThe harness compacts context on every turn with a summariser that has 4 slots.MediumSolve
  5. Harness progress notesThe harness writes progress notes and a checkpoint after every turn to a store with 6 slots.MediumSolve
  6. Harness test runsA coding harness runs the test suite in a sandbox every turn. 8 sandboxes, about 1.2 s per run.HardSolve
  7. Harness loses its modelA long-running harness depends on one model provider, which is down for 20 seconds mid-run.MediumSolve
  8. The planner model is the bottleneckA planner model plans each step for worker agents, but its deployment allows only 20 streams.MediumSolve
  9. The lead agent over-plansA lead agent re-plans 10 times per task before delegating, and each plan calls a flaky tool.MediumSolve
  10. Fragile subagentsEight subagents each call a tool that fails 1% of the time; any subagent failure fails the question.HardSolve
  11. Research agents share few browsersA deep-research system fans out to 5 subagents that each browse the web; the browser pool has 16 browsers.HardSolve
  12. Harness traces over quotaEvery harness turn is traced to an API capped at 600 spans a minute.EasySolve

AI security and governance 14

  1. Prompt injection on a support botAt 8 s, attackers send 400 messages a second with hidden instructions (“ignore previous instructions…”). Nothing screens them.MediumSolve
  2. Injected instructions in emailAn email agent reads inbound messages and can call tools. Attackers send 200 emails a second containing instructions to forward data.HardSolve
  3. Poisoned web pages in RAGA web-search assistant retrieves pages that contain hidden instructions. 300 poisoned requests a second arrive alongside real users.HardSolve
  4. Malicious issue textA coding harness reads issue descriptions. Attackers file issues with instructions to exfiltrate secrets, 100 a second.HardSolve
  5. Injection filter blocks real usersA strict prompt-injection filter flags 4% of real users as attackers.MediumSolve
  6. Guardrails too strictInput guardrails block 3% of legitimate questions as policy violations.MediumSolve
  7. DDoS on an LLM APIAt 8 s a 20,000 req/s flood hits your public LLM API. Every flood request costs GPU time.MediumSolve
  8. Bots abuse the free tierBots farm free-tier accounts and send 2,000 requests a second to your model.EasySolve
  9. Spend governor set too lowA spend governor caps model calls at 1,200 a minute to protect the budget, but legitimate traffic is 30 a second.EasySolve
  10. Credential stuffing on an AI appAttackers replay leaked passwords against your AI app’s login at 3,000 a second.MediumSolve
  11. Redacting every agent stepEvery agent step passes through a PII redactor that handles 2 calls at a time, about 40 ms each.MediumSolve
  12. Policy service on the chat pathEvery chat checks a data-access policy service with 8 slots and 40 ms per check.EasySolve
  13. Audit logging every model callEvery model call is written to an audit log (SIEM) that accepts 30 writes in flight.MediumSolve
  14. Two attacks at onceA DDoS flood and a prompt-injection campaign arrive together.HardSolve

AI products 16

  1. Support bot launch dayA customer-support bot with a semantic cache, retrieval and a hosted model launches to 60 questions a second. The token quota wasn’t raised.MediumSolve
  2. IDE code completionsInline completions: 400 requests a second, short outputs, and only 64 streams on the completion model.MediumSolve
  3. Meeting summaries time outSummaries are 1,000 tokens at 80 tokens a second; the app times out after 10 seconds.EasySolve
  4. AI search engine retrievalAn AI search engine rewrites the query, runs hybrid search and answers. Hybrid search has 30 slots at 1,200 queries a second.MediumSolve
  5. Translation with per-language adaptersA translation product serves 40 language adapters from one multi-LoRA server with 24 slots.MediumSolve
  6. Content moderation queueEvery post is checked by a moderation model with 30 slots, about 600 ms per post, at 80 posts a second.MediumSolve
  7. Invoice extractionInvoices are OCR’d and extracted on upload: 20 OCR slots, 1.2 s per page, 25 pages a second.EasySolve
  8. Voice replies lagA voice assistant’s text-to-speech has 20 slots and takes about 600 ms per reply at 50 replies a second.MediumSolve
  9. Shopping agent: payments downA shopping agent checks out through an external payment API that is down from 15 s to 30 s.HardSolve
  10. Travel agent: airline partner overloadedA travel agent books flights through an airline’s agent over A2A, which handles 15 tasks at once.MediumSolve
  11. Data analyst agent and the warehouseA data-analyst agent writes SQL against a warehouse that runs 8 queries at once, about 800 ms each.MediumSolve
  12. SRE agent runs too longAn on-call SRE agent investigates incidents in 15 steps of model calls and tool queries.MediumSolve
  13. Medical scribe: redaction bottleneckA medical scribe transcribes visits, redacts PII and drafts notes. The redactor has 1 vCPU and 20 ms per note, at 100 notes a second.HardSolve
  14. Legal research with GraphRAGLegal questions traverse a citation graph: 12 ms of CPU each on 4 vCPUs, at 500 questions a second.HardSolve
  15. Sales agent reads hostile repliesA sales agent reads inbound replies and updates the CRM. A campaign of replies with injected instructions arrives at 150 a second.HardSolve
  16. Video generation farmText-to-video takes about 20 s per clip on 40 GPUs; users request 3 clips a second.MediumSolve