119 components, all simulated

Spreads requests across replicas.

Did you know Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header.

Rule of thumb Least-connections beats round-robin when request times vary a lot.

Breaks when Forwards everything; the tier behind it is what breaks.

Token bucket in front of a service.

Did you know Most limiters use a token bucket: tokens refill at the rate, and the bucket size is the burst you allow.

Rule of thumb Return 429 with a Retry-After header so well-behaved clients back off.

Breaks when Rejects real users when the limit sits below peak traffic.

Serves cacheable reads at the edge; storage, TTL and popularity decide the hit rate.

Did you know A CDN keeps copies at edge locations near users, so a cache hit skips the trip to your origin.

Rule of thumb Purge by path or tag. Purging everything sends every request to the origin at once.

Breaks when Miss storms after a purge; origin overload.

Replicas with slots, CPU, timeouts, retries.

Did you know Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.

Rule of thumb Keep retries under about 10% of calls, or a slow dependency turns into a retry storm.

Breaks when Slot and queue exhaustion; retries amplify a slow dependency; crash on CPU.

Consumes a queue or stream.

Did you know Queues deliver at least once, so a worker can see the same message twice.

Rule of thumb Make handlers idempotent: record a processed-message id before doing side effects.

Breaks when Duplicate side effects without idempotency; backlog when slower than producers.

Routing, auth checks and per-key limits at the door.

Did you know Gateways centralise auth, rate limits and routing so each service does not reimplement them.

Rule of thumb Every request pays the gateway's latency: keep its plugins lean.

Breaks when Adds latency to every call; its own quota rejects real users.

Resolves names; failover and geo routing.

Did you know DNS answers are cached for their TTL, so a change can take that long to reach everyone.

Rule of thumb Lower the TTL a day before a planned failover.

Breaks when Slow resolution; wrong region during failover.

Sidecar proxy: retries, mTLS and tracing between services.

Did you know A mesh runs a sidecar proxy beside each service to handle mTLS, retries and telemetry.

Rule of thumb Retries in the mesh and in the app multiply: configure them in one place.

Breaks when Sidecar latency on every hop; retry storms.

Fails fast when a dependency keeps failing, then retries after a cooldown.

Did you know A breaker trips after too many failures and fails fast, giving the dependency room to recover.

Rule of thumb Pair it with a fallback, such as cached data, so users still see something useful.

Breaks when Trips too early (flapping) or too late (cascade).

Token checks, sessions and sign-in.

Did you know Short-lived access tokens plus refresh tokens limit the damage if a token leaks.

Rule of thumb Verify tokens locally with a public key instead of calling the auth service each time.

Breaks when Every request waits on it; a slow token check slows the whole site.

A third-party API with its own latency, quota and outages.

Did you know Third-party APIs have their own rate limits and outages that you cannot fix.

Rule of thumb Always set a timeout, retry with backoff, and cache what you can.

Breaks when Quota 429s; outage takes your feature down with it.

Card authorisations; slow, must never run twice.

Did you know Payment providers accept an idempotency key so a retried charge is not taken twice.

Rule of thumb Treat webhooks as the source of truth: they can arrive late, twice or out of order.

Breaks when Duplicate charges on retry; declines counted as errors.

Push, email and SMS fan-out.

Did you know Email, SMS and push providers all throttle senders and can delay delivery.

Rule of thumb Queue notifications and send them in the background; never block a user request on them.

Breaks when Provider quota; bursts back up.

Long-lived connections for realtime updates.

Did you know WebSockets hold a connection per client, so memory grows with connected users, not requests.

Rule of thumb Add jitter to client reconnects to avoid a storm after every deploy.

Breaks when Connection limit per node; reconnect storms.

Runs jobs on a timer; each job calls other components.

Did you know Cron jobs tend to start on the hour, so many systems see a spike at :00.

Rule of thumb Add jitter to schedules and make every job safe to run twice.

Breaks when Thundering herd at the top of the hour; missed runs.

Config and flags read on every request.

Did you know Flags let you ship code dark and turn it on gradually for a share of users.

Rule of thumb Evaluate flags locally from cached rules, not with a network call per request.

Breaks when A slow flag lookup slows everything; a bad flag rollout.

Nginx-style front door: TLS termination, compression, routing.

Did you know Reverse proxies like Nginx terminate TLS, compress responses and route to backends.

Rule of thumb Reuse upstream connections with keep-alive; a new connection per request wastes time.

Breaks when Connection limits; a single proxy becomes the bottleneck.

Scales per request; pays a cold start when idle.

Did you know Serverless functions scale per request, but a cold start adds latency to the first call.

Rule of thumb Watch the account concurrency limit: past it, requests are throttled.

Breaks when Cold-start latency spikes; account concurrency limit; per-call cost at scale.

Small code at the CDN edge: rewrites, auth, A/B.

Did you know Edge functions run in many locations close to users, often in lightweight isolates.

Rule of thumb Keep them small: a call back to a distant database erases the latency win.

Breaks when CPU time limit per request; no local state.

One query fans out to many services (federation).

Did you know GraphQL lets a client ask for exactly the fields it needs in one request.

Rule of thumb Limit query depth and cost; one nested query can fan out into thousands of lookups.

Breaks when N+1 fan-out; one slow resolver slows the whole query.

Binary RPC between internal services.

Did you know gRPC uses HTTP/2 and Protocol Buffers for compact, typed calls between services.

Rule of thumb Long-lived HTTP/2 connections defeat connection-level balancing: balance per request.

Breaks when Deadline propagation; connection reuse limits.

Composes several backend calls into one screen-shaped response.

Did you know A backend-for-frontend shapes data for one client, such as the mobile app.

Rule of thumb Call downstream services in parallel; serial calls add their latencies together.

Breaks when Sequential calls add up; it inherits every downstream failure.

Resizes and converts uploads on demand.

Did you know Resizing images is CPU-heavy, so it is usually done once and cached.

Rule of thumb Generate common sizes on upload, or cache on-the-fly results behind a CDN.

Breaks when CPU-bound; large uploads block workers.

Long jobs that turn uploads into streamable renditions.

Did you know Transcoding turns one video into several resolutions for adaptive streaming.

Rule of thumb Run it as queued background jobs; it can take longer than the video itself.

Breaks when Minutes per job; backlog grows faster than it drains.

Kubernetes-style cluster: pods, scheduling, rollouts.

Did you know Orchestrators like Kubernetes restart failed containers and spread them across machines.

Rule of thumb Set resource requests and limits, or one noisy service starves its neighbours.

Breaks when Scheduling delay on scale-out; node pressure evictions.

VMs that scale on CPU; slow to react, cheap when quiet.

Did you know Autoscalers add instances when a metric stays high, but new instances take minutes to be ready.

Rule of thumb Scale ahead of known peaks: reactive scaling lags a sudden spike.

Breaks when Scales minutes after the spike; thrashing on short bursts.

Where services find each other (Consul, Eureka).

Did you know A registry tracks which instances are healthy so callers can find them.

Rule of thumb Cache lookups in the client so a registry outage does not stop all traffic.

Breaks when Stale entries route to dead instances; registry outage.

Runs heavy jobs off the request path.

Did you know Batch jobs process large data sets on a schedule rather than per request.

Rule of thumb Checkpoint progress so a failed job resumes instead of starting over.

Breaks when Jobs pile up; retries re-run side effects.

Memory, TTL and key popularity decide the hit rate.

Did you know An in-memory cache like Redis usually answers in well under a millisecond inside one region.

Rule of thumb A cold cache sends every read to the database: warm it before traffic arrives.

Breaks when Cold-cache miss storm; hot keys.

Connections, CPU, replicas, contracts.

Did you know Postgres runs one process per connection, which is why poolers like PgBouncer sit in front of busy databases.

Rule of thumb Read replicas scale reads, not writes, and can serve slightly stale data.

Breaks when Pool exhaustion, CPU crash, lost updates, stale replica reads.

Partitions with per-partition limits.

Did you know Key-value stores spread keys over partitions; one very popular key can overload a single partition.

Rule of thumb Cache hot keys or add a random suffix to spread them across partitions.

Breaks when Hot partition despite low average load.

At-least-once delivery, visibility timeout.

Did you know If a consumer does not finish within the visibility timeout, the message is delivered again.

Rule of thumb Send messages that keep failing to a dead-letter queue instead of retrying forever.

Breaks when Redelivery and dead letters; visibility-timeout duplicates.

Ordered partitions, consumer groups, retention.

Did you know Kafka keeps order only within a partition, so the partition key decides what stays in sequence.

Rule of thumb Consumers beyond the partition count sit idle: add partitions to scale reads.

Breaks when Consumer lag; data lost past retention.

Full-text search with shards and refresh delay.

Did you know Search engines build an inverted index: each word points to the documents that contain it.

Rule of thumb Index from a change stream in the background; do not make writes wait for search.

Breaks when Query latency spikes under load; stale results.

Blobs and files; high throughput, higher latency.

Did you know Object stores like S3 keep copies across devices and facilities for very high durability.

Rule of thumb Serve files through a CDN with signed URLs instead of streaming them through your API.

Breaks when Request-rate throttling (503s) on hot prefixes.

JSON documents, flexible schema, single-document atomicity.

Did you know Document databases store nested JSON-like records, so one read can fetch a whole aggregate.

Rule of thumb Model around your queries; unbounded arrays inside one document become slow reads.

Breaks when Index write amplification; large documents slow every read.

Metrics and events by time; downsampling and retention.

Did you know Time-series databases compress by time and downsample old data to keep storage small.

Rule of thumb High-cardinality tags, like user ids, explode the number of series.

Breaks when Cardinality explosion; ingest falls behind.

Relationships and traversals.

Did you know Graph databases follow relationships directly, so multi-hop queries avoid large joins.

Rule of thumb Traversals through super-nodes with millions of edges are the usual slow query.

Breaks when Deep traversals burn CPU; supernodes.

Analytics queries on big tables, limited query slots.

Did you know Warehouses store data by column, so analytical queries read only the columns they need.

Rule of thumb Keep dashboards off the production database: copy the data to the warehouse.

Breaks when Queueing on slots; dashboards go stale.

Coordination service (etcd, ZooKeeper): leases and leader election.

Did you know A distributed lock needs a lease that expires, or a crashed holder blocks everyone.

Rule of thumb Use a fencing token so a holder whose lease expired cannot still write.

Breaks when Lock contention serialises work; lease expiry during GC pauses.

Fan-out messaging to many subscribers.

Did you know Pub/sub delivers each message to every subscriber, so producers never know who listens.

Rule of thumb Give each subscriber its own retry and dead-letter handling.

Breaks when Slow subscriber backs up the topic; no ordering.

Where poison messages go after too many retries.

Did you know A dead-letter queue holds messages that failed too often, so they stop blocking the rest.

Rule of thumb Alert when it grows, and keep a way to replay messages once the bug is fixed.

Breaks when Fills silently; nobody re-drives it.

Streams database changes to other systems.

Did you know Change data capture reads the database's write-ahead log to stream every change.

Rule of thumb Watch replication-slot lag: an unread slot makes the database keep old logs and fill its disk.

Breaks when Lag during bulk writes; schema changes break the stream.

Stateful processing over streams (Flink, Kafka Streams).

Did you know Stream processors like Flink compute over moving windows of events in real time.

Rule of thumb Plan for late events with watermarks and an allowed lateness.

Breaks when Checkpoint stalls; state grows without TTL.

Scheduled batch loads into the warehouse.

Did you know ETL extracts data, transforms it and loads it into a warehouse, usually in batches.

Rule of thumb Make each run rerunnable for a given date so backfills are safe.

Breaks when Runs overlap when one is late; partial loads.

Fast aggregations over wide tables (ClickHouse-style).

Did you know Columnar storage compresses well because each column holds similar values.

Rule of thumb Load in batches; many tiny single-row writes are slow.

Breaks when Many small inserts; concurrent heavy queries.

Append-only, verifiable records for money and audits.

Did you know Ledgers are append-only: a correction is a new entry, never an edit.

Rule of thumb Use double-entry records so every movement balances to zero.

Breaks when Write serialisation; slow proofs under load.

Fast store for login sessions and carts.

Did you know A shared session store lets any app server handle a user's next request.

Rule of thumb Keep sessions small and give them a TTL; they are read on every request.

Breaks when Every request reads it; an outage logs everyone out.

Nearby searches and map tiles.

Did you know Geo indexes split the map into cells, such as geohashes, to find nearby points fast.

Rule of thumb Dense cities overload single cells: use smaller cells where points cluster.

Breaks when Dense areas are hot cells; large radius queries.

Snapshots on a schedule; restores take minutes.

Did you know A backup only counts once you have restored from it successfully.

Rule of thumb Test restores regularly and keep a copy in another region or account.

Breaks when Snapshot I/O slows the primary; untested restores.

Ships writes to another region over a slow link.

Did you know Asynchronous replication is fast but can lose the last few writes if the primary fails.

Rule of thumb Monitor replication lag: it is the window of data you could lose.

Breaks when Lag during write bursts; split brain on failover.

Keys and credentials fetched at start-up or per call.

Did you know A secrets manager stores keys encrypted and can rotate them on a schedule.

Rule of thumb Cache secrets in memory for a few minutes; fetching per request hits API limits.

Breaks when Per-call fetches hit its quota; outage blocks deploys.

Absorbs floods at the edge before they reach you.

Did you know Volumetric floods are absorbed at the network edge, spread over many locations, before they reach you.

Rule of thumb Hide your origin's address so attackers cannot go around the shield.

Breaks when Flood gets through when off; scrubbing adds latency.

Blocks injection, scraping and known-bad requests by rule.

Did you know A WAF inspects HTTP requests for attacks such as SQL injection and cross-site scripting.

Rule of thumb Run new rules in log-only mode first to see false positives before blocking.

Breaks when Rules off means attacks pass; strict rules block real users.

Tells humans from bots with fingerprints and challenges.

Did you know Bot managers score clients by behaviour and fingerprint, not by IP address alone.

Rule of thumb Challenge suspicious clients instead of blocking them, to spare real users.

Breaks when Bots scrape or stuff credentials; challenges annoy real users.

Rejects requests without a valid token before they reach the app.

Did you know Checking credentials at the edge stops credential stuffing before it reaches the app.

Rule of thumb Limit login attempts per account and per IP, and offer MFA.

Breaks when Credential stuffing without lockout; token checks add latency.

Mutual TLS between services: only trusted peers connect.

Did you know Mutual TLS makes both sides prove identity with certificates, not just the server.

Rule of thumb Automate certificate rotation; expired certificates are a classic outage.

Breaks when Handshake CPU under load; expired certificates.

Collects security events for detection and audits.

Did you know A SIEM collects logs from everywhere and correlates them to spot attacks.

Rule of thumb Alert on a few high-signal rules; noisy alerts get ignored.

Breaks when Ingest lag hides an attack; log volume cost.

OIDC / SAML sign-in for people and services.

Did you know Identity providers handle sign-in with standards like OpenID Connect and SAML.

Rule of thumb Cache the provider's signing keys; checking a token should not need a network call.

Breaks when Login storms after an outage; token refresh floods.

Encrypts and decrypts with managed keys.

Did you know A KMS keeps master keys in hardware and encrypts your data keys for you (envelope encryption).

Rule of thumb Encrypt locally with a data key; calling KMS for every record hits its rate limits.

Breaks when Per-call quota; envelope encryption done per request.

Scans content for secrets and personal data before it leaves.

Did you know Data loss prevention scans content for patterns such as card numbers and national IDs.

Rule of thumb Scan in the background where you can; inline scanning adds latency to every request.

Breaks when Scan latency; false positives block real content.

Signature and anomaly detection on traffic.

Did you know An IDS alerts on suspicious traffic; an IPS sits inline and blocks it.

Rule of thumb Tune signatures to your own traffic, or false positives block real users.

Breaks when Misses novel attacks; alert fatigue.

Every request authenticated and authorised, no network trust.

Did you know Zero trust checks identity and device on every request instead of trusting the network.

Rule of thumb Roll it out one application at a time, starting with internal admin tools.

Breaks when Policy evaluation latency; a misrule locks everyone out.

Issues and rotates short-lived certificates.

Did you know A certificate authority signs the certificates that prove a service's identity.

Rule of thumb Prefer short-lived certificates with automatic renewal.

Breaks when Rotation storms; an expired root breaks everything.

Scores payments and sign-ups for risk.

Did you know Fraud models score each transaction in milliseconds from behaviour and history.

Rule of thumb Decide in advance what happens when scoring times out: allow or decline.

Breaks when Model latency on the checkout path; false declines cost sales.

Answer cache: a hit skips the model.

Did you know A semantic cache matches questions by meaning using embeddings, not exact text.

Rule of thumb Tune the similarity threshold: too loose returns wrong answers, too strict never hits.

Breaks when Low hit rate means every question reaches the model.

Tokens per request, TPM quota.

Did you know Embeddings turn text into vectors where similar meanings sit close together.

Rule of thumb Batch many texts into one call: per-request overhead dominates small inputs.

Breaks when Token quota 429s under bursts.

Index memory, query latency, QPS.

Did you know Approximate indexes like HNSW trade a little recall for much faster search.

Rule of thumb Raw size ≈ vectors × dimensions × 4 bytes. One million 1,536-d vectors is about 6 GB before index overhead.

Breaks when Index larger than memory; QPS limit.

Time to first token, tokens/s, quotas.

Did you know Latency has two parts: time to the first token, then a steady stream of tokens per second.

Rule of thumb Stream answers so they feel fast, and cap output tokens to bound cost.

Breaks when Quota 429s; time to first token under load; cost.

One door to every model: keys, quotas, caching, fallback.

Did you know An AI gateway sits in front of model providers to add keys, quotas, logging and fallbacks.

Rule of thumb Track spend per team or feature at the gateway, not after the bill arrives.

Breaks when Gateway quota; added latency per call.

Exact-match cache for repeated prompts.

Did you know Providers can cache a long repeated prompt prefix, cutting cost and time to first token.

Rule of thumb Put stable content first (instructions, documents) and the changing question last.

Breaks when Low hit rate on personalised prompts.

Cross-encoder that reorders retrieved chunks.

Did you know A reranker reads each retrieved passage together with the question and reorders by relevance.

Rule of thumb Retrieve broadly, rerank, then send only the top few passages to the model.

Breaks when Adds 50–200 ms per query; GPU concurrency limit.

Input and output checks: injection, PII, policy.

Did you know Guardrails check inputs and outputs for unsafe content, prompt injection or leaked data.

Rule of thumb Run cheap checks first and model-based checks only when they are needed.

Breaks when Latency on every call; misses injections at low detection.

Checks model output against a schema; retries on failure.

Did you know Validating model output against a schema catches malformed answers before users see them.

Rule of thumb On failure, retry once with the error message, then fall back.

Breaks when Retry amplification when the invalid-output rate is high.

Sends easy requests to a cheap model, hard ones to a big one.

Did you know A router sends easy requests to small, cheap models and hard ones to larger models.

Rule of thumb Measure quality per route: savings that hurt answers are not savings.

Breaks when Misrouting; the cheap path fails and falls back.

Collects requests and runs them later at a discount.

Did you know Batch APIs trade latency for price: results arrive later but cost less.

Rule of thumb Use it for evals, backfills and nightly jobs, never for a user waiting on a reply.

Breaks when Deadline misses when the batch window is long.

Your own GPUs with continuous batching.

Did you know GPU servers batch requests together; throughput rises with batch size until memory runs out.

Rule of thumb Long prompts use more KV-cache memory, so fewer requests fit at once.

Breaks when KV-cache pressure; queueing when all GPUs are busy.

Parses and chunks documents before embedding.

Did you know Ingestion splits documents into chunks before they are embedded for retrieval.

Rule of thumb Chunk on headings and paragraphs with a little overlap, not on fixed character counts.

Breaks when Backlog on large files; poisoned content.

Keyword and vector retrieval merged with reciprocal rank fusion.

Did you know Hybrid search combines keyword matching (BM25) with vector similarity.

Rule of thumb Keywords catch exact names and codes that embeddings often miss.

Breaks when Two indexes to keep in sync; merge latency.

Assembles retrieved chunks, memory and instructions into the prompt.

Did you know Context building assembles retrieved passages, history and instructions into the prompt.

Rule of thumb More context is not always better: irrelevant passages lower quality and raise cost.

Breaks when Context overflow; stale chunks; prompt-injection carried in.

Samples answers and scores quality, drift and cost.

Did you know Online evals score a sample of live answers to catch quality drops early.

Rule of thumb Score a small sample; grading every answer can double model spend.

Breaks when Scoring cost at high sample rates; silent quality drops.

Streams audio into transcripts.

Did you know Streaming speech recognition returns partial words while the user is still talking.

Rule of thumb Send audio in small chunks to keep the transcript close to real time.

Breaks when Concurrency caps; latency on long utterances.

Turns answers into audio, often streamed.

Did you know Speech synthesis can start playing before the whole sentence is generated.

Rule of thumb Cache common phrases; synthesising the same greeting again is wasted money.

Breaks when Time to first audio; character quotas.

Diffusion model behind a queue.

Did you know Image generation takes seconds per image and runs on GPUs.

Rule of thumb Run it in the background and show progress instead of holding a request open.

Breaks when Seconds per image; GPU queueing; content filters.

Extracts text and tables from scans and PDFs.

Did you know OCR turns scanned pages into text; layout and tables are the hard part.

Rule of thumb Process pages in parallel and cache results by document hash.

Breaks when Page-rate quotas; large files time out.

Trains an adapter on your data; hours per job.

Did you know Fine-tuning adapts a model to your examples; it changes style and format more than knowledge.

Rule of thumb Improve prompts and retrieval first, and fine-tune when they plateau.

Breaks when GPU queue; data drift makes it stale.

GPUs that only process prompts (prefill), split from decoding; sets time to first token.

Did you know Splitting prefill (reading the prompt) from decode (writing the answer) onto separate GPUs stops long prompts from stalling everyone else’s output.

Rule of thumb Size prefill for prompt tokens per second and decode for output tokens per second; they scale differently.

Breaks when Long prompts queue for prefill and time to first token climbs.

GPUs that only generate output tokens, batched; sets tokens per second.

Did you know Decoding is memory-bound: each request holds its KV cache in GPU memory for as long as it generates.

Rule of thumb Watch KV-cache memory, not just GPU utilisation; it sets how many requests decode at once.

Breaks when KV-cache memory fills; batches shrink and output slows.

Keeps attention KV blocks in CPU memory or SSD so repeated prefixes skip prefill.

Did you know Reusing the KV cache of a shared prefix (system prompt, chat history) skips most of the prefill work.

Rule of thumb Put the stable part of the prompt first so more requests share a prefix.

Breaks when Low hit rate when prompts do not share prefixes; transfer time on misses.

A small draft model proposes tokens the big model verifies in one pass.

Did you know A small draft model guesses several tokens; the large model checks them in one pass and keeps the ones it agrees with.

Rule of thumb It only pays off when most drafted tokens are accepted; measure the acceptance rate.

Breaks when Low acceptance rate wastes GPU time; draft model becomes the bottleneck.

Serves many fine-tuned adapters on one base model in shared batches.

Did you know Multi-LoRA serving runs one base model for many fine-tuned adapters, batching requests for different adapters together.

Rule of thumb Cap adapters per batch; swapping adapters in and out of GPU memory costs time.

Breaks when Adapter swaps and too many adapters per batch slow everyone.

A small model rewrites or expands questions before retrieval.

Did you know Rewriting a vague question into a precise search query often improves retrieval more than a bigger model does.

Rule of thumb Use a small, fast model; it runs on every question.

Breaks when Adds a model call to every question; quota and latency.

A knowledge graph built from documents for multi-hop questions.

Did you know GraphRAG builds a knowledge graph from documents so multi-hop questions can follow relationships.

Rule of thumb Build cost is high: refresh incrementally and keep vector search for simple lookups.

Breaks when Graph traversals are CPU-heavy; expensive to rebuild.

A model that grades answers for quality and safety (online evals).

Did you know An LLM judge grades outputs at scale, but it is another model call with its own latency, cost and biases.

Rule of thumb Judge a sample asynchronously instead of every answer inline.

Breaks when Doubles model calls; slow judges back up the pipeline.

Collects spans for every model and tool call (LLM observability).

Did you know Agent traces record every model and tool call with inputs, outputs, tokens and timing.

Rule of thumb Sample traces at high volume, but keep every failed one.

Breaks when Ingest limits drop traces exactly when you need them.

Screens user input and retrieved text for instructions that try to hijack the model.

Did you know Prompt injection hides instructions in input or retrieved content; no filter catches all of them.

Rule of thumb Combine a filter with design: keep untrusted data out of the control flow and limit what tools can do.

Breaks when Misses novel injections; false positives block real users.

Removes personal data from prompts and logs before they leave your system.

Did you know Redacting personal data before it reaches a model or a log reduces privacy and compliance risk.

Rule of thumb Redact before logging too; traces are a common leak.

Breaks when CPU-bound on long prompts; adds latency to every call.

Plans and loops: each request makes several model and tool calls.

Did you know An agent loops: think, call a tool, read the result, repeat until done.

Rule of thumb Cap steps and tokens per task; runaway loops are the costliest failure.

Breaks when Call amplification; runaway loops burn quota and money.

Executes tool calls for agents; may call other systems.

Did you know Tools give the model actions such as search or database queries; each call is another round trip.

Rule of thumb Return short, structured results; long tool output eats the context window.

Breaks when Tool outage; retried side effects.

Auth, allow-lists and quotas for tool servers.

Did you know MCP (Model Context Protocol) is an open standard for connecting models to tools and data.

Rule of thumb Allow-list the tools each agent may call, and log every call.

Breaks when Scope creep; quota rejects agent calls.

Isolated containers that run agent-written code.

Did you know Sandboxes run model-written code in an isolated container with limited network and time.

Rule of thumb Set CPU, memory and wall-clock limits; generated code can loop forever.

Breaks when Cold starts; sandbox queueing; timeouts.

A person approves risky actions; the agent waits.

Did you know Human-in-the-loop review holds risky actions until a person approves them.

Rule of thumb Ask only for irreversible or costly actions, or reviewers start rubber-stamping.

Breaks when Minutes-long waits; deadline misses; rejected actions.

Short- and long-term memory reads and writes.

Did you know Agent memory keeps facts across sessions so the agent does not start from scratch.

Rule of thumb Store short summaries rather than raw transcripts, and let memories expire.

Breaks when Write amplification; poisoned memories.

Another organisation's agent, reached over the agent-to-agent protocol.

Did you know Agent-to-agent protocols let agents from different teams or vendors hand work to each other.

Rule of thumb Treat another agent like an external API: timeouts, retries and a budget per call.

Breaks when Cascading delegation; remote quota; partial failure.

Long-running agent tasks that survive restarts (Temporal-style).

Did you know Workflow engines like Temporal persist every step, so long processes survive crashes.

Rule of thumb Keep each activity idempotent: the engine may retry it.

Breaks when History grows; replay cost; stuck workflows.

Hands each task to the specialist agent that fits it.

Did you know An agent router picks which specialised agent should handle a request.

Rule of thumb Log every routing decision; a wrong route looks like a bad answer.

Breaks when Misrouted tasks bounce; every hop adds a model call.

Versioned tool and skill definitions agents load at run time.

Did you know A skill registry lists reusable instructions and tools an agent can load when needed.

Rule of thumb Load skills on demand so unused ones do not fill the context window.

Breaks when Unpinned skills change behaviour; registry outage.

Headless browsers the agent drives to read sites.

Did you know Browser tools let agents read and act on web pages, which is slow and brittle.

Rule of thumb Use an API when one exists and keep the browser as the last resort.

Breaks when Seconds per page; bot blocks; memory per browser.

Compacts long histories into short memories with a model call.

Did you know Summarising older turns keeps long sessions inside the context window.

Rule of thumb Keep key facts and decisions word for word; summaries lose detail.

Breaks when Lossy summaries; summarisation cost grows with activity.

Token and dollar budgets per task; cuts off runaway agents.

Did you know A spend governor enforces budgets per user, task or team before calls go out.

Rule of thumb Warn at a soft limit, then stop at a hard one.

Breaks when Budget too low fails real tasks; too high lets loops burn money.

Buffers agent work so bursts drain at a steady rate.

Did you know Queuing agent tasks smooths bursts and caps how many run at once.

Rule of thumb Give each task a deadline so stale work does not run hours later.

Breaks when Tasks wait minutes; retries re-run tool side effects.

A lead agent fans work out to parallel subagents, each with its own context.

Did you know Multi-agent research systems fan a task out to parallel subagents and use many times more tokens than a single agent.

Rule of thumb Use them for broad, parallel work; give each subagent a clear goal, tools and output format.

Breaks when Token use multiplies with subagents; tools saturate under the fan-out.

Summarises old turns and tool results so long sessions fit the context window.

Did you know Compaction summarises a long context so the agent can keep going without hitting the window limit.

Rule of thumb Keep decisions, open tasks and file paths verbatim; summarise the rest.

Breaks when A model call per compaction; loses details it should keep.

Reuses results of identical, read-only tool calls across steps and agents.

Did you know Agents often repeat the same read-only tool call within and across tasks.

Rule of thumb Cache only read-only tools, and key the cache on normalised arguments.

Breaks when Stale results; low hit rate when arguments vary.

Durable execution: saves agent state after each step so work resumes after a crash.

Did you know Durable execution saves state after each step so a crash resumes instead of starting over.

Rule of thumb Checkpoint before side effects, and make side effects idempotent.

Breaks when Write volume on every step; large states are slow to save.

A virtual desktop an agent drives with screenshots, clicks and typing.

Did you know Computer-use agents act through screenshots and clicks, which takes seconds per action.

Rule of thumb Prefer APIs and MCP tools; use the desktop only when nothing else exists.

Breaks when Slow (seconds per action); few concurrent desktops; flaky UIs.

Checks every tool call against permissions and data-flow rules before it runs.

Did you know A policy engine checks each tool call against what this agent, user and data may do.

Rule of thumb Deny by default for side-effecting tools; log every decision.

Breaks when Over-strict policies stall agents; policy store becomes a hot path.

Runs a long-lived agent loop: context, tools, checkpoints and retries around the model.

Did you know A harness is everything around the model in an agent: context management, tools, checkpoints, retries and evaluation.

Rule of thumb Externalise progress to durable notes so a fresh session can pick up where the last one stopped.

Breaks when Too many turns per task; state lost without checkpoints.

Finds the few tools a step needs instead of loading every definition into context.

Did you know Loading hundreds of tool definitions into every prompt wastes context and confuses the model.

Rule of thumb Search for the few relevant tools per step and load only those.

Breaks when Misses the right tool; another hop per step.