Learnings

• Data
• Data Platforms — Digest
• Data Platforms — Article
• Data Warehouse — Digest
• Data Warehouse — Article
• Data Warehouse Use Cases — Needed vs. Overkill
• Elasticsearch Engineering – Digest
• Elasticsearch Engineering
• Prefect — Components Deep Dive
• RustFS — Components Deep Dive
• Miscs
• Diagram Types — When to Use Each
• Decision-Making & Thinking Methods — When to Use Each
• LLMs
• LangChain — Components Deep Dive
• LangGraph — Components Deep Dive

Data

Data Platforms — Digest

Definition

A data platform is the infrastructure and set of tools an organization uses to collect, store, process, and serve data so applications and people can reliably use it. It's the plumbing that moves data from where it's generated to where it's needed, in usable form.

Real-life usage

• E-commerce: clickstream, orders, and inventory land in a warehouse (Snowflake/BigQuery/Redshift) and feed both BI dashboards and ML models (recommendations, fraud detection) from the same data.
• Fintech: transaction data flows via streaming pipelines (Kafka) into systems supporting both real-time fraud scoring and end-of-day regulatory reporting.
• Healthcare: patient records, device telemetry, and claims data are unified under strict governance (HIPAA) while remaining queryable for research.
• SaaS internal analytics: app DB → ETL/ELT → warehouse → dashboards for product/finance teams.
• Analogy: a city utility grid — treatment plant (ingestion/cleaning), pipes/reservoirs (storage), pumping stations (processing), taps (serving). Break any layer, everything downstream is affected.

5 Whys — Root Cause

• Why do platforms exist? Raw source systems are wrong for two independent reasons.
• Reason 1 — contention: OLTP DBs are optimized for row-based point lookups. Large analytical scans evict the transactional working set from buffer cache and hold locks longer, causing cache misses and slow transactional reads (e.g., checkout).
• Read replicas solve contention but not query shape — row-store/3NF schemas are structurally bad at large scans/aggregations, hence column-store, star schemas, and pre-aggregation.
• Reason 2 — incoherence: each source system encodes its own implicit business logic (e.g., three different definitions of "active customer" across CRM, billing, and app DB). With N sources and M consumers and no shared contract, definitions drift — it's structurally guaranteed, not a one-off mistake.
• Platforms fix this by centralizing logic into governed, computed-once definitions everyone reads from.

Value vs. Risk

• Value: understanding the "why" lets you choose stack components by the guarantee a workload needs (ordering, replayability, latency, managed vs. self-hosted) instead of by popularity or familiarity — avoiding redundant tools solving the same problem.
• Risk — performance/consistency: black-boxing the platform reproduces the original problems (contention, definitional drift) plus new complexity/understanding debt (unclear why each tool is in the stack).
• Risk — trust erosion: when dashboards disagree and no one can explain why, people stop trusting the platform and quietly bypass it for gut-feel or spreadsheets — a costly, hard-to-reverse adoption failure.

Data Platforms — Article

Definition

A data platform is the infrastructure and set of tools an organization uses to collect, store, process, and serve data so that applications and people can reliably use it. Stripped of jargon: it's the plumbing that gets data from where it's produced to where it's needed, in a form someone can actually use. When you hear "data platform," think less of a single product and more of a layered system — ingestion, storage, processing, and serving — each layer doing a distinct job.

Real-life usage

You've already touched pieces of this: Debezium and NiFi doing change-data-capture and flow-based ingestion, Kafka as the durable transport backbone, Airflow and Azure Data Factory orchestrating batch pipelines, Elasticsearch and MongoDB serving specific access patterns. That's a real, if overlapping, data platform stack.

Beyond your own stack, the pattern repeats everywhere at scale:

• E-commerce platforms capture clickstream, orders, and inventory changes, land them in a warehouse (Snowflake, BigQuery, Redshift), and serve both BI dashboards and ML models (recommendations, fraud detection) from the same underlying data.
• Fintech systems stream transactions through Kafka into pipelines that support both millisecond-latency fraud scoring and end-of-day regulatory batch reporting — same data, wildly different latency contracts.
• Healthcare platforms unify patient records, device telemetry, and claims data while satisfying strict compliance (HIPAA), yet still need to be queryable for research.
• Even a mid-size SaaS company runs a scaled-down version: app DB → ETL/ELT → warehouse → dashboards for product and finance.

A useful mental model: a data platform is like a city's utility grid. A treatment plant cleans and prepares water (ingestion/cleaning), reservoirs and pipes store and move it (storage), pumping stations push it where it's needed (processing), and taps deliver it on demand (serving/access). Nobody turning on a faucet thinks about the treatment plant — but if any layer fails, everything downstream breaks.

5 Whys — Why Data Platforms Exist

The natural question is: why not just query the production database directly? Working through this honestly gets you to two independent root causes.

Root cause 1: resource contention. OLTP databases are optimized for row-based, indexed point lookups — fast reads/writes of single rows, like fetching one customer's cart. Analytical queries instead scan and aggregate across millions of rows. When a large aggregation runs against the same database, it evicts the transactional working set from the buffer cache. A checkout read that used to be an instant in-memory hit becomes a slow disk read. Combine that with longer-held locks from the scan, and the aggregation job and the checkout flow — which have nothing to do with each other logically — end up fighting over the same finite memory, I/O, and lock resources.

The obvious fix, "just add a read replica," solves contention: reads are offloaded to a copy, so the analytical workload stops starving the transactional one. But it doesn't solve the second problem — query shape. A replica has the exact same row-store, normalized (3NF) schema as production, which is structurally bad at scanning and aggregating across huge row counts, replica or not. This is why the industry didn't stop at replicas: it built column-store engines, star/denormalized schemas, and pre-aggregation — physically reshaping the data to match the access pattern of the consumer, not just relocating the same shape to different hardware. Vertical scaling (bigger single machine) eventually hits a ceiling too, which is part of why horizontal separation into distinct, purpose-built systems became the standard architecture rather than just buying a bigger box.

Root cause 2: definitional incoherence. Imagine three teams need "active customer": marketing pulls it from the CRM (logged in within 30 days), finance pulls it from billing (has a paid, non-cancelled subscription), and a data scientist queries the app DB directly (any tracked event in 7 days). All three are querying the same underlying reality, but each source system encodes its own local business logic implicitly — in application code, in how a status field gets set, in team-specific workflow assumptions. There's no shared, enforced contract anywhere. With N source systems and M independent consumers, you get up to N×M different interpretations of the same concept, and nothing structurally prevents that drift — it isn't a one-off mistake someone could have avoided by being careful, it's guaranteed by the absence of a shared definition layer.

Put together: data platforms exist because raw source systems are wrong for two independent reasons — they can't serve analytical access patterns without degrading transactional performance, and they can't produce consistent meaning across consumers without centralizing business logic. Every component of a modern data platform — ingestion tools, warehouses, orchestration, semantic/governance layers — exists to solve one or both of these.

Value vs. Risk

Value. Understanding the "why" rather than just the "how to configure X" changes how you make architectural decisions. Instead of picking a tool because it's popular or because you already know it, you pick based on the guarantee the workload actually needs: does it need ordering and replayability (Kafka), low-latency row-level change capture (Debezium), flexible flow-based routing over heterogeneous data (NiFi), managed Azure-native batch orchestration (ADF), or Python-native complex DAG logic (Airflow)? If two tools in your stack solve the same guarantee — say, two orchestrators — that's a signal of accidental complexity worth questioning directly, not a neutral fact about the stack.

Risk of treating the platform as a black box. The immediate risks are a recurrence of the original two problems — contention and inconsistency — plus a new one: complexity debt, where nobody can explain why a given tool is in the pipeline or what it specifically contributes. But there's a third, more expensive risk that's easy to miss because it isn't a technical failure: trust erosion. When two dashboards disagree and no one can explain why, people stop trusting the platform itself, not just that one number. The failure mode isn't a wrong query — it's leadership quietly reverting to gut feel or personal spreadsheets because they've learned the platform can't be relied on. That's a behavioral, org-wide cost, and by the time it's visible, the damage — bad decisions already made, the platform already abandoned in practice — is largely done. It's far more expensive to reverse than a performance regression, because it's an adoption and credibility problem, not an engineering one.

Data Warehouse — Digest

Definition

A data warehouse is a centralized store of structured, historical data — typically modeled with facts and dimensions (star/snowflake schema) — organized specifically to support analytical querying and reporting, as opposed to transactional operations.

Real-life usage

• Retail/e-commerce: fact_sales (quantity, revenue) joined to dim_product, dim_store, dim_date, dim_customer — slice revenue any way, on demand.
• Banking/fintech: regulatory reporting (risk exposure, AML) needs consistent historical snapshots, not just current state.
• Airlines: fact_bookings/fact_flights joined to dim_route, dim_aircraft, dim_time to analyze load factors and revenue per route over years.
• Marketing: fact_impressions/fact_conversions joined to dim_campaign, dim_channel, dim_audience.
• Analogy: a physical warehouse doesn't manufacture goods — it receives finished goods and organizes them onto labeled shelves by category (dimensions) with clear counts (facts), so anyone can pull exactly what they need without digging through the factory floor.

5 Whys — Root Cause

• Why not just use a regular RDBMS with a star schema? Row-store databases store full rows together on disk — a query needing 2 of 30 columns still has to read all 30 off disk.
• Root cause 1 — columnar storage: warehouse engines store each column contiguously, so aggregation queries read only the columns they need. This also enables better compression and vectorized execution.
• Why also separate facts and dimensions if columnar storage already solves read efficiency? Flattening everything into one wide table causes massive redundancy (a customer's address repeated across every transaction row) and update anomalies (changing one dimension value means rewriting/missing rows across a billion-row fact table).
• Root cause 2 — fact/dimension separation: avoids redundancy and update anomalies by separating high-volume immutable events (facts) from low-volume mutable reference data (dimensions), each managed according to its actual rate of change (see: slowly changing dimensions).

Value vs. Risk

• Value: understanding columnar storage + fact/dimension separation changes how you design any analytical store — not just SQL warehouses. E.g., a medallion architecture built on doc-oriented stores (like Elasticsearch) still needs to factor out dimension keys instead of embedding full dimension attributes in every aggregate document, or it silently rebuilds the flat-table problem.
• Risk — no dedicated warehouse layer: flat/embedded dimension data means a dimension change (e.g., product recategorized) can't be propagated without reindexing every historical record referencing it — leading to silent inconsistency between old and new records with no single source of truth to detect or correct the drift.

Data Warehouse — Article

Definition

A data warehouse is a centralized store of structured, historical data — typically modeled with facts and dimensions using a star or snowflake schema — organized specifically to support analytical querying and reporting, as opposed to the transactional operations a production database handles. It's not just "a database with reporting tables"; the storage engine itself and the data model are shaped around analytical access patterns.

Real-life usage

The clearest way to see this is through fact and dimension tables. A retail warehouse might have fact_sales — one row per line item, carrying measures like quantity and revenue — joined to dim_product, dim_store, dim_date, and dim_customer. Anyone in finance can then slice revenue by region, by month, by category, in combinations nobody had to pre-build.

The same pattern shows up everywhere analytics matters at scale:

• Banking/fintech: regulatory reporting (daily risk exposure, AML) depends on consistent historical snapshots — "what was our exposure on March 31st" — not just current state, which is exactly what a warehouse's historical fact tables are built to answer.
• Airlines: fact_bookings/fact_flights joined to dim_route, dim_aircraft, dim_time let analysts study load factors, delays, and revenue per route across years of history.
• Marketing: fact_impressions/fact_conversions joined to dim_campaign, dim_channel, dim_audience answer "which channel actually drives conversions" across time — something no single operational system tracks end to end.

A useful analogy: a physical warehouse doesn't manufacture goods — it receives finished goods from many sources and organizes them onto labeled shelves by category (dimensions), with clear counts (facts), so anyone can walk in and pull exactly what they need without digging through the factory floor. A data warehouse does the same thing with data instead of goods.

5 Whys — Why Data Warehouses Exist as a Distinct Thing

The natural challenge is: you could build a star schema in any relational database — Postgres, MySQL — so why does the industry build dedicated warehouse engines (Snowflake, BigQuery, Redshift) at all?

Root cause 1: columnar storage matches the analytical access pattern. In a normal row-store database, a table's rows are stored together on disk — all of row 1's columns, then all of row 2's columns, and so on. A typical warehouse query like SELECT SUM(revenue) FROM fact_sales WHERE region = 'EU' only needs 2 of maybe 30 columns, but a row-store still has to pull every column of every matching row off disk into memory before it can discard the ones it doesn't need. That's wasted I/O and wasted cache space, proportional to the columns you don't care about.

Warehouse engines fix this with columnar storage: each column is stored contiguously on disk, so a query reads only the columns it actually references. This also unlocks two side benefits — far better compression (similar values stored together compress much more efficiently than mixed-type rows) and vectorized execution (operating on whole columns of values at once instead of row by row). This is the physical, mechanical reason a warehouse engine behaves differently from a row-store RDBMS running the exact same schema.

Root cause 2: fact/dimension separation avoids redundancy and update anomalies. Given columnar storage already solves read efficiency, it's fair to ask why not just put everything — customer name, product category, store region, revenue, quantity — into one big flat, columnar table. The answer is what happens to descriptive attributes under that design. If "customer address" lives inline in every sales row, and that customer has 500 transactions, the same address string is stored 500 times. At billions of rows, that's significant redundancy even under compression, and it creates an update anomaly: if the customer moves, do you rewrite all 500 (or 5 million) rows referencing them? Miss one, and the data is now silently inconsistent.

The deeper issue is conceptual: a "sale" and a "customer's current address" are different kinds of data that change at fundamentally different rates. Facts are an append-only event log; dimensions are comparatively static reference data that occasionally changes (which is exactly why "slowly changing dimensions" exists as a defined technique — dimensions do change, just rarely, and need an explicit strategy for handling it). Separating facts from dimensions keeps each kind of data managed according to its actual rate of change, instead of conflating an immutable event stream with mutable reference data in one structure.

Put together: data warehouses exist because (1) columnar storage matches the "few columns, many rows" read pattern of analytical queries in a way row-stores structurally cannot, and (2) fact/dimension modeling avoids the redundancy and update anomalies that come from mixing high-volume immutable events with low-volume mutable reference data in a single flat structure.

This root cause generalizes beyond SQL warehouses. A doc-oriented store like Elasticsearch actually does provide columnar-style reads for aggregations, via a structure called doc_values — a separate, purpose-built columnar layer distinct from its inverted index for text search. So root cause 1 can genuinely be satisfied outside a traditional warehouse engine. But root cause 2 is a modeling discipline, not a storage-engine feature — it has to be deliberately designed in. A medallion-style agg_ index built from flat JSON documents that embed both facts and dimension attributes in every document reproduces the exact flat-table problem: dimension values are duplicated across every document that references them, and because a document store has no native join to factor dimensions out, a dimension change either goes unpropagated (silent inconsistency) or requires reindexing every historical document that referenced it.

Value vs. Risk

Value. Understanding these two root causes changes how you evaluate any analytical data store, not just SQL warehouses. It tells you precisely what to check before trusting a design: does the storage layer read only the columns/fields a query needs (columnar or equivalent), and are high-volume facts kept separate from low-volume, referenced dimension data — or are they flattened together for convenience? For a system like a medallion-on-Elasticsearch setup, this means factoring dimension keys/IDs into fact-like documents and maintaining separate dimension indices, resolving them at query or ingest time, rather than embedding full dimension attributes into every aggregate document.

Risk. Keeping facts and dimensions flattened together, whether in a wide SQL table or in JSON documents, defers a cost rather than avoiding it. As data volume grows and dimension values change more often, the concrete failure looks like this: a report segmenting historical sales "by current product category" silently returns wrong numbers, because only some historical records were ever updated to reflect the new category — and there's no single source of truth to detect or correct the drift from, since nothing enforces referential consistency across the duplicated copies. This isn't a performance problem; it's a correctness problem that erodes trust in the numbers exactly the way inconsistent metric definitions do at the platform level.

Data Warehouse Use Cases — Needed vs. Overkill

When a data warehouse is genuinely needed

• BI/reporting — ad hoc slicing across dimensions (region, time, product) at speed; row-stores choke on this at scale.
• Historical/point-in-time analysis — cohort retention, trend analysis, "what did exposure look like on a given date." Needs immutable historical snapshots that production systems don't retain.
• ML feature engineering / training data prep — building wide feature tables by joining large historical fact/dimension sources; needs efficient scans/joins at volume.
• Regulatory & compliance reporting — audits need reproducible, consistent historical figures with lineage, not a moving production DB.
• Cross-domain analytics / data mesh consumption — joining multiple business domains' data together; exactly the "N sources, M consumers, no shared contract" problem centralized modeling solves.
• Forecasting & capacity planning — needs large historical time series at consistent granularity.
• Aggregate-level anomaly/trend detection — e.g. "revenue by region dropped 20% week over week"; inherently an aggregation-over-time problem.

Common thread: all need to scan/aggregate across large historical volumes with consistent, centrally-defined meaning — the two root causes (columnar reads + fact/dimension governance).

When a data warehouse is overkill or the wrong tool

• Single-record, low-latency lookups (get this customer's cart/profile) — the OLTP point-lookup pattern; columnar storage actually hurts here. Use the operational DB.
• Real-time operational decisions inside a live transaction (fraud scoring at checkout, inventory decrement) — needs millisecond latency and current state; warehouses are typically minutes-to-hours behind via batch/CDC.
• Full-text search / relevance ranking — not built for inverted indices or scoring; use Elasticsearch/OpenSearch.
• Unstructured/semi-structured blob storage — raw images, PDFs, logs, un-modeled JSON; forcing this into a star schema is premature structure — use a data lake (or the raw layer of a lakehouse) first.
• High-frequency small writes — IoT sensor ingestion event-by-event; warehouses are read-optimized/batch-oriented. Use a time-series DB or streaming store.
• Simple, low-volume reporting for a small team — a few thousand rows, one monthly summary; the ETL/governance/infra overhead isn't justified. A well-indexed Postgres view or spreadsheet suffices.
• Graph-shaped queries — path-finding, fraud rings, network traversal; star schemas/columnar engines aren't built for relationship traversal. Use a graph database.

Common thread: low-latency point access, unstructured content, full-text relevance, high-frequency writes, or relationship traversal are all problems a warehouse wasn't designed to solve.

Grouped by business domain

Telecom

• Needs it: CDR analysis across billions of records, churn prediction feature tables, network capacity planning, regulatory usage/coverage reporting.
• Overkill/wrong tool: real-time call routing, live network fault detection (streaming/time-series), subscriber lookup during a support call (OLTP).

Retail / e-commerce

• Needs it: sales analysis across store/product/time, inventory trend forecasting, customer segmentation/LTV, marketing attribution.
• Overkill/wrong tool: checking current stock for one SKU at checkout, product search/autocomplete (search engine), real-time cart/session state.

Consulting

• Needs it: multi-client benchmarking and longitudinal engagement analysis, internal utilization/profitability reporting across practices and time.
• Overkill/wrong tool: a single short engagement's ad hoc analysis on a one-off dataset — a spreadsheet or lightweight Postgres/DuckDB is enough.

Industry / manufacturing

• Needs it: production yield analysis across plants/lines/time, supply chain and demand forecasting, quality trend analysis joining defects to equipment/shift/supplier.
• Overkill/wrong tool: real-time sensor/IoT ingestion and shop-floor anomaly alerts (time-series/streaming), machine-level current-status lookups.

Finance / banking

• Needs it: regulatory historical reporting, risk exposure trend analysis, AML pattern detection across large transaction histories.
• Overkill/wrong tool: fraud scoring at the moment of a transaction (real-time), account balance lookup (OLTP).

Healthcare

• Needs it: population health trend analysis, claims/cost analysis across providers and time, research cohort analysis.
• Overkill/wrong tool: pulling a single patient's current chart during a visit (OLTP), clinical decision support needing sub-second current data.

Pattern across every domain: warehouse = historical, cross-dimensional, aggregatable analysis at real volume. Anything current-state, single-record, sub-second, or small-scale is the wrong job for it, regardless of industry.

Elasticsearch Engineering – Digest

The short companion to Elasticsearch Engineering: a glossary, then the facts worth memorising for the Elastic Certified Engineer exam (Elasticsearch 9.3), domain by domain. Use it for revision; go back to the full guide for explanations, examples and doc links.

---

A. Glossary

Term	Meaning (one line)
Alias	A secondary name pointing to one or more indices or data streams; can filter, route and have one write index
Allocation (shard)	The master's decision about which node holds each shard copy, governed by deciders
Allocation awareness	Spreads shard copies across node attributes (zone, rack); forced awareness refuses to overload surviving zones
Allocation explain API	GET _cluster/allocation/explain: says why a shard is (un)assigned
Analyzer	Char filters → tokenizer → token filters; turns text into terms at index and search time
API key	Credential with its own (restricted) privileges; Authorization: ApiKey <encoded>
Auto-follow pattern	CCR rule that automatically follows new leader indices matching patterns
Bool query	Compound query with must, should, filter, must_not
Bootstrap checks	Startup checks enforced in production mode (heap, file descriptors, vm.max_map_count, memory lock…)
Bucket aggregation	Groups documents into buckets (terms, date_histogram, range, filters, composite…)
Bulk API	NDJSON batch of index/create/update/delete actions in one request
CCR	Cross-cluster replication: read-only follower indices pull changes from leader indices on a remote cluster
CCS	Cross-cluster search: query remote_alias:index from the local cluster
Classic stream	A Streams-managed view over an existing data stream (no migration)
Cluster state	Metadata (indices, mappings, settings, routing table) managed by the elected master
Component template	Reusable settings/mappings/aliases block referenced by index templates
Composable index template	_index_template: patterns + priority + composed_of + own template; can create data streams
Coordinating node	The node receiving a request; fans it out and merges results (a node with roles: [] does only this)
copy_to	Copies a field's value into another field at index time
Cross-cluster API key	API key created on the remote (/_security/cross_cluster/api_key) that scopes CCS/CCR access
Data stream	Append-only name over hidden backing indices; needs @timestamp and a template with data_stream: {}
Data stream lifecycle (DSL)	Simpler alternative to ILM for data streams: retention (+ downsampling)
Data tier	Node role groups data_hot/warm/cold/frozen/content; placement via _tier_preference
Dense vector	Fixed-length float embedding; searched with knn (HNSW)
Dissect	Delimiter-based parsing (faster, simpler than grok)
DLS / FLS	Document-level security (role query) / field-level security (field_security.grant/except)
Doc values	On-disk columnar structure used for sorting, aggregations and scripts
Dynamic mapping	Auto-mapping of unknown fields; controlled by dynamic: true/false/strict/runtime
Dynamic template	Rule mapping new fields by detected type, name or path; first match wins
ELSER	Elastic Learned Sparse EncodeR: Elastic's sparse-embedding model for semantic search
Enrich policy / processor	Lookup table built from a source index (_execute) and applied at ingest
ES|QL	Piped query language: FROM … | WHERE … | EVAL … | STATS … BY … | SORT … | LIMIT …
Failure store	Per-data-stream store that keeps documents that failed ingestion (<ds>::failures)
Filter context	Yes/no matching with no scoring, cacheable (filter, must_not)
Flattened	Maps an entire object as one field of keywords (avoids mapping explosion)
Flood-stage watermark	95% disk: indices on that node get read_only_allow_delete
Follower / leader index	CCR copy (read-only) and its source
Grok	Regex-based parsing with named patterns (%{IP:client.ip})
Health report	GET _health_report: indicator-based health with diagnoses
Hybrid search	Combining lexical (BM25) and semantic results, e.g. with an RRF retriever
ILM	Index lifecycle management: hot → warm → cold → frozen → delete phases with actions
Index	Logical collection of documents with one mapping and settings, split into shards
Index mode	standard, logsdb, time_series (TSDS), lookup
Inference endpoint	_inference/<task>/<id>: a configured model/service used to create embeddings
Ingest pipeline	Ordered processors that transform documents before indexing
Keystore	elasticsearch-keystore: encrypted store for secure settings
kNN	k-nearest-neighbour vector search (knn section or query)
LOOKUP JOIN	ES|QL join against an index with index.mode: lookup
Mapping	Schema of an index: fields, types, analyzers, parameters
Master-eligible node	Can be elected master; production uses 3
Metric aggregation	Computes values (avg, sum, cardinality, percentiles, stats, top_hits…)
Multi-field	Same value indexed in several ways (fields: { raw: keyword })
Nested	Field type keeping array-of-object members as separate hidden docs; needs nested query/agg
Normalizer	Analyzer-like chain for keyword fields (no tokenizer): lowercase, asciifolding
Pipeline aggregation	Aggregation over other aggregations' outputs (derivative, bucket_selector, max_bucket…)
Point in time (PIT)	Frozen view of indices for consistent deep pagination with search_after
Primary / replica shard	The original / copy of a shard; never on the same node
Query context	Scored matching ("how well"): must, should
Realm	Authentication source: native, file, LDAP, AD, SAML, OIDC, PKI, Kerberos, JWT
Refresh	Makes recent writes searchable (default 1s)
Reindex	Copies documents from a source to a destination index (optionally with query/script/pipeline)
Remote cluster	Named connection (sniff or proxy mode; API key on port 9443, or certificate) used by CCS/CCR
Reroute processor	Ingest processor sending a doc to another data stream
Retriever	Composable search building block: standard, knn, rrf, linear, text_similarity_reranker
Role / role mapping	Named bundle of privileges / rule assigning roles to externally authenticated users
Rollover	Creating a new write index when conditions (age, size, docs) are met
RRF	Reciprocal Rank Fusion: merges ranked lists without score normalisation
Runtime field	Field computed at query time from a script; no reindex needed
search_after	Cursor-based deep pagination using the last hit's sort values
Searchable snapshot	Index mounted from a snapshot repository (fully mounted = cold, partially = frozen)
Segment	Immutable Lucene sub-index inside a shard; merged over time
semantic_text	Field type that auto-chunks and embeds text through an inference endpoint
Service account	Built-in identity for Elastic services (e.g. elastic/kibana) using service tokens
Shard	A Lucene index holding part of an Elasticsearch index
SLM	Snapshot lifecycle management: scheduled snapshots with retention
Snapshot	Incremental backup of indices or cluster state into a repository
Sparse vector	Token → weight embedding (e.g. ELSER)
Streams	Kibana feature to process, partition, map, retain and monitor log data from one UI/API
Synonyms API	_synonyms/<set>: manage synonym sets usable by updateable search-time analyzers
Term-level query	Exact, non-analysed matching (term, terms, range, exists, prefix…)
Translog	Per-shard write-ahead log for durability between flushes
Update By Query	Updates matching docs in place (script or pipeline)
Watermarks (disk)	Low 85% / high 90% / flood 95%
Wired stream	Streams-managed hierarchy rooted at logs (9.2–9.3) with partitioning and inheritance

---

B. Key things to remember

0. Exam mechanics

• ES 9.3, about 3 hours, about 10–12 hands-on tasks, partial credit, only the Elastic docs allowed.
• Grading is based on final cluster state: exact names matter, don't delete extras, and verify with GET after every task.
• The Console shortcut Ctrl/Cmd+/ opens the docs for the API under the cursor.

1. Architecture

• The stack is Collect (Agent/Fleet, Beats, OTel, Logstash) → Process (pipelines, Logstash, Streams) → Store/Search (Elasticsearch) → Explore (Kibana).
• Routing is hash(_routing) % shards, so the primary count is fixed. Change it with _split, _shrink or reindex.
• 3 master-eligible nodes for quorum; cluster.initial_master_nodes is used only on the first bootstrap.
• Shard size 10–50 GB, < 200M docs. cluster.max_shards_per_node = 1000.
• Watermarks: 85 / 90 / 95%. Flood stage sets index.blocks.read_only_allow_delete.
• Forced awareness keeps replicas unassigned rather than overloading a surviving zone, so a yellow cluster can be correct.
• Allocation filters: include = any, exclude = none, require = all. Attributes: _name, _ip, _host, _tier, and custom node.attr.*.
• index.unassigned.node_left.delayed_timeout = 1m. For rolling restarts, use cluster.routing.allocation.enable: primaries.
• Content indices go to data_content; data stream backing indices go to data_hot.

2. Data Management

• Static settings: shards, codec, index.mode, sort, and analysis (close → update → open). Dynamic: replicas, refresh, allocation, blocks, default/final pipeline.
• dynamic: strict rejects, false keeps in _source but doesn't index, runtime creates runtime fields.
• Template precedence: components (in order) < index template template < create-index request.
• One template per index: the highest priority wins. Equal priority with overlapping patterns is an error.
• Built-in templates at priority 100 (logs-*-*, metrics-*-*, logs.otel*…). Use > 500 to override, or avoid overlapping.
• Test with _index_template/_simulate_index/<name>. Templates affect new indices only; roll over data streams.
• Dynamic templates: first match wins, so put specific rules first and the catch-all last. match = leaf name, path_match = full path. There's no integer detection (long/double).
• Data streams: need a template with data_stream: {} and @timestamp. Append-only (create op). Deleting the stream deletes its backing indices.
• Failure store: data_stream_options.failure_store.enabled: true; read <ds>::failures.
• ILM: min_age counts from rollover. Rollover conditions are OR-ed; empty indices never roll over. Poll interval is 10m.
• Rollover needs a data stream or rollover_alias + a bootstrap -000001 with is_write_index: true.
• migrate is implicit in warm and cold; with no tier nodes, the shards don't move.
• Frozen requires searchable_snapshot. Shrink and forcemerge in hot only with rollover.
• Debug with _ilm/explain, then _ilm/retry.
• DSL vs ILM: when both are set, ILM wins (prefer_ilm). DSL is for simple retention only.
• Aliases: _aliases actions are atomic (add, remove, remove_index). Multi-index aliases need one is_write_index to accept writes. Filtered aliases use filter + routing.

3. Ingestion & Processing

• Choose the ingest method: Agent + Fleet for managed hosts and integrations, Logstash for heavy transforms, buffering or multiple outputs, client + _bulk for applications, Streams for parsing logs from the UI, file upload for one-offs.
• Bulk: NDJSON with a final newline; check errors; 5–15 MB batches.
• Immutable once mapped: type, analyzer, index/doc_values, nested vs object. You can add fields, add multi-fields, and add runtime fields.
• New multi-field → _update_by_query to populate existing docs.
• text for full text, keyword for exact/sort/aggs, both via multi-field. Use a normalizer for case-insensitive keyword.
• nested keeps object pairs together; it needs a nested query/agg. Limits: 50 nested fields, 10k nested objects per doc.
• Limits: total_fields.limit 1000, depth.limit 20. Use flattened for unbounded keys.
• scaled_float for prices, constant_keyword for per-index constants, wildcard for grep-style search, match_only_text for logs, date_nanos for ns precision.
• Analyzer order: char_filter → tokenizer → filter. Test with _analyze. Put edge n-grams at index time only, with search_analyzer: standard.
• Search-time synonyms: synonym_graph + Synonyms API + updateable: true.
• Reindex: create the destination mapping first; op_type: create for data streams; conflicts: proceed; wait_for_completion=false → _tasks; remote reindex needs reindex.remote.whitelist.
• Painless: ctx._source.x in reindex/UBQ scripts vs ctx.x in ingest; ctx.op = 'noop'|'delete'; null-safe ?..
• Pipelines: if conditions, ignore_failure, ignore_missing, and processor or pipeline on_failure (_ingest.on_failure_message). Always _simulate?verbose.
• default_pipeline can be overridden by ?pipeline=; final_pipeline always runs.
• _ingest.timestamp is the ingest time. The reroute processor changes the target data stream. Use @custom pipelines to customise integrations.
• Enrich: create the policy → _execute → enrich processor. Policies are immutable (delete and recreate), and must be re-executed when the source changes.
• Streams: classic = existing data streams in place; wired = hierarchy rooted at logs (9.2–9.3) with partitioning and inheritance. The tabs are Processing (with preview), Schema, Data lifecycle (inherit / custom / ILM), Data quality (degraded vs failed → failure store) and Advanced. Don't edit managed components with raw APIs. The API lives under kbn:/api/streams.

4. Searching Data

• match is analysed (OR by default; operator: and, minimum_should_match, fuzziness: AUTO). term is exact. Never use term on a text field; use .keyword.
• match_phrase + slop; multi_match with fields: ["title^3"], types best_fields (default), most_fields, cross_fields, phrase, bool_prefix.
• Bool: must (scored), filter (unscored, cached), should (scored, optional if a must/filter exists, otherwise at least 1 required), must_not (unscored).
• post_filter filters hits but not aggregations (facets).
• from + size ≤ 10,000 (max_result_window); for deeper pages, use search_after + PIT. You can't sort on text.
• Aggregations: set size: 0; use terms on keyword only; terms counts are approximate (shard_size); use composite to page through all buckets.
• date_histogram: calendar_interval (1d, 1M, week…) vs fixed_interval (30m, 7d); min_doc_count: 0 + extended_bounds for empty buckets; time_zone.
• range agg: from is inclusive, to exclusive.
• buckets_path: > for nesting, . for a metric's value, _count for doc count. bucket_selector works like HAVING; bucket_sort sorts and truncates buckets.
• ES|QL: POST _query (?format=txt). The default LIMIT is 1000 (max 10k). STATS … BY, EVAL, KEEP/DROP/RENAME, DISSECT/GROK, ENRICH, LOOKUP JOIN (the lookup index needs index.mode: lookup), BUCKET(), MATCH() or : for full text, METADATA _index, _score, ? params, and remote:index for CCS.
• Semantic search: an inference endpoint (a default ELSER one exists) + a semantic_text field. Query with match or semantic on it; highlight with type: semantic. Hybrid uses the rrf retriever (rank_window_size, rank_constant). Manual path: dense_vector + knn (k, num_candidates, filter) or sparse_vector query. It needs ML capacity and a licence (trial in the lab).
• CCS: alias:index, *:index, exclusions -alias:*. skip_unavailable defaults to true; ccs_minimize_roundtrips defaults to true. _resolve/cluster checks connectivity. _clusters in the response shows per-cluster status.

5. Cluster Management

• Green / yellow / red = all assigned / replica(s) unassigned / primary unassigned.
• The workflow: _cluster/health → _cat/shards → _cluster/allocation/explain → fix → _cluster/reroute?retry_failed=true if max retries was hit.
• Common causes: too few nodes for the replicas, allocation filters, awareness, disk watermarks, a missing tier, allocation.enable set to none/primaries, max retries.
• Red with all copies lost: restore from a snapshot. allocate_empty_primary / allocate_stale_primary (with accept_data_loss) is the last resort.
• persistent cluster settings survive restarts; transient is deprecated. null resets a setting.
• Monitor heap (> 85% is bad), breakers, thread-pool rejections, disk, slow logs and hot threads. Use Stack Monitoring (Agent/Metricbeat → monitoring cluster) or AutoOps.
• Snapshots: fs repos need path.repo on every node. Snapshots are incremental. Restore needs the target closed or absent; use rename_pattern/rename_replacement otherwise. include_global_state and feature_states control what's restored.
• SLM: Quartz cron (seconds first), date-math names <nightly-{now/d}>, retention (expire_after, min_count, max_count), _execute to run now.
• Remote clusters: the API-key model is recommended (remote port 9443, remote_cluster_server.enabled, cross-cluster API key with access.search/access.replication, local keystore cluster.remote.<alias>.credentials). The certificate model is deprecated (port 9300, same-named roles on the remote). Modes are sniff (default) and proxy. Check with GET _remote/info. Local nodes need remote_cluster_client.
• The local role grants remote access through remote_indices.
• CCR: licence on both clusters; run on the follower cluster: PUT <follower>/_ccr/follow {remote_cluster, leader_index}. Auto-follow uses leader_index_patterns + follow_index_pattern: "{{leader_index}}-x" and covers new indices only. Followers are read-only. Failover (promote a follower): pause_follow → _close → unfollow → _open. Soft deletes must be on (the default).

6. Security & Access Control

• A role has cluster + indices (names, privileges, field_security, query) + applications (Kibana: kibana-.kibana, feature_x.read/all, space:<id>) + remote_indices.
• A user's roles are a union: one role with unrestricted read cancels another role's DLS.
• Key privileges: read + view_index_metadata for Kibana data views; create_doc + auto_configure for shippers writing to data streams; manage_security is effectively admin.
• DLS templating: {{_user.username}}. FLS: grant / except with wildcards.
• API keys get the intersection of their role_descriptors and the creator's privileges; use expiration; invalidate with DELETE _security/api_key.
• Role mappings use rules (field/any/all/except on groups, dn, username, realm.name) for LDAP, AD, SAML and OIDC users.
• Test with _security/_authenticate, _security/user/_has_privileges and the es-security-runas-user header.
• Production security: security on; transport TLS is mandatory for multi-node clusters with security; HTTPS for clients and Kibana; secrets in the keystore; Kibana via a service account token; no daily use of elastic.
• Tools: elasticsearch-certutil ca|cert|http, elasticsearch-reset-password -u <user>, elasticsearch-create-enrollment-token -s kibana|node, elasticsearch-service-tokens create, elasticsearch-keystore add, POST _nodes/reload_secure_settings.
• Bootstrap checks (production mode starts when you bind a non-loopback network.host): heap Xms = Xmx, ≤ 50% RAM, < ~31 GB; file descriptors ≥ 65,535; threads ≥ 4,096; vm.max_map_count ≥ 262,144; memory lock works if enabled; discovery configured; swap disabled.
• Audit logging (xpack.security.audit.enabled, events.include), IP filtering (xpack.security.transport|http.filter.allow/deny), and action.destructive_requires_name: true.
• Kibana needs a 32+ character xpack.encryptedSavedObjects.encryptionKey for alerting and encrypted saved objects.

---

C. Ten-second API recall

Need	API
Health & diagnosis	_cluster/health, _health_report, _cat/shards, _cluster/allocation/explain, _cluster/reroute?retry_failed=true
Templates	_component_template, _index_template, _index_template/_simulate_index/<n>
Data streams	_data_stream/<n>, <ds>/_rollover, <ds>::failures, _data_stream/<n>/_lifecycle
ILM	_ilm/policy/<p>, <idx>/_ilm/explain, <idx>/_ilm/retry
Aliases	POST _aliases {actions:[add/remove/remove_index]}
Mapping & analysis	<idx>/_mapping, <idx>/_analyze, _synonyms/<set>
Move data	_reindex, <idx>/_update_by_query, <idx>/_delete_by_query, _tasks/<id>
Pipelines	_ingest/pipeline/<p>, …/_simulate?verbose, _ingest/_simulate, _enrich/policy/<p>/_execute
Search	<idx>/_search, _async_search, <idx>/_pit, _query (ES|QL), _resolve/cluster/<pattern>
Semantic	_inference/<task>/<id>, semantic_text, semantic query, rrf retriever
Snapshots	_snapshot/<repo>, _snapshot/<repo>/<snap>/_restore, _slm/policy/<p>/_execute
Remote / CCR	_remote/info, _security/cross_cluster/api_key, <f>/_ccr/follow, _ccr/auto_follow/<n>, <f>/_ccr/unfollow
Security	_security/role/<r>, _security/user/<u>, _security/api_key, _security/role_mapping/<m>, _security/_authenticate, _security/user/_has_privileges

Elasticsearch Engineering

A study guide for the Elastic Certified Engineer exam (Elasticsearch 9.3), organised by the official exam objectives.

How to read this guide. Each objective gets (1) a mental model, often an everyday analogy, (2) the APIs you'll actually type, (3) the traps graders like, and (4) links to the exact Elastic documentation pages. The Elastic documentation is the only resource available during the exam, so the 📖 links double as a map of where to go on exam day. Exercises with solution notes close each domain. The companion page, Elasticsearch Engineering – Digest, holds the glossary and the must-remember facts.

---

0. The exam and your preparation

0.1 Format

Item	Detail
Version	Elasticsearch 9.3 (the 8.x exam was retired on 1 Sep 2026)
Type	Hands-on, performance-based, on live Elasticsearch and Kibana clusters, remote-proctored
Duration	About 3 hours
Tasks	About 10–12 practical tasks, often combining several objectives, with partial credit
Allowed resources	Only the Elastic Documentation
Grading	The final state of your clusters: indices, templates, pipelines, roles, and so on

📖 Exam page · Certification FAQ

Analogy. It's a practical driving test, not a written theory test. Nobody asks you to recite the highway code. The examiner checks whether the car ends up correctly parked. The docs are your satnav: allowed, but only useful if you already know roughly where you're going.

0.2 The six domains

#	Domain	Objectives
1	Elasticsearch Architecture	Stack architecture; shard sizing and allocation strategies
2	Data Management	Index definition; dynamic templates; ILM for time series; index template that creates a data stream; aliases
3	Data Ingestion & Processing	Ingest methods; mappings; field data types; Reindex and Update By Query; ingest pipelines; Streams
4	Searching Data	Term and phrase queries; bool combinations; metric and bucket aggregations; ES|QL; semantic search; cross-cluster search
5	Cluster Management	Monitor and maintain cluster health; cross-cluster operations
6	Security & Access Control	RBAC; securing a cluster for production

0.3 Build a lab first

You can't prepare for a hands-on exam by reading. Use 9.3.x so the APIs and UI match the exam.

Option A: single node in one command (good for Domains 2–4)

curl -fsSL https://elastic.co/start-local | sh

📖 Local installation quickstart

Option B: three-node Docker Compose (needed for allocation, awareness, snapshots, and Domain 5)

📖 Multi-node cluster with Docker Compose

Settings to vary per node (as elasticsearch.yml or environment variables):

cluster.name: lab
node.name: es01                                  # es02, es03
node.roles: [ master, data_hot, data_content, ingest, remote_cluster_client ]
node.attr.zone: zone-a                           # zone-b, zone-c → awareness exercises
discovery.seed_hosts: [ es01, es02, es03 ]
cluster.initial_master_nodes: [ es01, es02, es03 ]   # first bootstrap ONLY, then remove
path.repo: [ /usr/share/elasticsearch/snapshots ]    # shared FS repo for snapshots

For Domain 5, add a second cluster (e.g. remote) so you can practise remote clusters, cross-cluster search (CCS) and cross-cluster replication (CCR).

Enable a trial license. CCR, searchable snapshots, some security features and ML-based semantic search need it:

POST _license/start_trial?acknowledge=true

Load sample data. In Kibana: Home → Add data → Sample data (eCommerce, Flights, Web logs). Many exam tasks look exactly like work on these datasets.

0.4 Exam-day habits that earn points

1. Read each task twice and copy names (indices, fields, pipelines, roles) straight from the task text. A typo scores zero.
2. Verify everything. GET what you created. Compare _count before and after a reindex. _simulate a pipeline before using it.
3. Don't destroy state the task didn't mention. Graders inspect the final state.
4. Prototype on scratch indices (tmp-*), then clean up.
5. Time-box. Roughly 15 minutes per task. At 20 minutes, move on and come back later.
6. Know the "immutable" things. You can't change shard count, field types or a field's analyzer in place. The answer is a new index plus _reindex, or _split/_shrink for shards.
7. Use Console shortcuts. Ctrl/Cmd+Enter runs a request, Ctrl/Cmd+/ opens the docs for the API under the cursor, and Ctrl/Cmd+I auto-indents.

📖 Console (Dev Tools)

---

1. Elasticsearch Architecture

1.1 Describe the Elastic Stack architecture

The big picture

Analogy: a postal system. Collectors (Elastic Agent, Beats, OpenTelemetry collectors, Logstash) pick up letters everywhere. Sorting centres (Logstash, ingest pipelines, Streams) open, stamp and classify them. The warehouse (Elasticsearch) stores every letter so it can be found in milliseconds. The front desk (Kibana) is where people ask questions and see dashboards. Fleet is the dispatch office that tells every collector what to pick up.

Layer	Components	Role
Collect	Elastic Agent (+ Fleet Server), Beats (Filebeat, Metricbeat…), EDOT / OpenTelemetry Collector, APM agents, language clients, connectors	Ship logs, metrics, traces and documents
Process	Logstash, Elasticsearch ingest pipelines, Streams, integrations' built-in pipelines	Parse, enrich, route, transform
Store / search / analyse	Elasticsearch	Distributed document store, search engine, analytics and vector database built on Apache Lucene
Explore / manage	Kibana (Discover, Dashboards, Dev Tools, Stack Management, Fleet UI, Streams UI)	Visualise, query, administer
Solutions	Search, Observability, Security	Opinionated apps on top of the same data

Deployment options: Elastic Cloud Hosted (ECH), Elastic Cloud Serverless (no nodes or shards to manage), Elastic Cloud Enterprise (ECE), Elastic Cloud on Kubernetes (ECK), and self-managed. The exam uses self-managed-style clusters, so everything below assumes you can touch nodes and settings.

📖 Elastic fundamentals · Deployment comparison · Ingest tools overview

Inside Elasticsearch: cluster → node → index → shard → segment

Analogy: a library chain. The cluster is the whole chain. Each node is a branch building. An index is a book collection (for example "novels"). Because the collection is too big for one shelf, it's cut into volumes: the primary shards. Each volume has photocopies (replica shards), always kept in a different branch. Inside each volume, pages are glued in batches (segments) that are never edited, only replaced when batches are merged.

Concept	What to know
Cluster	One or more nodes sharing a cluster.name. One elected master manages the cluster state (metadata: indices, mappings, settings, shard routing table).
Node	One Elasticsearch process. Its roles decide what it does (table below).
Index	A logical namespace of JSON documents with one mapping and one set of settings.
Shard	A Lucene index. Primary shards hold the data; replicas are copies that give high availability and extra read throughput. A primary and its replica are never on the same node.
Segment	An immutable, self-contained piece of a shard. Deletes are marked; merges physically drop them.
Document routing	shard = hash(_routing) % number_of_routing_shards → mapped to primaries. _routing defaults to _id. This is why the primary count is fixed at creation.

Node roles (node.roles in elasticsearch.yml):

Role	Purpose
master	Eligible to be elected master. Run 3 dedicated master-eligible nodes in production.
voting_only	Votes in elections but never becomes master (a tie-breaker).
data	Generic data role (holds all tiers).
data_content	Non-time-series content (product catalogues and so on).
data_hot / data_warm / data_cold / data_frozen	Data tiers for time-series data, from fastest and most expensive to cheapest.
ingest	Runs ingest pipelines.
ml	Runs machine-learning jobs and trained models (e.g. ELSER).
remote_cluster_client	Can connect to remote clusters (needed for CCS and CCR).
transform	Runs transforms.
[] (empty)	Coordinating-only node: routes requests and merges results.

Every node is also a coordinating node. Whichever node receives the request fans it out and gathers the results.

Write path: client → coordinating node → routes to the primary shard → primary indexes (in-memory buffer + translog) → forwards to replicas in parallel → acknowledges once the required copies succeed (wait_for_active_shards, default 1 = primary only).

Near real-time search: a refresh (default every 1s, only on indices that have been searched recently) turns the buffer into a searchable segment. A flush makes segments durable on disk and trims the translog. Merges combine small segments in the background.

Read path (query then fetch): the coordinating node sends the query to one copy of each shard. Each shard returns its top IDs and scores (query phase). The coordinator merges them, then fetches the actual documents (fetch phase).

Cluster formation: master-eligible nodes elect a master by quorum (majority of the voting configuration). Three master-eligible nodes tolerate one failure. cluster.initial_master_nodes is used only for the very first bootstrap; remove it afterwards.

📖 Clusters, nodes and shards · Node roles · Reading and writing documents · Near real-time search · Discovery and cluster formation · Quorum-based decisions

Data tiers

Analogy: a wardrobe. Hot is the chair where today's clothes lie: instant access, most expensive space. Warm is the wardrobe. Cold is the suitcase under the bed. Frozen is the storage unit across town: you can still get anything, but it takes a trip. Content is your bookshelf. It's not "time-based", it's just always there.

• New indices go to data_content by default; data stream backing indices go to data_hot.
• Tier placement uses index.routing.allocation.include._tier_preference (e.g. data_warm,data_hot), which ILM's migrate action sets automatically.
• Frozen uses partially mounted searchable snapshots: a local cache plus the object store. Cold can use fully mounted searchable snapshots, which lets it drop replicas.

📖 Data tiers · Searchable snapshots

1.2 Describe shard sizing and allocation strategies

Shard sizing: the rules of thumb

Analogy: moving boxes. If boxes are too big, nobody can lift them: slow recovery, slow rebalancing. If there are too many tiny boxes, the truck spends its time on paperwork: every shard costs heap, file handles and cluster-state overhead. Aim for boxes one person can carry.

Guideline	Value
Target shard size	10 GB – 50 GB (time-series often aims for about 50 GB via rollover max_primary_shard_size)
Documents per shard	Keep under 200 million
Max shards per node (non-frozen)	cluster.max_shards_per_node = 1000 by default. It's a soft limit that blocks index creation.
Heap overhead	Mostly driven by mapped fields and segments. Avoid mapping explosions and thousands of tiny indices.
Searches	Each search runs one thread per shard copy. Many small shards means lots of queuing.
Time series	Use data streams + rollover instead of daily indices to control size, not calendar.
Too many tiny shards?	_shrink, _forcemerge, reindex into fewer indices, raise rollover thresholds.
Shard too big?	_split, or lower rollover thresholds for new data.

GET _cat/shards?v&s=store:desc&h=index,shard,prirep,state,docs,store,node
GET _cat/indices?v&s=pri.store.size:desc
GET _cluster/stats?filter_path=indices.shards,indices.count

📖 Size your shards · Scaling considerations

How shard allocation works

The master's allocator decides where every shard copy lives, filtered by deciders. Each decider can say YES, NO or THROTTLE. The common ones:

Decider	Rule
Same shard	Never two copies of the same shard on one node
Disk threshold	Watermarks: low 85% (no new shards to this node), high 90% (move shards away), flood stage 95% (indices on it become read-only: index.blocks.read_only_allow_delete)
Awareness	Spread copies across attribute values (zone or rack)
Filtering	include / exclude / require rules at index or cluster level
Data tier	_tier_preference must match a node's role
Max retries	After 5 failed allocations (index.allocation.max_retries) the shard gives up until retried
Enable	cluster.routing.allocation.enable: all / primaries / new_primaries / none
Total shards per node	index.routing.allocation.total_shards_per_node hard limit

The single best diagnostic API:

GET _cluster/allocation/explain
{ "index": "orders", "shard": 0, "primary": false }

Strategy 1: allocation awareness (survive losing a zone or rack)

Analogy. Don't keep your passport and its photocopy in the same bag.

# elasticsearch.yml on each node
node.attr.zone: zone-a        # zone-b on others

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone",
    "cluster.routing.allocation.awareness.force.zone.values": "zone-a,zone-b"
  }
}

• Awareness (first setting): Elasticsearch spreads copies across zones where it can.
• Forced awareness (second setting): if zone-b disappears, Elasticsearch won't squeeze all replicas into zone-a (which could overload it). The replicas stay unassigned instead. That's intended behaviour, so a yellow cluster can be the correct answer.

📖 Shard allocation awareness

Strategy 2: allocation filtering (pin or evict shards)

// Index-level: keep "hot-orders" only on nodes tagged box_type=ssd
PUT hot-orders/_settings
{ "index.routing.allocation.require.box_type": "ssd" }

// Index-level by node name (e.g. to prepare a shrink)
PUT legacy/_settings
{ "index.routing.allocation.require._name": "es01" }

// Cluster-level: drain a node before decommissioning it
PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.exclude._name": "es03" } }

include means at least one of the values, exclude means none of them, and require means all of them. Built-in attributes: _name, _host_ip, _publish_ip, _ip, _host, _id, _tier.

📖 Index-level shard allocation · Shard allocation, relocation and recovery

Strategy 3: delayed allocation and rolling restarts

When a node leaves, its replicas are reallocated elsewhere only after index.unassigned.node_left.delayed_timeout (default 1m), which avoids expensive reshuffles when a node is just rebooting. For planned maintenance:

PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": "primaries" } }
POST _flush
// …restart the node, wait for it to join…
PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": null } }

📖 Delaying allocation when a node leaves · Rolling restart

Strategy 4: resilience design

• Small clusters: at least 3 master-eligible nodes, and 1 replica for every index you can't afford to lose. A 2-node cluster isn't resilient: losing the wrong node leaves no voting majority. Add a third node (it can be a small voting_only, master-eligible tie-breaker).
• Larger clusters: dedicated masters, spread across 3 zones, with awareness enabled.

📖 Resilience in small clusters · Resilience in larger clusters

1.3 Exercises: Architecture

1. On your three-node lab, give the nodes attributes zone: a, zone: a, zone: b. Configure forced awareness on zone with values a,b. Create index aware with 1 shard and 1 replica. Stop the zone: b node and explain the resulting health.
2. Create index ssd-only whose shards may only live on nodes with attribute disk: ssd (set it on one node). Prove it with _cat/shards.
3. Decommission node es03: move all its shards off without stopping it.
4. Explain why an index created with 1 primary shard can't be given 3 primaries in place, and name the two ways to get there.

Solution notes

1. Settings as in Strategy 1. With zone b gone, the replica stays unassigned because forced awareness forbids placing both copies in zone a, so health is yellow. _cluster/allocation/explain names the awareness decider.
2. node.attr.disk: ssd on one node, then PUT ssd-only {"settings":{"index.routing.allocation.require.disk":"ssd","number_of_replicas":0}}. With 1 replica, the replica can't allocate because only one node qualifies.
3. PUT _cluster/settings {"persistent":{"cluster.routing.allocation.exclude._name":"es03"}}, then watch _cat/shards until nothing remains on es03.
4. Routing is hash(_routing) % shards, so changing the count would misplace every document. Use _split (by multiples of the routing shards; needs a write block) or _reindex into a new index.

---

2. Data Management

Analogy for the whole domain: a filing cabinet factory. An index is a filing cabinet. Its settings are the hardware: how many drawers (primary shards) and how many photocopied backups (replicas). Its mapping is the labelling scheme for folders. Templates are the factory spec sheet, so every new cabinet named logs-* ships pre-configured. Aliases are nicknames on the door ("current-orders"), so people don't need the serial number. ILM is the records-retention policy: new files on the desk, older ones in the basement, ancient ones in the shredder.

2.1 Define an index that satisfies a given set of requirements

An index creation request has three optional parts: settings, mappings and aliases.

PUT products
{
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "refresh_interval": "30s",
    "analysis": {
      "analyzer": {
        "folded": { "type": "custom", "tokenizer": "standard",
                    "filter": ["lowercase", "asciifolding"] }
      }
    }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "sku":      { "type": "keyword" },
      "name":     { "type": "text", "analyzer": "folded",
                    "fields": { "raw": { "type": "keyword", "ignore_above": 256 } } },
      "price":    { "type": "scaled_float", "scaling_factor": 100 },
      "in_stock": { "type": "boolean" },
      "created":  { "type": "date", "format": "yyyy-MM-dd||epoch_millis" },
      "tags":     { "type": "keyword" }
    }
  },
  "aliases": { "catalog": {} }
}

Static vs dynamic settings. Know which is which:

Static (at creation, or on a closed index)	Dynamic (PUT <index>/_settings any time)
index.number_of_shards	index.number_of_replicas
index.codec (e.g. best_compression)	index.refresh_interval (-1 disables refresh)
index.mode (standard, logsdb, time_series, lookup)	index.routing.allocation.*
index.sort.*	index.blocks.write / read_only / read
analysis definitions (close → update → open)	index.max_result_window, index.default_pipeline, index.final_pipeline

PUT products/_settings
{ "index": { "number_of_replicas": 0, "refresh_interval": "-1" } }   // bulk-load mode

Adding an analyzer to an existing index: POST products/_close → PUT products/_settings with the analysis block → POST products/_open. You still can't change the analyzer of an existing field. Add a new field or multi-field, or reindex.

Changing the shard count:

• _split: more shards, by a multiple. Needs index.blocks.write: true.
• _shrink: fewer shards, to a factor. Needs a write block and a copy of every shard on one node (index.routing.allocation.require._name).
• Otherwise _reindex.

PUT products/_settings { "index.blocks.write": true }
POST products/_split/products-4sh
{ "settings": { "index.number_of_shards": 4, "index.blocks.write": null } }

dynamic mapping modes, a frequent grader trap:

Value	Unknown field in a document…
true (default)	Added to the mapping automatically
runtime	Added as a runtime field (not indexed, computed at query time)
false	Accepted and kept in _source, but not indexed or searchable
strict	Document rejected with an error

Task phrasing to watch for: "unknown fields must be rejected" means strict; "must be stored but not searchable" means false.

Index modes (9.x). logsdb is the storage-optimised mode for logs, with synthetic _source and index sorting. It's the default for logs-*-* data streams in 9.x. time_series (TSDS) is for metrics. lookup is required for the right-hand index of an ES|QL LOOKUP JOIN.

📖 Create index API · Index settings reference · Index fundamentals · Dynamic mapping · Logs data streams (logsdb)

2.2 Index templates and component templates

Composable index templates (_index_template), built from reusable component templates (_component_template), are the standard. Legacy _template templates have been deprecated since 7.8; composable ones always win over them.

Analogy: Lego. Component templates are bricks: a "1 shard" brick, an "HTTP fields" brick. The index template is the instruction sheet saying which bricks to snap together for anything named web-*, plus a few extra pieces of its own.

PUT _component_template/ct-settings-small
{ "template": { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } } }

PUT _component_template/ct-mappings-http
{
  "template": {
    "mappings": {
      "properties": {
        "http.status": { "type": "short" },
        "url":         { "type": "keyword" }
      }
    }
  }
}

PUT _index_template/web-template
{
  "index_patterns": ["web-*"],
  "priority": 200,
  "composed_of": ["ct-settings-small", "ct-mappings-http"],
  "template": {
    "settings": { "number_of_replicas": 2 },
    "aliases":  { "web-all": {} }
  },
  "_meta": { "owner": "exam" }
}

Precedence, from lowest to highest: component templates in composed_of order (later overrides earlier), then the index template's own template block, then settings in the actual create-index request. In the example, replicas end up at 2.

Priority rules:

• Only one index template applies to a new index: the matching one with the highest priority.
• Two templates with equal priority and overlapping patterns are rejected when you create the second.
• Built-in templates use priority 100 for logs-*-*, metrics-*-*, synthetics-*-*, profiling-*, security_solution-*-*, and the Streams patterns logs.otel* and logs.ecs*. Fleet integrations go up to 200. To deliberately override a built-in pattern, the docs recommend priority > 500. Otherwise choose non-overlapping patterns.

Verify before you commit:

POST _index_template/_simulate_index/web-2026.09.28   // what would this index get?
POST _index_template/_simulate/web-template            // resolved template
GET  _index_template/web-template

Templates apply only when an index is created. Changing a template never alters existing indices. For data streams, roll over to pick up changes.

📖 Templates · Simulate multi-component templates · Put index template API · Put component template API

2.3 Define and use a dynamic template that satisfies a given set of requirements

Dynamic templates control how newly detected fields get mapped. You match on the JSON-detected type (match_mapping_type), the field name (match / unmatch, wildcards or "match_pattern": "regex"), or the dotted path (path_match / path_unmatch).

Analogy. A mail-room rulebook: "anything that looks like a parcel and whose label ends in _id goes to shelf K; every other parcel goes to shelf T". The first rule that fits wins.

PUT _index_template/sensor-template
{
  "index_patterns": ["sensor-*"],
  "template": {
    "mappings": {
      "dynamic_templates": [
        { "ids_as_keyword": {
            "match_mapping_type": "string",
            "match": "*_id",
            "mapping": { "type": "keyword" } } },
        { "desc_english": {
            "match_mapping_type": "string",
            "match": "desc_*",
            "mapping": { "type": "text", "analyzer": "english" } } },
        { "counts_as_integer": {
            "match_mapping_type": "long",
            "match": "*_count",
            "mapping": { "type": "integer" } } },
        { "raw_not_indexed": {
            "path_match": "raw.*",
            "mapping": { "type": "keyword", "index": false, "doc_values": false } } },
        { "strings_default": {
            "match_mapping_type": "string",
            "mapping": { "type": "text",
                         "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }
      ]
    }
  }
}

Rules to remember:

• Order matters: first match wins. Put specific rules first and the catch-all (strings_default) last. A catch-all placed first silently shadows the others, which is a classic hidden bug in exam tasks.
• match_mapping_type values are JSON-detected types: string, long, double, boolean, date, object, binary, or *. There's no integer or float detection.
• Placeholders: {name} (field name) and {dynamic_type} (detected type), e.g. "analyzer": "{name}".
• Use "runtime": { "type": "keyword" } instead of "mapping" to create runtime fields dynamically.
• path_match matches the full dotted path (raw.a.b); match matches only the leaf name.
• Dynamic templates apply to new fields only. Fields already mapped are untouched.
• Test it: index one sample document, then GET sensor-1/_mapping.

📖 Dynamic templates · Dynamic field mapping

2.4 Define an index template that creates a new data stream

A data stream is one name (e.g. logs-app-prod) in front of a series of hidden backing indices (.ds-logs-app-prod-2026.09.28-000001, …). It's for append-only, time-stamped data and requires an @timestamp field (date or date_nanos).

Analogy: a notebook with detachable pages. You always write on the current page (the write index). When a page fills up (rollover), a new page is added. You can read all pages at once, and you tear out and throw away old pages individually (ILM delete).

Only a matching index template with "data_stream": {} creates a data stream. Without it, writing to logs-app-prod creates an ordinary index.

PUT _index_template/logs-app-template
{
  "index_patterns": ["logs-app-*"],
  "data_stream": { },
  "priority": 501,
  "composed_of": [],
  "template": {
    "settings": { "index.lifecycle.name": "logs-app-policy" },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message":    { "type": "text" },
        "level":      { "type": "keyword" }
      }
    },
    "data_stream_options": { "failure_store": { "enabled": true } }
  }
}

// Created implicitly on first write…
POST logs-app-prod/_doc
{ "@timestamp": "2026-09-28T10:00:00Z", "message": "started", "level": "INFO" }

// …or explicitly
PUT _data_stream/logs-app-prod

Operations and constraints:

Want to…	Do
Add documents	POST <ds>/_doc, or _bulk with the create action. Data streams are append-only; index only works with op_type=create.
Update or delete documents	_update_by_query / _delete_by_query on the stream, or single-doc ops on the backing index with if_seq_no / if_primary_term
Roll over now	POST logs-app-prod/_rollover
Inspect	GET _data_stream/logs-app-prod, GET _data_stream/logs-app-prod/_stats
Delete	DELETE _data_stream/logs-app-prod (deletes its backing indices too)
Change mappings or settings	Update the template, then roll over. Some changes can also be applied to the stream via the update mapping API.
Reindex into a stream	"dest": { "index": "logs-app-prod", "op_type": "create" }
Convert an alias with a write index into a stream	POST _data_stream/_migrate/<alias>
Capture rejected documents	Failure store (data_stream_options.failure_store.enabled). Query it with FROM logs-app-prod::failures or GET logs-app-prod::failures/_search.

Naming scheme. Elastic's convention is type-dataset-namespace (e.g. logs-nginx.access-prod). Any name works for your own templates, but anything matching logs-*-* hits the built-in template (priority 100) unless yours has higher priority.

📖 Data streams · Set up a data stream · Use a data stream · Modify a data stream · Failure store · Time series data streams

2.5 Define an Index Lifecycle Management policy for a time-series index

ILM moves each index through phases: hot → warm → cold → frozen → delete. Each phase has a min_age (measured from rollover, or from creation if the index never rolled over) and a set of actions.

Analogy: a restaurant kitchen's food rotation. Fresh produce is on the counter (hot). After a few days it moves to the walk-in fridge (warm), then the freezer (cold or frozen), and finally the bin (delete). The chef doesn't move things by hand; the rotation rules do.

PUT _ilm/policy/logs-app-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink":     { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate":   { "number_of_replicas": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": { "searchable_snapshot": { "snapshot_repository": "my-repo" } }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

Where each action is allowed:

Action	Hot	Warm	Cold	Frozen	Delete
rollover	✓
set_priority	✓	✓	✓
unfollow (CCR)	✓	✓	✓	✓
readonly	✓*	✓	✓
shrink, forcemerge	✓*	✓
downsample (TSDS)	✓*	✓	✓
allocate, migrate		✓	✓
searchable_snapshot	✓*		✓	✓ (required)
wait_for_snapshot, delete					✓

* In hot, only when rollover is also configured.

Key facts:

• Rollover conditions are OR-ed (max_age, max_docs, max_size, max_primary_shard_size, max_primary_shard_docs). min_* conditions are AND-ed floors. Elasticsearch never rolls over an empty index.
• Rollover needs a data stream or a write alias: index.lifecycle.rollover_alias plus a bootstrap index name-000001 with "is_write_index": true.
• migrate is implicit in warm and cold. It sets _tier_preference to the phase's tier. With no nodes of that tier, the shards stay where they are, which is a common "why isn't it moving?" moment. Disable it with "migrate": {"enabled": false} when using attribute-based allocate.
• ILM checks every indices.lifecycle.poll_interval (10m default). In a lab, set it to 10s to see transitions quickly.
• Attach a policy to new indices via the template (index.lifecycle.name). For existing indices, use PUT <index>/_settings {"index.lifecycle.name": "…"}.
• Changing a policy: an index already in a phase keeps the cached phase definition and picks up the new one at the next phase.

Bootstrap pattern without data streams (a classic task):

PUT _index_template/orders-tpl
{
  "index_patterns": ["orders-*"],
  "template": { "settings": {
    "index.lifecycle.name": "orders-policy",
    "index.lifecycle.rollover_alias": "orders" } }
}
PUT orders-000001
{ "aliases": { "orders": { "is_write_index": true } } }

Debugging ILM:

GET logs-app-prod/_ilm/explain             // phase, action, step, failed_step, step_info
POST logs-app-prod/_ilm/retry               // after fixing the cause of an ERROR step
POST _ilm/move/<index>                      // force a step (rare)
GET _ilm/status          POST _ilm/start    POST _ilm/stop
PUT _cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "10s" } }

Data stream lifecycle (DSL): the lighter alternative

For data streams you can skip ILM and just declare retention (plus optional downsampling). Elasticsearch handles rollover and deletion itself.

PUT _index_template/metrics-app-template
{
  "index_patterns": ["metrics-app-*"],
  "data_stream": {},
  "template": { "lifecycle": { "data_retention": "30d" } }
}
PUT _data_stream/metrics-app-prod/_lifecycle   { "data_retention": "7d" }
GET _data_stream/metrics-app-prod/_lifecycle/explain

If both are configured, ILM wins (index.lifecycle.prefer_ilm defaults to true). Choose ILM when you need tiers (warm, cold, frozen), shrink, forcemerge or searchable snapshots. Choose DSL when retention is all that matters.

📖 Index lifecycle management · ILM phases and actions · Rollover · Create an ILM policy · Tutorial: time series with data streams · Tutorial: time series without data streams · Check ILM status · Data stream lifecycle

2.6 Define and use index aliases

An alias is a secondary name for one or more indices or data streams.

Analogy: a phone's contact name. You call "Mum", not the number. When Mum changes numbers, you edit the contact once and nobody else has to change anything. Apps query products, and you swap which physical index sits behind it.

Core uses:

1. Zero-downtime reindex: build products-v2, then atomically move the alias.
2. Filtered views: an alias that only shows region: EU documents.
3. Rollover write target: is_write_index: true.
4. Search many indices under one name: logs-all → logs-2026.09.*.

// Atomic swap. All actions succeed or none do.
POST _aliases
{
  "actions": [
    { "remove": { "index": "products-v1", "alias": "products" } },
    { "add":    { "index": "products-v2", "alias": "products", "is_write_index": true } }
  ]
}

// Filtered alias with routing
POST _aliases
{
  "actions": [
    { "add": {
        "index": "orders",
        "alias": "orders-eu",
        "filter": { "term": { "region": "EU" } },
        "routing": "eu"
    } }
  ]
}

// Remove an index entirely and point the alias elsewhere, atomically
POST _aliases
{ "actions": [
  { "remove_index": { "index": "products-old" } },
  { "add": { "index": "products-v3", "alias": "products" } } ] }

GET _alias/products          GET products-v2/_alias          GET _cat/aliases?v

Rules:

• An alias pointing to several indices is read-only unless exactly one of them has is_write_index: true.
• Aliases can point to data streams, but you can't mix data streams and indices in one alias.
• Filtered aliases apply the filter to searches, counts and delete-by-query run through the alias.
• Aliases can be defined in index templates (template.aliases) so every new index joins automatically.
• An alias name can't equal an existing index name.

📖 Aliases · Aliases API · Privileges for data streams and aliases

2.7 Exercises: Data Management

1. Create component templates ct-1shard (1 shard, 0 replicas) and ct-app-mappings (@timestamp date, service keyword, msg text). Create index template app-logs for app-logs-* that creates a data stream, uses both components, and has priority 300. Index a document into app-logs-api and confirm the backing index has 1 shard.
2. Add to (1) a dynamic template: new strings ending in _id become keyword; all other new strings become text with a keyword sub-field named raw. Prove it with one document (remember to roll over).
3. Create ILM policy app-policy: roll over at 1 GB primary shard size or 1 day; delete 3 days after rollover. Attach it via the template, roll over manually and check _ilm/explain.
4. Index legacy has 5 shards. Produce legacy-1 with 1 shard without reindexing.
5. Indices movies-v1 and movies-v2 exist. Alias movies points to v1. Atomically repoint it to v2 and make v2 the write index. Then create filtered alias movies-recent showing only year >= 2020.

Solution notes

1. composed_of: ["ct-1shard","ct-app-mappings"], data_stream: {}, priority: 300. Check with GET _data_stream/app-logs-api, then GET .ds-app-logs-api-*/_settings?filter_path=**.number_of_shards.
2. Put the *_id rule before the catch-all string rule. The catch-all uses "mapping": {"type":"text","fields":{"raw":{"type":"keyword"}}}. Existing backing indices don't change, so POST app-logs-api/_rollover before testing.
3. Hot: rollover: {max_primary_shard_size: "1gb", max_age: "1d"}; delete: min_age: "3d". Add index.lifecycle.name: app-policy to the template's settings. Existing backing indices need PUT .ds-app-logs-api-*/_settings, or a rollover.
4. PUT legacy/_settings {"index.blocks.write": true, "index.routing.allocation.require._name": "es01"}, wait for relocation, then POST legacy/_shrink/legacy-1 {"settings":{"index.number_of_shards":1,"index.routing.allocation.require._name":null,"index.blocks.write":null}}.
5. One _aliases call with remove + add (is_write_index: true), then add with filter: {range: {year: {gte: 2020}}}.

---

3. Data Ingestion & Processing

Analogy for the whole domain: a restaurant's back office. Deliveries arrive by different routes (ingest methods). Each crate is checked against the inventory sheet that says what goes on which shelf (mapping and field types). Some produce is washed, chopped and labelled before storage (ingest pipelines / Streams). When the shelving layout changes, the stock is moved to new shelves (Reindex) or relabelled in place (Update By Query).

3.1 Select and configure appropriate ingest methods

Method	Best for	Processing	Notes
Elastic Agent + Fleet (integrations)	Logs, metrics and security data from hosts and cloud services; centrally managed	Integration's own ingest pipelines + @custom hooks	One agent, many inputs; policies managed in Kibana. Writes to type-dataset-namespace data streams.
Standalone Elastic Agent	Same, without Fleet	Same	Configured by YAML
Beats (Filebeat, Metricbeat…)	Lightweight single-purpose shippers	Beat processors + ingest pipelines	Legacy but still supported
EDOT / OpenTelemetry Collector, OTLP endpoint	OpenTelemetry-native logs, metrics and traces	Collector processors, Streams	Elastic's direction of travel for observability
Logstash	Heavy transformation, many inputs and outputs, buffering (persistent queue), routing to several clusters	Logstash filters (grok, mutate, translate…)	Extra infrastructure to run
Language clients + _bulk API	Application data, product catalogues, custom ETL	Optional pipeline= parameter	Most control; you manage batching and retries
Kibana file upload	One-off CSV, NDJSON or log files	Auto-generated pipeline and mapping	Size-limited; good for quick labs
Connectors / crawler	Content sources (SharePoint, databases, websites)	Connector sync rules + pipelines	Search use cases
Streams (Kibana)	Logs, especially unstructured ones, managed from one UI	Streams processors (see 3.6)	9.x feature
_reindex from remote	Migrating data between clusters	Optional pipeline	Needs reindex.remote.whitelist

How to choose (exam-style reasoning):

• "Collect system logs from 200 hosts, centrally managed" → Elastic Agent with Fleet and the System integration.
• "Enrich events from a Kafka topic and send them to two clusters" → Logstash.
• "Index product data from our application" → client + _bulk, with an ingest pipeline for light cleanup.
• "Parse free-text logs quickly without writing grok by hand" → Streams.
• "Transform data already in Elasticsearch" → ingest pipeline + _reindex or _update_by_query.

The Bulk API is the workhorse behind almost every method:

POST _bulk?pipeline=clean-products
{ "index":  { "_index": "products", "_id": "1" } }
{ "sku": "A-1", "name": "Café crème", "price": 3.5 }
{ "create": { "_index": "logs-app-prod" } }
{ "@timestamp": "2026-09-28T10:00:00Z", "message": "hello" }
{ "update": { "_index": "products", "_id": "1" } }
{ "doc": { "price": 3.9 } }
{ "delete": { "_index": "products", "_id": "2" } }

The format is NDJSON: every line ends with a newline, including the last one. The response has a top-level errors: true/false, and each item carries its own status, so always check errors. Aim for batches of about 5–15 MB.

📖 Ingest overview · Ingest tools overview · Ingest architectures · Bulk API · Upload data files · OTLP endpoint · Tune for indexing speed

3.2 Define a mapping that satisfies a given set of requirements

A mapping is the schema: which fields exist, their types, and how each is indexed and analysed.

Analogy: a customs form. Each box says what the item is ("date", "exact code", "free text"). Customs (Lucene) stores it in the right way. A date written in the "free text" box can't be range-searched later, and you can't change the form once the parcel is shipped.

Explicit vs dynamic. Dynamic mapping guesses, for example "2026-09-28" becomes date and "hello" becomes text + .keyword. The exam almost always wants explicit mappings.

PUT articles
{
  "mappings": {
    "dynamic": "strict",
    "_source": { "excludes": ["internal_notes"] },
    "properties": {
      "title":    { "type": "text", "analyzer": "english",
                    "fields": { "raw": { "type": "keyword" },
                                "suggest": { "type": "search_as_you_type" } } },
      "author":   { "type": "keyword", "normalizer": "lowercase" },
      "body":     { "type": "text", "copy_to": "all_text" },
      "summary":  { "type": "text", "copy_to": "all_text" },
      "all_text": { "type": "text" },
      "views":    { "type": "integer", "null_value": 0 },
      "rating":   { "type": "half_float", "ignore_malformed": true },
      "published":{ "type": "date", "format": "strict_date_optional_time||epoch_millis" },
      "status":   { "type": "keyword", "doc_values": true, "index": true },
      "raw_html": { "type": "text", "index": false },
      "comments": { "type": "nested",
                    "properties": { "user": { "type": "keyword" },
                                    "text": { "type": "text" },
                                    "stars": { "type": "byte" } } },
      "labels":   { "type": "flattened" },
      "internal_notes": { "type": "text" }
    }
  }
}

Mapping parameters you must know:

Parameter	Meaning	Typical task wording
index: false	Not searchable (still in _source; aggregatable if it has doc values)	"stored but not searchable"
doc_values: false	Can't sort or aggregate; saves disk	"never used for aggregations"
enabled: false (object only)	Whole object kept in _source, not parsed at all	"ignore the contents of payload"
store: true	Stored separately from _source (retrieve with stored_fields)	rare
copy_to	Copies the value into another field (a custom "all fields" field)	"search title and body as one field"
fields	Multi-fields: same value indexed several ways	"full-text and exact-match / sortable"
analyzer / search_analyzer	Index-time vs query-time analysis	"autocomplete", "synonyms at search time"
normalizer	Analyzer for keyword (lowercase, asciifolding)	"case-insensitive exact match"
ignore_above	Don't index keyword strings longer than N	"skip values longer than 256"
null_value	Index a replacement for explicit null	"treat null as 0 / NONE"
ignore_malformed	Skip bad values instead of rejecting the document	"don't fail on bad numbers"
coerce	Convert "5" to 5 (default true)	"reject strings in numeric fields" → coerce: false
format	Date formats (yyyy-MM-dd, epoch_second…)	"dates like 28/09/2026" → dd/MM/yyyy
dynamic (per object)	true/false/strict/runtime at any level	"strict except in meta"

What you can and can't change on an existing index:

✅ Allowed via PUT <index>/_mapping	❌ Needs a new index + reindex
Add new fields	Change a field's type
Add a multi-field to an existing field, then run _update_by_query to populate it	Change a field's analyzer
Change ignore_above, search_analyzer (with care), mapping _meta	Change index, doc_values or nested vs object
Add runtime fields	Rename a field (use a field alias or reindex)

PUT articles/_mapping
{ "properties": { "author": { "type": "keyword", "fields": { "text": { "type": "text" } } } } }
POST articles/_update_by_query?conflicts=proceed      // populates author.text for existing docs

Text analysis: custom analyzers

Analogy: a kitchen processing line. Character filters wash the vegetables (strip HTML, map characters). The tokenizer chops them into pieces (words). Token filters season each piece (lowercase, remove stop words, stem, add synonyms). The same line must be used for indexing and searching, or the pieces won't match.

PUT recipes
{
  "settings": {
    "analysis": {
      "char_filter": {
        "amp_to_and": { "type": "mapping", "mappings": ["& => and"] }
      },
      "filter": {
        "my_stop":  { "type": "stop", "stopwords": "_english_" },
        "my_syn":   { "type": "synonym_graph", "synonyms": ["courgette, zucchini", "aubergine, eggplant"] },
        "edge_3_10":{ "type": "edge_ngram", "min_gram": 3, "max_gram": 10 }
      },
      "analyzer": {
        "recipe_text": {
          "type": "custom",
          "char_filter": ["html_strip", "amp_to_and"],
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "my_stop", "porter_stem"]
        },
        "recipe_search": {
          "type": "custom", "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "my_syn", "porter_stem"]
        },
        "autocomplete": {
          "type": "custom", "tokenizer": "standard",
          "filter": ["lowercase", "edge_3_10"]
        }
      },
      "normalizer": {
        "lower_fold": { "type": "custom", "filter": ["lowercase", "asciifolding"] }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text", "analyzer": "recipe_text", "search_analyzer": "recipe_search",
        "fields": {
          "ac":  { "type": "text", "analyzer": "autocomplete", "search_analyzer": "standard" },
          "raw": { "type": "keyword", "normalizer": "lower_fold" }
        }
      }
    }
  }
}

POST recipes/_analyze
{ "analyzer": "recipe_text", "text": "<b>Courgette & Crème</b> brûlée recipes" }

POST recipes/_analyze
{ "field": "name.ac", "text": "Courgette" }

Analysis facts to remember:

• The default analyzer is standard (standard tokenizer + lowercase). Built-in analyzers include simple, whitespace, keyword, stop, pattern and language analyzers (english, french…).
• Synonyms at search time use synonym_graph in the search_analyzer. To update synonyms without reindexing, use the Synonyms API (PUT _synonyms/my-set) with "synonyms_set": "my-set" and "updateable": true on a search-time filter.
• Edge n-grams go at index time only; search with the standard analyzer, otherwise the query is n-grammed too.
• keyword fields are not analysed. Use a normalizer for case-insensitivity.
• Always test with _analyze before creating the real index.

📖 Mapping · Explicit mapping · Update mapping examples · Mapping parameters · Text analysis · Anatomy of an analyzer · Create a custom analyzer · Test an analyzer · Index and search analysis · Synonyms

3.3 Define and use appropriate field data types for indexing, search and aggregation

Analogy. Pick the container for what you'll do with the contents. Soup goes in a bowl (text, for searching within), coins in a counting tray (numeric, for maths and ranges), keys on labelled hooks (keyword, for exact lookup, sorting and grouping).

Requirement	Type	Why
Full-text search (words, relevance)	text	Analysed into tokens
Exact match, sorting, terms aggregation (IDs, status, tags, emails)	keyword	Stored as-is, with doc values
Both of the above	text + fields.keyword (multi-field)	Default for dynamic strings
Same value in every document of an index	constant_keyword	Nearly free, great for data streams
Grep-like *foo* on huge high-cardinality strings	wildcard	Fast wildcard and regexp
Log messages, scoring not needed	match_only_text	Much smaller than text
Integers	byte / short / integer / long / unsigned_long	Smallest type that fits
Decimals	float / half_float / double / scaled_float (prices: scaling_factor: 100)
Dates and times	date (ms) / date_nanos	Range queries, date histograms
True/false	boolean
IPv4/IPv6 and CIDR queries	ip	term: {"ip": "10.0.0.0/8"} works
Lat/lon, distance, bounding box	geo_point	Geo queries and aggs
Polygons, shapes	geo_shape
Numeric or date intervals	integer_range, date_range, ip_range…	"Is 42 inside the doc's range?"
Arrays of objects queried together	nested	Keeps each object's fields bound together
Simple JSON objects	object (default)	Flattened into dotted fields
Unknown or unbounded keys (labels, tags maps)	flattened	One field, avoids mapping explosion
Parent/child in the same index	join	Costly; use sparingly
Semantic search (managed embeddings)	semantic_text	Inference handled for you (see 4.5)
Your own embeddings	dense_vector (kNN) / sparse_vector
Autocomplete as you type	search_as_you_type / completion
Software versions (1.10 > 1.9)	version	Correct semantic ordering
Pre-aggregated metrics	histogram, aggregate_metric_double
Another name for an existing field	alias	"Rename" without reindex (for search)

Arrays. There's no array type. Any field can hold several values of the same type.

object vs nested: the most tested pitfall

// Document:
{ "comments": [ { "user": "alice", "stars": 5 }, { "user": "bob", "stars": 1 } ] }

With object, Lucene stores comments.user: [alice, bob] and comments.stars: [5, 1], so the pairing is lost. A search for "alice AND stars 1" wrongly matches.

With nested, each comment is a hidden sub-document. You must use the nested query and the nested aggregation:

GET articles/_search
{
  "query": {
    "nested": {
      "path": "comments",
      "query": { "bool": { "must": [
        { "term": { "comments.user": "alice" } },
        { "range": { "comments.stars": { "gte": 4 } } } ] } },
      "inner_hits": {}
    }
  }
}

The limits are index.mapping.nested_fields.limit (50) and index.mapping.nested_objects.limit (10,000 per document). Other mapping explosion guards: index.mapping.total_fields.limit (1000) and index.mapping.depth.limit (20).

Runtime fields: schema on read

A field computed at query time from _source or other fields. There's nothing to reindex, but it's slower.

PUT articles/_mapping
{
  "runtime": {
    "day_of_week": {
      "type": "keyword",
      "script": { "source": "emit(doc['published'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ENGLISH))" }
    }
  }
}

// Or only for one search:
GET articles/_search
{
  "runtime_mappings": {
    "views_k": { "type": "double", "script": "emit(doc['views'].value / 1000.0)" }
  },
  "fields": ["views_k"],
  "aggs": { "by_day": { "terms": { "field": "day_of_week" } } }
}

📖 Field data types · keyword · text · Numeric · date · nested · flattened · Multi-fields · Runtime fields · Mapping limit settings

3.4 Use the Reindex API and Update By Query API

Analogy. Reindex is moving house: you pack every item from the old flat and unpack it into a new one that has a different layout (new mapping, new analyzer, new shard count). Update By Query is redecorating in place: same flat, you walk from room to room and change what matches.

Reindex

POST _reindex?wait_for_completion=false&slices=auto
{
  "source": {
    "index": "articles",
    "query": { "range": { "published": { "gte": "2025-01-01" } } },
    "_source": ["title", "author", "published", "views"],
    "size": 1000
  },
  "dest": {
    "index": "articles-v2",
    "pipeline": "normalize-articles",
    "op_type": "create"
  },
  "script": {
    "lang": "painless",
    "source": "ctx._source.views = ctx._source.views == null ? 0 : ctx._source.views; ctx._source.remove('internal_notes')"
  },
  "conflicts": "proceed",
  "max_docs": 100000
}

Option	Use
source.query	Copy only matching documents
source._source	Copy only some fields
dest.pipeline	Transform with an ingest pipeline on the way in
script	Painless on ctx._source (also ctx._id, ctx._index; set ctx.op = "noop" to skip, or "delete")
dest.op_type: create	Only create missing documents; required when the destination is a data stream
dest.version_type: external	Keep newer versions already in the destination
conflicts: proceed	Count version conflicts instead of aborting
max_docs	Limit the copy
slices=auto	Parallelise
requests_per_second	Throttle (change later with _reindex/<task>/_rethrottle)
wait_for_completion=false	Returns a task ID. Poll GET _tasks/<id>, cancel with POST _tasks/<id>/_cancel.
source.remote	Reindex from another cluster. The destination node needs reindex.remote.whitelist: "host:9200" in elasticsearch.yml.

The destination is NOT created from the source's mapping. Create articles-v2 (or a template for it) first, otherwise dynamic mapping guesses the types.

Zero-downtime migration recipe: create v2 with the new mapping → _reindex → verify _count → atomic alias swap (2.6) → delete v1 if asked.

Update By Query

POST articles/_update_by_query?conflicts=proceed&wait_for_completion=false
{
  "query": { "term": { "status": "draft" } },
  "script": {
    "lang": "painless",
    "source": "ctx._source.status = params.s; ctx._source.tags = ctx._source.tags ?: []; ctx._source.tags.add('migrated')",
    "params": { "s": "archived" }
  }
}

// Re-run all docs through a pipeline (e.g. to populate a new field)
POST articles/_update_by_query?pipeline=add-reading-time
{ "query": { "bool": { "must_not": { "exists": { "field": "reading_time" } } } } }

// Pick up a newly added multi-field for all docs
POST articles/_update_by_query?refresh

Related APIs:

• POST <index>/_update/<id> with doc (partial), script, upsert or doc_as_upsert
• POST <index>/_delete_by_query
• ?refresh to make results immediately visible

Painless essentials: ctx._source.field, params.x, ctx._source.remove('f'), ctx._source.containsKey('f'), null-safe ?. and elvis ?:, ctx.op = 'noop' | 'delete'. In ingest pipelines, fields are on ctx.field directly (no _source).

📖 Reindex API · Update by query API · Delete by query API · Update API · Migrate data using reindex · Painless · Update documents with scripts · Tasks API

3.5 Define and use an ingest pipeline that satisfies a given set of requirements

An ingest pipeline is an ordered list of processors that runs on ingest nodes before a document is indexed.

Analogy: an airport baggage belt. Each station (processor) scans, relabels or reroutes the bag. If a station jams, the bag is either sent to a lost-luggage office (on_failure) or the whole belt stops (default: the document is rejected).

PUT _ingest/pipeline/web-logs
{
  "description": "Parse Apache-style access logs",
  "processors": [
    { "grok": {
        "field": "message",
        "patterns": ["%{IPORHOST:client.ip} %{USER:ident} %{USER:user.name} \\[%{HTTPDATE:ts}\\] \"%{WORD:http.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.status:int} (?:%{NUMBER:bytes:long}|-)"]
    } },
    { "date":   { "field": "ts", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"], "target_field": "@timestamp" } },
    { "remove": { "field": ["ts", "ident"], "ignore_missing": true } },
    { "set":    { "field": "event.ingested", "value": "{{{_ingest.timestamp}}}" } },
    { "lowercase": { "field": "http.method" } },
    { "geoip":  { "field": "client.ip", "target_field": "client.geo", "ignore_failure": true } },
    { "user_agent": { "field": "user_agent.original", "ignore_missing": true } },
    { "set": {
        "if": "ctx.http?.status != null && ctx.http.status >= 500",
        "field": "event.outcome", "value": "failure" } },
    { "script": {
        "lang": "painless",
        "source": "ctx.bytes_kb = ctx.bytes == null ? 0 : ctx.bytes / 1024.0"
    } },
    { "drop": { "if": "ctx.url?.original == '/healthz'" } }
  ],
  "on_failure": [
    { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } },
    { "set": { "field": "_index", "value": "failed-{{ _index }}" } }
  ]
}

Frequently used processors:

Processor	Purpose
set, remove, rename, append, convert, lowercase/uppercase, trim, split, join, gsub	Field manipulation
grok, dissect, kv, csv, json	Parse text into fields (dissect is faster, for fixed delimiters; grok uses regex)
date, date_index_name	Parse dates; route to date-based index names
geoip, user_agent, uri_parts, community_id, network_direction, registered_domain	Enrichment
enrich	Lookup-join against another index
inference	Run an ML model (e.g. embeddings)
script	Painless for anything else
pipeline	Call another pipeline (modular pipelines)
reroute	Send a document to another data stream (dataset or namespace)
foreach	Apply a processor to each array element
drop, fail	Discard a document; raise a custom error
fingerprint, redact	Deduplication IDs; mask PII

Common processor options: if (Painless condition, null-safe with ?.), ignore_failure, ignore_missing, on_failure (per processor), tag, description.

Error handling, in order of scope:

1. Processor-level ignore_failure: true skips silently.
2. Processor-level on_failure: [...] handles the error locally.
3. Pipeline-level on_failure is the catch-all. Metadata: _ingest.on_failure_message, _ingest.on_failure_processor_type, _ingest.on_failure_processor_tag.
4. Nothing catches it → the document is rejected. For data streams with a failure store, it goes to ::failures instead.

Testing, which you should always do:

POST _ingest/pipeline/web-logs/_simulate?verbose=true
{ "docs": [ { "_source": { "message": "1.2.3.4 - - [28/Sep/2026:10:00:00 +0000] \"GET /a HTTP/1.1\" 503 12" } } ] }

// Simulate with the pipelines that WOULD run for an index (defaults, final, templates)
POST _ingest/_simulate
{ "docs": [ { "_index": "web-2026", "_source": { "message": "…" } } ] }

Ways to apply a pipeline:

How	Scope
?pipeline=name on index, bulk, reindex (dest.pipeline) or update-by-query	One request
index.default_pipeline	Default for an index; overridden by ?pipeline=
index.final_pipeline	Always runs last, even if a request pipeline was given
Index template setting	Every new index or backing index
pipeline processor	Nested pipelines
Integrations' @custom pipelines (e.g. logs-nginx.access@custom)	Customise Fleet integrations safely

PUT web-2026/_settings { "index.default_pipeline": "web-logs" }

Enrich processor: a lookup table at ingest time

PUT users-ref/_doc/1   { "email": "a@x.com", "dept": "finance", "manager": "zoe" }

PUT _enrich/policy/users-policy
{ "match": { "indices": "users-ref", "match_field": "email", "enrich_fields": ["dept", "manager"] } }

POST _enrich/policy/users-policy/_execute        // builds the .enrich-* index; re-run after the source changes

PUT _ingest/pipeline/add-user-info
{ "processors": [ { "enrich": {
    "policy_name": "users-policy", "field": "user.email",
    "target_field": "user.info", "max_matches": 1, "ignore_missing": true } } ] }

Policy types: match (exact term), range (value inside a range), geo_match (point in shape). An enrich policy can't be updated. Delete it and recreate it; you have to remove the processors that use it first.

📖 Ingest pipelines · Processor reference · Error handling · Example: parse logs · Data enrichment · Set up an enrich processor · Simulate pipeline API · Grok debugger · Painless in ingest

3.6 Ingest and process data using Streams

Streams is Kibana's UI and API layer for managing log data end to end: getting data in, parsing it, routing it, mapping fields, setting retention and watching data quality. Behind the scenes it creates and manages ordinary Elasticsearch objects (data streams, index templates, ingest pipelines, lifecycle settings).

Analogy: a smart sorting office with a control panel. Instead of wiring each conveyor belt (template, pipeline, ILM) yourself, you use one panel: "parse these letters like this, send the ones from Paris to that shelf, keep them for 30 days". The panel builds the belts for you. Don't rewire the belts by hand behind its back.

Two kinds of streams (9.2–9.3)

	Classic streams	Wired streams (tech preview)
What	Your existing data streams (e.g. logs-nginx.access-default), adopted in place	A managed hierarchy rooted at a special endpoint
Migration	None; data keeps flowing as before	Send data to the root stream: logs in 9.2–9.3 (logs.otel / logs.ecs from 9.4)
Processing	✓ (added without touching the underlying index template)	✓
Retention	✓ (inherit from template, custom period, or ILM policy)	✓ (inherit from parent, custom, or ILM)
Partitioning (routing into child streams)	✗	✓, e.g. logs → logs.nginx → logs.nginx.errors
Field mapping (Schema)	✓ (add mappings)	✓ (children inherit from the parent)
Inheritance	n/a	Children inherit mappings, lifecycle and processing from their parent

A third kind, query streams (virtual, defined by an ES|QL query), appears in newer versions.

Typical workflow in the Streams UI

1. Get data in. For wired streams, turn on wired streams in Streams settings, then point Elastic Agent, Filebeat, the OTel Collector or Logstash at the logs index/stream. No template setup is needed. For classic streams, just open an existing data stream in Streams.
2. Partition (wired only). Define routing conditions such as service.name == "nginx" → logs.nginx. Partition by logical groups (team, technology, environment), not by high-cardinality fields. Aim for tens of partitions, not hundreds. Each child is a real data stream.
3. Process. Build a processing pipeline in the Processing tab from processors that mirror ingest processors (grok, dissect, date, set, rename, convert, remove, append, drop, math, json extract, redact, user agent, URI parts, enrich…). Add conditions to steps. The Data preview simulates results on sample documents before you save. With a generative-AI connector configured, Suggest a pipeline can generate grok/dissect rules for you.
4. Map fields (Schema tab). Promote extracted fields to proper mapped types (keyword, date, long…).
5. Configure retention (Data lifecycle tab). Choose inherit, a custom retention period (data stream lifecycle), or an existing ILM policy.
6. Monitor data quality (Data quality tab). Degraded documents are ones with ignored or malformed fields (_ignored); failed documents are kept in the failure store so you can fix the processor using the real failing documents.
7. Advanced tab. Inspect the managed index templates, component templates and pipelines Streams created.

Rules to remember:

• Streams-managed components are managed. Change them through the Streams UI or API, not with raw Elasticsearch APIs.
• Streams is a Kibana feature: its API is under kbn:/api/streams/... (Kibana API), not the Elasticsearch REST API. In Console, prefix with kbn:.
• Required privileges to manage streams include cluster manage_index_templates, manage_ingest_pipelines, manage_pipeline and read_pipeline, plus data-stream-level read, write, create, manage, monitor, manage_data_stream_lifecycle, read_failure_store, manage_failure_store and manage_ilm.
• Don't reroute or migrate existing data streams into the wired root; classic streams work in place.
• Because Streams is new and moving fast, exam tasks are most likely to use the UI flow: add a processor, verify in preview, set retention, check the failure store. Practise it on your lab with sample logs.

# Example Kibana API calls from Console (verify paths against the 9.3 Streams API reference)
GET kbn:/api/streams
GET kbn:/api/streams/logs

📖 Streams overview · Get data into Streams · Wired streams · Partition (organise) data · Process documents · Configure data lifecycle · Map fields · Manage data quality · Advanced settings · Streamlang processors · Streams Kibana API

3.7 Exercises: Ingestion & Processing

1. Create index shop with: name (full-text with the English analyzer, plus a case-insensitive exact sub-field name.exact), sku (exact), price (two decimals, cheap to store), tags (exact, not aggregatable), specs (arbitrary key/values, no mapping explosion), reviews (array of {user, stars} that must be queried together). Unknown fields must be rejected.
2. Create pipeline shop-clean that trims and upper-cases sku, converts price from string to float, sets ingested_at, drops docs whose status is test, and on any failure writes the error to ingest_error instead of rejecting. Simulate it with 2 docs.
3. Reindex shop into shop-v2 (identical mapping except name uses a custom analyzer with asciifolding) through shop-clean, only for docs with price > 0. Then point alias shop-live at shop-v2.
4. Add a name.ac autocomplete multi-field to shop-v2 and populate it for existing docs without reindexing.
5. Create an enrich policy from index suppliers (sku → supplier, country) and add it to shop-clean.
6. In Streams, open the logs-generic-default classic stream (index some raw log lines first), add a dissect processor that extracts level and msg, map level as keyword, set retention to 14 days, and inspect the failure store.

Solution notes

1. dynamic: strict; name: text/english + fields.exact: keyword with a lowercase normalizer; sku: keyword; price: scaled_float, scaling_factor: 100; tags: keyword, doc_values: false; specs: flattened; reviews: nested.
2. Processors: trim + uppercase on sku; convert {type: float} on price; set ingested_at = {{{_ingest.timestamp}}}; drop with if: "ctx.status == 'test'". Pipeline-level on_failure sets ingest_error = {{ _ingest.on_failure_message }}. Verify with _simulate?verbose.
3. Create shop-v2 first. _reindex with source.query: range price gt 0 and dest.pipeline: shop-clean. Compare _count, then use _aliases add.
4. PUT shop-v2/_mapping adding fields.ac with an edge-ngram analyzer. The analyzer must already exist in the index settings; if not, close → add analysis → open. Then POST shop-v2/_update_by_query.
5. PUT _enrich/policy/... (match), _execute, then add an enrich processor to the pipeline. Re-PUT the whole pipeline; the PUT replaces it.
6. The UI flow covered in 3.6. Check the result on the Advanced tab and in GET logs-generic-default/_mapping.

---

4. Searching Data

Analogy for the whole domain: a library. Full-text queries are asking the librarian "books about sailing". She interprets the words and ranks the answers by relevance. Term-level queries are the card catalogue: exact codes, no interpretation. Bool combines requests: "must be about sailing, must be in French, should be recent, must not be a children's book". Aggregations are the statistics desk: "how many books per genre, average page count". ES|QL is a spreadsheet-style pipeline over the same shelves. Semantic search is a librarian who understands meaning ("boat trips" finds "sailing"). Cross-cluster search asks sister libraries in other cities at the same time.

4.1 Search for terms and/or phrases in one or more fields

Full-text queries (analysed: the query text goes through the field's analyzer)

// Single field, any word (OR by default)
GET articles/_search
{ "query": { "match": { "title": "elastic search guide" } } }

// All words required, or a minimum
{ "query": { "match": { "title": { "query": "elastic search guide", "operator": "and" } } } }
{ "query": { "match": { "title": { "query": "elastic search guide", "minimum_should_match": "2" } } } }

// Typo tolerance
{ "query": { "match": { "title": { "query": "elastik", "fuzziness": "AUTO" } } } }

// Exact phrase, with slop (words may be up to N positions apart)
{ "query": { "match_phrase": { "body": { "query": "shard allocation", "slop": 2 } } } }

// Phrase prefix, "search as you type" style
{ "query": { "match_phrase_prefix": { "title": "quick bro" } } }

// Several fields, with boosting
{ "query": { "multi_match": {
    "query": "shard sizing",
    "fields": ["title^3", "summary^2", "body"],
    "type": "best_fields",
    "tie_breaker": 0.3 } } }

multi_match types:

Type	Behaviour	Use when
best_fields (default)	Score of the best-matching field (+ tie_breaker × others)	Words expected together in one field
most_fields	Sum of all field scores	Same text indexed several ways (multi-fields)
cross_fields	Treats the fields as one big field (term-centric)	"first_name + last_name" style data
phrase / phrase_prefix	match_phrase on each field	Phrases across fields
bool_prefix	For search_as_you_type fields	Autocomplete

Also know combined_fields (like cross_fields for fields sharing an analyzer), query_string (Lucene syntax with AND/OR/NOT, wildcards and field names; throws on bad syntax) and simple_query_string (forgiving, safe for user input: + | - " * ( )).

Term-level queries (NOT analysed: exact values on keyword, numeric, date, ip, boolean)

{ "query": { "term":  { "status": "published" } } }
{ "query": { "term":  { "status": { "value": "Published", "case_insensitive": true } } } }
{ "query": { "terms": { "tags": ["elasticsearch", "lucene"] } } }
{ "query": { "range": { "published": { "gte": "now-30d/d", "lt": "now/d" } } } }
{ "query": { "range": { "price": { "gt": 10, "lte": 100 } } } }
{ "query": { "exists": { "field": "author" } } }
{ "query": { "prefix": { "sku": "ABC-" } } }
{ "query": { "wildcard": { "sku": { "value": "A*-9?" } } } }
{ "query": { "regexp": { "sku": "A[0-9]{3}" } } }
{ "query": { "fuzzy": { "author": { "value": "jonh", "fuzziness": 1 } } } }
{ "query": { "ids": { "values": ["1", "7"] } } }
{ "query": { "terms_set": { "skills": { "terms": ["java","go","rust"], "minimum_should_match_field": "required" } } } }

The number-one trap: term on a text field. "Published" was indexed as published, so term: "Published" finds nothing. Use match on text, or term on the .keyword sub-field.

Date math: now-1d/d (rounded to the day), 2026-09-01||+1M, plus a time_zone parameter on range queries.

Shaping results: pagination, sorting, source filtering, highlighting

GET articles/_search
{
  "from": 0, "size": 10,
  "track_total_hits": true,
  "_source": ["title", "published"],
  "fields": ["title", "published", "day_of_week"],
  "sort": [ { "published": "desc" }, { "title.raw": "asc" }, "_score" ],
  "highlight": {
    "pre_tags": ["<mark>"], "post_tags": ["</mark>"],
    "fields": { "body": { "fragment_size": 150, "number_of_fragments": 3 } }
  },
  "query": { "match": { "body": "shard allocation" } }
}

• from + size is capped by index.max_result_window (10,000). For deep paging use search_after with a point in time (POST <index>/_pit?keep_alive=1m). Scroll is legacy for this purpose.
• You can't sort on text fields. Sort on a keyword sub-field, a number or a date.
• track_total_hits defaults to accurate counting up to 10,000; set true for an exact count.
• explain: true shows the score calculation. profile: true shows per-shard timings (or use Kibana's Search Profiler).

📖 Full-text queries · match · match_phrase · multi_match · query_string · Term-level queries · term · range · Paginate results · Sort results · Highlighting · The _search API

4.2 Boolean combinations of queries and filters

GET articles/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "body": "elasticsearch" } }
      ],
      "should": [
        { "match_phrase": { "title": { "query": "certified engineer", "boost": 2 } } },
        { "term": { "tags": "exam" } }
      ],
      "minimum_should_match": 1,
      "filter": [
        { "term":  { "status": "published" } },
        { "range": { "published": { "gte": "2025-01-01" } } }
      ],
      "must_not": [
        { "term": { "author": "spam-bot" } }
      ]
    }
  }
}

Clause	Must match?	Affects score?	Cached?
must	Yes	Yes	No
filter	Yes	No (filter context)	Yes
should	See below	Yes (adds to the score)	No
must_not	Must not match	No (filter context)	Yes

should semantics (subtle, and heavily tested):

• If the bool has no must or filter, at least one should must match (minimum_should_match defaults to 1).
• If there is a must or filter, should clauses are optional and only boost the score, unless you set minimum_should_match.

Query vs filter context rule of thumb: "how well does it match?" (words, relevance) goes in must/should. "Does it match, yes or no?" (status, dates, ranges, IDs) goes in filter/must_not. Filters are faster and cached.

Other compound queries:

• constant_score: wrap a filter and give every hit the same score.
• boosting: positive query, with a negative query that demotes (not excludes) matches.
• dis_max: best of several queries, plus tie_breaker.
• function_score / script_score: modify scores by recency, popularity (field_value_factor), decay functions or scripts.
• Named queries ("_name": "q1") report which clauses matched each hit.

Nesting example: "(title contains X OR summary contains X) AND year ≥ 2020 AND NOT draft":

{ "query": { "bool": {
    "must": [ { "bool": { "should": [
        { "match": { "title": "X" } }, { "match": { "summary": "X" } } ] } } ],
    "filter": [ { "range": { "year": { "gte": 2020 } } } ],
    "must_not": [ { "term": { "status": "draft" } } ] } } }

post_filter filters the hits but not the aggregations. Use it for faceted navigation, where facet counts shouldn't shrink when a facet is selected.

📖 Query and filter context · bool query · Compound queries · function_score · Filter search results (post_filter)

4.3 Write and execute metric and bucket aggregations

Analogy: sorting sweets. Bucket aggregations tip the sweets into bowls by colour, by size range, or by the day they were bought. Metric aggregations count, weigh or average what's in each bowl. Pipeline aggregations compare the bowls ("which bowl is heaviest?", "running total"). Bowls can hold smaller bowls (sub-aggregations).

GET kibana_sample_data_ecommerce/_search
{
  "size": 0,
  "query": { "range": { "order_date": { "gte": "now-30d/d" } } },
  "aggs": {
    "per_day": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day",
        "time_zone": "Europe/Paris",
        "min_doc_count": 0,
        "extended_bounds": { "min": "now-30d/d", "max": "now/d" }
      },
      "aggs": {
        "revenue":     { "sum": { "field": "taxful_total_price" } },
        "unique_buyers": { "cardinality": { "field": "customer_id" } }
      }
    },
    "top_categories": {
      "terms": { "field": "category.keyword", "size": 5, "order": { "revenue": "desc" } },
      "aggs": {
        "revenue":   { "sum": { "field": "taxful_total_price" } },
        "price_stats": { "stats": { "field": "taxful_total_price" } },
        "p95":       { "percentiles": { "field": "taxful_total_price", "percents": [50, 95, 99] } },
        "best_order": { "top_hits": { "size": 1, "sort": [{ "taxful_total_price": "desc" }],
                                      "_source": ["order_id", "customer_full_name"] } }
      }
    },
    "price_bands": {
      "range": { "field": "taxful_total_price",
                 "ranges": [ { "to": 50 }, { "from": 50, "to": 150 }, { "from": 150 } ] }
    },
    "best_day": { "max_bucket": { "buckets_path": "per_day>revenue" } },
    "avg_daily_revenue": { "avg_bucket": { "buckets_path": "per_day>revenue" } }
  }
}

Metric aggregations: avg, sum, min, max, value_count, stats, extended_stats (std deviation, variance), cardinality (approximate distinct count, HyperLogLog++, precision_threshold), percentiles, percentile_ranks, median_absolute_deviation, weighted_avg, top_hits, top_metrics, geo_bounds, geo_centroid, scripted_metric.

Bucket aggregations:

Aggregation	Groups by	Key options
terms	Distinct values (keyword, number, ip…)	size (default 10), order (_count, _key or a sub-agg), min_doc_count, missing, include/exclude, shard_size
multi_terms	Combinations of several fields	terms: [{field}, {field}]
rare_terms, significant_terms	Rare or unusually frequent terms
histogram	Numeric intervals	interval, min_doc_count, extended_bounds
date_histogram	Time intervals	calendar_interval (minute, hour, day, week, month, quarter, year, or 1d, 1M) vs fixed_interval (30m, 7d; multiples allowed); time_zone, format, offset
range / date_range / ip_range	Custom ranges (from inclusive, to exclusive)	keyed: true
filter / filters	One or several query-defined buckets	other_bucket
composite	Paginated combinations of sources	after key: iterate all buckets
nested / reverse_nested	Enter or leave nested documents	path
global	Ignores the query scope
missing	Documents without the field
geohash_grid, geotile_grid, auto_date_histogram, variable_width_histogram, sampler	Specialised

Pipeline aggregations reference other aggregations by buckets_path (> for nesting, . for a metric's value, _count for doc count):

• Sibling (a new bucket next to the others): avg_bucket, max_bucket, min_bucket, sum_bucket, stats_bucket, percentiles_bucket.
• Parent (inside each histogram bucket): derivative, cumulative_sum, moving_fn, bucket_script, bucket_selector (keeps only matching buckets, like HAVING), bucket_sort (sort or truncate buckets), serial_diff.

"aggs": {
  "per_month": {
    "date_histogram": { "field": "order_date", "calendar_interval": "month" },
    "aggs": {
      "revenue":  { "sum": { "field": "taxful_total_price" } },
      "cum_rev":  { "cumulative_sum": { "buckets_path": "revenue" } },
      "big_only": { "bucket_selector": {
          "buckets_path": { "r": "revenue" }, "script": "params.r > 10000" } }
    }
  }
}

Aggregation facts:

• Set "size": 0 when you only need aggregations.
• You can't run terms on a text field (it has no doc values). Use .keyword.
• terms counts are approximate across shards (doc_count_error_upper_bound). Increase shard_size for accuracy. Use composite to list all buckets.
• Aggregations run on the documents matched by query. Use global to escape that scope, or post_filter to filter hits only.
• For nested fields, wrap sub-aggs in a nested aggregation.

📖 Aggregations · Aggregations tutorial (eCommerce) · Metric aggregations · Bucket aggregations · Pipeline aggregations · terms · date_histogram · composite

4.4 Write and execute ES|QL queries

ES|QL (Elasticsearch Query Language) is a piped language. Each command transforms a table and passes it to the next, like a Unix pipeline or a spreadsheet you keep refining.

Analogy: a kitchen assembly line. FROM grabs the ingredients, WHERE throws away what you don't need, EVAL chops new pieces, STATS weighs and groups, SORT arranges the plates, LIMIT serves the first N.

POST _query?format=txt
{
  "query": """
    FROM kibana_sample_data_logs
    | WHERE @timestamp >= NOW() - 7 days AND response.keyword != "200"
    | EVAL kb = bytes / 1024.0, day = DATE_TRUNC(1 day, @timestamp)
    | STATS errors = COUNT(*), avg_kb = AVG(kb), hosts = COUNT_DISTINCT(host.keyword) BY day, response.keyword
    | SORT day DESC, errors DESC
    | LIMIT 20
  """
}

format can be json (default), txt, csv, tsv, yaml and more. For long queries use POST _query/async and poll with GET _query/async/<id>. In Kibana, use Discover → ES|QL mode.

Commands you must know:

Type	Command	Example
Source	FROM	FROM logs-*, remote:logs-* METADATA _index, _id, _score
Source	ROW, SHOW INFO	ROW a = 1, b = "x"
Source	TS (time series, newer)	TS metrics-* | STATS …
Filter	WHERE	WHERE status IN ("500","503") AND bytes > 1000
Compute	EVAL	EVAL mb = bytes / 1048576
Aggregate	STATS … BY	STATS c = COUNT(*), p95 = PERCENTILE(latency, 95) BY service
Shape	KEEP, DROP, RENAME	KEEP @timestamp, host, message
Order	SORT, LIMIT	SORT c DESC NULLS LAST | LIMIT 10
Parse	DISSECT, GROK	DISSECT message "%{ip} - %{user} [%{ts}]"
Enrich	ENRICH	ENRICH users-policy ON email WITH dept
Join	LOOKUP JOIN	LOOKUP JOIN hosts-lookup ON host.name (the right-hand index must use index.mode: lookup)
Arrays	MV_EXPAND	Turn a multi-value column into rows
Analytics	CHANGE_POINT, SAMPLE, FORK, RERANK, COMPLETION	Newer commands; know they exist

Functions to know: aggregations COUNT, COUNT_DISTINCT, SUM, AVG, MIN, MAX, MEDIAN, PERCENTILE, VALUES, TOP; grouping BUCKET(@timestamp, 1 hour); date DATE_TRUNC, DATE_EXTRACT, DATE_DIFF, DATE_FORMAT, NOW(); string CONCAT, TO_LOWER, SUBSTRING, LENGTH, SPLIT, STARTS_WITH, LIKE, RLIKE; conditionals CASE(cond, a, cond2, b, default), COALESCE; conversion TO_INTEGER, TO_DATETIME, TO_IP, :: casts; multi-value MV_COUNT, MV_AVG; full-text MATCH(field, "text"), the : operator (message:"timeout"), QSTR("…"), KQL("…"), plus METADATA _score for relevance.

ES|QL facts:

• Default LIMIT is 1000 rows, with a maximum of 10,000 by default. Always add an explicit LIMIT.
• Case matters for values, not for command names.
• Use parameters to avoid injection: "query": "FROM x | WHERE a == ?v", "params": [{"v": 5}].
• Cross-cluster: FROM cluster_a:logs-*, logs-* (requires remote clusters, see 5.2).
• ES|QL doesn't return _source documents like _search; it returns columns and rows.
• STATS without BY returns one row. Columns not in BY or not aggregated are dropped.

Query DSL → ES|QL translation practice:

Query DSL	ES|QL
bool.filter.term status: published	WHERE status == "published"
range price gte 10 lt 100	WHERE price >= 10 AND price < 100
terms aggregation on category	STATS c = COUNT(*) BY category | SORT c DESC | LIMIT 10
date_histogram 1d + sum	STATS s = SUM(x) BY d = BUCKET(@timestamp, 1 day)
match body: "timeout"	WHERE MATCH(body, "timeout") or WHERE body:"timeout"

📖 ES|QL overview · ES|QL reference · ES|QL source commands · ES|QL processing commands · ES|QL functions and operators · LOOKUP JOIN · ES|QL REST API · ES|QL for search · Using ES|QL in Discover

4.5 Implement semantic search

Semantic search matches meaning rather than exact words. Text is turned into embeddings (vectors) by an ML model, and queries are compared to documents in that vector space.

Analogy: a map of meaning. Each passage becomes a pin on a huge map where similar ideas sit close together. "How do I fix a flat tyre?" lands next to "puncture repair". Keyword search would miss that, because none of the words overlap.

Two kinds of embeddings:

	Sparse (e.g. ELSER, Elastic's learned sparse encoder)	Dense (e.g. E5, Jina, OpenAI, Cohere)
Representation	Weighted tokens (expanded vocabulary)	Fixed-length float vector
Field type	sparse_vector	dense_vector
Query	sparse_vector query	knn (HNSW approximate nearest neighbour)
Strength	Great out-of-domain English, explainable	Multilingual, multimodal, many models

The easy path: semantic_text (recommended and most exam-friendly)

semantic_text hides all the plumbing: chunking, calling the inference endpoint at ingest time, storing embeddings, and embedding the query at search time.

Step 1: an inference endpoint. There's a preconfigured default ELSER endpoint, so this step is optional. To create your own:

PUT _inference/sparse_embedding/my-elser
{
  "service": "elasticsearch",
  "service_settings": {
    "model_id": ".elser_model_2",
    "adaptive_allocations": { "enabled": true, "min_number_of_allocations": 0, "max_number_of_allocations": 4 },
    "num_threads": 1
  }
}

// Dense example with a built-in multilingual model
PUT _inference/text_embedding/my-e5
{ "service": "elasticsearch",
  "service_settings": { "model_id": ".multilingual-e5-small", "num_allocations": 1, "num_threads": 1 } }

GET _inference/_all
POST _inference/sparse_embedding/my-elser { "input": "test sentence" }

The cluster needs an ML node (or Elastic Inference Service on Cloud) and a license that allows ML models, so use the trial on your lab.

Step 2: a mapping with semantic_text:

PUT docs-semantic
{
  "mappings": {
    "properties": {
      "title":   { "type": "text" },
      "content": { "type": "semantic_text", "inference_id": "my-elser" },
      "body":    { "type": "text", "copy_to": "content" }
    }
  }
}

Omit inference_id to use the default endpoint. copy_to into a semantic_text field lets you keep a normal text field for BM25 alongside.

Step 3: index documents normally. Embeddings are generated automatically. Bulk-loading many documents is slow on small ML nodes, so be patient or use a small batch.

Step 4: query.

// Simplest: the match query works on semantic_text fields
GET docs-semantic/_search
{ "query": { "match": { "content": "how do I keep my cluster available when a zone fails?" } } }

// Dedicated semantic query
GET docs-semantic/_search
{ "query": { "semantic": { "field": "content", "query": "zone failure resilience" } },
  "highlight": { "fields": { "content": { "type": "semantic", "number_of_fragments": 2 } } } }

Hybrid search: BM25 + semantic, fused with RRF

GET docs-semantic/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        { "standard": { "query": { "match": { "title": "zone failure" } } } },
        { "standard": { "query": { "semantic": { "field": "content", "query": "zone failure" } } } }
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  }
}

Retrievers are the composable search abstraction: standard, knn, rrf (Reciprocal Rank Fusion), linear (weighted score combination), text_similarity_reranker (semantic reranking), rule.

The manual path (know it exists)

• dense_vector mapping (dims, similarity: cosine | dot_product | l2_norm | max_inner_product, index_options such as int8_hnsw/bbq_hnsw quantisation) + knn search (k, num_candidates, filter, query_vector or query_vector_builder).
• An inference processor in an ingest pipeline to generate embeddings, or bring your own vectors.
• sparse_vector field + sparse_vector query with inference_id.

GET my-vectors/_search
{
  "knn": {
    "field": "embedding",
    "query_vector_builder": { "text_embedding": { "model_id": "my-e5", "model_text": "zone failure" } },
    "k": 10, "num_candidates": 100,
    "filter": { "term": { "lang": "en" } }
  }
}

📖 Semantic search overview · Semantic search with semantic_text · semantic_text field · semantic query · Semantic search with the inference API · Inference API: default endpoints · ELSER · Hybrid search with semantic_text · Retrievers · kNN search · Dense vector · Sparse vector · Semantic reranking

4.6 Write and execute a query that searches across multiple clusters

Cross-cluster search (CCS) runs one request from the local cluster against indices on one or more remote clusters. Setting up the remote connection is covered in 5.2; here is the querying side.

Analogy: interlibrary loan. You ask your local branch; it forwards your request to the branches in Lyon and Berlin and hands you one merged answer.

// Syntax: <remote_alias>:<index_pattern>
GET cluster_b:logs-*/_search
{ "query": { "match": { "message": "timeout" } } }

// Local + several remotes in one request
GET logs-*,cluster_b:logs-*,cluster_c:logs-*/_search
{
  "size": 20,
  "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now-1h" } } } ] } },
  "aggs": { "by_cluster": { "terms": { "field": "_index" } } }
}

// All remotes: wildcard on the alias; exclude with a minus sign
GET *:logs-*,-cluster_c:*/_search

// ES|QL across clusters
POST _query?format=txt
{ "query": "FROM logs-*, cluster_b:logs-* METADATA _index | STATS c = COUNT(*) BY _index | LIMIT 50" }

// Async CCS for long searches
POST cluster_b:logs-*/_async_search?wait_for_completion_timeout=2s
{ "query": { "match_all": {} } }

// Check reachability and matching indices before searching
GET _resolve/cluster/cluster_b:logs-*,cluster_c:logs-*

What to know:

• Hits show the origin in _index as cluster_b:logs-2026…. The response's _clusters section reports total, successful, skipped, partial and failed.
• skip_unavailable (per remote cluster setting, default true): an unreachable remote is skipped instead of failing the whole search.
• ccs_minimize_roundtrips (default true): each remote reduces its own results, which means fewer network round trips.
• Security: the user needs read privileges on the remote indices. With the API-key model, that's the cross-cluster API key's access.search plus the local user's role (remote_indices). With the certificate model, it's a role of the same name on the remote cluster.
• Kibana data views can use cluster_b:logs-* patterns too.
• CCS works across versions within the compatibility window (see the CCS version compatibility table in the docs).

📖 Cross-cluster search · Resolve cluster before CCS · ES|QL cross-cluster · Async search

4.7 Exercises: Searching Data

Use kibana_sample_data_ecommerce, kibana_sample_data_flights and kibana_sample_data_logs. Check field names and types first with GET <index>/_mapping, since some sample fields are text + .keyword and others are plain keyword.

1. Flights: find flights whose OriginCityName or DestCityName contains "New York" (both fields, boost the destination ×2), that are not cancelled, with FlightDelayMin between 30 and 120 and a timestamp in the last 30 days. Return only FlightNum, Carrier, FlightDelayMin, sorted by delay descending.
2. eCommerce: find orders whose products.product_name contains the exact phrase "leather jacket" (slop 1). Highlight the matches.
3. eCommerce: revenue per week (taxful_total_price) for the last 8 weeks, with the week-over-week difference and a running total; keep only weeks above 5,000.
4. eCommerce: the top 5 manufacturer.keyword values by number of orders, each with average basket value and the 3 most frequent category.keyword values.
5. Logs, in ES|QL: per host, the count of 4xx and 5xx responses in the last 24 hours, the 95th-percentile bytes, sorted by error count, top 10.
6. Create a semantic_text index faq, index 5 short Q&A documents, and run a hybrid RRF query combining BM25 on question with semantic search on answer.
7. Across clusters: count documents per index on lab and remote for logs-* in one request, and make the search succeed even if remote is down.

Solution notes

1. bool.should with match_phrase on both city fields (boost: 2 on Dest) and minimum_should_match: 1. filter: term Cancelled: false, range FlightDelayMin gte 30 lte 120, range timestamp gte now-30d/d. Add _source and sort.
2. match_phrase with slop: 1 on products.product_name, plus highlight.fields.
3. date_histogram with calendar_interval: week and extended_bounds, then sum → derivative → cumulative_sum → bucket_selector (params.r > 5000). The query uses range order_date gte now-8w/w.
4. terms manufacturer.keyword size 5 → avg taxful_total_price + terms category.keyword size 3.
5. FROM kibana_sample_data_logs | WHERE @timestamp > NOW() - 24 hours | EVAL code = TO_INTEGER(response.keyword) | WHERE code >= 400 | STATS errors = COUNT(*), p95 = PERCENTILE(bytes, 95) BY host.keyword | SORT errors DESC | LIMIT 10
6. Mapping answer: semantic_text, question: text. Use an rrf retriever with a match on question and a semantic query on answer.
7. GET logs-*,remote:logs-*/_search {"size":0,"aggs":{"i":{"terms":{"field":"_index"}}}}, with cluster.remote.remote.skip_unavailable: true (the default).

---

5. Cluster Management

5.1 Monitor and maintain cluster health

Analogy: a hospital ward. Green means every patient (shard) has a bed and a backup nurse (replica). Yellow means every patient has a bed, but some backup nurses are missing: no data lost, but less safe. Red means at least one patient has no bed at all: a primary shard is unassigned and its data is unavailable. The allocation explain API is the doctor's chart that says why.

The monitoring toolkit

GET _cluster/health                                  // status, nodes, unassigned_shards, relocating…
GET _cluster/health/orders?level=shards              // per index or shard
GET _cluster/health?wait_for_status=green&timeout=30s
GET _health_report                                   // 9.x health indicators with diagnoses & remediation
GET _health_report/shards_availability

GET _cat/health?v
GET _cat/nodes?v&h=name,node.role,master,heap.percent,ram.percent,cpu,load_1m,disk.used_percent
GET _cat/indices?v&health=yellow&s=index
GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node&s=state
GET _cat/allocation?v                               // disk per node, shard counts
GET _cat/recovery?v&active_only=true
GET _cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected
GET _cat/pending_tasks?v
GET _nodes/stats/jvm,fs,indices,breaker
GET _cluster/stats?human
GET _tasks?actions=*reindex*&detailed
GET _cluster/allocation/explain                     // first unassigned shard, or specify one

📖 Cluster health API · Health report API · Cluster allocation explain API · cat APIs · Troubleshoot

Diagnosing yellow and red: the decision tree

1. GET _cluster/health: which status, and how many unassigned_shards?
2. GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED (in Console, use &s=state).
3. GET _cluster/allocation/explain with the index, shard and primary flag. Read allocate_explanation and each node's deciders.
4. Fix according to the cause:

Symptom or explanation	Cause	Fix
Replica unassigned, "a copy of this shard is already allocated to this node"	Single-node cluster or too few nodes	Add nodes or number_of_replicas: 0 (only if the task allows it)
awareness decider says NO	Forced awareness with a zone missing	Expected. Restore the zone, or relax the awareness values.
filter decider says NO	require/include/exclude rule matches no node	Fix or clear the index.routing.allocation.* settings
disk_threshold decider says NO	Node above the low (85%) or high (90%) watermark	Free disk, delete old indices, add nodes, or (temporarily) adjust cluster.routing.allocation.disk.watermark.*
Index is read-only, writes fail with FORBIDDEN/12/index read-only / allow delete	Flood stage (95%) reached	Free disk. The block is removed automatically once usage drops below the high watermark, or clear it: PUT idx/_settings {"index.blocks.read_only_allow_delete": null}
max_retry decider: "shard has exceeded the maximum number of retries"	Repeated failures (e.g. an earlier disk or mapping problem)	Fix the cause, then POST _cluster/reroute?retry_failed=true
data_tier decider says NO	_tier_preference asks for a tier with no nodes	Add a node with that role or change _tier_preference
enable decider says NO	cluster.routing.allocation.enable set to none or primaries	Reset it to null (all)
Primary unassigned, "no valid shard copy" (red)	All copies lost (node or disk gone)	Bring the node back, restore from snapshot, or, as a last resort, allocate_stale_primary / allocate_empty_primary via _cluster/reroute with accept_data_loss: true
total_shards_per_node limit	Too restrictive a per-node cap	Raise or remove index.routing.allocation.total_shards_per_node
NODE_LEFT + delayed	Node left recently; waiting delayed_timeout (1m)	Wait, or lower the timeout

POST _cluster/reroute?retry_failed=true

PUT _cluster/settings
{ "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } }

// Manual moves / last-resort recovery
POST _cluster/reroute
{ "commands": [
    { "move": { "index": "orders", "shard": 0, "from_node": "es01", "to_node": "es02" } },
    { "allocate_replica": { "index": "orders", "shard": 1, "node": "es03" } } ] }

Cluster settings: persistent survives a full restart; transient is deprecated. Set a setting to null to reset it to the default. GET _cluster/settings?include_defaults=true&flat_settings=true shows everything.

📖 Red or yellow cluster health · Diagnose unassigned shards · Fix watermark errors · Cluster-level shard allocation and routing settings · Cluster reroute API

Performance signals to watch

Signal	Where	Worry when…
JVM heap	_cat/nodes heap.percent, _nodes/stats/jvm	Consistently > 85%, long old-GC pauses
Circuit breakers	_nodes/stats/breaker	tripped counts increasing (requests rejected with 429s)
Thread-pool rejections	_cat/thread_pool	rejected > 0 on write or search → back-pressure
Disk	_cat/allocation	Approaching 85/90/95%
Slow queries or indexing	Slow logs: index.search.slowlog.threshold.query.warn: 5s, index.indexing.slowlog.threshold.index.warn: 10s	Repeated slow entries
Hot threads	GET _nodes/hot_threads	CPU-bound nodes
Pending tasks	_cat/pending_tasks	Master overloaded (too many shards or mapping updates)

Monitoring solutions:

• Stack Monitoring: Elastic Agent or Metricbeat collects metrics into a (preferably separate) monitoring cluster; view them in Kibana → Stack Monitoring. Legacy internal collection is deprecated.
• AutoOps: Elastic's hosted, automated diagnosis with recommendations. Available for ECH, and via Cloud Connect for self-managed clusters.
• Kibana alerting rules on cluster health, CPU, disk and so on.

📖 Monitoring · Stack monitoring · Collect monitoring data with Elastic Agent · AutoOps · AutoOps vs Stack Monitoring · Slow logs · Update logging levels

Maintaining the cluster: snapshots, SLM and routine operations

Analogy: snapshots are incremental photo backups of your phone. The first one copies everything; later ones only copy new photos (new segments), yet every snapshot can restore the full album on its own.

// 1) Register a repository (shared FS: every node needs path.repo pointing to the same mount)
PUT _snapshot/my-repo
{ "type": "fs", "settings": { "location": "/usr/share/elasticsearch/snapshots/my-repo", "compress": true } }
POST _snapshot/my-repo/_verify

// 2) Take a snapshot
PUT _snapshot/my-repo/snap-2026.09.28?wait_for_completion=true
{ "indices": "orders,logs-app-*", "include_global_state": false,
  "metadata": { "taken_by": "exam", "reason": "before migration" } }

GET _snapshot/my-repo/_all        GET _snapshot/my-repo/snap-2026.09.28/_status
GET _cat/snapshots/my-repo?v

// 3) Restore (the target index must not exist and be open; rename or close/delete first)
POST _snapshot/my-repo/snap-2026.09.28/_restore
{
  "indices": "orders",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "include_aliases": false,
  "index_settings": { "index.number_of_replicas": 0 },
  "ignore_index_settings": ["index.refresh_interval"]
}

Snapshot lifecycle management (SLM):

PUT _slm/policy/nightly
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "my-repo",
  "config": { "indices": ["*"], "include_global_state": true },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}
POST _slm/policy/nightly/_execute          // run now
GET  _slm/policy/nightly                   // last_success / last_failure
GET  _slm/stats
POST _slm/_execute_retention

The SLM schedule uses Quartz cron (seconds first: sec min hour day month weekday), and the name supports date math.

Other routine operations:

• Searchable snapshots: mount an index straight from a repository. Use POST _snapshot/my-repo/snap/_mount?storage=full_copy (cold) or shared_cache (frozen), or let ILM do it.
• Repository types: fs, url (read-only), s3, gcs, azure, source (source-only).
• Restore a whole cluster or feature states: include_global_state, feature_states (e.g. security, kibana).
• Close and open indices to cut resource use: POST idx/_close.
• Force merge read-only indices: POST idx/_forcemerge?max_num_segments=1.
• Clear caches: POST idx/_cache/clear. Refresh/flush: POST idx/_refresh, POST idx/_flush.
• Rolling restarts and upgrades: disable replica allocation → flush → restart → re-enable (see 1.2).

📖 Snapshot and restore · Manage repositories · Shared file system repository · Create snapshots and SLM · Restore a snapshot · Searchable snapshots · Add and remove nodes

5.2 Perform cross-cluster operations

Cross-cluster operations depend on remote clusters: a named connection from the local cluster (which starts the requests) to a remote cluster (which holds the data). Two features use it:

• Cross-cluster search (CCS): query remote data (queries in 4.6).
• Cross-cluster replication (CCR): continuously copy leader indices on the remote into read-only follower indices on the local cluster. It needs a Platinum/Enterprise license or the trial.

Analogy. CCS is a phone call to another library: you ask and hear the answer, but nothing is copied. CCR is a newspaper subscription: the publisher (leader) keeps sending every new edition, and your copy (follower) is read-only.

Step 1: connect a remote cluster

	API key authentication (recommended)	TLS certificate authentication (deprecated)
Port on the remote	9443 (dedicated remote cluster server)	9300 (transport)
Trust	Local trusts the remote's CA; access is limited by a cross-cluster API key	Mutual TLS; both clusters trust each other's transport CA
Remote privileges	Defined in the API key (access.search, access.replication)	Roles with the same names must exist on the remote

Connection modes: sniff (default; connects directly to seed nodes and discovers gateway nodes) or proxy (a single address, e.g. through a load balancer). ECH, ECE and ECK effectively use proxy mode.

API key setup, self-managed:

# On the REMOTE cluster (elasticsearch.yml): enable the remote cluster server interface
remote_cluster_server.enabled: true
remote_cluster.port: 9443
xpack.security.remote_cluster_server.ssl.enabled: true
xpack.security.remote_cluster_server.ssl.key: certs/remote.key
xpack.security.remote_cluster_server.ssl.certificate: certs/remote.crt

// On the REMOTE: create a cross-cluster API key that limits what the local cluster may do
POST _security/cross_cluster/api_key
{
  "name": "from-lab",
  "access": {
    "search":      [ { "names": ["logs-*", "orders"] } ],
    "replication": [ { "names": ["orders", "metrics-*"] } ]
  }
}

# On EVERY node of the LOCAL cluster: store the encoded key in the keystore, then reload
bin/elasticsearch-keystore add cluster.remote.cluster_b.credentials

POST _nodes/reload_secure_settings

// On the LOCAL cluster (it also needs to trust the remote's CA:
// xpack.security.remote_cluster_client.ssl.certificate_authorities)
PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.cluster_b.seeds": ["remote-es01:9443"],
    "cluster.remote.cluster_b.skip_unavailable": true
  }
}

// Proxy mode alternative
PUT _cluster/settings
{ "persistent": { "cluster.remote.cluster_b.mode": "proxy",
                  "cluster.remote.cluster_b.proxy_address": "lb.example.com:9443" } }

GET _remote/info          // "connected": true, mode, num_nodes_connected, cluster_credentials

The same can be done in Kibana: Stack Management → Remote Clusters. The local nodes that make the connections need the remote_cluster_client role, which is included by default when no roles are set explicitly.

Local user privileges for the API key model: add remote_indices to the user's role on the local cluster:

PUT _security/role/ccs-reader
{
  "cluster": [],
  "indices": [ { "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] } ],
  "remote_indices": [
    { "clusters": ["cluster_b"], "names": ["logs-*"], "privileges": ["read", "read_cross_cluster", "view_index_metadata"] }
  ]
}

For CCR, the local user's role needs remote_indices with cross_cluster_replication and cross_cluster_replication_internal privileges, plus the local manage_ccr cluster privilege. The docs list the exact privileges per model.

📖 Remote clusters · Security models · Connection modes · Self-managed remote clusters · Add remote clusters with API key auth · Add remote clusters with TLS certificate auth · Migrate from certificate to API key · Create cross-cluster API key

Step 2a: cross-cluster search

See 4.6 for the queries. Per-remote options: skip_unavailable (default true); cluster.remote.<alias>.transport.compress. Check connectivity with GET _remote/info and GET _resolve/cluster/<alias>:*.

Step 2b: cross-cluster replication

Prerequisites: licence (Platinum/Enterprise or trial) on both clusters; the remote connection from the follower (local) cluster to the leader; soft deletes enabled on the leader index (the default since 7.0); compatible versions (the follower may be the same or newer).

// Follow one specific leader index (run on the FOLLOWER / local cluster)
PUT orders-copy/_ccr/follow?wait_for_active_shards=1
{
  "remote_cluster": "cluster_b",
  "leader_index": "orders",
  "max_read_request_operation_count": 5120,
  "settings": { "index.number_of_replicas": 0 }
}

// Auto-follow: automatically follow NEW leader indices (and data streams) that match patterns
PUT _ccr/auto_follow/logs-follower
{
  "remote_cluster": "cluster_b",
  "leader_index_patterns": ["logs-*", "metrics-*"],
  "leader_index_exclusion_patterns": ["logs-debug-*"],
  "follow_index_pattern": "{{leader_index}}-dr"
}

Managing replication:

GET orders-copy/_ccr/stats           GET orders-copy/_ccr/info           GET _ccr/stats
POST orders-copy/_ccr/pause_follow
POST orders-copy/_ccr/resume_follow

// Turn a follower into a normal writable index (e.g. failover), in order:
POST orders-copy/_ccr/pause_follow
POST orders-copy/_close
POST orders-copy/_ccr/unfollow
POST orders-copy/_open

GET _ccr/auto_follow            POST _ccr/auto_follow/logs-follower/pause
DELETE _ccr/auto_follow/logs-follower

CCR facts:

• Followers are read-only. Writing to them fails.
• Auto-follow only picks up leader indices created after the pattern exists. Follow existing ones explicitly.
• Replication is pull-based: the follower fetches operations from the leader's shard history (soft deletes, retained by shard history retention leases).
• Mapping and settings changes on the leader are replicated automatically. Some follower settings can't be changed.
• Data streams are replicated through auto-follow patterns, which replicate the backing indices and create a matching follower data stream.
• Uses: disaster recovery (active–passive), bringing data closer to users (geo read replicas), and centralised reporting.
• Bi-directional replication is possible with two auto-follow setups and distinct write targets (e.g. aliases or data streams per region).
• The ILM action unfollow converts followers automatically as they age.

📖 Cross-cluster replication · Set up CCR · CCR prerequisites · Configure privileges for CCR · Create a follower index · Create an auto-follow pattern · Manage CCR · Uni-directional disaster recovery · Bi-directional disaster recovery

Other cross-cluster operations to recognise:

• Reindex from remote: _reindex with source.remote (HTTP, port 9200; needs reindex.remote.whitelist). This one doesn't use remote-cluster connections.
• Restore a snapshot into another cluster: register the same repository (read-only, "readonly": true, on the second cluster) and restore.
• Cross-cluster ES|QL and EQL, and Kibana data views with cluster:index patterns.

5.3 Exercises: Cluster Management

1. Your cluster is yellow. Find which shards are unassigned and why, then fix it without adding nodes and without deleting data.
2. Index hot-data is red after node es03 died permanently. A snapshot from last night exists. Recover it as hot-data.
3. Register an fs repository backups, snapshot only indices matching orders* without global state, and create an SLM policy running every day at 02:15 that keeps between 3 and 10 snapshots for 14 days.
4. A node shows disk.used_percent: 96, and writes to logs-x fail with index read-only / allow delete. Explain what happened and fix it.
5. Connect cluster lab to cluster remote as dr using API-key authentication, permitting search on logs-* and replication of orders.
6. On lab, make orders-dr follow remote's orders. Then fail over: turn orders-dr into a writable index.
7. Configure auto-follow so every new logs-* index on remote is replicated to lab with the suffix -copy.

Solution notes

1. _cat/shards → _cluster/allocation/explain. Typical fixes: remove a bad index.routing.allocation.require.*; reset cluster.routing.allocation.enable; _cluster/reroute?retry_failed=true; or (on a 1-node lab) set number_of_replicas: 0 if the task says data only needs one copy.
2. DELETE hot-data (or close it), then POST _snapshot/<repo>/<snap>/_restore {"indices":"hot-data"}. Verify with _cat/indices and _count.
3. PUT _snapshot/backups {"type":"fs","settings":{"location":"…"}} (the path must be under path.repo). Snapshot with "indices":"orders*","include_global_state":false. SLM: "schedule":"0 15 2 * * ?", retention: {expire_after:"14d", min_count:3, max_count:10}.
4. The flood-stage watermark (95%) applied index.blocks.read_only_allow_delete. Free disk (delete, move or shrink data, or add capacity). The block auto-releases below the high watermark, or remove it explicitly with null.
5. On remote: enable the remote cluster server and create a cross-cluster API key with search: logs-*, replication: orders. On lab: add the key to the keystore as cluster.remote.dr.credentials on every node, _nodes/reload_secure_settings, set cluster.remote.dr.seeds, then check GET _remote/info.
6. PUT orders-dr/_ccr/follow {"remote_cluster":"dr","leader_index":"orders"}. Failover: pause_follow → _close → _ccr/unfollow → _open.
7. PUT _ccr/auto_follow/logs {"remote_cluster":"dr","leader_index_patterns":["logs-*"],"follow_index_pattern":"{{leader_index}}-copy"}. Only new leader indices are picked up.

---

6. Security & Access Control

Analogy for the whole domain: an office building. Authentication is the badge check at the door: who are you? (native users, API keys, LDAP/AD, SAML, OIDC, PKI, service accounts). Authorization / RBAC is which doors your badge opens: roles bundle privileges on the cluster, on indices (even specific documents and fields inside a room), and in Kibana features and spaces. TLS is the armoured corridor between buildings (nodes) and to visitors (clients). Audit logs are the CCTV.

6.1 Implement role-based access control (RBAC)

The model

user ──(has)──► roles ──(grant)──► privileges on resources
   ▲                                   ├─ cluster:   monitor, manage, manage_ilm, manage_index_templates, manage_security…
   │                                   ├─ indices:   read, write, create_doc, manage, view_index_metadata, all…
   │                                   │             + field-level security (FLS) + document-level security (DLS)
   │                                   ├─ applications: Kibana features & spaces
   │                                   └─ remote_indices / remote_cluster (for CCS/CCR)
role mapping (LDAP groups, SAML attributes, PKI DN…) ──► roles

Built-in roles worth knowing: superuser, kibana_admin, kibana_system (for Kibana's own service connection), viewer (read-only, including Kibana), editor, monitoring_user, remote_monitoring_agent, ingest_admin, snapshot_user, machine_learning_admin/_user, transform_admin, beats_system, logstash_system. Built-in users: elastic (superuser), kibana_system, logstash_system, beats_system, apm_system, remote_monitoring_user.

Privileges, the most-used ones:

Cluster privilege	Allows
monitor	Read-only cluster info (health, stats, nodes)
manage	Everything cluster-level except security
manage_index_templates, manage_ilm, manage_slm, manage_pipeline, manage_ingest_pipelines, manage_enrich, manage_ccr, manage_transform, manage_ml	The named feature
manage_security	Users, roles, API keys (effectively admin)
manage_own_api_key	Create or invalidate your own API keys
read_ccr, cross_cluster_search, cross_cluster_replication	Cross-cluster operations

Index privilege	Allows
read	Search, get, count
view_index_metadata	Get mappings, settings, aliases (Kibana needs this for data views)
write	Index, update, delete (includes index, create, delete)
create_doc	Only create new documents (ideal for shippers, data streams)
create_index, delete_index, manage (settings, aliases, refresh…), manage_ilm, auto_configure, maintenance	Index administration
monitor	Index stats, recovery, segments
all	Everything on those indices

Create roles and users

PUT _security/role/sales_analyst
{
  "description": "Read EU sales, hide PII, use Discover & Dashboards in the 'sales' space",
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["sales-*", "orders"],
      "privileges": ["read", "view_index_metadata"],
      "field_security": {
        "grant": ["*"],
        "except": ["customer.email", "customer.phone", "card.*"]
      },
      "query": { "term": { "region": "EU" } }
    },
    {
      "names": ["sales-reports"],
      "privileges": ["read", "write", "create_index"]
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["feature_discover.read", "feature_dashboard.all"],
      "resources": ["space:sales"]
    }
  ],
  "metadata": { "owner": "exam" }
}

POST _security/user/jane
{
  "password": "Str0ng-Passw0rd!",
  "roles": ["sales_analyst"],
  "full_name": "Jane Doe",
  "email": "jane@example.com",
  "metadata": { "team": "sales" }
}

POST _security/user/jane/_password    { "password": "An0ther-Str0ng-One!" }
POST _security/user/jane/_disable     POST _security/user/jane/_enable
GET  _security/user/jane              GET _security/role/sales_analyst
DELETE _security/role/old_role

Field-level security (FLS) uses field_security.grant / except. Wildcards are allowed. Metadata fields such as _id are always visible.

Document-level security (DLS) uses a query, which can be a templated query based on the user:

"query": { "template": { "source": { "term": { "owner": "{{_user.username}}" } } } }

When a user has several roles, the permissions are the union of all of them. A role without DLS on an index grants access to all documents of that index, even if another role restricts it. That's a classic exam trap.

Kibana privileges: application: "kibana-.kibana". Use "privileges": ["all"] or ["read"] for base access, or per-feature ones (feature_discover.read, feature_dashboard.all, feature_visualize.all…) for specific features. resources holds the spaces: space:<id>, or * for all. It's easiest to create these in Kibana → Stack Management → Roles and then check the result with GET _security/role/<name>.

Test what you granted

GET _security/_authenticate                          // who am I, which roles, which realm

GET _security/user/_has_privileges
{ "cluster": ["monitor"], "index": [ { "names": ["sales-2026"], "privileges": ["read", "write"] } ] }

// Run a request AS another user (requires the run_as privilege) to verify their view
GET sales-*/_search
# Header:  es-security-runas-user: jane

The fastest real-world check is to log into Kibana as the user, or call the API with -u jane:password, and try the allowed and the forbidden actions.

API keys

POST _security/api_key
{
  "name": "ingest-shipper",
  "expiration": "30d",
  "role_descriptors": {
    "writer": {
      "cluster": ["monitor"],
      "indices": [ { "names": ["logs-app-*"], "privileges": ["create_doc", "auto_configure"] } ]
    }
  },
  "metadata": { "app": "shipper" }
}
// → returns id, api_key, encoded   → header:  Authorization: ApiKey <encoded>

GET _security/api_key?name=ingest-shipper
DELETE _security/api_key  { "name": "ingest-shipper" }             // invalidate
POST _security/api_key/_query { "query": { "term": { "name": "ingest-shipper" } } }

An API key's privileges are the intersection of the role_descriptors and the creator's privileges at creation time. Without role_descriptors, it gets a snapshot of the owner's permissions.

Role mappings (external realms)

PUT _security/role_mapping/ldap-admins
{
  "enabled": true,
  "roles": ["superuser"],
  "rules": { "field": { "groups": "cn=admins,ou=groups,dc=example,dc=com" } }
}

PUT _security/role_mapping/saml-readers
{
  "enabled": true,
  "roles": ["viewer"],
  "rules": { "all": [ { "field": { "realm.name": "saml1" } },
                      { "field": { "groups": "readers" } } ] }
}

Rules support field, any, all and except. Fields: username, dn, groups, realm.name, metadata.*. Use role_templates for dynamic role names.

Realms, in order of evaluation (order): reserved → file → native (both on by default) → ldap, active_directory, pki, saml, oidc, kerberos, jwt. SAML and OIDC work through Kibana. Service accounts (e.g. elastic/kibana, elastic/fleet-server) use service tokens instead of passwords.

📖 Users and roles · User roles · Defining roles · Role structure · Built-in roles · Security privileges · Kibana privileges · Kibana role management · Document and field level security · Privileges for data streams and aliases · Role mapping · Native realm · Built-in users · Service accounts · Elasticsearch API keys · Submit requests on behalf of other users

6.2 Secure a cluster for production

Analogy: moving from a house party to a bank. At a party the door is open and everyone knows each other (development mode). A bank has locked doors (authentication), badges per room (RBAC), armoured transport between branches (transport TLS), bulletproof glass at the counter (HTTP TLS), a vault for secrets (keystore), cameras (audit logs), and a building inspection before opening (bootstrap checks).

Layer 1: security on, with TLS everywhere

Since 8.0, a fresh self-managed install auto-configures security: it generates certificates, enables TLS on the transport and HTTP layers, prints the elastic password, and gives you an enrollment token for Kibana and additional nodes. Know how to do it manually too:

# 1. CA + transport (node-to-node) certificates
bin/elasticsearch-certutil ca --out elastic-stack-ca.p12
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --out elastic-certificates.p12
# 2. HTTP certificates (interactive wizard: CSR or CA-signed, hostnames, IPs)
bin/elasticsearch-certutil http
# 3. Store keystore passwords
bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

# elasticsearch.yml
xpack.security.enabled: true
xpack.security.enrollment.enabled: true

xpack.security.transport.ssl.enabled: true               # REQUIRED for multi-node clusters with security
xpack.security.transport.ssl.verification_mode: certificate   # or full (also checks hostnames)
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true                    # HTTPS for REST clients & Kibana
xpack.security.http.ssl.keystore.path: certs/http.p12

# Passwords & tokens
bin/elasticsearch-reset-password -u elastic                 # or -i for interactive
bin/elasticsearch-reset-password -u kibana_system
bin/elasticsearch-create-enrollment-token -s kibana         # or -s node
bin/elasticsearch-service-tokens create elastic/kibana my-token

# kibana.yml (secured connection to Elasticsearch)
elasticsearch.hosts: ["https://es01:9200"]
elasticsearch.serviceAccountToken: "AAEAAWVs..."            # preferred over username/password
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/http_ca.crt"]
server.ssl.enabled: true                                     # HTTPS for browsers → Kibana
server.ssl.certificate: /etc/kibana/certs/kibana.crt
server.ssl.key: /etc/kibana/certs/kibana.key
xpack.encryptedSavedObjects.encryptionKey: "<32+ chars>"     # needed for alerting and saved-object secrets
xpack.security.encryptionKey: "<32+ chars>"
xpack.reporting.encryptionKey: "<32+ chars>"

Rules:

• With security enabled, a multi-node cluster must have transport TLS. Otherwise nodes refuse to form a cluster once they bind to a non-loopback address (production mode).
• Secrets belong in the keystore (elasticsearch-keystore), never in plain elasticsearch.yml. Reloadable secure settings can be refreshed with POST _nodes/reload_secure_settings.
• Don't use the elastic superuser for daily work or for Kibana. Create personal admin users, and use service accounts or API keys for applications.
• Anonymous access stays disabled unless explicitly required.

Layer 2: production-mode node configuration and bootstrap checks

As soon as network.host binds to a non-loopback address, Elasticsearch runs in production mode and enforces bootstrap checks. A failing check stops the node from starting.

Check / setting	Requirement
Heap size	-Xms = -Xmx (set in jvm.options.d/*.options or ES_JAVA_OPTS). By default Elasticsearch sizes the heap automatically; manually, ≤ 50% of RAM and below the compressed-oops threshold (~31 GB).
File descriptors	≥ 65,535 (ulimit -n)
Max threads	≥ 4,096
Virtual memory	vm.max_map_count ≥ 262,144 (check the docs for the value recommended for your version)
Memory lock	If bootstrap.memory_lock: true, the lock must succeed. Also disable swap.
Discovery	discovery.seed_hosts / cluster.initial_master_nodes (first bootstrap only) or equivalent must be configured
Others	Max file size, max virtual memory, JVM checks (client JVM, serial GC), system call filter, OnError/OnOutOfMemoryError

cluster.name: prod-search
node.name: es-prod-01
node.roles: [ master, data_hot, ingest ]
network.host: 10.0.1.11            # _site_, _local_, specific IP…
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["10.0.1.11", "10.0.1.12", "10.0.1.13"]
cluster.initial_master_nodes: ["es-prod-01", "es-prod-02", "es-prod-03"]   # remove after first start
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
action.destructive_requires_name: true     # no DELETE on wildcards / _all (the default is true in 8+)

Layer 3: least privilege, network restrictions and auditing

Least privilege checklist:

• Personal admin accounts instead of the shared elastic account.
• Shippers (Agent, Beats, Logstash) get API keys with create_doc + auto_configure on their own data streams only.
• Applications get dedicated users or API keys limited to their indices.
• Analysts get read + view_index_metadata, with FLS/DLS where needed, and Kibana privileges limited to specific spaces.
• action.destructive_requires_name: true prevents DELETE *.

IP filtering (self-managed; on Cloud use traffic filters or network security):

xpack.security.transport.filter.allow: ["10.0.1.0/24"]
xpack.security.transport.filter.deny: _all
xpack.security.http.filter.allow: ["10.0.0.0/16"]
xpack.security.http.filter.deny: _all

Audit logging (needs a Platinum/Enterprise licence or the trial):

xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.include: ["access_denied", "authentication_failed", "connection_denied", "tampered_request", "run_as_denied", "security_config_change"]
xpack.security.audit.logfile.events.emit_request_body: false

The output goes to <cluster>_audit.json in the logs directory. Ignore policies (xpack.security.audit.logfile.events.ignore_filters.*) reduce noise. Kibana has its own audit log (xpack.security.audit.enabled in kibana.yml).

Other production hardening:

• Encrypt data at rest via the file system or disk (Elasticsearch doesn't encrypt shards itself). On Cloud, use customer-managed keys.
• Keep Elasticsearch off the public internet. Put it behind a firewall, load balancer or VPN, and expose only Kibana over HTTPS if needed.
• Set a session timeout in Kibana (xpack.security.session.idleTimeout, lifespan).
• Rotate certificates before they expire. The docs cover rotating with the same CA and with a different CA.
• Snapshot the security feature state (feature_states: ["security"]) so users and roles can be restored.
• For FIPS environments, use FIPS 140-2 mode with approved algorithms and keystores.

📖 Security overview · Self-managed security setup · Automatic security setup · Minimal security setup · Set up transport TLS · Set up HTTPS · Configure security in Kibana · Manage TLS encryption · Update TLS certificates · Secure settings (keystore) · IP filtering (self-managed) · Audit logging · Enable audit logging · Important system configuration · Bootstrap checks · Important settings · Kibana session management · Tutorial: self-managed secure install

6.3 Exercises: Security

1. Create role logs_reader: read and view metadata on logs-*, no access to the field user.password_hash, only documents where service.environment is production, plus Discover (read) in the Kibana space ops. Create user ana with this role and verify with _has_privileges and by logging in.
2. Create role ingest_writer that can only append documents to data streams matching logs-app-*, and create an API key limited to it that expires in 7 days.
3. User bob has roles A (DLS: team: blue on projects) and B (plain read on projects). Which documents can he see, and why? Fix it so he sees only team: blue.
4. Map LDAP group cn=sre,ou=groups,dc=corp to roles monitoring_user and logs_reader.
5. On a three-node lab started without security, enable security with transport TLS (own CA) and HTTPS, reset the elastic password, connect Kibana with a service account token, and turn on audit logging for authentication failures and denied access.
6. List the bootstrap checks that would fail on a VM with vm.max_map_count=65530, ulimit -n 4096 and swap enabled with bootstrap.memory_lock: true, and fix each one.

Solution notes

1. indices: [{names:["logs-*"], privileges:["read","view_index_metadata"], field_security:{grant:["*"], except:["user.password_hash"]}, query:{term:{"service.environment":"production"}}}], applications: [{application:"kibana-.kibana", privileges:["feature_discover.read"], resources:["space:ops"]}].
2. Role: indices: [{names:["logs-app-*"], privileges:["create_doc","auto_configure"]}]. API key: POST _security/api_key with expiration: "7d" and the same role_descriptors.
3. He sees all documents: role permissions are a union, and B's unrestricted read wins. Fix by removing B from bob, or adding the same DLS query to B.
4. PUT _security/role_mapping/sre {"roles":["monitoring_user","logs_reader"],"enabled":true,"rules":{"field":{"groups":"cn=sre,ou=groups,dc=corp"}}}.
5. certutil ca → certutil cert (copy to all nodes) → certutil http → transport and HTTP SSL settings + keystore passwords → rolling or full restart → elasticsearch-reset-password -u elastic → elasticsearch-service-tokens create elastic/kibana kibana-token → kibana.yml elasticsearch.serviceAccountToken + CA → xpack.security.audit.enabled: true with the events.include list.
6. Max map count → sysctl -w vm.max_map_count=262144 (persist it in /etc/sysctl.d). File descriptors → raise the limit to 65535 (limits.conf or systemd LimitNOFILE). Memory lock → LimitMEMLOCK=infinity (systemd) or ulimit -l unlimited. Disable swap (swapoff -a) as good practice.

---

7. Final checklist and study plan

7.1 A four-week plan

Week	Focus	Deliverable
1	Domains 1–2: lab build, templates, dynamic templates, data streams, ILM, aliases	Every Domain 2 exercise done twice without notes, just the docs
2	Domain 3: mappings, analyzers, field types, reindex/UBQ, pipelines, enrich, Streams	A "migration" drill: v1 → v2 with a new mapping, pipeline and alias swap in < 20 min
3	Domain 4: Query DSL, bool, aggregations, ES|QL, semantic search, CCS	20 queries on the sample data; translate 5 of them to ES|QL
4	Domains 5–6 + mock exams: health, snapshots, SLM, remote clusters, CCR, RBAC, TLS	Two timed mock exams of 10 mixed tasks in 3 hours

7.2 "Can I do this from a blank Console with only the docs?"

• ☐ Explain node roles, tiers, primaries vs replicas, and the write and read paths
• ☐ Configure forced awareness and allocation filtering; explain a yellow cluster
• ☐ Create an index with custom analyzers, multi-fields, nested fields and strict mapping
• ☐ Component + index templates with priority; simulate them
• ☐ Dynamic templates with correct ordering
• ☐ A data stream from a template, with ILM rollover/warm/delete; _ilm/explain
• ☐ Alias swap, filtered alias, write index
• ☐ Choose an ingest method for a scenario
• ☐ Reindex with query, script and pipeline; update-by-query to populate a new multi-field
• ☐ A pipeline with grok/dissect, date, set, conditionals, on_failure, enrich; simulate it
• ☐ Streams: processing, retention and data quality in the UI
• ☐ match, match_phrase, multi_match, term/range, bool with filter/should logic
• ☐ Terms/date_histogram/range buckets with metrics and pipeline aggs
• ☐ ES|QL with WHERE, EVAL, STATS…BY, SORT, LIMIT, DISSECT and LOOKUP JOIN
• ☐ semantic_text + inference endpoint + semantic/match query + RRF hybrid
• ☐ Remote cluster (API key) + CCS + CCR follower + auto-follow + failover
• ☐ Diagnose unassigned shards with allocation explain; fix watermark blocks
• ☐ Snapshot repository, snapshot, restore with rename, SLM policy
• ☐ Roles with FLS/DLS and Kibana privileges, users, API keys, role mappings
• ☐ TLS (transport + HTTP), keystore, bootstrap checks, audit logging

7.3 Further references

• 📖 Elastic Documentation home: the only resource allowed in the exam
• 📖 Elasticsearch REST API reference
• 📖 Kibana API reference
• 📖 Elasticsearch release notes: skim 9.0 → 9.3 for new features and breaking changes
• 🎓 Elastic Training: Elasticsearch Engineer course: the official course aligned to the exam
• 🧪 Elasticsearch Labs: tutorials on semantic search, ES|QL and retrievers
• 💬 Elastic Discuss forum: exam-experience threads
• 📝 Community practice exercises by Guido Lena Cota (written for the 8.x exam, still useful for templates, mappings and search): Store data · Deploy and operate a cluster

Prefect — Components Deep Dive

A study guide to Prefect 3 (Python workflow orchestrator), one component at a time: what it is, the code you will type, when to use it, the traps, and where the official docs live.

How to read this guide. Each component has (1) a mental model, usually an analogy, (2) the code, (3) a when to use / when not to verdict, (4) traps and (5) 📖 doc links. The decision tables for “which way do I run this?” (§5-6) and “when is Prefect the wrong tool?” (§13) are the parts to re-read before choosing an orchestrator. The sibling page RustFS — Components Deep Dive covers an S3-compatible store that Prefect can use for results and code. Version note: written 23 Oct 2026; the docs index lists Prefect 3.9 as the latest OSS line. ⚑ marks details I wrote from prior knowledge rather than re-reading the page this session (decorator parameters, exact import paths, CLI flags). Confirm those against the linked docs and your installed version (prefect version) before copying.

---

0. The map: what Prefect is

Analogy: an airline operations centre. Your flights are ordinary Python functions. The operations centre does not fly the plane; it tracks every flight, reschedules on delay (retries), reroutes when an airport closes (infrastructure), logs everything, and alerts when something goes wrong. Planes (your code) fly on your own aircraft (your infrastructure).

• Workflows are plain Python. A flow is a decorated function; a task is a decorated function called inside it. There is no DAG file to declare: dependencies come from data flow and normal control flow (if, for, try), so a workflow can branch and loop dynamically.
• Orchestration is hybrid. Your code runs on your infrastructure; the Prefect API (self-hosted server or Prefect Cloud) stores state, schedules, events and automations. Task runs are orchestrated client-side and their updates are logged in batches, so task state in the UI is eventually consistent.
• Code is not stored by Prefect. A deployment is server-side metadata (which flow, where its code lives, which infrastructure, schedule, parameters). Deployments reference flows defined in Python files; they do not contain code.
• Prefect 3 changes (old tutorials differ): workers and work pools replace agents; block-based deployments and storage/infrastructure blocks are the legacy path; assets, transactions and background tasks are first-class. The 2.x to 3.0 upgrade and agents-to-workers guides exist under how-to-guides/migrate.

Piece	Where it lives	Job
Flow / task code	Your repo or image	The actual work
Prefect API + database	Cloud or self-hosted server (Postgres)	State, schedules, events, UI
Deployment	API (metadata only)	Run this flow, from here, on that pool, on this schedule
Work pool (+ queues)	API	A channel between deployments and infrastructure
Worker	Your infra (or none for push/managed pools)	Polls the pool and starts runs

📖 Concepts · Quickstart · Upgrade to 3.0 · Migrate from Airflow

---

1. Flows

Analogy: a recipe card. The card lists the steps; any cook with a kitchen can follow it. The kitchen is infrastructure; the order ticket is a flow run.

from prefect import flow, task, get_run_logger

@flow(name='daily-etl', retries=1, retry_delay_seconds=60, timeout_seconds=3600, log_prints=True)
def daily_etl(day: str, full_refresh: bool = False) -> int:
    rows = extract(day)
    return load(rows)

if __name__ == '__main__':
    daily_etl('2026-10-22')          # runs locally, recorded in the API if one is configured

• A flow is callable like a function. Calling it creates a flow run with its own state history. ⚑ Common options: name, retries, retry_delay_seconds, timeout_seconds, log_prints, validate_parameters (type-checks arguments against hints), persist_result, task_runner, on_completion/on_failure/on_cancellation/on_crashed hooks.
• Subflows: a flow may call another flow; it becomes a child run with its own state. Flows and tasks can call each other with no required nesting pattern.
• Flow versus task: a flow is the unit you deploy, schedule and parameterise; a task is a unit you retry, cache and run concurrently. Do not put a deployment boundary inside a task.
• Flow retries rerun the whole flow; task retries rerun one step. Prefer task retries plus caching so a flow retry does not redo finished work.

Traps. Parameters must be serialisable for deployments (they travel as JSON). Top-level code outside @flow runs on every import, including when a worker loads the file. Keep imports light and put heavy setup inside tasks.

📖 Flows · Write and run a workflow · Pass inputs

---

2. Tasks

Analogy: one station on the line. Small, with a clear input and output, and the supervisor logs how long it took and whether it passed.

Tasks are cacheable, retryable units of work with transactional semantics. A task is identified by a task key (a hash of its name and fully qualified function name). The docs recommend small tasks: each should be a logical step or a significant side effect, so the UI narrates your workflow.

from datetime import timedelta
from prefect import task
from prefect.cache_policies import INPUTS, TASK_SOURCE

@task(retries=3, retry_delay_seconds=[10, 60, 300], timeout_seconds=120,
      cache_policy=INPUTS + TASK_SOURCE, cache_expiration=timedelta(hours=1), log_prints=True)
def extract(url: str) -> dict:
    ...

2.1 Three ways to run a task

Call	Blocks?	Returns	Use for
t(x)	Yes	The result	Simple sequential steps
t.submit(x)	No	PrefectFuture, resolve with .result() or .wait() from the same context	Concurrency inside a flow
t.delay(x)	No	Future you need not resolve from the same context	Fire-and-forget background work on separate workers (§10)

• Dependencies are implicit: passing an upstream result or future into a downstream task means it cannot start until the upstream is Completed. Add wait_for=[...] for explicit ordering. t.map(items) fans out over an iterable ⚑.
• Parameter resolution trap (documented): Prefect walks parameters to resolve nested futures and states. If a dataclass or named tuple field changes, Prefect rebuilds it with dataclasses.replace(), which calls __init__ and __post_init__ again; Pydantic models use model_construct() and skip validators. Use quote(obj) to pass a fully resolved object untouched (no future resolution, no dependency tracking), or opaque(future) to resolve a top-level future without traversing its contents (from prefect.utilities.annotations).
• Task run life cycle: Pending → Running → Completed; on error Retrying/AwaitingRetry → Running; retries exhausted → Failed; infrastructure failure → Crashed; a cached result jumps Pending → Completed. Background tasks start as Scheduled.

When to make something a task: it has a side effect, may fail transiently, is worth caching, or should run concurrently. When not to: trivial pure helpers, and anything so fine-grained that per-call overhead and UI noise exceed the value.

📖 Tasks · Retries · Run work concurrently

---

3. Task runners and concurrency control

Analogy: how many cooks you hire and whether they share a kitchen. A task runner decides whether submitted tasks run in threads, processes, or on a cluster. Concurrency limits are the 'only two people at the fryer' rules.

• Task runners control how .submit()/.map() tasks execute: concurrent threads (the default ⚑), parallel processes, or distributed (prefect-dask, prefect-ray integrations). Set per flow with @flow(task_runner=...).
• Choose: I/O-bound work (HTTP, DB, S3) → threads. CPU-bound Python → processes or Dask/Ray. Data too large for one machine → Dask or Ray, or push the work into the warehouse and orchestrate only.
• Global concurrency limits are named limits you acquire in code and share across flows and deployments, for protecting a database or an API; rate limits use the same mechanism with a refill rate. Tag-based limits cap concurrent task runs carrying a tag.

from prefect.concurrency.sync import concurrency, rate_limit      # ⚑ import path

@task
def write_db(rows):
    with concurrency('warehouse-writes', occupy=1):   # blocks if the limit is full
        ...
@task
def call_api(x):
    rate_limit('vendor-api')                           # waits for a slot at the configured rate
    ...

Traps. A task runner only affects tasks started with .submit()/.map(); direct calls are sequential. Threads do not speed up CPU-bound code. Forgetting to create the named global limit means the code has nothing to enforce. Worker-level limits (work pool and queue concurrency, §6) limit flow runs, not tasks.

📖 Task runners · Global concurrency limits · Tag-based limits · Per-worker task concurrency example

---

4. States, results, caching and transactions

Analogy: a parcel tracking page, a saved receipt and an undo button. State says where the run is, results are the saved output, caching says 'I already did this exact job', and transactions say 'do these steps all-or-nothing'.

• States are rich objects (Pending, Running, Completed, Failed, Cancelled, Crashed, Paused, Scheduled, Retrying/AwaitingRetry...). They carry data and messages, drive state-change hooks and are what automations trigger on. Crashed means the infrastructure failed (killed process, out of memory), not that your code raised.
• Results are the data a flow or task returns. Persisting them is off by default for most runs and is what enables caching and resumption. Choose result_storage (local path, or an S3-compatible bucket via an integration block), persist_result=True, and the serialiser. ⚑ Persisted results need storage reachable by every worker that may read them.
• Caching skips a task when a cache key matches. Policies combine with +: INPUTS (hash of arguments), TASK_SOURCE (the task's code), FLOW_PARAMETERS, RUN_ID and others ⚑; plus cache_expiration. Use a custom key_fn when arguments are not hashable or should be partly ignored. A cache hit moves the task straight to Completed.
• Transactions give tasks transactional semantics: group steps so they commit together, and register rollback hooks that run if a later step fails.

from prefect.transactions import transaction      # ⚑

@task
def write_staging(batch): ...
@write_staging.on_rollback
def drop_staging(txn): ...                         # runs if the transaction rolls back

@flow
def load(batch):
    with transaction():
        write_staging(batch)
        swap_into_prod()                           # if this raises, drop_staging runs

Use caching for expensive, deterministic steps (downloads, parsing, model scoring on fixed input) so retries and reruns are cheap. Do not cache steps whose result depends on time or external state unless the key includes it, or you will happily serve stale data. Idempotency still matters: a retry or a worker crash can run a task twice, so design writes as upserts or partition overwrites.

📖 States · Caching · Customize caching · Persist and retrieve results · Transactional workflows

---

5. Deployments: three ways to run a flow

Analogy: a standing order versus cooking at home. Running the function yourself is cooking at home. A deployment is a standing order registered with the operations centre: it says what, where the recipe lives, which kitchen, and when.

Approach	How	Infrastructure	Pick it when
Run the function	python flow.py	Wherever you run it	Development and tests
flow.serve()	A long-lived Python process listens for runs and runs schedules	That process (container, VM)	Simple, single environment, no pool to manage
flow.from_source(...).deploy() or prefect deploy with prefect.yaml	Deployment bound to a work pool	Dynamically provisioned per run	Many deployments, different environments, isolation per run, platform-team guardrails

# 1. serve: simplest
daily_etl.serve(name='daily-etl', cron='0 6 * * *', parameters={'day': 'today'})

# 2. deploy to a work pool (code pulled from git at run time)
from prefect import flow
flow.from_source(source='https://github.com/acme/pipelines.git', entrypoint='flows/etl.py:daily_etl').deploy(
    name='daily-etl-prod', work_pool_name='k8s-pool', cron='0 6 * * *',
    parameters={'day': 'today'}, job_variables={'image': 'acme/pipelines:1.4'}, tags=['etl'])     # ⚑ argument names

# trigger on demand
from prefect.deployments import run_deployment
run_deployment(name='daily-etl/daily-etl-prod', parameters={'day': '2026-10-22'})

• Three ways to define a deployment: Python SDK (.deploy()), YAML (prefect.yaml plus prefect deploy), or Terraform. All populate the same schema: name, flow, parameters, schedules, tags, work pool, job variables, version.
• Code storage: bake the code into the Docker image (pool types that use images), pull it from git or a cloud bucket at run time, or use a local path with Process pools. Do not rely on a path that exists only on your laptop.
• Versioning: deployments carry a version and can be rolled back; tag flows and tasks with version too. Override per-run infrastructure with job variables.
• CI/CD: deploy from CI on merge, one deployment per environment, with the image tag as a job variable.

Traps. A deployment does not contain code, so a worker that cannot reach the repo or image fails the run. Changing the flow signature without redeploying breaks scheduled runs that carry old parameters. serve ties execution to one process that must stay up.

📖 Deployments · Create deployments · prefect.yaml · Store flow code · Versioning · CI/CD

---

6. Work pools, workers and queues

Analogy: pub/sub topics. A work pool is a topic. Deployments publish runs to it; workers subscribe and start the runs. Switch a deployment to another pool and a different worker runs it, which is how you promote between environments or debug locally.

Pool kind	Who starts runs	You run a worker?	Examples
Hybrid	A worker in your infrastructure	Yes	Process, Docker, Kubernetes, AWS ECS, Azure Container Instances, Google Cloud Run (v1, v2), Vertex AI
Push (Cloud)	Prefect submits to your serverless provider	No	ECS push, Cloud Run push, ACI push, Modal push
Managed (Cloud)	Prefect-managed infrastructure	No	Prefect Managed; Coiled is also available

• Why pools at all: dynamic infrastructure per run (scale to zero), default job configuration that every deployment inherits and can override, and opinionated, enforced interfaces for platform teams.
• Workers: prefect worker start --pool my-pool ⚑ polls and submits runs; a hybrid worker needs outbound access to the API and the permissions to create infrastructure. Run it as a supervised service (systemd, a Deployment) and use the worker healthcheck documented under advanced guides.
• Work queues sit inside a pool. Each has a priority (a unique positive integer; 1 is highest) and an optional concurrency limit that cannot exceed the pool's global limit. Runs are submitted from the highest priority queue in waterfall order. A queue is READY if a worker polled it in the last 60 seconds, PAUSED when you pause it, otherwise NOT_READY.
• Example policy: low (priority 10, unlimited), high (priority 5, limit 3), critical (priority 1, limit 1) for urgent backfills.
• Job variables and the base job template define per-pool defaults (image, resources, env vars, namespace); override them on the deployment or even per run. Customising Kubernetes job templates has its own guide.

Pick a pool: laptop or single VM → Process. Containerised isolation on one host → Docker. Production on a cluster → Kubernetes. Serverless without operating a worker → push pool on Cloud. Avoid a Process pool for untrusted or conflicting dependencies.

Traps. The worker's own credentials decide what it can launch, so scope them. A pool with no live worker leaves runs Late. Mixing images with different Python or Prefect versions between client and worker causes confusing serialisation errors; keep versions aligned (see the versioning and compatibility page).

📖 Work pools · Workers · Manage work pools · Kubernetes · Docker · Serverless · Job variables · Worker healthchecks

---

7. Schedules, events and automations

Analogy: an alarm clock versus a motion sensor. A schedule fires at times you choose; an event trigger fires when something happens, such as a file arriving or a flow failing.

• Schedules tell the API to create flow runs automatically: cron, interval or RRule, attached to a deployment, with timezone support. Pause, resume or remove them per deployment.
• Events record activity in your stack: Prefect emits them for state changes and you can emit custom ones (emit_event(...) ⚑). They carry a resource, an event name and a payload.
• Automations pair a trigger (event, metric or compound) with actions: run a deployment, pause a schedule, send a notification, cancel a run. Deployment triggers make event-driven pipelines; chaining deployments with events gives dependencies between separate workflows without coupling the code; you can pass the event payload as flow parameters.
• Debounce with the within parameter so a burst of events does not create a run per event. Zombie flows (runs that stopped reporting) can be detected with an automation.

Choose a schedule for predictable batch windows; choose an event trigger for 'when upstream finished' or 'when data landed'. Prefer events over polling loops inside a flow.

Traps. Cron is in UTC unless a timezone is given. A schedule creates runs even if the previous one is still running unless concurrency limits stop it. Triggers that match their own outputs can loop.

📖 Schedules · Events · Automations · Event triggers · Chaining deployments · Debouncing · Zombie flows

---

8. Configuration: variables, blocks, secrets, settings

Thing	Holds	Use for	Not for
Variables	Dynamically-named, mutable, non-sensitive JSON values	Shared config such as a table name or a flag	Secrets
Blocks	Typed, saved configuration with a UI (credentials, buckets, notification targets)	Reusable connections and secrets managed centrally	Per-run ephemeral data
Secret blocks	Encrypted sensitive strings	API keys, passwords	Large documents
Settings and profiles	PREFECT_* settings per environment	API URL, logging, timeouts, server tuning	Application config
Runtime context	Info about the current run (prefect.runtime)	Flow run ID, parameters, deployment name inside code	Persistent state

from prefect.variables import Variable
from prefect.blocks.system import Secret
from prefect.runtime import flow_run

limit = Variable.get('batch_size', default='500')
token = Secret.load('vendor-api-token').get()
print(flow_run.id, flow_run.parameters)          # ⚑ runtime attribute names

Switch between environments with profiles (prefect profile use prod ⚑) rather than editing code. Self-hosted servers have security settings (basic auth and others) documented under advanced guides; do not expose an unauthenticated API to a network.

📖 Variables · Blocks · Settings and profiles · Runtime context · Store secrets

---

9. Observability, artifacts, assets and human input

• Logging: log_prints=True turns print into run logs; get_run_logger() gives a structured logger. Logging config is customisable. Cloud offers AI summaries of failed-run logs.
• Artifacts are persisted outputs for humans, rendered in the UI (markdown, tables, links, progress): create_markdown_artifact(...), create_table_artifact(...) ⚑. Use them for data-quality reports and run summaries, not as a results store.
• Assets represent objects your workflows produce (a table, a file, a model) and are tracked as lineage across runs. Declare them on tasks with @materialize(...) ⚑ and add metadata and explicit dependencies where inference is not enough. Check your version supports them.
• State change hooks run code when a flow or task enters a state (on failure, on completion): the place for notifications and cleanup.
• Human in the loop: a run can pause and wait for input or approval, or suspend and free its infrastructure, then resume from the UI or API. Interactive workflows have a dedicated guide. Cancellation of running flows is a documented workflow too.

📖 Logging · Artifacts · Assets · State hooks · Interactive workflows · Cancel workflows

---

10. Background tasks

Analogy: a drop-box and a back-office team. The web app drops a job in the box and returns to the customer immediately. Back-office workers pick it up whenever they are free.

task.delay(...) pushes a task run onto a server-side topic; a pool of task workers executes it. The docs position this as a replacement for Celery plus a broker for queue functionality. Use it when a web request or service must dispatch heavy work without waiting and you want to scale task execution independently. Do not use it for work that needs a flow-level schedule, deployments and per-run infrastructure; use a deployment for that. The caller does not need to resolve the returned future.

📖 Run background tasks · Web app powered by background tasks

---

11. Hosting: Cloud versus self-hosted

	Prefect Cloud	Self-hosted server
You operate	Workers and your code only	API, database, background services, workers
Extras	Push and managed pools, workspaces, SSO, service accounts, ACLs, audit logs, webhooks, SLAs, PrivateLink	Open source; fewer pool types (hybrid ones)
Limits	Documented API rate limits and data retention	Yours to size

• Run a server: prefect server start locally; Docker; Docker Compose with Postgres, Redis, server, background services and a worker; Helm chart for Kubernetes. Point clients at it with PREFECT_API_URL.
• Scale: run multiple server instances for HA (needs Postgres and Redis for messaging); keep the database maintained (monitoring, vacuuming, retention of old runs).
• Secure: enable authentication via the security settings, restrict network access, and see the page on endpoints and ports for restricted networks.

📖 Server · Docker Compose · Helm · Scale self-hosted · Database maintenance · Security settings

---

12. Testing and integrations

• Test: a flow is a function, so unit-test the logic; the docs describe a test harness and pytest fixtures for running flows and tasks against a temporary backend. Test deployment definitions by loading them in CI.
• Integrations are PyPI packages: prefect-aws (S3, ECS worker, Secrets Manager, Lambda, Glue, Batch), prefect-gcp, prefect-azure, prefect-kubernetes, prefect-docker, prefect-dbt (runner with asset lineage, or per-node orchestration), prefect-snowflake, prefect-sqlalchemy, prefect-databricks, prefect-dask, prefect-ray, prefect-redis, prefect-slack, prefect-email, prefect-github, prefect-gitlab, prefect-bitbucket, prefect-shell.
• AI assist: a Prefect MCP server lets assistants inspect your workspace.

📖 Test workflows · Integrations · prefect-aws · prefect-dbt · MCP server

---

13. When is Prefect the right tool?

Situation	Verdict
Python-heavy data or ML pipelines with retries, caching, scheduling, dynamic fan-out	Good fit. Flows are ordinary code
Event-driven pipelines (file landed, upstream finished)	Good fit (events and automations)
A team that wants per-run isolated infrastructure without operating a scheduler fleet	Good fit (work pools, push or managed)
You need a declarative, reviewable DAG with a large existing operator ecosystem and heavy SQL-first teams	Airflow may fit better; Prefect has a migration guide, so weigh the rewrite
The product is the asset graph with software-defined data assets and a data catalogue	Compare with asset-first tools; Prefect's assets are newer, so check maturity for your needs
Strict durable execution of business transactions over days with exactly-once semantics	A durable-execution engine such as Temporal is a closer match
One cron job calling one script	Overkill. Use cron or a platform scheduler until you need retries, visibility or dependencies
Heavy transformation inside a warehouse	Orchestrate it (dbt or SQL tasks) and let the warehouse do the work

Challenge before adopting: orchestration adds an API, a database and workers to run, so ask what failure you are actually solving (silent failures, no retries, no visibility) and whether a simpler scheduler plus alerting would do. Also confirm where results and logs may be stored if your data is regulated.

---

14. Trap checklist and exercises

1. Expecting a deployment to carry code; worker cannot fetch the repo or image.
2. Client, worker and image on different Prefect or Python versions.
3. Non-idempotent writes with retries; caching a time-dependent task with a code-only key.
4. Using threads for CPU-bound work; using .submit() but never resolving futures, so errors go unnoticed.
5. Dataclass fields rebuilt by parameter resolution (use quote or opaque).
6. Cron in UTC surprises; schedules overlapping long runs; trigger loops.
7. Secrets in variables instead of secret blocks; unauthenticated self-hosted API.
8. A pool with no worker, or worker credentials broader than needed.
9. Persisting results to storage that other workers cannot reach.

Exercises. (1) Write a three-task flow with retries=3 on a flaky task and caching on an expensive one; run it twice and read the states. (2) Deploy it with serve, then to a Process work pool, and compare where the code runs. (3) Add a global concurrency limit and submit 20 tasks. (4) Add a deployment trigger that runs a second flow when the first completes. (5) Wrap two writes in a transaction() with a rollback hook and force the second to fail.

Solution notes

1. A cached task shows Completed immediately on the second run; flow retries rerun the flow but cached tasks are skipped.
2. serve runs in your process; the pool path needs a worker polling the pool and a reachable code source.
3. Create the limit first (CLI or UI), then acquire it in code with concurrency(name); excess tasks wait.
4. Use an automation with a flow-run-completed trigger and a run-deployment action, optionally passing the event payload.
5. The rollback hook of the first write runs, leaving no partial output.

---

Sources: docs.prefect.io (v3): tasks, work pools, deployments, documentation index. Other sections were written from the index summaries plus prior knowledge; items marked ⚑ should be verified.

RustFS — Components Deep Dive

A study guide to RustFS, an open-source, S3-compatible distributed object store written in Rust: what each component is, the commands and settings you will actually use, when to use it, the traps, and where the docs live.

How to read this guide. Each component has (1) a mental model, (2) the code or config, (3) a when to use / when not to verdict, (4) traps and (5) 📖 doc links. The decision tables are in §2 (which deployment mode) and §11 (should I use RustFS at all). The sibling page Prefect — Components Deep Dive covers an orchestrator that can use an S3-compatible store for results and code. Version note: written 23 Oct 2026 from docs.rustfs.com. The S3 compatibility page was snapshotted by RustFS against a commit dated 9 Aug 2026, so treat counts as a moving target. ⚑ marks items from prior knowledge or inference rather than the pages I read (boto3 usage, exact Prefect wiring). Read §11 before adopting: this is a young project, so verify production readiness yourself.

---

0. The map: what RustFS is

Analogy: a warehouse of labelled bins with a front desk that speaks Amazon's language. Customers (applications) hand over boxes (objects) with a bin label (bucket) and a shelf label (key). The front desk (the S3 API) uses the same forms as Amazon's warehouse, so existing tools just work. Behind the desk, each box is split into pieces and spread over many shelves (disks) so a few broken shelves do not lose anything.

• What: a distributed object storage system in Rust, Apache 2.0 licensed, S3-compatible, aimed at private cloud, data lakes, AI/ML and cloud-native workloads. It runs on Linux, Windows, macOS, FreeBSD and containers across x86, ARM and RISC-V.
• Positioning from the project: the permissive licence is the headline contrast with AGPL-licensed alternatives; it claims migration and coexistence with S3-compatible platforms such as MinIO and Ceph. The README also claims 2.3x faster than MinIO for 4 KB payloads. That is a vendor benchmark; reproduce it on your hardware and object sizes before it informs a decision.
• Components: the rustfs server (S3 API on port 9000), the web Console (port 9001), the rc command-line client, optional protocol gateways (WebDAV, FTP/FTPS, SFTP, OpenStack Swift), background services (scanner for bit-rot, healing, lifecycle, replication), Kubernetes Helm chart and Operator, and an MCP server for AI assistants.
• Object-store vocabulary: an object is a blob plus metadata, addressed by bucket/key. Keys look like paths but the namespace is flat; 'folders' are just key prefixes.

Addressing. Path-style by default (http://host:9000/my-bucket/path/to/hello.txt). For virtual-host style (https://my-bucket.s3.example.com/hello.txt) set RUSTFS_SERVER_DOMAINS=s3.example.com and create wildcard DNS (*.s3.example.com) pointing at RustFS.

📖 Docs home · S3 protocol · Virtual-host access · GitHub

---

1. Installing it

Goal	Path
Try it locally	One-command install script, or a container
Single-server production	SNSD (one disk) or SNMD (several disks), on Linux
Multi-server production	MNMD after the pre-installation checklists
Kubernetes	Helm chart (standalone or distributed) or the RustFS Operator (tenants, pools, KMS)
Windows or macOS	Native install; the Windows build is for development and debugging only

1.1 Docker, the fastest start

docker volume create rustfs-data
docker run -d --name rustfs --restart unless-stopped \
  -p 9000:9000 -p 9001:9001 -v rustfs-data:/data \
  -e RUSTFS_ACCESS_KEY='<your-access-key>' -e RUSTFS_SECRET_KEY='<your-secret-key>' \
  -e RUSTFS_ADDRESS=':9000' -e RUSTFS_CONSOLE_ADDRESS=':9001' -e RUSTFS_CONSOLE_ENABLE=true \
  rustfs/rustfs:latest /data
curl --fail http://localhost:9000/health         # S3 API health; Console on :9001

• Change the default credentials immediately. Never use the well-known rustfsadmin value. On Linux set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY in /etc/default/rustfs and restart; with containers pass them at creation (recreate the container if it started without them; the volume survives).
• The container runs as non-root user rustfs, UID 10001. A bind-mounted host directory must be owned by 10001 (chown -R 10001:10001 /path) or you get permission denied. Named volumes need no setup.
• Flags win over env vars when both are given (--access-key, --address, --console-enable).
• Compose: the repository ships a docker-compose.yml with optional Grafana, Prometheus, OpenTelemetry collector and Jaeger under --profile observability; start only the server with docker compose up -d rustfs. An init container fixes volume ownership.
• Multi-node containers need --network host (the default bridge network does not support it) and name resolution (/etc/hosts on every node): RUSTFS_VOLUMES='http://node{1...4}:9000/data/rustfs{0...3}'.
• TLS: mount a cert directory and set RUSTFS_TLS_PATH=/certs. For HTTPS in front, the docs cover Nginx, Traefik, Caddy, Envoy, HAProxy and Apache with separate hostnames for S3 and Console.

📖 Installation · Docker · Linux · Kubernetes · Checklists · TLS · Reverse proxy

---

2. Deployment modes: which one?

Mode	Nodes	Disks	Fault tolerance	Use for
SNSD	1	1	None. Rely on backups	Dev, test, small non-critical
SNMD	1	Several	Up to M parity disks inside the node; the node itself is a single point of failure	Mid-sized non-critical on one server
MNMD	4 or more	Several per node	Disk- and node-level via erasure coding across servers	Production

Decision rule: if losing a server or its data would hurt, you need MNMD plus a backup or replication strategy (erasure coding protects against disk failures, not against deletion, corruption by software, or a lost site). SNSD is a development tool, not a storage strategy.

Disk preparation (Linux). Use dedicated data disks, format, mount and verify them before installing (the docs refer to XFS and a verification step); keep identical service environment configuration on every node. Work through the hardware, network, software and security checklists first.

📖 SNSD · SNMD · MNMD · Disk preparation · Hardware selection

---

3. Erasure coding and capacity

Analogy: a jigsaw with spare pieces. Each object is cut into N-M data pieces plus M extra 'parity' pieces, then spread across N disks. Any N-M of the N pieces rebuild the object, so up to M pieces can be lost.

• Reed-Solomon erasure coding. Each pool is divided into one or more erasure sets. For a set of N drives with M parity shards: data shards = N-M; tolerable unavailable shards = M; capacity efficiency about (N-M)/N. A 16-drive set with EC:4 stores 12 data and 4 parity shards, about 75% efficiency before file-system, metadata and operational overhead.
• Failure-domain caveat (from the docs): parity covers unavailable shards within one set. Node-level tolerance depends on how a set's drives are spread over nodes; do not assume EC:4 survives four whole nodes, because one node may hold several drives of the same set.

Drives per set (N)	Automatic STANDARD parity	Approx. efficiency
1	EC:0	100%
2-3	EC:1	50-67%
4-5	EC:2	50-60%
6-7	EC:3	50-57%
8-16	EC:4	50-75%

# /etc/default/rustfs
RUSTFS_STORAGE_CLASS_STANDARD=          # recommended: leave empty for automatic per-pool parity
# or pin explicitly (must be valid for the NARROWEST pool):
RUSTFS_STORAGE_CLASS_STANDARD=EC:4
RUSTFS_STORAGE_CLASS_RRS=EC:1           # REDUCED_REDUNDANCY writes; default 1 parity on multi-drive sets
RUSTFS_ERASURE_SET_DRIVE_COUNT=16       # pin set width only if you verified the whole layout

• Rules for explicit parity: M <= N/2 in every target set; STANDARD parity >= RRS parity when both non-zero; valid for every pool. An invalid value makes startup fail rather than silently weakening parity.
• Set widths range from 2 to 16 drives, selected automatically from the topology. Validate against the set width, not the cluster's total drive count.
• Capacity estimate: usable is roughly raw x (N-M)/N. Sixteen 10 TiB drives as one 16-drive set with EC:4 gives about 120 TiB, and you must still reserve space for metadata, versioning, incomplete multipart uploads, healing and headroom.
• Verify after start: Console Status page or rc admin info cluster shows layout, parity, drive availability and pool membership.

Trap. More parity buys durability at the price of usable capacity and write cost; do not pin high parity on small sets without checking the efficiency. Leaving the default is the documented recommendation unless you have a validated requirement.

📖 EC configuration · Status check · Storage env vars

---

4. Configuration surface

Setting	Purpose
RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY	Root credentials. Set unique values
RUSTFS_ADDRESS	S3 API bind address (default :9000)
RUSTFS_CONSOLE_ADDRESS, RUSTFS_CONSOLE_ENABLE	Console bind address and switch (port 9001)
RUSTFS_VOLUMES	The storage endpoints; supports {1...4} range expansion for multi-node
RUSTFS_SERVER_DOMAINS	Domain for virtual-host style access
RUSTFS_TLS_PATH	Directory with certificates
RUSTFS_STORAGE_CLASS_*, RUSTFS_ERASURE_SET_DRIVE_COUNT	Erasure coding (§3)
RUSTFS_OBS_*	Logging and observability (level, log directory)

The reference page also lists variables for KMS, the scanner, healing and event targets. rustfs has subcommands and server flags with environment equivalents (CLI reference). Keep the same environment file on every node.

📖 Environment variables · CLI reference · CORS

---

5. The S3 API and compatibility

Analogy: speaking a dialect. RustFS speaks Amazon S3 well enough for most everyday conversation, but not every idiom. The project says so itself: it is a tested subset, not a claim of full coverage.

# AWS CLI against RustFS: just add --endpoint-url
aws --endpoint-url "$RUSTFS_ENDPOINT" s3 cp hello.txt s3://my-bucket/path/to/hello.txt
aws --endpoint-url "$RUSTFS_ENDPOINT" s3 ls s3://my-bucket/
aws --endpoint-url "$RUSTFS_ENDPOINT" s3api head-object --bucket my-bucket --key path/to/hello.txt
aws --endpoint-url "$RUSTFS_ENDPOINT" s3 presign s3://my-bucket/path/to/hello.txt

# Python (boto3)  ⚑
import boto3
s3 = boto3.client('s3', endpoint_url='http://localhost:9000',
                  aws_access_key_id=KEY, aws_secret_access_key=SECRET, region_name='us-east-1')
s3.upload_file('hello.txt', 'my-bucket', 'path/to/hello.txt')

Area	Status in the compatibility gate
Bucket create/delete/list/inspect, tagging, policies, public access block	Tested
Object put/get/copy/head/delete, listing (prefix, delimiter, marker, max-keys; V1 and V2)	Tested
Range and conditional reads, user metadata, object tags, presigned GET/PUT	Tested
Multipart create/upload/complete/abort; selected copy and checksum cases	Tested
Selected versioning, Object Lock, CORS and lifecycle cases; SSE-C and selected SSE-KMS	Tested (only the cases in the implemented lists)
Bucket access logging, bucket ownership controls, POST Object form checksum handling	Planned (not implemented)
ACL authorisation	Excluded: intentionally unsupported
Multipart upload listing and part-lookup edge cases	Excluded from the default gate

• Gate size (Aug 2026 snapshot): 455 implemented cases, 5 lifecycle cases, 17 planned, 270 excluded, from Ceph s3-tests lists. Read the numbers as 'a lot is covered, but 270 are excluded', not as 'complete'.
• Encrypted objects are not portable. A passing encryption test means RustFS can read what RustFS encrypted; it does not mean it can read an encrypted object copied raw from another implementation.
• S3 Select currently accepts only uncompressed input.
• Other protocols: WebDAV, FTP/FTPS, SFTP and (optional build) OpenStack Swift gateways over the same buckets.

Practical advice: if you depend on ACLs, bucket access logging, or exotic multipart listing behaviour, test those exact calls against RustFS before committing. Use rclone or your SDK's integration test suite as a smoke test.

📖 S3 compatibility matrix · SDKs · Python · SFTP · WebDAV

---

6. Data management features

Feature	What it does	Use it for	Gotcha
Versioning	Keeps prior versions; deletes create markers	Recover from overwrites and accidental deletes	Storage grows; required by replication
Object Lock	Retention periods and legal holds on versioned objects	WORM and compliance retention	Needs versioning; test retention modes before relying on them
Lifecycle rules	Expire or transition objects by age	Cost control, log retention	Managed with the rc client; verify rules fire
Tiered storage	Lifecycle-driven transition to a remote storage tier	Move cold data to cheaper storage	Reads of tiered objects depend on the remote tier
Bucket quota	Hard storage quota per bucket	Multi-tenant limits	Hard limit: writes fail at the cap
Bucket replication	Asynchronous replication between versioned buckets	Copies in another cluster	Async, so a failover can lose the last writes; both ends need versioning
Site replication	Multi-site replication managed with rc	Disaster recovery across sites	Plan network and consistency expectations
Multipart upload	Parts for large objects; Console task monitor	Big files, resumable uploads	Incomplete uploads consume space; clean up
Scanner	Background object scan with a bit-rot cycle and metrics	Detect silent corruption	Watch its metrics to confirm scans still run
S3 Tables	Table buckets with a built-in Iceberg REST catalog	Lakehouse without a separate catalog	Newer feature; test with your engines (PyIceberg, Spark docs exist)
Event notifications	Filtered bucket events to external targets (webhook and others)	React to uploads, trigger pipelines	Targets and queue directories need persistent storage

📖 Versioning · Object Lock · Lifecycle · Tiering · Quota · Bucket replication · Site replication · Scanner · S3 Tables · Event notifications

---

7. Security

Analogy: badges, keys and a locked cabinet. IAM decides who may enter which room (badges). Encryption keeps the contents unreadable even if someone steals a disk (locked cabinet). TLS protects the hallway.

• IAM: users, groups and policies; policy attachment and evaluation semantics are documented. Service accounts and STS issue scoped, short-lived credentials for applications. Access keys are managed per identity. Prefer service accounts or STS for applications and keep the root key out of code.
• Bucket policies are applied through the S3 API; public access block exists. ACLs are intentionally unsupported.
• OIDC: Console single sign-on with providers such as Keycloak and GitLab.
• Encryption at rest: SSE-S3 (server-managed keys), SSE-C (customer-provided AES-256 keys) and SSE-KMS with a local or HashiCorp Vault backend; the Operator can configure KMS for tenants. Note: the project README marks its KMS as 'under testing' while the docs describe KMS configuration, so validate your key-management path and recovery procedure before putting regulated data on it.
• TLS with RUSTFS_TLS_PATH; mTLS and cert-manager are documented for Helm deployments.
• Audit targets deliver request-level audit records to external systems. Credential management has its own page; rotate the root credentials.

Traps. Default credentials left in place. Exposing the Console (9001) to the internet. Putting RustFS behind a reverse proxy that rewrites the Host header without enabling the matching virtual-host settings. Losing the KMS key means losing the data.

📖 Security and compliance · IAM · Service accounts and STS · Encryption · OIDC · Audit · Credentials

---

8. Operations

• The rc client: a RustFS command-line client to connect to, inspect and administer a cluster, including lifecycle rules, replication, heal tasks and rc admin info cluster.
• Health and status: /health on the S3 port; the Console Status page; the status-check workflow covers network access, health endpoints, cluster status and capacity.
• Observability: export metrics, traces and logs over OTLP; the docs list recommended alerts and metric names, and integrations with Prometheus, Grafana, Loki, Tempo and Thanos exist.
• Scaling: expand by adding a storage pool; rebalance to redistribute existing objects over active pools; decommission to drain and retire a pool and verify the data moved. Pools are the unit of growth, so plan pool sizes and symmetry up front.
• Healing and failures: replace a failed drive and let RustFS heal the missing shards onto it; for a lost node prepare a replacement, update its configuration, restore cluster membership and watch healing. Manual heal tasks can be bounded with rc.
• Upgrades: binary, container and Kubernetes procedures that preserve availability. Rehearse on a non-production cluster first.
• Kubernetes: the Helm chart (standalone or distributed, probes, hardening, pool expansion) or the Operator (multi-tenant isolation, pool expansion, KMS).

Runbook items to write before go-live: disk replacement, node replacement, credential rotation, certificate renewal, capacity alert thresholds, upgrade order, restore test from backup or replica.

📖 rc client · Observability · Scaling · Node healing · Drive failures · Node failures · Upgrade · Cluster lifecycle

---

9. Integrations relevant to a data platform

RustFS documents step-by-step integrations (mostly Docker Compose based) for tools that speak S3. Grouped by purpose:

Purpose	Tools with a RustFS guide
Orchestration and ingestion	Airflow (S3 provider), DolphinScheduler, SeaTunnel, Kafka Connect S3 sink, AutoMQ
Lakehouse and query	Apache Iceberg (Spark, REST catalog), PyIceberg via S3 Tables, Delta Lake (delta-rs), Hudi, Paimon, Hive, Spark, Flink, Trino, DuckDB (httpfs), ClickHouse (S3 disk), Databend, Doris, lakeFS
Vector and ML	LanceDB, Milvus, MLflow artifact store, Ray Data, vLLM model weights
Observability storage	Loki, Tempo, Thanos, Cortex, OpenObserve, GreptimeDB, Fluentd, OpenTelemetry, VictoriaMetrics backups
Backup and DR	Restic, Kopia, Velero, Longhorn, Vitess
DevOps and platform	Terraform state with locking, Elasticsearch and OpenSearch snapshots, Gitea LFS, Jenkins artifacts, Harbor and Docker Registry, Flux, Nextcloud
Filesystem views	s3fs-fuse, rclone, JuiceFS, ZeroFS (NFS), Alluxio cache, SFTPGo, OpenDAL

9.1 Using it with Prefect ⚑

Prefect can keep results and deployment code in S3-compatible storage through the prefect-aws integration. Because RustFS speaks S3, the wiring is the integration's S3 bucket block with a custom endpoint URL pointing at RustFS and a service-account key. I wrote this from how prefect-aws normally works, not from a tested RustFS example; check the prefect-aws docs for the current block and client-parameter names and run a write/read round trip before trusting it.

• Results: point result storage at a bucket such as prefect-results; every worker must reach the endpoint and credentials.
• Code: a pull step can fetch flow code from a bucket at run time.
• Hygiene: a dedicated service account with a bucket-scoped policy, a lifecycle rule to expire old results, TLS on the endpoint.

📖 Integrations index · Airflow · Iceberg · DuckDB · Spark · lakeFS · Terraform state · rclone · prefect-aws

---

10. Moving data in and out

• Migration and coexistence: the project says it supports running alongside other S3 platforms such as MinIO and Ceph. The practical tools are rclone sync, bucket replication (versioned buckets) and application-level dual writes.
• Validate a migration: compare object counts, sizes and checksums; test presigned URLs, multipart behaviour, listing order and any ACL or logging features you used, because those are the documented gaps.
• Do not copy encrypted objects raw between implementations (not portable); decrypt and re-encrypt through the S3 API.
• Exit plan: because the API is S3, leaving RustFS later is an rclone job, which is a point in its favour. Keep that tested.

---

11. Should you use RustFS? (the part to challenge)

Situation	Verdict
Local dev and CI needing an S3 endpoint	Good fit. One container, quick to start
Self-hosted S3 for logs, artefacts, backups, ML datasets where a permissive licence matters	Reasonable, after your own load and failure tests
Data lake or lakehouse storage with Iceberg or Delta	Plausible; documented integrations exist, but pilot with your engines and table sizes
Regulated or business-critical primary storage	Proceed carefully. Confirm KMS maturity, Object Lock behaviour, support terms and your recovery drills
Need full AWS S3 feature parity (ACLs, access logging, every multipart listing edge)	Not a fit where those are documented as excluded or planned
Cloud-native workload already on a managed S3	Staying on the managed service removes operations; self-host only for cost, sovereignty or latency reasons

Questions to answer before adopting: (1) Is the benefit the licence, the cost, performance, or Rust itself, and have you measured it? (2) Who is on call when a node dies at 3 a.m.? (3) Have you rehearsed a disk and node replacement and a restore? (4) What is the backup strategy beyond erasure coding? (5) How mature is the release you pin, and what is the upgrade policy? Pin a version rather than using :latest in production.

---

12. Trap checklist and exercises

1. Default credentials; Console exposed publicly.
2. Bind-mount directory not owned by UID 10001.
3. Believing EC:4 equals four-node tolerance.
4. Pinning parity that the narrowest pool cannot satisfy (startup fails), or setting a set width that does not divide the topology.
5. SNSD or SNMD treated as production, with no backup.
6. Replication configured without versioning; async replication assumed to be lossless.
7. Relying on ACLs, access logging or encrypted-object portability.
8. Using :latest; no upgrade rehearsal; no restore test.
9. Windows build used in production.

Exercises. (1) Start the Docker command with unique credentials, create a bucket and upload a file with the AWS CLI using --endpoint-url. (2) Enable versioning, overwrite the file, and recover the first version. (3) Compute usable capacity for 8 nodes x 4 x 8 TiB drives if sets are 16 wide with EC:4, and say what a node loss does to a set. (4) Add a lifecycle rule to expire objects under tmp/ after one day and verify. (5) Run a write/read round trip from a Python script with boto3 and a presigned URL.

Solution notes

1. Use aws --endpoint-url http://localhost:9000 s3 mb s3://demo, then cp; configure the CLI with the access key pair you set.
2. Versioning is a per-bucket setting; list versions to see both, then fetch by version ID or remove the delete marker.
3. Raw is 8 x 4 x 8 = 256 TiB; with a 16-wide set and EC:4 efficiency is 75%, so about 192 TiB before overhead. Whether a node loss is survivable depends on how many of a set's 16 drives sit on that node: if a node holds 4 drives of each set, losing it removes 4 shards, which EC:4 can reconstruct, with no margin left.
4. Lifecycle rules are managed with rc; confirm by checking the object disappears after the scanner or lifecycle cycle runs.
5. boto3 needs endpoint_url; generate_presigned_url('get_object', ...) gives a URL you can fetch with curl.

---

Sources: docs.rustfs.com: docs home, installation, Docker, EC configuration, S3 compatibility matrix, S3 protocol, docs index; GitHub README (claims). Sections 6-8 summarise the documentation index descriptions rather than every page body; items marked ⚑ are unverified.

Miscs

Diagram Types — When to Use Each

Structure & relationships

• Architecture — Shows the components of a system (services, databases, queues, external APIs) and the connections between them. Use it when someone needs to understand how the pieces of a system fit together at a glance — onboarding a new engineer, documenting a service before a migration, or explaining a design in a review. Skip it if you only have 2-3 boxes with one connection; that's a sentence, not a diagram.
• ER / data model — Entities (tables/objects) with their fields and the relationships (one-to-many, many-to-many) between them. Use it when designing or documenting a database schema, or when you need to reason about how data is normalized and connected before writing migrations. Most useful early in a project, before the schema calcifies.
• Layer stack — Abstraction layers drawn as horizontal bands, stacked so lower layers support higher ones (e.g. hardware → OS → runtime → application, or OSI network layers). Use it to explain dependency direction between layers of abstraction — what sits on top of what, and what breaks if a lower layer changes.
• Nested — Hierarchy expressed by literally drawing boxes inside boxes (an org inside a division inside a company, or a VPC inside a region inside a cloud account). Use it when containment itself is the point — when something is physically or logically inside something else, not just related to it.
• Venn — Two or more overlapping circles showing what's shared vs. unique between sets. Use it for comparing scope or membership — feature overlap between two products, skill overlap between two roles, shared vs. distinct requirements between teams. Gets unreadable past 3 circles; don't force a 4-way Venn.
• Dependency graph — Nodes and directed edges showing what depends on what, including fan-in (many things depending on one), fan-out (one thing many others depend on), and cycles. Use it to find risky central dependencies, plan safe refactor order, or explain why one library change broke six unrelated services.
• UML class — Classes with their attributes, methods, and typed relationships (inheritance, composition, association) between them. Use it for object-oriented design docs, before writing code for a non-trivial class hierarchy, or when reviewing someone else's OOP design and you need to see the shape of it.
• Database schema — Physical tables with actual column names, types, and foreign key lines between them — more concrete and implementation-level than an ER diagram. Use it once the schema is built or close to final, as reference documentation for whoever queries or migrates the database next.

Flow & process

• Flowchart — Decision logic expressed as boxes and diamonds, with branches for different outcomes. Use it for any process with conditional branches — an approval workflow, a troubleshooting procedure, business logic with multiple if/else paths. If there's no branching, it's just a list — use a numbered list instead.
• Sequence — Actors (services, users, systems) as vertical lifelines, with horizontal arrows showing messages passed between them over time, top to bottom. Use it for anything where order and timing between parties matters — an API request/response cycle, an OAuth token refresh flow, a distributed transaction across microservices.
• State machine — States as nodes, transitions as labeled edges between them, showing what triggers a move from one state to another. Use it for anything with a finite, well-defined lifecycle — an order's status (pending → paid → shipped → delivered), a connection's states, a UI component's modes. If a system can be in infinitely many configurations, this isn't the right tool.
• Swimlane — A flowchart split into horizontal or vertical lanes, one per team/role/system, showing who does what and where work crosses a boundary. Use it when a process spans multiple owners and the handoffs between them are the interesting part — e.g. an incident response process moving between on-call, engineering, and comms.
• Process — A multi-actor sequential workflow, similar to swimlane but framed around ordered steps rather than lane ownership. Use it for step-by-step operational workflows involving more than one participant, where you want the sequence to read left-to-right or top-to-bottom rather than lane-by-lane.
• Data flow — Pipeline steps scoped to specific roles or systems, showing how data moves and transforms from source to destination. Use it to document an ETL/ELT pipeline, a data ingestion path, or any process where the main story is data being transformed as it passes through stages.
• Sankey — Flow width is proportional to quantity, and flows can split and merge — the classic "where did the volume go" diagram. Use it when quantity itself carries meaning: budget allocation across categories, user funnel volume splitting into different paths, energy or material flow through a system. Wrong tool if the flows aren't quantitative.
• Fishbone — A central "spine" (the effect/problem) with angled "bones" grouping contributing causes into categories. Use it for root-cause analysis — postmortems, quality issues, "why did this fail" investigations where causes cluster into categories (people, process, tools, environment, etc.).
• Deployment — Physical or logical deployment topology: zones, hosts, and the artifacts running on each. Use it to document where things actually run — which services live in which network zone, on which hosts or clusters — useful for infra reviews, security audits, and incident diagnosis.

Hierarchy & ranking

• Tree — Parent-to-children branching, no cross-links. Use it for anything strictly hierarchical with a single path down from any node to the root — file system structure, a decision tree, a taxonomy, a component breakdown.
• Org chart — People or teams with reporting lines and, optionally, routing/ownership indicators. Use it for team structure, escalation paths, or "who owns what" documentation — anywhere the hierarchy is about people/teams rather than abstract concepts.
• Pyramid / funnel — A ranked hierarchy (base = most foundational/most volume) or a funnel narrowing toward a smaller output (e.g. conversion funnel). Use a pyramid for priority or maturity models (Maslow-style); use the funnel framing specifically when you're showing drop-off between stages, like a sales or signup funnel.
• Story map — A horizontal "backbone" of user activities, with vertical slices showing what's included in each release. Use it for product/agile planning — mapping a user journey against release scope, so everyone can see what's in v1 vs. later.
• Kanban — Columns representing workflow states (to do / in progress / done, or more granular), with cards moving across them. Use it to show current work-in-progress distribution across a pipeline, or as a snapshot of a team's board at a point in time for a status update.

Comparison & positioning

• Quadrant — Two axes crossing to form four quadrants, items placed by where they score on each axis. Use it for prioritization (impact vs. effort), positioning (build vs. buy, cost vs. value), or any comparison that reduces cleanly to two independent dimensions.
• Radar / spider — Multiple axes radiating from a center, one shape per item being compared, showing relative strength across many dimensions at once. Use it when you're comparing 2-4 options across 4+ criteria simultaneously — vendor comparisons, skill assessments, product feature comparisons. Gets messy with more than a few overlapping shapes.
• Wardley map — Value chain position (how visible/close to the user) on one axis, evolution stage (genesis → custom-built → product → commodity) on the other. Use it for strategic technology or business planning — deciding what to build vs. buy vs. outsource based on how commoditized a capability is. Niche but powerful in strategy contexts; don't reach for it outside that use case.
• Polar chart — Similar to radar but encodes a cyclic magnitude with a linear radius — think of periodic/cyclical data (time of day, seasonal patterns) mapped around a circle. Use it specifically for cyclical quantitative data where the "wraparound" nature of the axis matters, unlike a radar chart's fixed set of categorical axes.

Time-based

• Timeline — Discrete events plotted along a single time axis. Use it for historical narratives, project milestones, or any story that's fundamentally "this happened, then this happened."
• Gantt — Tasks and phases as horizontal bars along a timeline, showing duration and overlap. Use it for project planning and scheduling — when you need to show not just that things happened in sequence but how long each took and where they overlapped.
• User journey — Stages of an experience, the user's actions at each stage, and their sentiment (frustration, delight) plotted alongside. Use it for UX research and product design — surfacing where in a flow users struggle, to prioritize what to fix.
• Loop / flywheel — A circular flow representing a reinforcing cycle, often with a shared hub in the middle that all the loop's stages feed back into. Use it for business models or systems that compound over time — growth loops, feedback loops, virtuous cycles — where the point is that the cycle strengthens itself with each pass.

Charts (quantitative)

• Bar chart — Categorical comparison, bar length/height proportional to value. Use it for comparing discrete categories against each other — sales by region, votes by candidate. The default choice for "compare these numbers."
• Line chart — Trend over a continuous axis, usually time. Use it whenever the trajectory matters more than any single value — revenue over quarters, a metric drifting up or down.
• Scatter plot — Individual data points plotted on two numeric axes, revealing distribution and correlation. Use it when you have many data points and want to see whether/how two variables relate, or to spot outliers and clusters.
• Treemap — Rectangles sized by value, nested to show part-of-whole relationships (e.g. storage used by folder, revenue by product line). Use it for hierarchical proportions where area itself needs to communicate scale — this only works if the area-to-value mapping is accurate, so it's less forgiving than a bar chart of small errors.

Data-platform specific

• Medallion — Multi-tier data storage architecture, typically bronze (raw) → silver (cleaned) → gold (aggregated/business-ready). Use it specifically when documenting a data lakehouse or similar layered data architecture — this is a domain-specific variant of the layer stack, tailored to data engineering vocabulary.
• DP integration — Sources feeding into a core platform, which feeds out to consumers. Use it to document a data platform's ingestion and distribution boundaries — what comes in, what the platform does, who/what consumes the output.
• DP security matrix — A grid of roles against resources/permissions, showing who can access what. Use it for access-control documentation and audits — a compliance or security review needs exactly this shape: role rows, resource columns, permission cells.

When not to diagram at all

• Quick throwaway visuals for a chat message or terminal output — plain text is faster and just as clear.
• A list of anything — a table or bullet list beats a diagram every time.
• Before/after comparisons — a table communicates this more precisely than boxes and arrows.
• A single labeled box — if there's only one shape, you don't have a diagram, you have a sentence. Write the sentence.

The underlying test: would a reader learn more from this than from a well-written paragraph? If not, don't draw it.

Decision-Making & Thinking Methods — When to Use Each

Elimination-based methods

• Process of elimination — Systematically ruling out options that fail a hard requirement until only viable ones remain. Use it when you have many options and at least one clear disqualifying criterion (budget cap, hard deadline, legal requirement) — it's fast because you never have to score the options you kill off. Bad fit when no option can be truly disqualified and everything is a matter of degree; then you need scoring, not elimination.
• MECE exclusion (Mutually Exclusive, Collectively Exhaustive) — Breaking a decision space into buckets that don't overlap and don't leave gaps, then eliminating whole buckets at once instead of individual items. Use it when the option space is large and unstructured (e.g. "how do we reduce costs") — forcing MECE buckets first prevents you from redundantly comparing options that are really the same idea in different words.
• Veto voting — Each participant can strike out options they find unacceptable, and whatever survives every veto wins. Use it in group decisions where avoiding a bad outcome matters more than optimizing for a great one — picking a restaurant for a group with dietary restrictions, or approving a shortlist where any single stakeholder can kill a candidate.

Branching / tree-based methods

• Decision trees — Mapping out sequential decision points as branches, with the outcomes and probabilities at each leaf. Use it when a decision unfolds in stages and later choices depend on earlier ones — should we invest now or wait for more data, given different ways the market could move. Overkill for one-shot decisions with no sequential dependency.
• Backward induction — Starting from the desired end state or last decision point and reasoning backward to figure out what has to be true at each earlier stage. Use it for planning problems with a known target and multiple paths to it — game theory, negotiation prep, or working out the last responsible moment to make a call. Only works cleanly when the end state is well-defined; useless for open-ended exploration.
• Minimax / worst-case branching — At each decision branch, assume the opponent or environment will pick the outcome worst for you, and choose the branch that minimizes that worst case. Use it in adversarial or highly uncertain situations where you can't assign real probabilities — competitive strategy, security threat modeling, negotiating with an unpredictable counterparty. Too conservative for situations where you actually have decent probability estimates; expected-value methods do better there.

Comparative / scoring methods

• Weighted decision matrix — List options as rows, criteria as columns, assign a weight to each criterion, score each option per criterion, and sum. Use it when you have 3+ options, multiple criteria that matter to different degrees, and you want a defensible, auditable comparison — vendor selection, hiring between finalists, picking between architecture options. Don't bother for a 2-option call with one dominant criterion; that's just a comparison sentence.
• Pairwise comparison — Comparing every option against every other option one-on-one, then tallying wins, instead of scoring all of them on an abstract scale at once. Use it when abstract scoring feels arbitrary or when direct comparison is easier for people than absolute rating — ranking a shortlist of candidates, or prioritizing a backlog where "how much better is A than B" is easier to answer than "rate A from 1-10."
• Pros/cons list — The simplest form: two columns, upsides and downsides, no weighting. Use it for quick, low-stakes personal decisions where you mainly need to externalize your thinking, not calculate an optimum. Weak for anything with more than a couple of criteria of very different importance — a long pros list can outweigh one severe con it shouldn't.
• Cost-benefit analysis — Converting the expected costs and benefits of each option into comparable units (usually money or time) and comparing net values. Use it for decisions with quantifiable costs and benefits — capital investment, whether to build vs. buy, whether a project is worth the resourcing. Breaks down when the biggest factors are genuinely unquantifiable (reputation, morale); forcing a number onto those can mislead more than it helps.

Root-cause & diagnostic methods

• Five Whys — Asking "why" repeatedly (usually 5 times) on an observed problem until you reach the root cause rather than a symptom. Use it right after an incident or failure, when the visible problem is probably a downstream effect of something deeper — a production outage, a recurring team conflict, a process that keeps breaking. Weak when a single problem has multiple independent root causes; the format's single-chain structure will make you stop at the first plausible-sounding answer instead of finding all of them.
• Fishbone / Ishikawa diagram — Mapping many possible contributing causes, grouped into categories (people, process, tools, environment), converging on one effect. Use it when a problem plausibly has several independent contributing factors and you want to explore breadth before committing to one root cause — quality defects, complex incident postmortems. More effort than Five Whys, but better suited to multi-causal problems.
• First principles thinking — Stripping a problem down to its fundamental truths and reasoning up from there, instead of reasoning by analogy to how it's normally done. Use it when the standard approach to a problem is expensive, slow, or just assumed rather than justified — redesigning a process everyone "always" does a certain way, or challenging an industry convention. Costly to do for routine decisions; reserve it for decisions where the payoff of a genuinely novel answer is high.

Framing & structuring methods

• SWOT analysis (Strengths, Weaknesses, Opportunities, Threats) — A four-quadrant framing that separates internal factors (strengths/weaknesses) from external ones (opportunities/threats). Use it early in strategic planning, when you need a shared, structured starting point for a discussion rather than an actual decision — it's a framing tool, not a decision procedure, so don't expect it to output an answer by itself.
• Six Thinking Hats — Deliberately switching between six modes of thinking (facts, feelings, caution, optimism, creativity, process) in turn, usually in a group. Use it when a group discussion keeps getting stuck because people default to one mode (usually caution or optimism) and talk past each other — it forces everyone to visit the same mode together instead of arguing across modes.
• Cynefin framework — Classifying a situation as simple, complicated, complex, or chaotic, and matching your decision approach to that classification (best practice, good practice, emergent practice, or rapid stabilizing action respectively). Use it as a first step before choosing any other method on this list — it tells you whether a decision even has a knowable right answer (simple/complicated) or whether you should be probing and adapting instead (complex/chaotic).

Iterative / adaptive methods

• OODA loop (Observe, Orient, Decide, Act) — A fast repeating cycle for decisions under time pressure and incomplete information, where you act, then immediately re-observe and adjust. Use it in fast-moving, high-uncertainty situations where waiting for full information means missing the window to act — incident response, competitive/adversarial situations, live operational decisions. Wrong tool for decisions where you have time to gather full information up front; the speed advantage isn't worth the lost rigor.
• PDCA / iterative improvement (Plan, Do, Check, Act) — Running small experiments in a loop, checking results, and adjusting before scaling up. Use it for process improvement where you can test cheaply before committing — a new workflow, a process change, an experimental feature. Not suited to one-shot, high-stakes, irreversible decisions where you don't get a second iteration.
• Satisficing — Choosing the first option that meets a "good enough" threshold, rather than searching for the optimum. Use it when the cost of continued search or analysis exceeds the expected gain from finding a marginally better option — routine purchasing decisions, hiring under time pressure, anything where "good enough, now" beats "perfect, eventually."

Group / consensus methods

• Delphi method — Anonymously collecting expert opinions in rounds, sharing an aggregated summary between rounds, and repeating until opinions converge. Use it for forecasting or expert judgment on questions with no hard data, where you want to avoid groupthink and status-driven bias — long-range technology forecasts, expert risk assessments. Slow; not for time-pressured decisions.
• RAPID / RACI — Frameworks that assign explicit decision roles (who Recommends, who Agrees, who Performs, who has Input, who Decides / who's Responsible, Accountable, Consulted, Informed). Use it when a decision keeps stalling because it's unclear who actually has the authority to decide, not because the content itself is unclear — organizational decisions with many stakeholders. Doesn't help at all with the substance of the decision, only the process of making it.
• Dot voting / brainstorming convergence — After open brainstorming generates many options, each participant places a limited number of votes (dots) on their preferred ones, and the highest-vote options move forward. Use it to converge a large, unstructured set of ideas from a group down to a shortlist quickly, without a full scoring exercise — workshop facilitation, feature prioritization sessions.

Heuristics / shortcuts

• Pareto principle (80/20) — Recognizing that a small fraction of causes/inputs typically drives most of the effect/output, and focusing effort there instead of treating all factors equally. Use it for prioritization when you have many candidate causes or actions and limited time — which bugs to fix first, which customers to focus retention efforts on.
• Occam's razor — Among competing explanations that fit the evidence equally well, preferring the simplest one. Use it during diagnosis or root-cause analysis when multiple explanations are consistent with what you've observed — don't reach for an exotic explanation when a mundane one accounts for the same facts.
• Premortem — Before committing to a plan, imagining it has already failed and working backward to figure out why. Use it right before locking in a significant decision, as a check against overconfidence and groupthink — it surfaces risks that "will this work" framing tends to suppress because everyone's already invested in saying yes.
• Regret minimization — Choosing the option that minimizes your future regret if it turns out badly, rather than the option with the best expected value. Use it for personal, high-stakes, largely irreversible decisions where the emotional cost of "what if" outweighs a purely numerical optimum — career changes, major life decisions.

LLMs

LangChain — Components Deep Dive

A study guide to building agents with LangChain 1.x (Python), one component at a time: what it is, the code you will actually type, when to use it (and when not to), the traps, and where the official docs live.

How to read this guide. Every component has (1) a mental model, usually an everyday analogy, (2) the code you will write, (3) a when to use / when not to verdict, (4) the traps, and (5) 📖 links to the official docs. The decision tables for “when do I use tools?” (§3) and “when do I use memory?” (§6) are the parts to re-read before a design review. The companion page is LangGraph — Components Deep Dive. Version note: written 17 Oct 2026 from docs.langchain.com. Features marked ⚑ need a minimum version (table in §14), and model names move fast, so run pip show langchain langgraph and pick current model IDs from your provider before copying anything.

---

0. The map: what LangChain is in 2026

Analogy: a car. LangChain is the cockpit: steering wheel, pedals and dashboard that feel the same in every car (any model provider). LangGraph is the engine and chassis: it keeps the car running, lets you stop and restart a long trip, and gives you low-level control. Deep Agents is a fully equipped rental: GPS, spare tyre and a trunk (planning, subagents, filesystem) already installed.

	Runtime	Framework	Harness
Example	LangGraph (also Temporal, Inngest)	LangChain (also Vercel AI SDK, CrewAI, OpenAI Agents SDK, Google ADK, LlamaIndex)	Deep Agents SDK (also Claude Agent SDK)
Value add	Durable execution, streaming, human-in-the-loop, persistence	Abstractions and integrations	Predefined tools, prompts, subagents, planning, filesystem
Use when	Low-level control, long-running stateful workflows, mixing deterministic and agentic steps	Getting started quickly, standardising how a team builds agents	Autonomous agents on complex, non-deterministic, multi-step tasks

Packages. langchain (agents, messages, tools, embeddings), langchain-core (base abstractions), one package per provider (install by extra: pip install 'langchain[anthropic]'), langgraph (runtime, checkpointers, stores), and langchain-classic (legacy chains, retrievers, hub and community re-exports, moved out of the main namespace in v1).

What changed in v1 (the things old tutorials get wrong):

• create_agent is the way to build an agent. The old prompt parameter is now system_prompt; dynamic prompts go through middleware.
• Middleware is the defining feature: hooks around the model call and tool call replace most custom graph surgery.
• Messages expose standard content blocks (content_blocks) that look the same across providers.
• The langchain namespace is deliberately small. If from langchain.chains import ... or from langchain import hub fails, you want pip install langchain-classic.
• Agent state schemas must be TypedDict (no Pydantic state in create_agent).

Choosing the layer (quick verdict): single model call such as classify, extract or summarise → a bare model (§1). Model plus tools in a loop → create_agent. Need branching, parallel fan-out, custom loops or durable multi-step flows → LangGraph (companion page), and you can still embed a create_agent agent as a node. Long autonomous work with files and subagents → Deep Agents.

📖 Runtimes, frameworks, harnesses · What's new in v1 · Migration guide · Philosophy

---

1. Models

Analogy: an engine you can swap. The model is the reasoning engine. LangChain gives every engine the same plug (invoke, stream, batch, bind_tools, structured output), so you can change provider without rewriting the car.

1.1 Initialise and call

from langchain.chat_models import init_chat_model

model = init_chat_model('anthropic:claude-sonnet-5', temperature=0.2, timeout=30, max_tokens=1000, max_retries=6)

model.invoke('Why do parrots talk?')             # one AIMessage
for chunk in model.stream('Tell me a story'):    # AIMessageChunk objects, sum them to get the full message
    print(chunk.text, end='', flush=True)
model.batch(['q1', 'q2', 'q3'], config={'max_concurrency': 5})   # client-side parallelism

• Two ways in: init_chat_model('provider:model'), or the provider class (ChatAnthropic, ChatOpenAI...). Model names are passed straight to the provider API, so a new model works without a LangChain update.
• Resilience is built in: max_retries defaults to 6 with exponential backoff and jitter for network errors, 429 and 5xx. 401 and 404 are not retried. For long agent runs on shaky networks raise it to 10-15 and add a checkpointer.
• batch() is not the provider's batch API. It parallelises calls client-side; provider batch endpoints are a separate product.
• Configurable models: init_chat_model(temperature=0) with no model name lets you pick the model per call via config={'configurable': {'model': '...'}}. Use config_prefix when a chain has several models.

1.2 Tool calling and structured output on a bare model

model_with_tools = model.bind_tools([get_weather])               # tool_choice='any' forces a call
ai = model_with_tools.invoke('Weather in Boston and Tokyo?')    # ai.tool_calls may hold several parallel calls

from pydantic import BaseModel, Field
class Movie(BaseModel):
    title: str = Field(description='Title')
    year: int
model.with_structured_output(Movie).invoke('Details on Inception')   # Movie(...)

With a bare model you run the tool and send a ToolMessage back. An agent (§4) does that loop for you. with_structured_output accepts Pydantic, TypedDict or JSON Schema; method is json_schema (provider-native), function_calling (forces a tool call) or json_mode (valid JSON, schema only in the prompt). include_raw=True returns the raw message too, which is how you get token counts.

1.3 Other model features worth knowing

Feature	What to know
Reasoning	Read content_blocks of type reasoning. ⚑ A standard reasoning_effort parameter exists for several providers; check model.profile['reasoning_effort_levels'].
Model profile ⚑	model.profile lists capabilities (max_input_tokens, tool_calling, structured_output, modalities). Summarisation middleware and structured-output auto-selection read it. It is beta and can be overridden with profile=.
Multimodal	Pass {'type': 'image', 'url': ...} or base64 plus mime_type as content blocks; not every model supports every type.
Prompt caching	Three levels: implicit (OpenAI, Gemini), explicit provider controls (cache_control blocks for Anthropic, cachePoint for Bedrock), and middleware that caches stable system prompt and tool content. Usually only engages above a minimum token count.
Server-side tools	Web search and code interpreters run at the provider. The response carries server_tool_call and server_tool_result blocks in one turn, with no ToolMessage for you to return.
Rate limiting	rate_limiter=InMemoryRateLimiter(requests_per_second=...) limits request count only, not request size.
Exceptions	Standard types (ModelRateLimitError, ModelTimeoutError, ContextOverflowError...) each carry is_retryable, which retry middleware respects.
Token usage	msg.usage_metadata, or aggregate across models with get_usage_metadata_callback().

When to use a bare model: one-shot generation, classification, extraction, routing decisions inside your own code, or evaluator calls. When not to: as soon as the model must choose tools and loop, use an agent.

Traps. (1) model.invoke returning a string means you built a legacy text-completion LLM, not a chat model. (2) A pre-bound model (bind_tools already called) cannot be used where create_agent needs structured output. (3) parallel_tool_calls=False exists on some providers if your tools are not safe to run concurrently.

📖 Models · Providers and models · Chat model integrations

---

2. Messages

Analogy: a transcript with stage directions. Each line has a speaker (role), the words (content) and margin notes (metadata such as IDs and token counts). The whole conversation is just a list of these lines.

Type	Role	Notes
SystemMessage	Instructions that prime behaviour	Always a SystemMessage inside ModelRequest.system_message, even when you passed a string.
HumanMessage	User input, text or multimodal	A plain string is shorthand for one HumanMessage.
AIMessage	Model output	.text, .content, .content_blocks, .tool_calls, .usage_metadata, .response_metadata, .id.
ToolMessage	Result of one tool call	tool_call_id must match the AIMessage's call ID. artifact carries data for your app that is not sent to the model.

Standard content blocks (msg.content_blocks): text (with annotations/citations), reasoning, image, audio, video, file, text-plain, tool_call, tool_call_chunk, invalid_tool_call, server_tool_call, server_tool_result and the escape hatch non_standard. content stays loosely typed (string, provider-native blocks or standard blocks); content_blocks lazily parses it into the standard form. Set output_version='v1' (or LC_OUTPUT_VERSION=v1) if another system must read the standard blocks from stored content.

Rules that bite. Every tool call in history needs a matching ToolMessage, and most providers want history to start with a user message. Streaming chunks (AIMessageChunk) add up with +. Serialisation uses dumpd and load, but never call load() on untrusted data: it instantiates Python objects and can trigger side effects.

📖 Messages

---

3. Tools

Analogy: hands for a brain. The model can only think and talk. A tool lets it reach out: read a database, call an API, write a file. The model decides whether and with what arguments to reach; your code does the reaching.

3.1 Define a tool

from langchain.tools import tool

@tool
def search_orders(user_id: str, status: str, limit: int = 10) -> str:
    '''Search a user's orders by status. Use when the user asks about order history.

    Args:
        user_id: Unique user ID
        status: pending, shipped or delivered
        limit: Max results
    '''
    ...

• Type hints are required (they become the input schema). The docstring is the description the model reads, so write it as a prompt: when to use it, what the arguments mean.
• Prefer snake_case names; spaces and special characters get rejected by some providers. Override with @tool('web_search') or description=....
• Richer inputs: @tool(args_schema=PydanticModel) or a JSON schema.
• Reserved argument names: config and runtime. Do not use them for your own parameters.

3.2 ToolRuntime: state, context, store, streaming

Add runtime: ToolRuntime to the signature. It is injected at call time and hidden from the model's schema.

Field	What it is	Use for
runtime.state	Short-term memory for this thread (messages, custom fields)	Read history, counters, auth flags
runtime.context	Immutable per-run config passed at invoke time	User ID, tenant, API keys, permissions
runtime.store	Long-term memory, persists across threads	Preferences, learned facts
runtime.stream_writer	Emit custom progress events	'Fetched 10/100 records'
runtime.tool_call_id	ID of this call	Building a matching ToolMessage
runtime.execution_info ⚑	thread_id, run_id, attempt number	Logging, retry-aware behaviour
runtime.server_info ⚑	Assistant/graph ID, authenticated user on LangGraph Server (else None)	Per-user scoping on the platform

from langchain.tools import tool, ToolRuntime
from langchain.messages import ToolMessage
from langgraph.types import Command

@tool
def set_user_name(new_name: str, runtime: ToolRuntime) -> Command:
    '''Save the user's name.'''
    return Command(update={
        'user_name': new_name,
        'messages': [ToolMessage(content='Name saved.', tool_call_id=runtime.tool_call_id)],
    })

3.3 What a tool may return

Return	Effect	Pick it when
str	Becomes a ToolMessage the model reads	Result is naturally text
dict/object	Serialised and returned; the model reads fields	Downstream reasoning needs explicit fields
Content-block list	Multimodal ToolMessage (text plus image...)	Screenshots, charts; model must support the modality
Command(update=..., goto=...)	Writes to state and can route	The tool must change state. Include a ToolMessage with the matching tool_call_id or ToolNode raises ValueError.
@tool(return_direct=True)	Skips the final model call and returns the tool output	Output is already the complete answer

Parallel gotchas. Models may call several tools in one step. If two tools write the same state key, define a reducer for it. return_direct only short-circuits the loop when every tool call in that step has it set; a mixed batch goes back to the model.

3.4 Errors, dynamic tools, headless tools, MCP

• Errors: use ToolErrorMiddleware ⚑ or a @wrap_tool_call that catches the exception and returns a ToolMessage so the model can self-correct. Return the exception type rather than the raw message when details could be sensitive. Put retry inner (later in the list) so exceptions reach the error handler after retries are exhausted.
• Dynamic tool selection: if all tools are known up front, register them all and filter per call with @wrap_model_call and request.override(tools=...) (by auth state, role, feature flag). If tools are discovered at runtime (MCP, registry), you need two hooks: wrap_model_call to add them and wrap_tool_call to execute them, otherwise the agent cannot run the tool it just advertised.
• Headless tools: schema-only tools whose implementation lives in the client (browser geolocation, clipboard). The run interrupts with a tool payload and resumes when the client returns a result.
• MCP ⚑ (beta): from langchain.mcp import MCPAdapter; async with MCPAdapter(url) as a: tools = await a.list_tools(), then pass to create_agent. Requires langchain[mcp]>=1.4.0; the standalone langchain-mcp-adapters package has a migration guide.

3.5 WHEN to use tools (decision table)

Situation	Use	Why
Needs fresh or private data (DB, API, files, web)	Tool	The model cannot know it
Needs exact arithmetic, code execution, or a guaranteed-correct lookup	Tool	Models are unreliable calculators
Has a side effect (send, write, delete, pay)	Tool plus human approval (§9)	Gate the dangerous ones
Step is always executed, in a fixed order, with no judgement needed	Plain code or a graph node, not a tool	Do not pay tokens and latency to have a model 'decide' the obvious
You only need the final answer in a fixed shape	Structured output (§8)	Cheaper and safer than a tool that just returns data
Static knowledge that fits in the prompt	Prompt or skill	No call needed
Big knowledge base	Retrieval tool (§12)	Fetch only relevant chunks
More than roughly 10 tools and the model picks wrong	Filter dynamically, LLMToolSelector, skills or subagents	Too many tools overload context and raise errors
Tool output is huge	Context editing or filesystem middleware	Stops results filling the window

Traps. Vague docstrings cause wrong calls. A tool that returns a 50k-token blob poisons every later turn. A tool that mutates state without returning a ToolMessage breaks message history. Tools run again if a node is retried or resumed, so make side effects idempotent.

📖 Tools · MCP · Headless tools · Prebuilt tools

---

4. Agents (create_agent)

Analogy: a new hire with a toolbox and a handbook. Agent = model + harness. The model does the thinking; the harness (prompt, tools, middleware) decides what it sees and may do. The loop is: think, pick a tool, see the result, think again, until it answers.

from langchain.agents import create_agent
from langgraph.checkpoint.memory import InMemorySaver
from langchain_core.utils.uuid import uuid7

agent = create_agent(
    model='anthropic:claude-sonnet-5',
    tools=[search_orders],
    system_prompt='You are a concise support agent.',
    checkpointer=InMemorySaver(),          # dev only
    name='support_agent',
)
cfg = {'configurable': {'thread_id': str(uuid7())}}
r1 = agent.invoke({'messages': [{'role': 'user', 'content': 'Where is order 42?'}]}, config=cfg)
r2 = agent.invoke({'messages': [{'role': 'user', 'content': 'And the one before?'}]}, config=cfg)   # same thread, remembers

Parameter	Purpose
model	'provider:model' string or model instance (not pre-bound if you use structured output)
tools	Callables, @tool objects or provider tool dicts
system_prompt	String or SystemMessage (use a SystemMessage for cache control)
middleware	The customisation primitive (§5)
response_format	Schema for the final structured answer (§8)
state_schema	Extra state fields (TypedDict only). Prefer defining them on the middleware that uses them
context_schema	Shape of per-run immutable context (runtime.context)
checkpointer	Thread persistence (short-term memory, HITL, time travel)
store	Cross-thread long-term memory
name	Identifier; vital when the agent is a subgraph, and surfaces as lc_agent_name in stream metadata

thread_id versus context. thread_id scopes the conversation (history and checkpoints). context= carries per-run data (user ID, keys). In production pass both on every call.

Result shape. result['messages'] is the full history; result['structured_response'] exists when response_format is set. With version='v2', invoke returns an object with .value and .interrupts (⚑ LangGraph 1.1+).

Limits. recursion_limit is a top-level key of config (not inside configurable); LangGraph 1.0.6+ defaults it to 1000 steps and raises GraphRecursionError beyond it. Add ModelCallLimitMiddleware so a looping agent cannot burn money.

When create_agent is enough: a model with tools, memory, guardrails and approvals. When to drop to LangGraph: explicit branching and fan-out you must control, deterministic steps interleaved with agentic ones, or custom loops.

📖 Agents · Quickstart · Runtime

---

5. Middleware

Analogy: airport security checkpoints along the agent's route. Before boarding (before_agent), before each flight (before_model), wrapped around the flight itself (wrap_model_call), after landing (after_model), around each baggage handoff (wrap_tool_call), and at the exit (after_agent). Each checkpoint can inspect, change or turn the passenger around.

Style	Hook	Runs
Node	before_agent / after_agent	Once per invocation
Node	before_model / after_model	Around each model response
Wrap	wrap_model_call	Around each model call (you choose how often to call handler: 0 = short-circuit, 1 = normal, N = retry)
Wrap	wrap_tool_call	Around each tool call
Convenience	@dynamic_prompt	Build the system prompt per call

Order. before_* run first to last; after_* run last to first; wrap_* nest like function calls (first middleware is outermost). Put critical guards first.

from langchain.agents.middleware import before_model, wrap_model_call, dynamic_prompt, AgentState, ModelRequest, ModelResponse
from langgraph.runtime import Runtime
from langchain.messages import AIMessage

@before_model(can_jump_to=['end'])
def stop_long_chats(state: AgentState, runtime: Runtime):
    if len(state['messages']) >= 50:
        return {'messages': [AIMessage('Conversation limit reached.')], 'jump_to': 'end'}

@dynamic_prompt
def role_prompt(request: ModelRequest) -> str:
    return f'You are a helpful assistant for a {request.runtime.context.user_role}.'

@wrap_model_call
def route_model(request: ModelRequest, handler) -> ModelResponse:
    model = big_model if len(request.messages) > 10 else small_model
    return handler(request.override(model=model))

• Jumps: return {'jump_to': 'end' | 'tools' | 'model'}, and declare the target with can_jump_to or @hook_config.
• State updates: node hooks return a dict applied through the graph's reducers. Wrap hooks return ExtendedModelResponse(model_response=..., command=Command(update=...)) for model calls, or a Command for tool calls. With several layers, commands compose: messages are additive, and for non-reducer keys the outermost value wins. If an outer layer retries, commands from earlier attempts are dropped.
• Middleware-scoped state: subclass AgentMiddleware, set state_schema = MyState (extra fields live and die with that middleware), and optionally ship tools (for example write_todos) and stream transformers ⚑.
• Class versus decorator: decorators for one hook; classes for several hooks, configuration, or when you need both sync and async (abefore_model...).
• Middleware is not a separate runtime. The hooks run inside the compiled LangGraph, so you can embed the whole agent as a node in a larger StateGraph and its HITL, summarisation and retries come with it.

5.1 Built-in middleware catalogue

Middleware	Use it for	Key options
ToolErrorMiddleware ⚑	Turn tool exceptions into error ToolMessages the model can recover from	on_error (return None to propagate), tools
ToolRetryMiddleware	Retry flaky tools with backoff	max_retries=2, backoff_factor=2.0, retry_on, on_failure='continue'|'error'
ModelRetryMiddleware	Retry model calls	Same options; ⚑ default respects is_retryable
ModelFallbackMiddleware	Fail over to other models/providers	Ordered list of fallback models
SummarizationMiddleware	Compress old history persistently	trigger (tokens/messages/fraction, AND within a dict, OR across a list), keep
ContextEditingMiddleware	Clear old tool outputs when context grows	ClearToolUsesEdit(trigger, keep=3, exclude_tools)
HumanInTheLoopMiddleware	Approve, edit, reject tool calls	interrupt_on, interrupt_mode, when ⚑
ModelCallLimitMiddleware	Cap model calls (cost, loops)	thread_limit (needs checkpointer), run_limit, exit_behavior
ToolCallLimitMiddleware	Cap total or per-tool calls	tool_name, thread_limit, run_limit, exit_behavior
PIIMiddleware	Detect/handle PII	strategy block|redact|mask|hash, custom detector, apply_to_input/output/tool_results
TodoListMiddleware	Planning via a write_todos tool	Custom prompt/description
LLMToolSelectorMiddleware	Pre-select relevant tools when you have 10+	model, max_tools, always_include
ProviderToolSearchMiddleware	Defer tools behind provider-side tool search	searchable_tools; Anthropic and recent OpenAI models only
ShellToolMiddleware	Persistent shell session	Execution policy: host, Docker or sandbox. Not compatible with interrupts yet
FilesystemFileSearchMiddleware	Glob and grep tools over a directory	root_path, use_ripgrep
FilesystemMiddleware, SubAgentMiddleware, MemoryMiddleware, SkillsMiddleware (deepagents)	Files as context, delegation, memory files, on-demand skills	Backends: StateBackend, StoreBackend, CompositeBackend
RubricMiddleware (beta)	Self-grade against a rubric and iterate	max_iterations
LLMToolEmulator	Fake tool results for tests	tools, model
Provider-specific	Anthropic prompt caching, bash, text editor, memory; Bedrock prompt caching; OpenAI moderation	See integrations

A sensible production stack (outer to inner): call limits → HITL → PII → summarisation or context editing → model retry/fallback → tool error → tool retry. The one ordering rule worth memorising: HITL must wrap retry and error middleware in per-call mode, or a retried call asks the reviewer again.

Traps. Keep middleware single-purpose. Hooks that run before a HITL pause run again on resume, so they must be side-effect free. thread_limit on any limiter silently does nothing without a checkpointer. Summarisation compresses text only; images are not shrunk.

📖 Overview · Prebuilt · Custom

---

6. Memory: when to use which

Analogy: a notepad on your desk versus a filing cabinet in the archive. The notepad (short-term memory) holds this conversation and is thrown away or archived with it. The cabinet (long-term memory) holds things worth knowing in any future conversation. The desk is small, so you also need a rule for when to summarise the notepad.

6.1 Decision table

You need…	Use	Mechanism
The agent to remember earlier turns of this conversation	Short-term memory	checkpointer plus a stable thread_id
Pause and resume later (approvals, crashes)	Short-term memory	Same checkpointer; required for HITL
The conversation is getting too long for the context window	Trim, delete or summarise	@before_model trim, RemoveMessage, or SummarizationMiddleware
Remember a user across sessions (preferences, profile, learned facts)	Long-term memory	store with namespace and key; read/write from tools or middleware
Find memories by meaning, not exact key	Store with semantic index	InMemoryStore(index={'embed':..., 'dims':...}) then store.search(query=...)
Per-request settings (user ID, tenant, keys)	Not memory: runtime context	context_schema plus context= on invoke
Large intermediate artefacts (search dumps, files)	Filesystem, not messages	FilesystemMiddleware; route /memories/ to a StoreBackend to persist
Static reference knowledge	Not memory: prompt, skills or retrieval	§12

6.2 Short-term memory (thread scope)

from langgraph.checkpoint.postgres import PostgresSaver     # pip install langgraph-checkpoint-postgres 'psycopg[binary]'
with PostgresSaver.from_conn_string(DB_URI) as cp:
    cp.setup()                                               # creates tables
    agent = create_agent('openai:gpt-5.5', tools=[...], checkpointer=cp)
    agent.invoke({'messages': 'Hi, I am Bob'}, {'configurable': {'thread_id': 'cust-17'}})

• State is read at the start of each step and written when a step completes. Custom fields: subclass AgentState (TypedDict) and pass it through state_schema or, better, a middleware's state_schema.
• InMemorySaver is for development; use Postgres, SQLite, Mongo or Cosmos-backed savers in production.

# Trim (transient is possible via wrap_model_call; this one rewrites state)
from langchain.messages import RemoveMessage
from langgraph.graph.message import REMOVE_ALL_MESSAGES
@before_model
def trim(state: AgentState, runtime: Runtime):
    m = state['messages']
    if len(m) <= 3: return None
    return {'messages': [RemoveMessage(id=REMOVE_ALL_MESSAGES), m[0], *m[-3:]]}

# Summarise (persistent)
SummarizationMiddleware(model='openai:gpt-5.4-mini', trigger=('tokens', 4000), keep=('messages', 20))

Choosing a strategy. Trim is cheap and lossy: fine for chit-chat. Delete is surgical: remove specific messages. Summarise costs an extra model call but keeps meaning: use for long support or research threads. Context editing clears old tool outputs only: use when tool results are what bloat the window. Always keep history valid: a tool-call AIMessage must be followed by its ToolMessage, and some providers want a user message first. RemoveMessage only works on a key with the add_messages reducer (the default messages key has it).

6.3 Long-term memory (cross-thread)

from langgraph.store.memory import InMemoryStore            # PostgresStore in production
store = InMemoryStore(index={'embed': embeddings, 'dims': 1536, 'fields': ['$']})
store.put(('user_123', 'memories'), 'm1', {'text': 'Prefers short, direct answers'})
hits = store.search(('user_123', 'memories'), query='communication style', limit=3)

@tool
def save_memory(text: str, runtime: ToolRuntime[Context]) -> str:
    '''Save a durable fact about the user.'''
    runtime.store.put((runtime.context.user_id, 'memories'), str(uuid7()), {'text': text})
    return 'saved'

agent = create_agent(model, tools=[save_memory], store=store, context_schema=Context)

• Model: JSON documents under a namespace tuple (like a folder, usually including user or org ID) and a key (like a file name). Cross-namespace search is by prefix and content filter.
• Nothing is injected automatically. You must read memories (a tool the model can call, or a @dynamic_prompt/wrap_model_call that injects them) and write them (a tool, or a background step after the turn). Decide deliberately between 'model decides what to remember' (tool, flexible, sometimes forgets) and 'we extract after every turn' (middleware, consistent, costs a call).
• Use user-scoped namespaces so one user can never read another's memories. Namespace prefix matching returns sub-namespaces too.
• On the managed platform the store is provisioned for you; semantic search needs index settings in langgraph.json.

Traps. (1) Reusing a thread_id across users leaks history. (2) A fresh thread_id means a blank notepad, not a lost cabinet: long-term memory survives because it is in the store. (3) InMemoryStore and InMemorySaver vanish on restart. (4) Summarisation permanently replaces messages; if you need the originals for audit, write them elsewhere first.

📖 Short-term memory · Long-term memory · Memory concepts · Checkpointer integrations · Store integrations

---

7. Context engineering

Analogy: packing the briefing folder for a consultant. The consultant is capable, but if the folder holds the wrong pages, the advice will be wrong. Most agent failures are the wrong folder, not a weak consultant.

Agents usually fail because the right context was not passed to the model, not because the model is too weak. LangChain splits what you control into three kinds:

Context type	You control	Lifetime
Model context	System prompt, messages, tools, model, response format for one call	Transient (does not change saved state)
Tool context	What tools read and write (state, store, runtime context)	Persistent
Life-cycle context	What happens between steps: summarise, guardrails, logging	Persistent

Data source	Also called	Scope	Examples
Runtime context	Static config	One run	User ID, role, API keys, environment
State	Short-term memory	One conversation	Messages, uploaded files, auth status
Store	Long-term memory	Across conversations	Preferences, extracted insights

• Transient versus persistent. wrap_model_call with request.override(...) changes only what this call sees. To change saved history, return a Command from the wrap hook or use before_model, after_model or a tool.
• Recency matters. Models pay more attention to the last messages, so injected context (compliance rules, style guide) is appended after history rather than before it.
• Typical moves: dynamic system prompt by role or environment; inject uploaded-file summaries; filter tools by authentication state; route to a bigger model for long threads; switch response_format by stage.
• Start simple: static prompt and tools first, add one dynamic behaviour at a time, and watch token use and latency.

📖 Context engineering in agents · Context overview

---

8. Structured output

Analogy: a form instead of an essay. You hand the model a form with typed boxes. You get back something your code can use without parsing prose.

from pydantic import BaseModel, Field
from langchain.agents.structured_output import ToolStrategy, ProviderStrategy

class Ticket(BaseModel):
    category: str = Field(description='billing, technical, account or product')
    priority: str
    summary: str

agent = create_agent(model='openai:gpt-5.5', tools=[], response_format=Ticket)   # auto strategy
r = agent.invoke({'messages': [{'role': 'user', 'content': 'My card was charged twice!'}]})
r['structured_response']      # Ticket(...)

Option	Behaviour	Pick it when
Pass a schema type	Auto: provider-native if the model supports it, else tool-calling	Default
ProviderStrategy(schema, strict=...) ⚑	Provider enforces the schema; most reliable	Provider supports it; need strict adherence
ToolStrategy(schema, handle_errors=...)	Schema becomes a tool; model must call it; validation errors are fed back for retry	Provider lacks native support; union types

• Schema types: Pydantic (returns validated instance), dataclass or TypedDict (return a dict), JSON Schema (needs top-level title and description, and must be wrapped in an explicit strategy), and Union[...] with ToolStrategy so the model picks one shape.
• handle_errors: True (default, retry with a template message), a string (always that message), an exception type or tuple (only those), a callable returning a message, or False (raise). If the model returns two structured outputs at once, the agent feeds back an error and asks for one.
• tool_message_content customises the confirmation line that lands in history.
• If you pass tools and a schema, the model must support using both together.

Structured output versus a tool: use response_format for the final answer's shape; use a tool when something must happen mid-run. Outside agents, use model.with_structured_output (§1).

📖 Structured output

---

9. Human-in-the-loop and guardrails

Analogy: a second signature on the cheque. The agent may prepare the transfer, but for large amounts a human signs before it goes out. The agent waits, with its place saved, as long as it takes.

from langchain.agents.middleware import HumanInTheLoopMiddleware
from langgraph.types import Command

agent = create_agent('openai:gpt-5.5', tools=[write_file, execute_sql, read_data],
    middleware=[HumanInTheLoopMiddleware(interrupt_on={
        'write_file': True,                                         # all decisions allowed
        'execute_sql': {'allowed_decisions': ['approve', 'reject']},
        'read_data': False,                                         # auto-approve
    })],
    checkpointer=InMemorySaver())                                   # REQUIRED
cfg = {'configurable': {'thread_id': 't1'}}
r = agent.invoke({'messages': [...]}, config=cfg, version='v2')
r.interrupts                                                        # what needs review
agent.invoke(Command(resume={'decisions': [{'type': 'approve'}]}), config=cfg, version='v2')   # same thread_id

Decision	Effect	Note
approve	Run as proposed
edit	Run with changed args (edited_action)	Edit conservatively; big edits make the model re-plan and may repeat the call
reject	Skip and send feedback message	Use this to deny. Say whether the agent may retry
respond	Your reply becomes the tool result	For 'ask user' tools only. The model sees a success, so never use it to deny

• Batched mode (default): one interrupt per model turn listing all gated calls; resume with an ordered decisions list. interrupt_mode='per_call' ⚑: one interrupt per call, each with a response_schema, resumed with a dict keyed by interrupt ID. The docs recommend per-call for new agents and plan to make it the default next major version.
• Conditional gating ⚑: 'when': predicate(request) pauses only risky calls (for example SQL that is not a SELECT). The predicate re-runs on resume and must give the same answer, or the reviewer's decision is ignored.
• In per-call mode an invalid answer is rejected before saving, so send a corrected answer to the same interrupt ID. If several answers are sent together and one is invalid, others may have run, so validate first or answer one at a time.
• Stream while pausing: stream = agent.stream_events(..., version='v3'), check stream.interrupted and stream.interrupts.

Guardrails are policies enforced deterministically regardless of what the model does: PIIMiddleware (redact/mask/hash/block, with custom detectors), call limits, and your own after_agent checks (including model-based safety checks using jump_to='end'). A system prompt asking the model to behave is not a guardrail.

📖 Human-in-the-loop · HITL concepts · Guardrails

---

10. Streaming

Analogy: a live sports ticker instead of the final score. You can show progress (each step), words as they are typed (tokens), or your own custom events ('Fetched 10/100 records').

Mode	Yields
updates	State update after each agent step (model, tools, model)
messages	(token, metadata) from any node that calls an LLM (metadata has langgraph_node, lc_agent_name)
custom	Anything written with runtime.stream_writer or get_stream_writer()
values	Full state after each step

# Classic modes: v2 format = one dict per chunk with type, ns, data
for chunk in agent.stream(inp, stream_mode=['updates', 'messages', 'custom'], version='v2'):
    if chunk['type'] == 'messages':
        token, meta = chunk['data']

# Event streaming (recommended for new apps, langchain 1.3+): one typed iterator per projection
stream = agent.stream_events(inp, config=cfg, version='v3')
for kind, item in stream.interleave('messages', 'tool_calls'):
    if kind == 'messages':
        for t in item.text: print(t, end='')
final = stream.output

• Reasoning tokens: iterate message.reasoning in event streaming, or filter content_blocks for reasoning.
• Subagents and nested graphs: pass subgraphs=True on classic streaming, or read stream.subgraphs in event streaming. Name every agent so you can tell who is talking.
• Tool streaming: tool calls arrive as tool_call_chunk blocks (partial JSON); complete calls come from updates or by summing chunks until chunk_position == 'last'.
• Turn it off for a model with streaming=False (or disable_streaming=True) when you do not want its tokens reaching the client.
• If a custom tool uses get_stream_writer, it can no longer run outside a LangGraph context.

📖 Streaming · Event streaming · Frontend

---

11. Multi-agent patterns

Analogy: a hospital. A generalist (single agent) can handle most cases. When the case needs a specialist's long reference books, you either call the specialist in (subagent), transfer the patient (handoff), look up a protocol (skill), or triage at the door (router).

First question: do you need multi-agent at all? A single agent with the right tools and prompt often suffices. Reach for multi-agent when (a) context management: specialised knowledge would overflow one prompt, (b) distributed development: separate teams own capabilities, or (c) parallelisation. Also when one agent has too many tools and picks poorly.

Pattern	How it works	Strengths
Subagents	Main agent calls subagents as tools; all routing goes through it	Isolation, parallel calls, large-context domains; stateless by default
Handoffs	Tool calls change a state variable; the active agent or its prompt/tools switch	Direct user conversation with the specialist; stateful
Skills	One agent loads specialised prompts on demand	Simple, directly conversational; context accumulates
Router	A step classifies input and sends it to one or more agents, then synthesises	Parallel fan-out across knowledge sources; stateless
Custom workflow	LangGraph graph mixing deterministic and agentic nodes	Full control; embed any other pattern as a node

Measured trade-offs from the docs: one-shot 'buy coffee' costs 4 model calls with subagents but 3 with handoffs, skills or router. A repeat request costs 8 total with subagents (stateless) versus 5 with handoffs or skills (stateful). Multi-domain comparison (three languages, about 2K tokens of docs each): subagents and router about 9K tokens, skills about 15K (context accumulates), handoffs 14K+ and sequential. Rule of thumb: parallel and big-context → subagents or router; repeat and conversational → handoffs or skills.

@tool
def ask_fruit_expert(question: str) -> str:
    '''Ask the fruit expert. Use for ALL fruit questions.'''
    r = fruit_agent.invoke({'messages': [{'role': 'user', 'content': question}]})
    return r['messages'][-1].content
supervisor = create_agent(model, tools=[ask_fruit_expert, ask_veggie_expert], checkpointer=cp)

Traps. A per-thread stateful subagent (checkpointer=True) breaks when the model calls it twice in parallel (checkpoint conflict): add ToolCallLimitMiddleware(tool_name=..., run_limit=1). The langgraph-supervisor package has a migration guide to this tool-wrapped pattern. Deep Agents ships subagents, skills, planning and filesystem if you would rather not assemble this yourself.

📖 Multi-agent · Subagents · Handoffs · Skills · Router · Custom workflow

---

12. Retrieval (RAG)

Analogy: an open-book exam. Instead of memorising the library, the model gets a librarian (the retriever) who fetches the right pages when asked.

Scope note: I did not re-read the RAG tutorials for this page; this section is the stable shape plus pointers. Verify names against the tutorials. The pipeline is: load documents → split into chunks → embed (init_embeddings) → store in a vector store → retrieve. Two ways to use it: agentic RAG (expose a retrieval tool; the model decides when to search and can search several times) and two-step RAG (always retrieve first, then answer; predictable and cheap). Return the passage as the ToolMessage content and put document IDs or page numbers in artifact so your UI can cite without bloating the prompt.

Use RAG when knowledge is large, changes often, or must be cited. Do not use it for facts that fit in the prompt, or as a substitute for structured data you can query exactly (use a SQL tool). Memory versus RAG: memory is about this user and these conversations; RAG is about shared documents.

📖 Build a semantic search engine · Agentic RAG with LangGraph · SQL agent · Vector stores

---

13. Production: reliability, cost, security, testing

• Reliability stack: model max_retries plus ModelRetryMiddleware, ModelFallbackMiddleware for provider outages, tool retry and error middleware, a durable checkpointer, and call limits as circuit breakers.
• Cost: prompt caching, summarisation or context editing, a cheap model for routing and summaries (a wrap_model_call router), tool filtering, and ModelCallLimitMiddleware.
• Security: treat tool output as untrusted input (prompt injection can arrive inside a web page or document); gate side effects with HITL; run shell tools under Docker or a sandbox policy, never the host policy on untrusted input; redaction rules do not stop exfiltration on the host; use PII middleware; sanitise error text returned to the model; never load() untrusted serialised messages; scope store namespaces per user.
• Observability: set up LangSmith tracing from day one; it records every turn, tool call and middleware span. Use a trace_policy ⚑ to omit large payloads.
• Testing: unit-test logic with fake chat models and in-memory persistence (no API calls); integration-test with real models (manage keys, flakiness, cost); evaluate trajectories with deterministic matching or LLM-as-judge; LLMToolEmulator fakes tools when real ones are expensive.
• Deployment: LangSmith deployment provisions checkpointer and store for you; locally pass them explicitly.

📖 Testing · Observability · Deployment · LangSmith Studio

---

14. Version gates and the trap list

⚑ Feature	Minimum
Model profiles, fraction triggers in summarisation	langchain 1.1
ProviderStrategy(strict=...)	langchain 1.2
Middleware stream transformers; PII output redaction in streams; event streaming	langchain 1.3.x (1.3.2 for transformers/PII)
HITL when predicate	1.3.3
ToolErrorMiddleware	1.3.14
Middleware trace_policy	1.3.15
Retry middleware respects is_retryable by default	1.3.16
langchain.mcp (beta)	langchain[mcp] 1.4.0
HITL interrupt_mode='per_call'	1.4.6
reasoning_effort standard parameter	langchain-core 1.5.2 plus provider package versions
runtime.execution_info, server_info; v2 stream format	langgraph 1.1.x

Trap checklist (read before every review):

1. No checkpointer, yet expecting memory or HITL. Or a reused thread_id across users.
2. InMemory* in production.
3. Tool with a vague docstring, or no ToolMessage after returning a Command.
4. Too many tools; huge tool outputs; no call limits.
5. Retry placed outside HITL; side effects before an interrupt.
6. Structured output with a pre-bound model, or a raw JSON Schema not wrapped in a strategy.
7. Old tutorial imports (prompt=, langchain.chains, Pydantic agent state) on a v1 install.
8. Trusting the system prompt as a security control.

---

15. Exercises

1. Build an agent with one @tool that reads runtime.context.user_id, a checkpointer, and a thread_id. Prove it remembers turn 1 on turn 2, then change only the thread_id and show it forgets.
2. Add SummarizationMiddleware with a tiny trigger (('messages', 6)), run 10 turns, and print the history to see the summary message replace the old ones.
3. Add a save_memory tool plus an InMemoryStore. In thread A tell the agent a preference; in thread B (new thread_id, same user_id) show it can recall it via a search tool.
4. Gate execute_sql with HITL so only non-SELECT queries pause (a when predicate). Approve one, reject one with a message, and observe how the model reacts to the rejection.
5. Write a wrap_model_call router that uses a cheap model for the first 6 messages and a stronger one after. Verify with usage_metadata or LangSmith.

Solution notes

1. Same thread_id reloads state from the checkpointer; a new one starts empty. context is separate and must be passed on every call.
2. Summarisation is persistent: the checkpointed state now holds a summary message plus the last keep messages. Trim, by contrast, only changes what is sent if done in wrap_model_call.
3. The store key space is (user_id, 'memories'), independent of thread. The agent will not use the memory unless a tool or prompt injection reads it.
4. when returns True to pause. On resume it is evaluated again and must match. A reject message should tell the agent whether to retry.
5. request.override(model=...) changes only this call; models should be created once outside the hook.

---

Sources: docs.langchain.com (Python): LangChain overview, products, agents, models, messages, tools, short-term memory, long-term memory, middleware, structured output, HITL, streaming, multi-agent, context engineering. RAG section: scope note above.

LangGraph — Components Deep Dive

A study guide to LangGraph 1.x (Python), taken one component at a time: state, nodes, edges, compile and run, persistence, interrupts, fault tolerance, subgraphs, workflow patterns, the Functional API and streaming. Each part says what it is, shows the code, says when to use it, and lists the traps.

How to read this guide. Each component has (1) a mental model, (2) the code you will type, (3) a when to use / when not to verdict, (4) traps, and (5) 📖 doc links. Section 12 is the decision table for “when do I use which kind of memory?” and section 9 for “which workflow pattern?”. The companion page is LangChain — Components Deep Dive (tools, middleware, memory from the agent side). Version note: written 17 Oct 2026 from docs.langchain.com. ⚑ marks features that need a minimum version (table in §14). The Functional API section (§10) draws on the graph-API docs and my prior knowledge more than on a full re-read, so verify it against the linked pages.

---

0. The map: what LangGraph is

Analogy: a factory floor with a conveyor and a clipboard. The clipboard is the shared state. Each station (node) reads the clipboard, does a job, and writes updates. Signs on the floor (edges) say which station comes next. A supervisor with a camera (the checkpointer) photographs the clipboard after every shift, so if the power fails you restart from the last photo, not from the start.

• Three primitives: State (shared snapshot plus the rules for merging updates), Nodes (functions that do the work and return updates), Edges (functions or fixed links that choose what runs next). Nodes do the work, edges say what to do next. Nodes and edges are plain functions: they may contain an LLM or just ordinary code.
• Execution model (Pregel-inspired message passing): the program runs in discrete super-steps. Nodes that run in parallel share a super-step; sequential nodes are separate ones. A node becomes active when it receives a message on an incoming edge, runs, and emits updates. The run ends when every node is inactive and nothing is in transit. Checkpoints are written at super-step boundaries, which explains most of the behaviour in §5-7.
• LangGraph is a low-level runtime. It gives you durable execution, streaming, human-in-the-loop and persistence. LangChain's create_agent is built on it, so you can start high-level and drop down. The docs' guidance: use LangGraph for fine-grained orchestration control, long-running stateful work, workflows mixing deterministic and agentic steps, and production-grade deployment.
• Two APIs: the Graph API (StateGraph, explicit nodes and edges; this page's main subject) and the Functional API (@entrypoint and @task on ordinary Python control flow; §10).

You need…	Use
A model plus tools in a loop, memory, approvals	create_agent (LangChain)
Branching, fan-out/fan-in, explicit loops, deterministic steps between model calls	LangGraph Graph API
Mostly linear Python with loops and ifs, but you want checkpointing and interrupts	Functional API
Planning, files and subagents out of the box for long autonomous tasks	Deep Agents (built on LangGraph)

📖 Overview · Thinking in LangGraph · Graph API overview · Choosing between Graph and Functional · LangGraph runtime

---

1. State

Analogy: a shared whiteboard with one marker per column. Each column (state key) has its own rule for what happens when someone writes: overwrite the old value, or append to it.

1.1 Schemas

• TypedDict is the main documented choice. Use a dataclass if you want default values. A Pydantic BaseModel adds recursive validation but is slower. (create_agent in LangChain only supports TypedDict state.)
• Input/output/private schemas. By default input and output equal the graph schema. Pass input_schema= and output_schema= to constrain what the graph accepts and returns; nodes can also write to extra private channels declared by a schema they use.

from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END

class InputState(TypedDict):
    user_input: str
class OutputState(TypedDict):
    graph_output: str
class OverallState(TypedDict):
    foo: str
    user_input: str
    graph_output: str
class PrivateState(TypedDict):
    bar: str

def node_1(state: InputState) -> OverallState:  return {'foo': state['user_input'] + ' name'}
def node_2(state: OverallState) -> PrivateState: return {'bar': state['foo'] + ' is'}
def node_3(state: PrivateState) -> OutputState:  return {'graph_output': state['bar'] + ' Lance'}

b = StateGraph(OverallState, input_schema=InputState, output_schema=OutputState)
b.add_node('node_1', node_1); b.add_node('node_2', node_2); b.add_node('node_3', node_3)
b.add_edge(START, 'node_1'); b.add_edge('node_1', 'node_2'); b.add_edge('node_2', 'node_3'); b.add_edge('node_3', END)
b.compile().invoke({'user_input': 'My'})      # {'graph_output': 'My name is Lance'}

Trap: private is not secret. Schemas constrain what a node reads and what invoke returns. When streaming with stream_mode='values' the graph emits all channels, private ones included. Pass output_keys=[...] to restrict, or use stream_mode='updates'.

1.2 Reducers

Each key has its own reducer: a binary function new = reducer(left=current, right=update). With no reducer, the update overwrites.

from operator import add
from typing import Annotated
class State(TypedDict):
    foo: int                          # overwrite
    bar: Annotated[list[str], add]    # append
# state {'foo':1,'bar':['hi']}; a node returns {'foo':2} then {'bar':['bye']} -> {'foo':2,'bar':['hi','bye']}

• Messages: use Annotated[list[AnyMessage], add_messages], or subclass MessagesState (class State(MessagesState): documents: list[str]). add_messages tracks message IDs, so updating an existing message replaces it and RemoveMessage deletes one; operator.add would just append duplicates. It also deserialises dicts ({'type': 'human', 'content': ...}) into message objects, so read attributes with dot notation.
• Parallel writes: two nodes writing the same key in one super-step need a reducer, or the update is ambiguous.
• Overwrite: a merging reducer cannot be cleared by returning an empty value ([] is merged in). Wrap the update: {'errors': Overwrite([])} (from langgraph.types). This matters for error buffers and retry counters.
• RemainingSteps: a managed value, remaining_steps: RemainingSteps, tells a node how many super-steps are left before the recursion limit (§4).

When to put something in state: if more than one node needs it or it must survive an interrupt or crash. When not to: connections, clients, secrets and per-run config belong in runtime context (§2), because state is checkpointed and must be serialisable.

📖 State · Use the graph API (how-tos)

---

2. Nodes

Analogy: a station on the floor. It receives the clipboard, does one job, writes back only the lines it changed.

A node is a sync or async Python function. It can accept state, then optionally config (a RunnableConfig with thread_id, tags) and runtime (a Runtime[Context]). A node returns a partial update, not the whole state.

from dataclasses import dataclass
from langgraph.runtime import Runtime

@dataclass
class Context:
    user_id: str

def my_node(state: State, runtime: Runtime[Context]):
    print(runtime.context.user_id, runtime.execution_info.thread_id)
    return {'results': 'Hello, {}!'.format(state['input'])}

builder = StateGraph(State, context_schema=Context)
builder.add_node('my_node', my_node)      # name defaults to the function name if omitted

Runtime field	Use
context	Per-run immutable config (context_schema; pass context= to invoke)
store	Long-term memory store
stream_writer	Emit custom stream events
execution_info ⚑	thread_id, run_id, checkpoint_id, task_id, node_attempt (1-indexed), node_first_attempt_time
server_info ⚑	Assistant/graph ID and authenticated user when running on LangGraph Server
heartbeat() ⚑	Reset an idle timeout manually (§7)
control / drain_requested ⚑	Graceful-shutdown signals (§7)

2.1 Re-execution and idempotency (the rule behind many bugs)

Checkpoints are saved at super-step boundaries, not mid-function. After an interrupt or a retry the affected node runs again from its first line, so any side effect before the pause runs again. Design nodes so a second run is harmless: upserts instead of inserts, idempotency keys, read-before-write. You may freely add or remove nodes and edges for existing threads; determinism rules apply only to tasks and interrupts inside a node.

2.2 Tasks inside a node

If a node does several expensive or non-idempotent operations, wrap each in a @task (from langgraph.func). Task results are checkpointed, so on resume completed tasks are skipped instead of re-run.

from langgraph.func import task
@task
def fetch(url: str): return requests.get(url).text[:100]
def call_api(state):
    futures = [fetch(u) for u in state['urls']]      # run concurrently
    return {'results': [f.result() for f in futures]}

2.3 Node caching

from langgraph.types import CachePolicy
from langgraph.cache.memory import InMemoryCache
builder.add_node('expensive', expensive, cache_policy=CachePolicy(ttl=3))   # ttl seconds; key_func defaults to a hash of the input
graph = builder.compile(cache=InMemoryCache())

Use caching for pure, expensive functions (embedding a fixed text, a slow lookup). Do not cache nodes with side effects or time-dependence. The result carries __metadata__: {'cached': True} on a hit.

📖 Nodes

---

3. Edges, Send and Command

Analogy: signs on the factory floor. Some arrows are painted on (normal edges). Some depend on what is on the clipboard (conditional edges). A foreman can also say 'make one copy of this job per item on the list' (Send) or 'do this update and go there next' (Command).

3.1 Edge types

graph.add_edge('a', 'b')                                          # always a -> b
graph.add_edge(START, 'a'); graph.add_edge('b', END)              # entry / finish
graph.add_conditional_edges('a', route_fn)                        # route_fn(state) returns a node name or list of names
graph.add_conditional_edges('a', route_fn, {True: 'b', False: 'c'})   # optional path map
graph.add_conditional_edges(START, route_fn)                      # conditional entry point

• Fan-out is automatic: a node with several outgoing edges runs all destinations in parallel in the next super-step. A conditional edge that returns a list does the same.
• One routing mechanism per node. Use normal edges or dynamic routing (conditional edges, Command, tool-driven goto), never both. Command only adds dynamic edges; static edges still fire, so you would run both destinations.

3.2 Command

Command(update=..., goto=..., graph=..., resume=...). Return it from a node to update state and route in one step. You must annotate the return type with the possible destinations, -> Command[Literal['next_node']], so rendering and validation know where it can go.

def decide(state: State) -> Command[Literal['approve', 'reject']]:
    return Command(update={'checked': True}, goto='approve' if state['score'] > 5 else 'reject')

• graph=Command.PARENT navigates from inside a subgraph to a node in the parent (the basis of multi-agent handoffs). If the key is shared by parent and subgraph schemas, the parent needs a reducer for it.
• Input to invoke: Command(resume=...) is the only Command intended as run input. Never send Command(update=...) to continue a conversation: any Command input resumes from the latest checkpoint, so a finished graph looks stuck. Send a plain input dict ({'messages': [...]}) to start a new turn on an existing thread.
• Use Command versus a conditional edge: Command when you must update state and route; conditional edge when you only route.

3.3 Send (map-reduce)

Analogy: a teacher hands each student a different page of the exam to mark, then collects the marks into one pile.

from langgraph.types import Send
def assign_workers(state: State):
    return [Send('write_section', {'section': s}) for s in state['sections']]   # one worker per item, each with its own input

class State(TypedDict):
    sections: list[Section]
    completed: Annotated[list, operator.add]     # workers write here in parallel, so it needs a reducer
graph.add_conditional_edges('orchestrator', assign_workers, ['write_section'])

Use Send when the number of parallel branches is unknown until runtime and each branch needs its own input. A Send can carry its own timeout= ⚑ that overrides the target node's default.

📖 Edges · Command · Map-reduce and Send how-to

---

4. Compile and run

You must compile() before use. Compilation checks structure (no orphan nodes) and is where you attach runtime pieces.

graph = builder.compile(
    checkpointer=InMemorySaver(),      # thread persistence, HITL, time travel
    store=my_store,                    # cross-thread memory
    cache=InMemoryCache(),             # node caching
    interrupt_before=['node_a'],       # static breakpoints (debugging only)
)
config = {'configurable': {'thread_id': 't1'}, 'recursion_limit': 50}
result = graph.invoke({'messages': [...]}, config, context={'llm_provider': 'anthropic'})

• thread_id lives in configurable. recursion_limit is a top-level key of config, not inside configurable. Context is a separate context= argument.
• Recursion limit = max super-steps in one run. Default 1000 since LangGraph 1.0.6; exceeding it raises GraphRecursionError. The step counter is config['metadata']['langgraph_step'].
• Proactive handling (recommended): a RemainingSteps state field plus a router that sends the graph to a fallback node when remaining_steps <= 2, so the run ends gracefully with partial results and checkpoints. Reactive: catch GraphRecursionError outside; simpler, but the run is terminated.
• Other metadata in config['metadata']: langgraph_node, langgraph_triggers, langgraph_path, langgraph_checkpoint_ns.
• Visualise: graph.get_graph().draw_mermaid_png() (use xray=True to expand subgraphs).
• v2 output ⚑: invoke(..., version='v2') returns an object with .value (state) and .interrupts, cleanly separating interrupts from state.

📖 Recursion limit · Quickstart · Run a local server

---

5. Persistence: checkpointers, threads, stores

Analogy: save slots in a video game. A thread is one save file. A checkpoint is a save point written at the end of every super-step. Loading a slot continues from the last save; you can also load an older save and branch (time travel). The store is the shared hall of fame that every save file can read.

• Checkpointer: compile with one and LangGraph snapshots state at every super-step, organised in threads. thread_id is the primary key: without it nothing is saved and nothing can resume after an interrupt.
• Pending writes: if one node fails mid-super-step, successful sibling nodes' writes are stored, so a retry does not redo them.
• What persistence unlocks: human-in-the-loop, conversational (short-term) memory, time-travel replay and forking, and fault-tolerant execution.
• Backends: InMemorySaver (dev), SQLite, Postgres (PostgresSaver, AsyncPostgresSaver), MongoDB, Azure Cosmos DB. Call .setup() once for SQL backends. On the Agent Server all of this is provisioned for you.
• Inspect: graph.get_state(config); add subgraphs=True to see nested state (only works when the subgraph is statically discoverable: added as a node or called inside a node, not when hidden inside a tool).

5.1 Stores (cross-thread)

A store holds JSON documents under a namespace tuple and a key, accessible from any thread. It is the right place for user preferences and learned facts.

from langgraph.store.memory import InMemoryStore       # PostgresStore / MongoDB / Redis in production
store = InMemoryStore(index={'embed': embeddings, 'dims': 1536, 'fields': ['food_preference', '$']})
ns = (user_id, 'memories')
store.put(ns, str(uuid7()), {'food_preference': 'I like pizza'}, index=['food_preference'])   # index=False: stored, not searchable
store.search(ns, query='What does the user like to eat?', limit=3)
store.list_namespaces(prefix=(user_id,), max_depth=2)

graph = builder.compile(checkpointer=checkpointer, store=store)
async def call_model(state: MessagesState, runtime: Runtime[Context]):
    hits = await runtime.store.asearch((runtime.context.user_id, 'memories'), query=state['messages'][-1].content, limit=3)

• search traps: namespace_prefix matches by prefix (('alice',) also returns ('alice','memories')); results beyond limit (default 10) are silently dropped, so paginate with offset; ordering differs by backend (Postgres: most recently updated first; in-memory: insertion order), so sort client-side if order matters.
• Custom store: subclass BaseStore and implement the five async methods (aput, aget, adelete, asearch, alist_namespaces). No conformance suite exists, so test against InMemoryStore.

Checkpointer versus store: the checkpointer saves the whole graph state for one thread; the store saves chosen facts for many threads. You normally want both.

Time travel (replay a past execution, or fork from a past checkpoint to explore another path) has its own how-to page; I did not re-read it for this guide.

📖 Persistence · Checkpointers · Stores · Time travel · Add memory

---

6. Interrupts and human-in-the-loop

Analogy: a bookmark with a note in the margin. The graph stops, marks its place, and waits as long as needed. When you hand back an answer, it re-reads the chapter from the start of the page and carries on.

Call interrupt(value) inside a node. LangGraph saves state, surfaces value (any JSON-serialisable object), and waits. Resume by invoking again with Command(resume=answer) on the same thread_id; answer becomes the return value of interrupt(). You need a checkpointer and a thread ID.

from langgraph.types import interrupt, Command

def approval(state) -> Command[Literal['proceed', 'cancel']]:
    ok = interrupt({'question': 'Proceed?', 'details': state['action']})
    return Command(goto='proceed' if ok else 'cancel')

stream = graph.stream_events(inputs, config, version='v3')
_ = stream.output                       # drive the run
if stream.interrupted: print(stream.interrupts)      # tuple of Interrupt(value=..., id=...)
graph.stream_events(Command(resume=True), config, version='v3').output

6.1 The rules (each one is a classic bug)

1. The node restarts from its first line on resume. Everything before interrupt() runs again.
2. Never wrap interrupt() in a bare try/except. It pauses by raising a special exception; swallowing it silently disables the pause. Catch specific exception types only.
3. Do not reorder, skip or loop interrupts non-deterministically. Resume values are matched by index within the node. No conditional skipping, no loops over data that may change.
4. No while True validation loops with interrupt() in one node. Each resume replays earlier iterations. Instead: call interrupt() once, store a pending_question in state on invalid input, and use a conditional edge to loop back.
5. Payloads must be serialisable (strings, numbers, dicts of simple values). No functions or class instances.
6. Side effects before interrupt() must be idempotent: upsert, not insert; or move the side effect after the interrupt or into a separate node.

6.2 Patterns

Pattern	Shape
Approve or reject	interrupt returns bool; route with Command(goto=...)
Review and edit state	edited = interrupt({'content': state['draft']}); return {'draft': edited}
Interrupt inside a tool	The tool calls interrupt({...}) before acting; resume with {'action': 'approve', 'subject': 'new subject'} to approve with edits
Validate human input	One interrupt per node call plus a conditional-edge loop (rule 4)
Several parallel interrupts	Resume with Command(resume={interrupt.id: answer, ...}) for all pending IDs at once
Typed answers ⚑	interrupt(payload, response_schema=Decision) (Pydantic, TypedDict or dataclass); invalid resume raises ValidationError and you can resend to the same thread. A raw JSON-schema dict is exposed to clients but not validated. Needs langgraph 1.2.12.

• Static breakpoints (interrupt_before/interrupt_after at compile or run time, resume with invoke(None, config)) are for debugging, not for human-in-the-loop. Use interrupt() for that.
• Subgraph called as a function: both the parent node and the subgraph node restart from their beginnings on resume.
• Drive a UI loop: call stream_events in a loop, stream tokens from stream.messages, check stream.interrupted, ask the user, then resume with Command(resume=...), until nothing is interrupted.

📖 Interrupts · HITL concepts

---

7. Fault tolerance

Analogy: a delivery driver with three escalating responses. Knock again (retry). Don't wait forever at one door (timeout). If the house is truly unreachable, follow the 'undeliverable' procedure (error handler): leave a note and return to depot.

⚑ Per-node timeouts, error handlers, set_node_defaults and graceful drain need langgraph 1.2+. The order is fixed: an attempt raises (including a timeout) → the retry policy decides → only after retries are exhausted does the error handler run.

7.1 Retries

from langgraph.types import RetryPolicy, default_retry_on
builder.add_node('call_api', call_api, retry_policy=RetryPolicy(max_attempts=3))
# custom: retry_on can be an exception type, a sequence of types, or a callable(exc) -> bool
def retry_on(exc): return False if isinstance(exc, MyBug) else default_retry_on(exc)

Parameter	Default
max_attempts (includes the first)	3
initial_interval / backoff_factor / max_interval	0.5 s / 2.0 / 128 s
jitter	True
retry_on	default_retry_on

Default is conservative: it retries any exception except ValueError, TypeError, ArithmeticError, ImportError, LookupError, NameError, SyntaxError, RuntimeError, ReferenceError, StopIteration, StopAsyncIteration, OSError (programming errors). For requests and httpx only 5xx are retried. NodeTimeoutError is retryable. Inside a node, runtime.execution_info.node_attempt lets you switch to a fallback API on attempt 2+.

7.2 Timeouts

from langgraph.types import TimeoutPolicy
builder.add_node('call_model', call_model, timeout=60)                                  # seconds, wall clock
builder.add_node('call_model', call_model, timeout=TimeoutPolicy(run_timeout=120, idle_timeout=30))

• Async nodes only. A sync node with a timeout is rejected at compile time. Wrap blocking I/O with asyncio.to_thread inside an async node.
• run_timeout: hard cap per attempt, never refreshed. idle_timeout: fires only if the node stops making progress; the clock resets on state writes, stream chunks, child-task scheduling, stream-writer calls and any LangChain callback event (tokens, tool calls). refresh_on='heartbeat' restricts refresh to explicit runtime.heartbeat() calls for long work with no natural progress signal.
• On timeout: NodeTimeoutError (with node, elapsed, kind), writes from that attempt are cleared, then the retry policy decides. The clock resets per attempt.

7.3 Error handlers (compensation, the Saga pattern)

from langgraph.errors import NodeError
def payment_failed(state: State, error: NodeError) -> Command:
    return Command(update={'status': 'compensated: ' + str(error.error)}, goto='finalize')

builder.add_node('charge_payment', charge_payment,
                 retry_policy=RetryPolicy(max_attempts=3, retry_on=ConnectionError),
                 error_handler=payment_failed)

• The handler gets the state and, if you annotate error: NodeError, the failing node and error. It may also take runtime or config. Return a state update or a Command to route elsewhere.
• Failure provenance is checkpointed: if the process crashes after the failure but before the handler finishes, the handler sees the same NodeError on resume.
• interrupt() bypasses retries and handlers (it is control flow, not failure). A failing subgraph surfaces to the parent node's handler. If the handler itself raises, the error propagates as if there were no handler. One handler per node.

7.4 Graph-wide defaults

graph = (StateGraph(State)
    .set_node_defaults(retry_policy=RetryPolicy(max_attempts=3), error_handler=mark_failed, timeout=TimeoutPolicy(run_timeout=30))
    .add_node('step_a', step_a)
    .add_node('charge', charge, error_handler=refund)     # per-node value overrides the default
    .compile())

Default	Regular nodes	Error-handler nodes
retry_policy	Yes	Yes
timeout	Yes	Yes
error_handler	Yes	No (a handler must not catch itself)
cache_policy	Yes	No (caching a handler is unsafe)

Defaults are not inherited by subgraphs. The same timeout= and retry_policy= exist on @task and @entrypoint in the Functional API.

7.5 Graceful shutdown (drain)

import signal
from langgraph.runtime import RunControl
from langgraph.errors import GraphDrained
control = RunControl()
signal.signal(signal.SIGTERM, lambda *_: control.request_drain('sigterm'))
try:
    graph.invoke(inputs, config, control=control)
except GraphDrained as e:
    pass                                   # checkpoint saved; resume later
graph.invoke(None, config)                 # resume with the same thread_id

Drain is cooperative and works between super-steps: a running node (including one mid-retry) finishes first. It does not cancel asyncio tasks or kill threads; pair it with a hard timeout. If the graph finishes on the same tick, it just returns normally (check control.drain_requested). Inside a node, runtime.drain_requested lets you skip expensive work.

7.6 Changing a deployed graph (migrations)

• Threads at the end of the graph: any topology change is fine.
• Threads interrupted: any change except renaming or removing a node the thread is about to enter.
• State keys: adding or removing is safe both ways; renaming a key loses its saved value in existing threads; incompatible type changes can break old threads.
• Changes that are technically compatible but alter behaviour (a new tool set, a restructured flow): pin a behavioural version in state so old threads keep the old path while new threads take the new one.

Traps. Retrying a non-idempotent node duplicates side effects. A timeout on a sync node fails at compile time. Putting the handler on every node via defaults and then also expecting handler nodes to be covered (they are not).

📖 Fault tolerance · Backward compatibility

---

8. Subgraphs

Analogy: a department inside the company. The company chart (parent graph) treats it as one box. Inside, it has its own stations. Whether the department remembers last week's work depends on its filing policy (checkpointer mode).

8.1 How parent and subgraph talk

Pattern	Use when	How
Call inside a node	Different state schemas (no shared keys), or you must transform state	A wrapper node maps parent state to subgraph input, calls subgraph.invoke(...), maps the result back
Add as a node	Shared keys (for example both use messages)	builder.add_node('agent', compiled_subgraph); no wrapper

# different schemas
def call_subgraph(state: Parent):
    out = subgraph.invoke({'bar': state['foo']})
    return {'foo': out['bar']}
# shared schema
builder.add_node('node_2', compiled_subgraph)

8.2 Persistence mode: the checkpointer argument of compile()

Mode	checkpointer=	Behaviour	Pick it for
Per-invocation (default)	None	Each call starts fresh; inherits the parent's checkpointer so interrupts and durable execution work within the call	Most apps; subagents handling independent requests
Per-thread	True	State accumulates across calls on the same thread	A subagent that needs multi-turn memory
Stateless	False	No checkpointing; runs like a function; no interrupts, no durability	Cheap pure helpers; accept that a crash means rerun

• Per-thread caveats: no parallel calls to the same per-thread subgraph (checkpoint conflict); several different per-thread subgraphs called inside one node get namespaces by call order, so reordering mixes up their states. Give each its own wrapper StateGraph with a unique node name for a stable namespace. Subgraphs added as nodes already get name-based namespaces.
• The parent must have a checkpointer for interrupts, state inspection and per-thread memory to work in a subgraph.
• Inspect: graph.get_state(config, subgraphs=True).tasks[0].state. Per-invocation shows only the current call; per-thread shows accumulated state.
• Stream: stream_events(..., version='v3') and iterate stream.subgraphs (each has graph_name, path, messages, values); or classic stream(..., subgraphs=True). Without that flag, stream_mode='messages' on a parent does not emit tokens from an inner agent.
• Navigate up: Command(goto='other', graph=Command.PARENT).

📖 Subgraphs

---

9. Workflow and agent patterns, one by one

Workflows follow predetermined code paths; agents choose their own path. All of these are just graphs. Pick the simplest that fits.

Pattern	Shape	Use when	Avoid when
Prompt chaining	A → gate → B → C; each call consumes the previous output	Well-defined task splittable into verifiable steps (translate, then verify)	Steps are not really sequential
Parallelization	START fans out to N nodes; all feed an aggregator	Independent subtasks for speed, or repeated runs for confidence	Branches depend on each other
Routing	Classifier node (structured output) → conditional edge → specialised node	Distinct input types needing different handling	Categories overlap heavily
Orchestrator-worker	Plan → Send one worker per subtask → synthesiser	Subtasks cannot be predefined (edit N unknown files)	Subtasks are fixed (use parallelization)
Evaluator-optimizer	Generate → evaluate → loop back with feedback until accepted	Clear success criteria that need iteration	No reliable evaluator; unbounded cost (add an iteration cap)
Agent	LLM node ↔ tool node loop until no tool calls	Open-ended problems where steps are unpredictable	You know the steps in advance

# Agent loop in the Graph API
def llm_call(state: MessagesState):
    return {'messages': [llm_with_tools.invoke([SystemMessage('You are a calculator.')] + state['messages'])]}
def tool_node(state: MessagesState):
    out = []
    for tc in state['messages'][-1].tool_calls:
        out.append(ToolMessage(content=tools_by_name[tc['name']].invoke(tc['args']), tool_call_id=tc['id']))
    return {'messages': out}
def should_continue(state) -> Literal['tool_node', '__end__']:
    return 'tool_node' if state['messages'][-1].tool_calls else END

b = StateGraph(MessagesState)
b.add_node('llm_call', llm_call); b.add_node('tool_node', tool_node)
b.add_edge(START, 'llm_call')
b.add_conditional_edges('llm_call', should_continue, ['tool_node', END])
b.add_edge('tool_node', 'llm_call')
agent = b.compile()

# Routing: structured output as the router
class Route(BaseModel):
    step: Literal['poem', 'story', 'joke']
router = llm.with_structured_output(Route)
def route_node(state): return {'decision': router.invoke(state['input']).step}
graph.add_conditional_edges('route_node', lambda s: {'story': 'write_story', 'joke': 'write_joke', 'poem': 'write_poem'}[s['decision']])

9.1 ToolNode

ToolNode (from langgraph.prebuilt) is the prebuilt node that runs tool calls: it handles parallel tool execution, error handling and state injection. Add it with builder.add_node('tools', ToolNode([search, calculator])). Tools read graph state and run context through the injected ToolRuntime argument. Tools only see the state passed to the ToolNode: if you call a ToolNode manually from another node, pass the full state (tool_node.invoke(state)), not just {'messages': ...}, or custom state fields are invisible. A tool returning Command must include a matching ToolMessage.

Hand-written loop versus create_agent: write the loop yourself only when you need control the middleware cannot give (custom routing, extra nodes between model and tools). Otherwise use create_agent and embed it as a node.

📖 Workflows and agents · Agentic RAG · SQL agent

---

10. The Functional API

Analogy: a recipe written as ordinary steps instead of a wiring diagram. You write normal Python (if, while, for); the runtime still saves progress after each marked step.

from langgraph.func import entrypoint, task

@task
def generate(topic: str): return llm.invoke('Write a joke about ' + topic).content
@task
def improve(joke: str):  return llm.invoke('Make this funnier: ' + joke).content

@entrypoint(checkpointer=InMemorySaver())
def workflow(topic: str):
    joke = generate(topic).result()          # tasks return futures; .result() waits
    return improve(joke).result()

• Tasks are the checkpoint unit. Their results are saved, so a resumed run skips completed tasks. Launch several before calling .result() to run in parallel.
• Determinism and idempotency rules are stricter here, because the whole entrypoint re-runs on resume: keep task and interrupt order stable, and keep side effects inside tasks.
• @task and @entrypoint accept retry_policy= and timeout=. An entrypoint compiles to a single node.
• Choose it when the flow reads naturally as code, you do not need a drawn graph, and you want minimal state plumbing. Choose the Graph API when you want explicit visualisation, fine-grained state reducers, parallel branches via edges, or per-node policies. The two can be mixed (a task inside a graph node, §2.2).

📖 Functional API · Use the functional API

---

11. Streaming

Mode	Yields
values	Full state after each step (includes private channels)
updates	Only what each node changed
messages	(token, metadata) from any LLM call inside nodes
custom	Whatever you write with runtime.stream_writer / get_stream_writer()

• Classic: graph.stream(inp, config, stream_mode=['updates', 'messages'], version='v2') yields dicts with type, ns, data. Interrupts appear under __interrupt__ in updates.
• Event streaming (recommended for new apps): stream = graph.stream_events(inp, config, version='v3') gives typed projections you consume independently: stream.messages, stream.values, stream.subgraphs, stream.interrupted, stream.interrupts, stream.output. Pass output_keys=[...] to restrict streamed state.
• Inside a node, get_stream_writer() only works within a LangGraph execution context.
• Disable token streaming for one model with streaming=False (or disable_streaming=True); useful on the platform for models whose output must not reach the client.

📖 Streaming · Event streaming · Frontend

---

12. Memory in LangGraph: when to use which

Analogy: the checkpointer is the save file of this playthrough; the store is the profile that follows the player across playthroughs; state is the HUD you see right now.

Need	Use	Where it lives
Continue the same conversation later	Checkpointer + thread_id	Thread (all graph state)
Pause for a human, survive a crash	Checkpointer	Thread
Keep history from overflowing the context window	Trim, RemoveMessage or a summarise node	State (rewritten) or model input (transient)
Remember facts about a user across threads	Store (runtime.store)	Cross-thread, namespaced
Find memories by meaning	Store with index config and search(query=...)	Cross-thread
Per-request configuration or clients	Runtime context	Not persisted
Data that should not be checkpointed (live connections)	Keep out of state; use context or closures	n/a

Pattern for long-term memory in a graph: a call_model node reads relevant memories from the store and adds them to the prompt; a separate update_memory node (or background job) writes new ones. Keep the write path in its own node so retries and interrupts do not duplicate writes, and use user-scoped namespaces. Memory design from the agent side (tools, middleware, summarisation) is on the LangChain page.

---

13. LangChain agents are LangGraph graphs

• create_agent returns a compiled graph. Embed it as a node or subgraph, and its middleware (HITL, summarisation, PII, retries) travels with it.
• As a subgraph, tokens from the inner model only appear in the parent's messages stream with subgraphs=True (or stream.subgraphs). Give each agent a name to see lc_agent_name in metadata.
• Multi-agent handoffs use Command(goto=..., graph=Command.PARENT) under the hood; the per-thread subagent rules in §8.2 apply.
• Deep Agents adds planning, a filesystem and subagents on top of this same runtime.

---

14. Production, version gates and the trap list

• Deploy: LangSmith Cloud or the Agent Server provisions checkpointer and store automatically and exposes the graph through an API; locally you pass them yourself. Use LangSmith Studio to inspect state, set breakpoints and replay runs. Trace with LangSmith.
• Test: unit-test nodes as plain functions, and graphs with in-memory persistence; use get_state to assert on checkpoints.
• Operate: graceful drain on SIGTERM, per-node timeouts and retries, a default error handler that marks the external job failed, recursion limits with proactive fallback.

⚑ Feature	Minimum
Default recursion limit 1000	langgraph 1.0.6
v2 stream/invoke format (GraphOutput)	1.1
runtime.execution_info, server_info	1.1.5
Per-node timeouts, node error handlers, set_node_defaults, RunControl drain, runtime.heartbeat	1.2
interrupt(..., response_schema=...)	1.2.12
Event streaming (stream_events(version='v3'))	langchain 1.3-era releases; check your install

Trap checklist:

1. No checkpointer yet using interrupt, threads or time travel; or no thread_id.
2. Non-idempotent code before an interrupt() or in a retried node.
3. Bare try/except around interrupt(); while True prompt loops; reordered interrupts.
4. Parallel writes to one key with no reducer; expecting an empty list to clear a reducer field.
5. Mixing static edges with Command/conditional routing from the same node.
6. A Command return annotation missing; a tool returning Command without a ToolMessage.
7. Sending Command(update=...) as input to continue a chat (the graph looks stuck).
8. recursion_limit placed inside configurable; no proactive fallback.
9. Timeout on a sync node; relying on defaults being inherited by subgraphs.
10. Renaming a state key or a node while threads are in flight.
11. Treating private state channels as hidden in values streams.
12. store.search silently truncated at the default limit of 10.

---

15. Exercises

1. Build the agent loop from §9 by hand with MessagesState, compile it with InMemorySaver, and show that two invoke calls on one thread_id share history.
2. Add a node that calls interrupt() before a tool runs. Resume with approval, then repeat and reject. Then deliberately put a counter-incrementing database call before the interrupt and observe the double run.
3. Make a state key errors: Annotated[list[str], add]. Write a node that tries to clear it by returning [], see that nothing changes, then fix it with Overwrite.
4. Add a flaky async node with RetryPolicy(max_attempts=3), a TimeoutPolicy(run_timeout=2), and an error_handler that routes to a finalize node. Make it fail on every attempt and read the final state.
5. Fan out with Send over a list of 5 items into a worker node writing to results: Annotated[list, add], then synthesise. Add a RemainingSteps guard.

Solution notes

1. The checkpointer reloads the whole state for that thread_id; the messages reducer appends the new turn.
2. On resume the node restarts from its first line, so the pre-interrupt side effect runs twice. Fix by upserting, or by moving the call after the interrupt or into its own node.
3. A merging reducer merges the empty list into the old value; {'errors': Overwrite([])} bypasses the reducer.
4. The order is attempts → retry policy → handler. The handler sees NodeError with the node name and last exception; status is whatever your handler wrote. Remember timeouts need an async node.
5. Each Send gets its own worker input; the reducer on results merges parallel writes. Without the reducer the parallel writes conflict.

---

Sources: docs.langchain.com (Python): LangGraph overview, Graph API, persistence, stores, interrupts, fault tolerance, subgraphs, workflows and agents, runtimes, frameworks, harnesses, LangGraph v1 release notes. Time travel and the Functional API were not fully re-read (see scope notes above).