Learnings Data Data Platforms — Digest Definition A data platform is the infrastructure and set of tools an organization uses to collect, store, process, and serve data so applications and people can reliably use it. It's the plumbing that moves data from where it's generated to where it's needed, in usable form. Real-life usage E-commerce: clickstream, orders, and inventory land in a warehouse (Snowflake/BigQuery/Redshift) and feed both BI dashboards and ML models (recommendations, fraud detection) from the same data. Fintech: transaction data flows via streaming pipelines (Kafka) into systems supporting both real-time fraud scoring and end-of-day regulatory reporting. Healthcare: patient records, device telemetry, and claims data are unified under strict governance (HIPAA) while remaining queryable for research. SaaS internal analytics: app DB → ETL/ELT → warehouse → dashboards for product/finance teams. Analogy: a city utility grid — treatment plant (ingestion/cleaning), pipes/reservoirs (storage), pumping stations (processing), taps (serving). Break any layer, everything downstream is affected. 5 Whys — Root Cause Why do platforms exist? Raw source systems are wrong for two independent reasons. Reason 1 — contention: OLTP DBs are optimized for row-based point lookups. Large analytical scans evict the transactional working set from buffer cache and hold locks longer, causing cache misses and slow transactional reads (e.g., checkout). Read replicas solve contention but not query shape — row-store/3NF schemas are structurally bad at large scans/aggregations, hence column-store, star schemas, and pre-aggregation. Reason 2 — incoherence: each source system encodes its own implicit business logic (e.g., three different definitions of "active customer" across CRM, billing, and app DB). With N sources and M consumers and no shared contract, definitions drift — it's structurally guaranteed, not a one-off mistake. Platforms fix this by centralizing logic into governed, computed-once definitions everyone reads from. Value vs. Risk Value: understanding the "why" lets you choose stack components by the guarantee a workload needs (ordering, replayability, latency, managed vs. self-hosted) instead of by popularity or familiarity — avoiding redundant tools solving the same problem. Risk — performance/consistency: black-boxing the platform reproduces the original problems (contention, definitional drift) plus new complexity/understanding debt (unclear why each tool is in the stack). Risk — trust erosion: when dashboards disagree and no one can explain why, people stop trusting the platform and quietly bypass it for gut-feel or spreadsheets — a costly, hard-to-reverse adoption failure. Data Platforms — Article Definition A data platform is the infrastructure and set of tools an organization uses to collect, store, process, and serve data so that applications and people can reliably use it. Stripped of jargon: it's the plumbing that gets data from where it's produced to where it's needed, in a form someone can actually use. When you hear "data platform," think less of a single product and more of a layered system — ingestion, storage, processing, and serving — each layer doing a distinct job. Real-life usage You've already touched pieces of this: Debezium and NiFi doing change-data-capture and flow-based ingestion, Kafka as the durable transport backbone, Airflow and Azure Data Factory orchestrating batch pipelines, Elasticsearch and MongoDB serving specific access patterns. That's a real, if overlapping, data platform stack. Beyond your own stack, the pattern repeats everywhere at scale: E-commerce platforms capture clickstream, orders, and inventory changes, land them in a warehouse (Snowflake, BigQuery, Redshift), and serve both BI dashboards and ML models (recommendations, fraud detection) from the same underlying data. Fintech systems stream transactions through Kafka into pipelines that support both millisecond-latency fraud scoring and end-of-day regulatory batch reporting — same data, wildly different latency contracts. Healthcare platforms unify patient records, device telemetry, and claims data while satisfying strict compliance (HIPAA), yet still need to be queryable for research. Even a mid-size SaaS company runs a scaled-down version: app DB → ETL/ELT → warehouse → dashboards for product and finance. A useful mental model: a data platform is like a city's utility grid. A treatment plant cleans and prepares water (ingestion/cleaning), reservoirs and pipes store and move it (storage), pumping stations push it where it's needed (processing), and taps deliver it on demand (serving/access). Nobody turning on a faucet thinks about the treatment plant — but if any layer fails, everything downstream breaks. 5 Whys — Why Data Platforms Exist The natural question is: why not just query the production database directly? Working through this honestly gets you to two independent root causes. Root cause 1: resource contention. OLTP databases are optimized for row-based, indexed point lookups — fast reads/writes of single rows, like fetching one customer's cart. Analytical queries instead scan and aggregate across millions of rows. When a large aggregation runs against the same database, it evicts the transactional working set from the buffer cache. A checkout read that used to be an instant in-memory hit becomes a slow disk read. Combine that with longer-held locks from the scan, and the aggregation job and the checkout flow — which have nothing to do with each other logically — end up fighting over the same finite memory, I/O, and lock resources. The obvious fix, "just add a read replica," solves contention: reads are offloaded to a copy, so the analytical workload stops starving the transactional one. But it doesn't solve the second problem — query shape . A replica has the exact same row-store, normalized (3NF) schema as production, which is structurally bad at scanning and aggregating across huge row counts, replica or not. This is why the industry didn't stop at replicas: it built column-store engines, star/denormalized schemas, and pre-aggregation — physically reshaping the data to match the access pattern of the consumer, not just relocating the same shape to different hardware. Vertical scaling (bigger single machine) eventually hits a ceiling too, which is part of why horizontal separation into distinct, purpose-built systems became the standard architecture rather than just buying a bigger box. Root cause 2: definitional incoherence. Imagine three teams need "active customer": marketing pulls it from the CRM (logged in within 30 days), finance pulls it from billing (has a paid, non-cancelled subscription), and a data scientist queries the app DB directly (any tracked event in 7 days). All three are querying the same underlying reality, but each source system encodes its own local business logic implicitly — in application code, in how a status field gets set, in team-specific workflow assumptions. There's no shared, enforced contract anywhere. With N source systems and M independent consumers, you get up to N×M different interpretations of the same concept, and nothing structurally prevents that drift — it isn't a one-off mistake someone could have avoided by being careful, it's guaranteed by the absence of a shared definition layer. Put together: data platforms exist because raw source systems are wrong for two independent reasons — they can't serve analytical access patterns without degrading transactional performance, and they can't produce consistent meaning across consumers without centralizing business logic. Every component of a modern data platform — ingestion tools, warehouses, orchestration, semantic/governance layers — exists to solve one or both of these. Value vs. Risk Value. Understanding the "why" rather than just the "how to configure X" changes how you make architectural decisions. Instead of picking a tool because it's popular or because you already know it, you pick based on the guarantee the workload actually needs: does it need ordering and replayability (Kafka), low-latency row-level change capture (Debezium), flexible flow-based routing over heterogeneous data (NiFi), managed Azure-native batch orchestration (ADF), or Python-native complex DAG logic (Airflow)? If two tools in your stack solve the same guarantee — say, two orchestrators — that's a signal of accidental complexity worth questioning directly, not a neutral fact about the stack. Risk of treating the platform as a black box. The immediate risks are a recurrence of the original two problems — contention and inconsistency — plus a new one: complexity debt, where nobody can explain why a given tool is in the pipeline or what it specifically contributes. But there's a third, more expensive risk that's easy to miss because it isn't a technical failure: trust erosion . When two dashboards disagree and no one can explain why, people stop trusting the platform itself, not just that one number. The failure mode isn't a wrong query — it's leadership quietly reverting to gut feel or personal spreadsheets because they've learned the platform can't be relied on. That's a behavioral, org-wide cost, and by the time it's visible, the damage — bad decisions already made, the platform already abandoned in practice — is largely done. It's far more expensive to reverse than a performance regression, because it's an adoption and credibility problem, not an engineering one. Data Warehouse — Digest Definition A data warehouse is a centralized store of structured, historical data — typically modeled with facts and dimensions (star/snowflake schema) — organized specifically to support analytical querying and reporting, as opposed to transactional operations. Real-life usage Retail/e-commerce: fact_sales (quantity, revenue) joined to dim_product , dim_store , dim_date , dim_customer — slice revenue any way, on demand. Banking/fintech: regulatory reporting (risk exposure, AML) needs consistent historical snapshots, not just current state. Airlines: fact_bookings / fact_flights joined to dim_route , dim_aircraft , dim_time to analyze load factors and revenue per route over years. Marketing: fact_impressions / fact_conversions joined to dim_campaign , dim_channel , dim_audience . Analogy: a physical warehouse doesn't manufacture goods — it receives finished goods and organizes them onto labeled shelves by category (dimensions) with clear counts (facts), so anyone can pull exactly what they need without digging through the factory floor. 5 Whys — Root Cause Why not just use a regular RDBMS with a star schema? Row-store databases store full rows together on disk — a query needing 2 of 30 columns still has to read all 30 off disk. Root cause 1 — columnar storage: warehouse engines store each column contiguously, so aggregation queries read only the columns they need. This also enables better compression and vectorized execution. Why also separate facts and dimensions if columnar storage already solves read efficiency? Flattening everything into one wide table causes massive redundancy (a customer's address repeated across every transaction row) and update anomalies (changing one dimension value means rewriting/missing rows across a billion-row fact table). Root cause 2 — fact/dimension separation: avoids redundancy and update anomalies by separating high-volume immutable events (facts) from low-volume mutable reference data (dimensions), each managed according to its actual rate of change (see: slowly changing dimensions). Value vs. Risk Value: understanding columnar storage + fact/dimension separation changes how you design any analytical store — not just SQL warehouses. E.g., a medallion architecture built on doc-oriented stores (like Elasticsearch) still needs to factor out dimension keys instead of embedding full dimension attributes in every aggregate document, or it silently rebuilds the flat-table problem. Risk — no dedicated warehouse layer: flat/embedded dimension data means a dimension change (e.g., product recategorized) can't be propagated without reindexing every historical record referencing it — leading to silent inconsistency between old and new records with no single source of truth to detect or correct the drift. Data Warehouse — Article Definition A data warehouse is a centralized store of structured, historical data — typically modeled with facts and dimensions using a star or snowflake schema — organized specifically to support analytical querying and reporting, as opposed to the transactional operations a production database handles. It's not just "a database with reporting tables"; the storage engine itself and the data model are shaped around analytical access patterns. Real-life usage The clearest way to see this is through fact and dimension tables. A retail warehouse might have fact_sales — one row per line item, carrying measures like quantity and revenue — joined to dim_product , dim_store , dim_date , and dim_customer . Anyone in finance can then slice revenue by region, by month, by category, in combinations nobody had to pre-build. The same pattern shows up everywhere analytics matters at scale: Banking/fintech : regulatory reporting (daily risk exposure, AML) depends on consistent historical snapshots — "what was our exposure on March 31st" — not just current state, which is exactly what a warehouse's historical fact tables are built to answer. Airlines : fact_bookings / fact_flights joined to dim_route , dim_aircraft , dim_time let analysts study load factors, delays, and revenue per route across years of history. Marketing : fact_impressions / fact_conversions joined to dim_campaign , dim_channel , dim_audience answer "which channel actually drives conversions" across time — something no single operational system tracks end to end. A useful analogy: a physical warehouse doesn't manufacture goods — it receives finished goods from many sources and organizes them onto labeled shelves by category (dimensions), with clear counts (facts), so anyone can walk in and pull exactly what they need without digging through the factory floor. A data warehouse does the same thing with data instead of goods. 5 Whys — Why Data Warehouses Exist as a Distinct Thing The natural challenge is: you could build a star schema in any relational database — Postgres, MySQL — so why does the industry build dedicated warehouse engines (Snowflake, BigQuery, Redshift) at all? Root cause 1: columnar storage matches the analytical access pattern. In a normal row-store database, a table's rows are stored together on disk — all of row 1's columns, then all of row 2's columns, and so on. A typical warehouse query like SELECT SUM(revenue) FROM fact_sales WHERE region = 'EU' only needs 2 of maybe 30 columns, but a row-store still has to pull every column of every matching row off disk into memory before it can discard the ones it doesn't need. That's wasted I/O and wasted cache space, proportional to the columns you don't care about. Warehouse engines fix this with columnar storage: each column is stored contiguously on disk, so a query reads only the columns it actually references. This also unlocks two side benefits — far better compression (similar values stored together compress much more efficiently than mixed-type rows) and vectorized execution (operating on whole columns of values at once instead of row by row). This is the physical, mechanical reason a warehouse engine behaves differently from a row-store RDBMS running the exact same schema. Root cause 2: fact/dimension separation avoids redundancy and update anomalies. Given columnar storage already solves read efficiency, it's fair to ask why not just put everything — customer name, product category, store region, revenue, quantity — into one big flat, columnar table. The answer is what happens to descriptive attributes under that design. If "customer address" lives inline in every sales row, and that customer has 500 transactions, the same address string is stored 500 times. At billions of rows, that's significant redundancy even under compression, and it creates an update anomaly: if the customer moves, do you rewrite all 500 (or 5 million) rows referencing them? Miss one, and the data is now silently inconsistent. The deeper issue is conceptual: a "sale" and a "customer's current address" are different kinds of data that change at fundamentally different rates. Facts are an append-only event log; dimensions are comparatively static reference data that occasionally changes (which is exactly why "slowly changing dimensions" exists as a defined technique — dimensions do change, just rarely, and need an explicit strategy for handling it). Separating facts from dimensions keeps each kind of data managed according to its actual rate of change, instead of conflating an immutable event stream with mutable reference data in one structure. Put together: data warehouses exist because (1) columnar storage matches the "few columns, many rows" read pattern of analytical queries in a way row-stores structurally cannot, and (2) fact/dimension modeling avoids the redundancy and update anomalies that come from mixing high-volume immutable events with low-volume mutable reference data in a single flat structure. This root cause generalizes beyond SQL warehouses. A doc-oriented store like Elasticsearch actually does provide columnar-style reads for aggregations, via a structure called doc_values — a separate, purpose-built columnar layer distinct from its inverted index for text search. So root cause 1 can genuinely be satisfied outside a traditional warehouse engine. But root cause 2 is a modeling discipline, not a storage-engine feature — it has to be deliberately designed in. A medallion-style agg_ index built from flat JSON documents that embed both facts and dimension attributes in every document reproduces the exact flat-table problem: dimension values are duplicated across every document that references them, and because a document store has no native join to factor dimensions out, a dimension change either goes unpropagated (silent inconsistency) or requires reindexing every historical document that referenced it. Value vs. Risk Value. Understanding these two root causes changes how you evaluate any analytical data store, not just SQL warehouses. It tells you precisely what to check before trusting a design: does the storage layer read only the columns/fields a query needs (columnar or equivalent), and are high-volume facts kept separate from low-volume, referenced dimension data — or are they flattened together for convenience? For a system like a medallion-on-Elasticsearch setup, this means factoring dimension keys/IDs into fact-like documents and maintaining separate dimension indices, resolving them at query or ingest time, rather than embedding full dimension attributes into every aggregate document. Risk. Keeping facts and dimensions flattened together, whether in a wide SQL table or in JSON documents, defers a cost rather than avoiding it. As data volume grows and dimension values change more often, the concrete failure looks like this: a report segmenting historical sales "by current product category" silently returns wrong numbers, because only some historical records were ever updated to reflect the new category — and there's no single source of truth to detect or correct the drift from, since nothing enforces referential consistency across the duplicated copies. This isn't a performance problem; it's a correctness problem that erodes trust in the numbers exactly the way inconsistent metric definitions do at the platform level. Data Warehouse Use Cases — Needed vs. Overkill When a data warehouse is genuinely needed BI/reporting — ad hoc slicing across dimensions (region, time, product) at speed; row-stores choke on this at scale. Historical/point-in-time analysis — cohort retention, trend analysis, "what did exposure look like on a given date." Needs immutable historical snapshots that production systems don't retain. ML feature engineering / training data prep — building wide feature tables by joining large historical fact/dimension sources; needs efficient scans/joins at volume. Regulatory & compliance reporting — audits need reproducible, consistent historical figures with lineage, not a moving production DB. Cross-domain analytics / data mesh consumption — joining multiple business domains' data together; exactly the "N sources, M consumers, no shared contract" problem centralized modeling solves. Forecasting & capacity planning — needs large historical time series at consistent granularity. Aggregate-level anomaly/trend detection — e.g. "revenue by region dropped 20% week over week"; inherently an aggregation-over-time problem. Common thread: all need to scan/aggregate across large historical volumes with consistent, centrally-defined meaning — the two root causes (columnar reads + fact/dimension governance). When a data warehouse is overkill or the wrong tool Single-record, low-latency lookups (get this customer's cart/profile) — the OLTP point-lookup pattern; columnar storage actually hurts here. Use the operational DB. Real-time operational decisions inside a live transaction (fraud scoring at checkout, inventory decrement) — needs millisecond latency and current state; warehouses are typically minutes-to-hours behind via batch/CDC. Full-text search / relevance ranking — not built for inverted indices or scoring; use Elasticsearch/OpenSearch. Unstructured/semi-structured blob storage — raw images, PDFs, logs, un-modeled JSON; forcing this into a star schema is premature structure — use a data lake (or the raw layer of a lakehouse) first. High-frequency small writes — IoT sensor ingestion event-by-event; warehouses are read-optimized/batch-oriented. Use a time-series DB or streaming store. Simple, low-volume reporting for a small team — a few thousand rows, one monthly summary; the ETL/governance/infra overhead isn't justified. A well-indexed Postgres view or spreadsheet suffices. Graph-shaped queries — path-finding, fraud rings, network traversal; star schemas/columnar engines aren't built for relationship traversal. Use a graph database. Common thread: low-latency point access, unstructured content, full-text relevance, high-frequency writes, or relationship traversal are all problems a warehouse wasn't designed to solve. Grouped by business domain Telecom Needs it: CDR analysis across billions of records, churn prediction feature tables, network capacity planning, regulatory usage/coverage reporting. Overkill/wrong tool: real-time call routing, live network fault detection (streaming/time-series), subscriber lookup during a support call (OLTP). Retail / e-commerce Needs it: sales analysis across store/product/time, inventory trend forecasting, customer segmentation/LTV, marketing attribution. Overkill/wrong tool: checking current stock for one SKU at checkout, product search/autocomplete (search engine), real-time cart/session state. Consulting Needs it: multi-client benchmarking and longitudinal engagement analysis, internal utilization/profitability reporting across practices and time. Overkill/wrong tool: a single short engagement's ad hoc analysis on a one-off dataset — a spreadsheet or lightweight Postgres/DuckDB is enough. Industry / manufacturing Needs it: production yield analysis across plants/lines/time, supply chain and demand forecasting, quality trend analysis joining defects to equipment/shift/supplier. Overkill/wrong tool: real-time sensor/IoT ingestion and shop-floor anomaly alerts (time-series/streaming), machine-level current-status lookups. Finance / banking Needs it: regulatory historical reporting, risk exposure trend analysis, AML pattern detection across large transaction histories. Overkill/wrong tool: fraud scoring at the moment of a transaction (real-time), account balance lookup (OLTP). Healthcare Needs it: population health trend analysis, claims/cost analysis across providers and time, research cohort analysis. Overkill/wrong tool: pulling a single patient's current chart during a visit (OLTP), clinical decision support needing sub-second current data. Pattern across every domain: warehouse = historical, cross-dimensional, aggregatable analysis at real volume. Anything current-state, single-record, sub-second, or small-scale is the wrong job for it, regardless of industry. Elasticsearch Engineering – Digest The short companion to Elasticsearch Engineering : a glossary, then the facts worth memorising for the Elastic Certified Engineer exam (Elasticsearch 9.3), domain by domain. Use it for revision; go back to the full guide for explanations, examples and doc links. A. Glossary Term Meaning (one line) Alias A secondary name pointing to one or more indices or data streams; can filter, route and have one write index Allocation (shard) The master's decision about which node holds each shard copy, governed by deciders Allocation awareness Spreads shard copies across node attributes (zone, rack); forced awareness refuses to overload surviving zones Allocation explain API GET _cluster/allocation/explain : says why a shard is (un)assigned Analyzer Char filters → tokenizer → token filters; turns text into terms at index and search time API key Credential with its own (restricted) privileges; Authorization: ApiKey Auto-follow pattern CCR rule that automatically follows new leader indices matching patterns Bool query Compound query with must , should , filter , must_not Bootstrap checks Startup checks enforced in production mode (heap, file descriptors, vm.max_map_count , memory lock…) Bucket aggregation Groups documents into buckets ( terms , date_histogram , range , filters , composite …) Bulk API NDJSON batch of index/create/update/delete actions in one request CCR Cross-cluster replication: read-only follower indices pull changes from leader indices on a remote cluster CCS Cross-cluster search: query remote_alias:index from the local cluster Classic stream A Streams-managed view over an existing data stream (no migration) Cluster state Metadata (indices, mappings, settings, routing table) managed by the elected master Component template Reusable settings/mappings/aliases block referenced by index templates Composable index template _index_template : patterns + priority + composed_of + own template; can create data streams Coordinating node The node receiving a request; fans it out and merges results (a node with roles: [] does only this) copy_to Copies a field's value into another field at index time Cross-cluster API key API key created on the remote ( /_security/cross_cluster/api_key ) that scopes CCS/CCR access Data stream Append-only name over hidden backing indices; needs @timestamp and a template with data_stream: {} Data stream lifecycle (DSL) Simpler alternative to ILM for data streams: retention (+ downsampling) Data tier Node role groups data_hot / warm / cold / frozen / content ; placement via _tier_preference Dense vector Fixed-length float embedding; searched with knn (HNSW) Dissect Delimiter-based parsing (faster, simpler than grok) DLS / FLS Document-level security (role query ) / field-level security ( field_security.grant/except ) Doc values On-disk columnar structure used for sorting, aggregations and scripts Dynamic mapping Auto-mapping of unknown fields; controlled by dynamic: true/false/strict/runtime Dynamic template Rule mapping new fields by detected type, name or path; first match wins ELSER Elastic Learned Sparse EncodeR: Elastic's sparse-embedding model for semantic search Enrich policy / processor Lookup table built from a source index ( _execute ) and applied at ingest ES|QL Piped query language: FROM … | WHERE … | EVAL … | STATS … BY … | SORT … | LIMIT … Failure store Per-data-stream store that keeps documents that failed ingestion ( ::failures ) Filter context Yes/no matching with no scoring, cacheable ( filter , must_not ) Flattened Maps an entire object as one field of keywords (avoids mapping explosion) Flood-stage watermark 95% disk: indices on that node get read_only_allow_delete Follower / leader index CCR copy (read-only) and its source Grok Regex-based parsing with named patterns ( %{IP:client.ip} ) Health report GET _health_report : indicator-based health with diagnoses Hybrid search Combining lexical (BM25) and semantic results, e.g. with an RRF retriever ILM Index lifecycle management: hot → warm → cold → frozen → delete phases with actions Index Logical collection of documents with one mapping and settings, split into shards Index mode standard , logsdb , time_series (TSDS), lookup Inference endpoint _inference// : a configured model/service used to create embeddings Ingest pipeline Ordered processors that transform documents before indexing Keystore elasticsearch-keystore : encrypted store for secure settings kNN k-nearest-neighbour vector search ( knn section or query) LOOKUP JOIN ES|QL join against an index with index.mode: lookup Mapping Schema of an index: fields, types, analyzers, parameters Master-eligible node Can be elected master; production uses 3 Metric aggregation Computes values ( avg , sum , cardinality , percentiles , stats , top_hits …) Multi-field Same value indexed in several ways ( fields: { raw: keyword } ) Nested Field type keeping array-of-object members as separate hidden docs; needs nested query/agg Normalizer Analyzer-like chain for keyword fields (no tokenizer): lowercase, asciifolding Pipeline aggregation Aggregation over other aggregations' outputs ( derivative , bucket_selector , max_bucket …) Point in time (PIT) Frozen view of indices for consistent deep pagination with search_after Primary / replica shard The original / copy of a shard; never on the same node Query context Scored matching ("how well"): must , should Realm Authentication source: native, file, LDAP, AD, SAML, OIDC, PKI, Kerberos, JWT Refresh Makes recent writes searchable (default 1s) Reindex Copies documents from a source to a destination index (optionally with query/script/pipeline) Remote cluster Named connection (sniff or proxy mode; API key on port 9443, or certificate) used by CCS/CCR Reroute processor Ingest processor sending a doc to another data stream Retriever Composable search building block: standard , knn , rrf , linear , text_similarity_reranker Role / role mapping Named bundle of privileges / rule assigning roles to externally authenticated users Rollover Creating a new write index when conditions (age, size, docs) are met RRF Reciprocal Rank Fusion: merges ranked lists without score normalisation Runtime field Field computed at query time from a script; no reindex needed search_after Cursor-based deep pagination using the last hit's sort values Searchable snapshot Index mounted from a snapshot repository (fully mounted = cold, partially = frozen) Segment Immutable Lucene sub-index inside a shard; merged over time semantic_text Field type that auto-chunks and embeds text through an inference endpoint Service account Built-in identity for Elastic services (e.g. elastic/kibana ) using service tokens Shard A Lucene index holding part of an Elasticsearch index SLM Snapshot lifecycle management: scheduled snapshots with retention Snapshot Incremental backup of indices or cluster state into a repository Sparse vector Token → weight embedding (e.g. ELSER) Streams Kibana feature to process, partition, map, retain and monitor log data from one UI/API Synonyms API _synonyms/ : manage synonym sets usable by updateable search-time analyzers Term-level query Exact, non-analysed matching ( term , terms , range , exists , prefix …) Translog Per-shard write-ahead log for durability between flushes Update By Query Updates matching docs in place (script or pipeline) Watermarks (disk) Low 85% / high 90% / flood 95% Wired stream Streams-managed hierarchy rooted at logs (9.2–9.3) with partitioning and inheritance B. Key things to remember 0. Exam mechanics ES 9.3 , about 3 hours, about 10–12 hands-on tasks, partial credit, only the Elastic docs allowed. Grading is based on final cluster state : exact names matter, don't delete extras, and verify with GET after every task. The Console shortcut Ctrl/Cmd+/ opens the docs for the API under the cursor. 1. Architecture The stack is Collect (Agent/Fleet, Beats, OTel, Logstash) → Process (pipelines, Logstash, Streams) → Store/Search (Elasticsearch) → Explore (Kibana). Routing is hash(_routing) % shards , so the primary count is fixed. Change it with _split , _shrink or reindex. 3 master-eligible nodes for quorum; cluster.initial_master_nodes is used only on the first bootstrap. Shard size 10–50 GB , < 200M docs . cluster.max_shards_per_node = 1000 . Watermarks: 85 / 90 / 95%. Flood stage sets index.blocks.read_only_allow_delete . Forced awareness keeps replicas unassigned rather than overloading a surviving zone, so a yellow cluster can be correct. Allocation filters: include = any, exclude = none, require = all. Attributes: _name , _ip , _host , _tier , and custom node.attr.* . index.unassigned.node_left.delayed_timeout = 1m . For rolling restarts, use cluster.routing.allocation.enable: primaries . Content indices go to data_content ; data stream backing indices go to data_hot . 2. Data Management Static settings : shards, codec, index.mode , sort, and analysis (close → update → open). Dynamic : replicas, refresh, allocation, blocks, default/final pipeline. dynamic : strict rejects, false keeps in _source but doesn't index, runtime creates runtime fields. Template precedence : components (in order) < index template template < create-index request. One template per index : the highest priority wins. Equal priority with overlapping patterns is an error. Built-in templates at priority 100 ( logs-*-* , metrics-*-* , logs.otel* …). Use > 500 to override, or avoid overlapping. Test with _index_template/_simulate_index/ . Templates affect new indices only; roll over data streams. Dynamic templates : first match wins, so put specific rules first and the catch-all last. match = leaf name, path_match = full path. There's no integer detection ( long / double ). Data streams : need a template with data_stream: {} and @timestamp . Append-only ( create op). Deleting the stream deletes its backing indices. Failure store : data_stream_options.failure_store.enabled: true ; read ::failures . ILM : min_age counts from rollover . Rollover conditions are OR-ed; empty indices never roll over. Poll interval is 10m. Rollover needs a data stream or rollover_alias + a bootstrap -000001 with is_write_index: true . migrate is implicit in warm and cold; with no tier nodes, the shards don't move. Frozen requires searchable_snapshot . Shrink and forcemerge in hot only with rollover. Debug with _ilm/explain , then _ilm/retry . DSL vs ILM : when both are set, ILM wins ( prefer_ilm ). DSL is for simple retention only. Aliases : _aliases actions are atomic ( add , remove , remove_index ). Multi-index aliases need one is_write_index to accept writes. Filtered aliases use filter + routing . 3. Ingestion & Processing Choose the ingest method : Agent + Fleet for managed hosts and integrations, Logstash for heavy transforms, buffering or multiple outputs, client + _bulk for applications, Streams for parsing logs from the UI, file upload for one-offs. Bulk : NDJSON with a final newline; check errors ; 5–15 MB batches. Immutable once mapped : type, analyzer, index/doc_values, nested vs object. You can add fields, add multi-fields, and add runtime fields. New multi-field → _update_by_query to populate existing docs. text for full text, keyword for exact/sort/aggs, both via multi-field. Use a normalizer for case-insensitive keyword. nested keeps object pairs together; it needs a nested query/agg. Limits: 50 nested fields, 10k nested objects per doc. Limits: total_fields.limit 1000 , depth.limit 20. Use flattened for unbounded keys. scaled_float for prices, constant_keyword for per-index constants, wildcard for grep-style search, match_only_text for logs, date_nanos for ns precision. Analyzer order: char_filter → tokenizer → filter . Test with _analyze . Put edge n-grams at index time only, with search_analyzer: standard . Search-time synonyms: synonym_graph + Synonyms API + updateable: true . Reindex : create the destination mapping first ; op_type: create for data streams; conflicts: proceed ; wait_for_completion=false → _tasks ; remote reindex needs reindex.remote.whitelist . Painless: ctx._source.x in reindex/UBQ scripts vs ctx.x in ingest; ctx.op = 'noop'|'delete' ; null-safe ?. . Pipelines : if conditions, ignore_failure , ignore_missing , and processor or pipeline on_failure ( _ingest.on_failure_message ). Always _simulate?verbose . default_pipeline can be overridden by ?pipeline= ; final_pipeline always runs . _ingest.timestamp is the ingest time. The reroute processor changes the target data stream. Use @custom pipelines to customise integrations. Enrich : create the policy → _execute → enrich processor. Policies are immutable (delete and recreate), and must be re-executed when the source changes. Streams : classic = existing data streams in place; wired = hierarchy rooted at logs (9.2–9.3) with partitioning and inheritance . The tabs are Processing (with preview), Schema, Data lifecycle (inherit / custom / ILM), Data quality (degraded vs failed → failure store) and Advanced. Don't edit managed components with raw APIs. The API lives under kbn:/api/streams . 4. Searching Data match is analysed (OR by default; operator: and , minimum_should_match , fuzziness: AUTO ). term is exact. Never use term on a text field; use .keyword . match_phrase + slop ; multi_match with fields: ["title^3"] , types best_fields (default), most_fields , cross_fields , phrase , bool_prefix . Bool : must (scored), filter (unscored, cached), should (scored, optional if a must / filter exists, otherwise at least 1 required), must_not (unscored). post_filter filters hits but not aggregations (facets). from + size ≤ 10,000 ( max_result_window ); for deeper pages, use search_after + PIT . You can't sort on text . Aggregations: set size: 0 ; use terms on keyword only; terms counts are approximate ( shard_size ); use composite to page through all buckets. date_histogram : calendar_interval (1d, 1M, week…) vs fixed_interval (30m, 7d); min_doc_count: 0 + extended_bounds for empty buckets; time_zone . range agg: from is inclusive, to exclusive. buckets_path : > for nesting, . for a metric's value, _count for doc count. bucket_selector works like HAVING; bucket_sort sorts and truncates buckets. ES|QL : POST _query ( ?format=txt ). The default LIMIT is 1000 (max 10k). STATS … BY , EVAL , KEEP / DROP / RENAME , DISSECT / GROK , ENRICH , LOOKUP JOIN (the lookup index needs index.mode: lookup ), BUCKET() , MATCH() or : for full text, METADATA _index, _score , ? params, and remote:index for CCS. Semantic search : an inference endpoint (a default ELSER one exists) + a semantic_text field. Query with match or semantic on it; highlight with type: semantic . Hybrid uses the rrf retriever ( rank_window_size , rank_constant ). Manual path: dense_vector + knn ( k , num_candidates , filter ) or sparse_vector query. It needs ML capacity and a licence (trial in the lab). CCS : alias:index , *:index , exclusions -alias:* . skip_unavailable defaults to true ; ccs_minimize_roundtrips defaults to true. _resolve/cluster checks connectivity. _clusters in the response shows per-cluster status. 5. Cluster Management Green / yellow / red = all assigned / replica(s) unassigned / primary unassigned. The workflow: _cluster/health → _cat/shards → _cluster/allocation/explain → fix → _cluster/reroute?retry_failed=true if max retries was hit. Common causes: too few nodes for the replicas, allocation filters, awareness, disk watermarks, a missing tier, allocation.enable set to none/primaries, max retries. Red with all copies lost: restore from a snapshot. allocate_empty_primary / allocate_stale_primary (with accept_data_loss ) is the last resort. persistent cluster settings survive restarts; transient is deprecated. null resets a setting. Monitor heap (> 85% is bad), breakers, thread-pool rejections, disk, slow logs and hot threads. Use Stack Monitoring (Agent/Metricbeat → monitoring cluster) or AutoOps. Snapshots : fs repos need path.repo on every node. Snapshots are incremental. Restore needs the target closed or absent; use rename_pattern / rename_replacement otherwise. include_global_state and feature_states control what's restored. SLM : Quartz cron (seconds first), date-math names , retention ( expire_after , min_count , max_count ), _execute to run now. Remote clusters : the API-key model is recommended (remote port 9443 , remote_cluster_server.enabled , cross-cluster API key with access.search / access.replication , local keystore cluster.remote..credentials ). The certificate model is deprecated (port 9300, same-named roles on the remote). Modes are sniff (default) and proxy . Check with GET _remote/info . Local nodes need remote_cluster_client . The local role grants remote access through remote_indices . CCR : licence on both clusters; run on the follower cluster: PUT /_ccr/follow {remote_cluster, leader_index} . Auto-follow uses leader_index_patterns + follow_index_pattern: "{{leader_index}}-x" and covers new indices only. Followers are read-only. Failover (promote a follower): pause_follow → _close → unfollow → _open . Soft deletes must be on (the default). 6. Security & Access Control A role has cluster + indices ( names , privileges , field_security , query ) + applications (Kibana: kibana-.kibana , feature_x.read/all , space: ) + remote_indices . A user's roles are a union : one role with unrestricted read cancels another role's DLS. Key privileges: read + view_index_metadata for Kibana data views; create_doc + auto_configure for shippers writing to data streams; manage_security is effectively admin. DLS templating: {{_user.username}} . FLS: grant / except with wildcards. API keys get the intersection of their role_descriptors and the creator's privileges; use expiration ; invalidate with DELETE _security/api_key . Role mappings use rules ( field / any / all / except on groups , dn , username , realm.name ) for LDAP, AD, SAML and OIDC users. Test with _security/_authenticate , _security/user/_has_privileges and the es-security-runas-user header. Production security : security on; transport TLS is mandatory for multi-node clusters with security; HTTPS for clients and Kibana; secrets in the keystore ; Kibana via a service account token ; no daily use of elastic . Tools: elasticsearch-certutil ca|cert|http , elasticsearch-reset-password -u , elasticsearch-create-enrollment-token -s kibana|node , elasticsearch-service-tokens create , elasticsearch-keystore add , POST _nodes/reload_secure_settings . Bootstrap checks (production mode starts when you bind a non-loopback network.host ): heap Xms = Xmx , ≤ 50% RAM, < ~31 GB; file descriptors ≥ 65,535 ; threads ≥ 4,096 ; vm.max_map_count ≥ 262,144 ; memory lock works if enabled; discovery configured; swap disabled. Audit logging ( xpack.security.audit.enabled , events.include ), IP filtering ( xpack.security.transport|http.filter.allow/deny ), and action.destructive_requires_name: true . Kibana needs a 32+ character xpack.encryptedSavedObjects.encryptionKey for alerting and encrypted saved objects. C. Ten-second API recall Need API Health & diagnosis _cluster/health , _health_report , _cat/shards , _cluster/allocation/explain , _cluster/reroute?retry_failed=true Templates _component_template , _index_template , _index_template/_simulate_index/ Data streams _data_stream/ , /_rollover , ::failures , _data_stream//_lifecycle ILM _ilm/policy/

, /_ilm/explain , /_ilm/retry Aliases POST _aliases {actions:[add/remove/remove_index]} Mapping & analysis /_mapping , /_analyze , _synonyms/ Move data _reindex , /_update_by_query , /_delete_by_query , _tasks/ Pipelines _ingest/pipeline/

, …/_simulate?verbose , _ingest/_simulate , _enrich/policy/

/_execute Search /_search , _async_search , /_pit , _query (ES|QL), _resolve/cluster/ Semantic _inference// , semantic_text , semantic query, rrf retriever Snapshots _snapshot/ , _snapshot///_restore , _slm/policy/

/_execute Remote / CCR _remote/info , _security/cross_cluster/api_key , /_ccr/follow , _ccr/auto_follow/ , /_ccr/unfollow Security _security/role/ , _security/user/ , _security/api_key , _security/role_mapping/ , _security/_authenticate , _security/user/_has_privileges Elasticsearch Engineering A study guide for the Elastic Certified Engineer exam (Elasticsearch 9.3), organised by the official exam objectives. How to read this guide. Each objective gets (1) a mental model, often an everyday analogy, (2) the APIs you'll actually type, (3) the traps graders like, and (4) links to the exact Elastic documentation pages. The Elastic documentation is the only resource available during the exam , so the 📖 links double as a map of where to go on exam day. Exercises with solution notes close each domain. The companion page, Elasticsearch Engineering – Digest , holds the glossary and the must-remember facts. 0. The exam and your preparation 0.1 Format Item Detail Version Elasticsearch 9.3 (the 8.x exam was retired on 1 Sep 2026) Type Hands-on, performance-based, on live Elasticsearch and Kibana clusters, remote-proctored Duration About 3 hours Tasks About 10–12 practical tasks, often combining several objectives, with partial credit Allowed resources Only the Elastic Documentation Grading The final state of your clusters: indices, templates, pipelines, roles, and so on 📖 Exam page · Certification FAQ Analogy. It's a practical driving test, not a written theory test. Nobody asks you to recite the highway code. The examiner checks whether the car ends up correctly parked. The docs are your satnav: allowed, but only useful if you already know roughly where you're going. 0.2 The six domains # Domain Objectives 1 Elasticsearch Architecture Stack architecture; shard sizing and allocation strategies 2 Data Management Index definition; dynamic templates; ILM for time series; index template that creates a data stream; aliases 3 Data Ingestion & Processing Ingest methods; mappings; field data types; Reindex and Update By Query; ingest pipelines; Streams 4 Searching Data Term and phrase queries; bool combinations; metric and bucket aggregations; ES|QL; semantic search; cross-cluster search 5 Cluster Management Monitor and maintain cluster health; cross-cluster operations 6 Security & Access Control RBAC; securing a cluster for production 0.3 Build a lab first You can't prepare for a hands-on exam by reading. Use 9.3.x so the APIs and UI match the exam. Option A: single node in one command (good for Domains 2–4) curl -fsSL https://elastic.co/start-local | sh 📖 Local installation quickstart Option B: three-node Docker Compose (needed for allocation, awareness, snapshots, and Domain 5) 📖 Multi-node cluster with Docker Compose Settings to vary per node (as elasticsearch.yml or environment variables): cluster.name: lab node.name: es01 # es02, es03 node.roles: [ master, data_hot, data_content, ingest, remote_cluster_client ] node.attr.zone: zone-a # zone-b, zone-c → awareness exercises discovery.seed_hosts: [ es01, es02, es03 ] cluster.initial_master_nodes: [ es01, es02, es03 ] # first bootstrap ONLY, then remove path.repo: [ /usr/share/elasticsearch/snapshots ] # shared FS repo for snapshots For Domain 5, add a second cluster (e.g. remote ) so you can practise remote clusters, cross-cluster search (CCS) and cross-cluster replication (CCR). Enable a trial license. CCR, searchable snapshots, some security features and ML-based semantic search need it: POST _license/start_trial?acknowledge=true Load sample data. In Kibana: Home → Add data → Sample data (eCommerce, Flights, Web logs). Many exam tasks look exactly like work on these datasets. 0.4 Exam-day habits that earn points Read each task twice and copy names (indices, fields, pipelines, roles) straight from the task text. A typo scores zero. Verify everything. GET what you created. Compare _count before and after a reindex. _simulate a pipeline before using it. Don't destroy state the task didn't mention. Graders inspect the final state. Prototype on scratch indices ( tmp-* ), then clean up. Time-box. Roughly 15 minutes per task. At 20 minutes, move on and come back later. Know the "immutable" things. You can't change shard count, field types or a field's analyzer in place. The answer is a new index plus _reindex , or _split / _shrink for shards. Use Console shortcuts. Ctrl/Cmd+Enter runs a request, Ctrl/Cmd+/ opens the docs for the API under the cursor, and Ctrl/Cmd+I auto-indents. 📖 Console (Dev Tools) 1. Elasticsearch Architecture 1.1 Describe the Elastic Stack architecture The big picture Analogy: a postal system. Collectors (Elastic Agent, Beats, OpenTelemetry collectors, Logstash) pick up letters everywhere. Sorting centres (Logstash, ingest pipelines, Streams) open, stamp and classify them. The warehouse (Elasticsearch) stores every letter so it can be found in milliseconds. The front desk (Kibana) is where people ask questions and see dashboards. Fleet is the dispatch office that tells every collector what to pick up. Layer Components Role Collect Elastic Agent (+ Fleet Server), Beats (Filebeat, Metricbeat…), EDOT / OpenTelemetry Collector, APM agents, language clients, connectors Ship logs, metrics, traces and documents Process Logstash, Elasticsearch ingest pipelines , Streams , integrations' built-in pipelines Parse, enrich, route, transform Store / search / analyse Elasticsearch Distributed document store, search engine, analytics and vector database built on Apache Lucene Explore / manage Kibana (Discover, Dashboards, Dev Tools, Stack Management, Fleet UI, Streams UI) Visualise, query, administer Solutions Search, Observability, Security Opinionated apps on top of the same data Deployment options: Elastic Cloud Hosted (ECH), Elastic Cloud Serverless (no nodes or shards to manage), Elastic Cloud Enterprise (ECE), Elastic Cloud on Kubernetes (ECK), and self-managed. The exam uses self-managed-style clusters, so everything below assumes you can touch nodes and settings. 📖 Elastic fundamentals · Deployment comparison · Ingest tools overview Inside Elasticsearch: cluster → node → index → shard → segment Analogy: a library chain. The cluster is the whole chain. Each node is a branch building. An index is a book collection (for example "novels"). Because the collection is too big for one shelf, it's cut into volumes: the primary shards . Each volume has photocopies ( replica shards ), always kept in a different branch. Inside each volume, pages are glued in batches ( segments ) that are never edited, only replaced when batches are merged. Concept What to know Cluster One or more nodes sharing a cluster.name . One elected master manages the cluster state (metadata: indices, mappings, settings, shard routing table). Node One Elasticsearch process. Its roles decide what it does (table below). Index A logical namespace of JSON documents with one mapping and one set of settings. Shard A Lucene index. Primary shards hold the data; replicas are copies that give high availability and extra read throughput. A primary and its replica are never on the same node. Segment An immutable, self-contained piece of a shard. Deletes are marked ; merges physically drop them. Document routing shard = hash(_routing) % number_of_routing_shards → mapped to primaries . _routing defaults to _id . This is why the primary count is fixed at creation. Node roles ( node.roles in elasticsearch.yml ): Role Purpose master Eligible to be elected master. Run 3 dedicated master-eligible nodes in production. voting_only Votes in elections but never becomes master (a tie-breaker). data Generic data role (holds all tiers). data_content Non-time-series content (product catalogues and so on). data_hot / data_warm / data_cold / data_frozen Data tiers for time-series data, from fastest and most expensive to cheapest. ingest Runs ingest pipelines. ml Runs machine-learning jobs and trained models (e.g. ELSER). remote_cluster_client Can connect to remote clusters (needed for CCS and CCR). transform Runs transforms. [] (empty) Coordinating-only node: routes requests and merges results. Every node is also a coordinating node. Whichever node receives the request fans it out and gathers the results. Write path: client → coordinating node → routes to the primary shard → primary indexes (in-memory buffer + translog ) → forwards to replicas in parallel → acknowledges once the required copies succeed ( wait_for_active_shards , default 1 = primary only). Near real-time search: a refresh (default every 1s, only on indices that have been searched recently) turns the buffer into a searchable segment. A flush makes segments durable on disk and trims the translog. Merges combine small segments in the background. Read path (query then fetch): the coordinating node sends the query to one copy of each shard . Each shard returns its top IDs and scores ( query phase ). The coordinator merges them, then fetches the actual documents ( fetch phase ). Cluster formation: master-eligible nodes elect a master by quorum (majority of the voting configuration). Three master-eligible nodes tolerate one failure. cluster.initial_master_nodes is used only for the very first bootstrap ; remove it afterwards. 📖 Clusters, nodes and shards · Node roles · Reading and writing documents · Near real-time search · Discovery and cluster formation · Quorum-based decisions Data tiers Analogy: a wardrobe. Hot is the chair where today's clothes lie: instant access, most expensive space. Warm is the wardrobe. Cold is the suitcase under the bed. Frozen is the storage unit across town: you can still get anything, but it takes a trip. Content is your bookshelf. It's not "time-based", it's just always there. New indices go to data_content by default; data stream backing indices go to data_hot . Tier placement uses index.routing.allocation.include._tier_preference (e.g. data_warm,data_hot ), which ILM's migrate action sets automatically. Frozen uses partially mounted searchable snapshots: a local cache plus the object store. Cold can use fully mounted searchable snapshots, which lets it drop replicas. 📖 Data tiers · Searchable snapshots 1.2 Describe shard sizing and allocation strategies Shard sizing: the rules of thumb Analogy: moving boxes. If boxes are too big, nobody can lift them: slow recovery, slow rebalancing. If there are too many tiny boxes, the truck spends its time on paperwork: every shard costs heap, file handles and cluster-state overhead. Aim for boxes one person can carry. Guideline Value Target shard size 10 GB – 50 GB (time-series often aims for about 50 GB via rollover max_primary_shard_size ) Documents per shard Keep under 200 million Max shards per node (non-frozen) cluster.max_shards_per_node = 1000 by default. It's a soft limit that blocks index creation. Heap overhead Mostly driven by mapped fields and segments. Avoid mapping explosions and thousands of tiny indices. Searches Each search runs one thread per shard copy . Many small shards means lots of queuing. Time series Use data streams + rollover instead of daily indices to control size, not calendar. Too many tiny shards? _shrink , _forcemerge , reindex into fewer indices, raise rollover thresholds. Shard too big? _split , or lower rollover thresholds for new data. GET _cat/shards?v&s=store:desc&h=index,shard,prirep,state,docs,store,node GET _cat/indices?v&s=pri.store.size:desc GET _cluster/stats?filter_path=indices.shards,indices.count 📖 Size your shards · Scaling considerations How shard allocation works The master's allocator decides where every shard copy lives, filtered by deciders . Each decider can say YES, NO or THROTTLE. The common ones: Decider Rule Same shard Never two copies of the same shard on one node Disk threshold Watermarks : low 85% (no new shards to this node), high 90% (move shards away), flood stage 95% (indices on it become read-only: index.blocks.read_only_allow_delete ) Awareness Spread copies across attribute values (zone or rack) Filtering include / exclude / require rules at index or cluster level Data tier _tier_preference must match a node's role Max retries After 5 failed allocations ( index.allocation.max_retries ) the shard gives up until retried Enable cluster.routing.allocation.enable : all / primaries / new_primaries / none Total shards per node index.routing.allocation.total_shards_per_node hard limit The single best diagnostic API: GET _cluster/allocation/explain { "index": "orders", "shard": 0, "primary": false } Strategy 1: allocation awareness (survive losing a zone or rack) Analogy. Don't keep your passport and its photocopy in the same bag. # elasticsearch.yml on each node node.attr.zone: zone-a # zone-b on others PUT _cluster/settings { "persistent": { "cluster.routing.allocation.awareness.attributes": "zone", "cluster.routing.allocation.awareness.force.zone.values": "zone-a,zone-b" } } Awareness (first setting): Elasticsearch spreads copies across zones where it can. Forced awareness (second setting): if zone-b disappears, Elasticsearch won't squeeze all replicas into zone-a (which could overload it). The replicas stay unassigned instead. That's intended behaviour, so a yellow cluster can be the correct answer. 📖 Shard allocation awareness Strategy 2: allocation filtering (pin or evict shards) // Index-level: keep "hot-orders" only on nodes tagged box_type=ssd PUT hot-orders/_settings { "index.routing.allocation.require.box_type": "ssd" } // Index-level by node name (e.g. to prepare a shrink) PUT legacy/_settings { "index.routing.allocation.require._name": "es01" } // Cluster-level: drain a node before decommissioning it PUT _cluster/settings { "persistent": { "cluster.routing.allocation.exclude._name": "es03" } } include means at least one of the values, exclude means none of them, and require means all of them. Built-in attributes: _name , _host_ip , _publish_ip , _ip , _host , _id , _tier . 📖 Index-level shard allocation · Shard allocation, relocation and recovery Strategy 3: delayed allocation and rolling restarts When a node leaves, its replicas are reallocated elsewhere only after index.unassigned.node_left.delayed_timeout (default 1m ), which avoids expensive reshuffles when a node is just rebooting. For planned maintenance: PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "primaries" } } POST _flush // …restart the node, wait for it to join… PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": null } } 📖 Delaying allocation when a node leaves · Rolling restart Strategy 4: resilience design Small clusters: at least 3 master-eligible nodes , and 1 replica for every index you can't afford to lose. A 2-node cluster isn't resilient: losing the wrong node leaves no voting majority. Add a third node (it can be a small voting_only , master-eligible tie-breaker). Larger clusters: dedicated masters, spread across 3 zones, with awareness enabled. 📖 Resilience in small clusters · Resilience in larger clusters 1.3 Exercises: Architecture On your three-node lab, give the nodes attributes zone: a , zone: a , zone: b . Configure forced awareness on zone with values a,b . Create index aware with 1 shard and 1 replica. Stop the zone: b node and explain the resulting health. Create index ssd-only whose shards may only live on nodes with attribute disk: ssd (set it on one node). Prove it with _cat/shards . Decommission node es03 : move all its shards off without stopping it. Explain why an index created with 1 primary shard can't be given 3 primaries in place, and name the two ways to get there. Solution notes Settings as in Strategy 1. With zone b gone, the replica stays unassigned because forced awareness forbids placing both copies in zone a, so health is yellow . _cluster/allocation/explain names the awareness decider. node.attr.disk: ssd on one node, then PUT ssd-only {"settings":{"index.routing.allocation.require.disk":"ssd","number_of_replicas":0}} . With 1 replica, the replica can't allocate because only one node qualifies. PUT _cluster/settings {"persistent":{"cluster.routing.allocation.exclude._name":"es03"}} , then watch _cat/shards until nothing remains on es03. Routing is hash(_routing) % shards , so changing the count would misplace every document. Use _split (by multiples of the routing shards; needs a write block) or _reindex into a new index. 2. Data Management Analogy for the whole domain: a filing cabinet factory. An index is a filing cabinet. Its settings are the hardware: how many drawers (primary shards) and how many photocopied backups (replicas). Its mapping is the labelling scheme for folders. Templates are the factory spec sheet, so every new cabinet named logs-* ships pre-configured. Aliases are nicknames on the door ("current-orders"), so people don't need the serial number. ILM is the records-retention policy: new files on the desk, older ones in the basement, ancient ones in the shredder. 2.1 Define an index that satisfies a given set of requirements An index creation request has three optional parts: settings , mappings and aliases . PUT products { "settings": { "number_of_shards": 2, "number_of_replicas": 1, "refresh_interval": "30s", "analysis": { "analyzer": { "folded": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "asciifolding"] } } } }, "mappings": { "dynamic": "strict", "properties": { "sku": { "type": "keyword" }, "name": { "type": "text", "analyzer": "folded", "fields": { "raw": { "type": "keyword", "ignore_above": 256 } } }, "price": { "type": "scaled_float", "scaling_factor": 100 }, "in_stock": { "type": "boolean" }, "created": { "type": "date", "format": "yyyy-MM-dd||epoch_millis" }, "tags": { "type": "keyword" } } }, "aliases": { "catalog": {} } } Static vs dynamic settings. Know which is which: Static (at creation, or on a closed index) Dynamic ( PUT /_settings any time) index.number_of_shards index.number_of_replicas index.codec (e.g. best_compression ) index.refresh_interval ( -1 disables refresh) index.mode ( standard , logsdb , time_series , lookup ) index.routing.allocation.* index.sort.* index.blocks.write / read_only / read analysis definitions (close → update → open) index.max_result_window , index.default_pipeline , index.final_pipeline PUT products/_settings { "index": { "number_of_replicas": 0, "refresh_interval": "-1" } } // bulk-load mode Adding an analyzer to an existing index: POST products/_close → PUT products/_settings with the analysis block → POST products/_open . You still can't change the analyzer of an existing field. Add a new field or multi-field, or reindex. Changing the shard count: _split : more shards, by a multiple. Needs index.blocks.write: true . _shrink : fewer shards, to a factor. Needs a write block and a copy of every shard on one node ( index.routing.allocation.require._name ). Otherwise _reindex . PUT products/_settings { "index.blocks.write": true } POST products/_split/products-4sh { "settings": { "index.number_of_shards": 4, "index.blocks.write": null } } dynamic mapping modes , a frequent grader trap: Value Unknown field in a document… true (default) Added to the mapping automatically runtime Added as a runtime field (not indexed, computed at query time) false Accepted and kept in _source , but not indexed or searchable strict Document rejected with an error Task phrasing to watch for: "unknown fields must be rejected" means strict ; "must be stored but not searchable" means false . Index modes (9.x). logsdb is the storage-optimised mode for logs, with synthetic _source and index sorting. It's the default for logs-*-* data streams in 9.x. time_series (TSDS) is for metrics. lookup is required for the right-hand index of an ES|QL LOOKUP JOIN . 📖 Create index API · Index settings reference · Index fundamentals · Dynamic mapping · Logs data streams (logsdb) 2.2 Index templates and component templates Composable index templates ( _index_template ), built from reusable component templates ( _component_template ), are the standard. Legacy _template templates have been deprecated since 7.8; composable ones always win over them. Analogy: Lego. Component templates are bricks: a "1 shard" brick, an "HTTP fields" brick. The index template is the instruction sheet saying which bricks to snap together for anything named web-* , plus a few extra pieces of its own. PUT _component_template/ct-settings-small { "template": { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } } } PUT _component_template/ct-mappings-http { "template": { "mappings": { "properties": { "http.status": { "type": "short" }, "url": { "type": "keyword" } } } } } PUT _index_template/web-template { "index_patterns": ["web-*"], "priority": 200, "composed_of": ["ct-settings-small", "ct-mappings-http"], "template": { "settings": { "number_of_replicas": 2 }, "aliases": { "web-all": {} } }, "_meta": { "owner": "exam" } } Precedence, from lowest to highest: component templates in composed_of order (later overrides earlier), then the index template's own template block, then settings in the actual create-index request. In the example, replicas end up at 2 . Priority rules: Only one index template applies to a new index: the matching one with the highest priority . Two templates with equal priority and overlapping patterns are rejected when you create the second. Built-in templates use priority 100 for logs-*-* , metrics-*-* , synthetics-*-* , profiling-* , security_solution-*-* , and the Streams patterns logs.otel* and logs.ecs* . Fleet integrations go up to 200. To deliberately override a built-in pattern, the docs recommend priority > 500 . Otherwise choose non-overlapping patterns. Verify before you commit: POST _index_template/_simulate_index/web-2026.09.28 // what would this index get? POST _index_template/_simulate/web-template // resolved template GET _index_template/web-template Templates apply only when an index is created. Changing a template never alters existing indices. For data streams, roll over to pick up changes. 📖 Templates · Simulate multi-component templates · Put index template API · Put component template API 2.3 Define and use a dynamic template that satisfies a given set of requirements Dynamic templates control how newly detected fields get mapped . You match on the JSON-detected type ( match_mapping_type ), the field name ( match / unmatch , wildcards or "match_pattern": "regex" ), or the dotted path ( path_match / path_unmatch ). Analogy. A mail-room rulebook: "anything that looks like a parcel and whose label ends in _id goes to shelf K; every other parcel goes to shelf T". The first rule that fits wins. PUT _index_template/sensor-template { "index_patterns": ["sensor-*"], "template": { "mappings": { "dynamic_templates": [ { "ids_as_keyword": { "match_mapping_type": "string", "match": "*_id", "mapping": { "type": "keyword" } } }, { "desc_english": { "match_mapping_type": "string", "match": "desc_*", "mapping": { "type": "text", "analyzer": "english" } } }, { "counts_as_integer": { "match_mapping_type": "long", "match": "*_count", "mapping": { "type": "integer" } } }, { "raw_not_indexed": { "path_match": "raw.*", "mapping": { "type": "keyword", "index": false, "doc_values": false } } }, { "strings_default": { "match_mapping_type": "string", "mapping": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } ] } } } Rules to remember: Order matters: first match wins. Put specific rules first and the catch-all ( strings_default ) last. A catch-all placed first silently shadows the others, which is a classic hidden bug in exam tasks. match_mapping_type values are JSON-detected types: string , long , double , boolean , date , object , binary , or * . There's no integer or float detection. Placeholders: {name} (field name) and {dynamic_type} (detected type), e.g. "analyzer": "{name}" . Use "runtime": { "type": "keyword" } instead of "mapping" to create runtime fields dynamically. path_match matches the full dotted path ( raw.a.b ); match matches only the leaf name . Dynamic templates apply to new fields only. Fields already mapped are untouched. Test it: index one sample document, then GET sensor-1/_mapping . 📖 Dynamic templates · Dynamic field mapping 2.4 Define an index template that creates a new data stream A data stream is one name (e.g. logs-app-prod ) in front of a series of hidden backing indices ( .ds-logs-app-prod-2026.09.28-000001 , …). It's for append-only, time-stamped data and requires an @timestamp field ( date or date_nanos ). Analogy: a notebook with detachable pages. You always write on the current page (the write index ). When a page fills up ( rollover ), a new page is added. You can read all pages at once, and you tear out and throw away old pages individually (ILM delete). Only a matching index template with "data_stream": {} creates a data stream. Without it, writing to logs-app-prod creates an ordinary index. PUT _index_template/logs-app-template { "index_patterns": ["logs-app-*"], "data_stream": { }, "priority": 501, "composed_of": [], "template": { "settings": { "index.lifecycle.name": "logs-app-policy" }, "mappings": { "properties": { "@timestamp": { "type": "date" }, "message": { "type": "text" }, "level": { "type": "keyword" } } }, "data_stream_options": { "failure_store": { "enabled": true } } } } // Created implicitly on first write… POST logs-app-prod/_doc { "@timestamp": "2026-09-28T10:00:00Z", "message": "started", "level": "INFO" } // …or explicitly PUT _data_stream/logs-app-prod Operations and constraints: Want to… Do Add documents POST /_doc , or _bulk with the create action. Data streams are append-only; index only works with op_type=create . Update or delete documents _update_by_query / _delete_by_query on the stream, or single-doc ops on the backing index with if_seq_no / if_primary_term Roll over now POST logs-app-prod/_rollover Inspect GET _data_stream/logs-app-prod , GET _data_stream/logs-app-prod/_stats Delete DELETE _data_stream/logs-app-prod (deletes its backing indices too) Change mappings or settings Update the template, then roll over. Some changes can also be applied to the stream via the update mapping API. Reindex into a stream "dest": { "index": "logs-app-prod", "op_type": "create" } Convert an alias with a write index into a stream POST _data_stream/_migrate/ Capture rejected documents Failure store ( data_stream_options.failure_store.enabled ). Query it with FROM logs-app-prod::failures or GET logs-app-prod::failures/_search . Naming scheme. Elastic's convention is type-dataset-namespace (e.g. logs-nginx.access-prod ). Any name works for your own templates, but anything matching logs-*-* hits the built-in template (priority 100) unless yours has higher priority. 📖 Data streams · Set up a data stream · Use a data stream · Modify a data stream · Failure store · Time series data streams 2.5 Define an Index Lifecycle Management policy for a time-series index ILM moves each index through phases : hot → warm → cold → frozen → delete . Each phase has a min_age (measured from rollover , or from creation if the index never rolled over) and a set of actions . Analogy: a restaurant kitchen's food rotation. Fresh produce is on the counter (hot). After a few days it moves to the walk-in fridge (warm), then the freezer (cold or frozen), and finally the bin (delete). The chef doesn't move things by hand; the rotation rules do. PUT _ilm/policy/logs-app-policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" }, "set_priority": { "priority": 100 } } }, "warm": { "min_age": "7d", "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 }, "allocate": { "number_of_replicas": 1 }, "set_priority": { "priority": 50 } } }, "cold": { "min_age": "30d", "actions": { "searchable_snapshot": { "snapshot_repository": "my-repo" } } }, "delete": { "min_age": "90d", "actions": { "delete": {} } } } } } Where each action is allowed: Action Hot Warm Cold Frozen Delete rollover ✓ set_priority ✓ ✓ ✓ unfollow (CCR) ✓ ✓ ✓ ✓ readonly ✓* ✓ ✓ shrink , forcemerge ✓* ✓ downsample (TSDS) ✓* ✓ ✓ allocate , migrate ✓ ✓ searchable_snapshot ✓* ✓ ✓ (required) wait_for_snapshot , delete ✓ * In hot, only when rollover is also configured. Key facts: Rollover conditions are OR-ed ( max_age , max_docs , max_size , max_primary_shard_size , max_primary_shard_docs ). min_* conditions are AND-ed floors. Elasticsearch never rolls over an empty index. Rollover needs a data stream or a write alias : index.lifecycle.rollover_alias plus a bootstrap index name-000001 with "is_write_index": true . migrate is implicit in warm and cold. It sets _tier_preference to the phase's tier. With no nodes of that tier, the shards stay where they are, which is a common "why isn't it moving?" moment. Disable it with "migrate": {"enabled": false} when using attribute-based allocate . ILM checks every indices.lifecycle.poll_interval ( 10m default). In a lab, set it to 10s to see transitions quickly. Attach a policy to new indices via the template ( index.lifecycle.name ). For existing indices, use PUT /_settings {"index.lifecycle.name": "…"} . Changing a policy: an index already in a phase keeps the cached phase definition and picks up the new one at the next phase. Bootstrap pattern without data streams (a classic task): PUT _index_template/orders-tpl { "index_patterns": ["orders-*"], "template": { "settings": { "index.lifecycle.name": "orders-policy", "index.lifecycle.rollover_alias": "orders" } } } PUT orders-000001 { "aliases": { "orders": { "is_write_index": true } } } Debugging ILM: GET logs-app-prod/_ilm/explain // phase, action, step, failed_step, step_info POST logs-app-prod/_ilm/retry // after fixing the cause of an ERROR step POST _ilm/move/ // force a step (rare) GET _ilm/status POST _ilm/start POST _ilm/stop PUT _cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "10s" } } Data stream lifecycle (DSL): the lighter alternative For data streams you can skip ILM and just declare retention (plus optional downsampling). Elasticsearch handles rollover and deletion itself. PUT _index_template/metrics-app-template { "index_patterns": ["metrics-app-*"], "data_stream": {}, "template": { "lifecycle": { "data_retention": "30d" } } } PUT _data_stream/metrics-app-prod/_lifecycle { "data_retention": "7d" } GET _data_stream/metrics-app-prod/_lifecycle/explain If both are configured, ILM wins ( index.lifecycle.prefer_ilm defaults to true ). Choose ILM when you need tiers (warm, cold, frozen), shrink, forcemerge or searchable snapshots. Choose DSL when retention is all that matters. 📖 Index lifecycle management · ILM phases and actions · Rollover · Create an ILM policy · Tutorial: time series with data streams · Tutorial: time series without data streams · Check ILM status · Data stream lifecycle 2.6 Define and use index aliases An alias is a secondary name for one or more indices or data streams. Analogy: a phone's contact name. You call "Mum", not the number. When Mum changes numbers, you edit the contact once and nobody else has to change anything. Apps query products , and you swap which physical index sits behind it. Core uses: Zero-downtime reindex: build products-v2 , then atomically move the alias. Filtered views: an alias that only shows region: EU documents. Rollover write target: is_write_index: true . Search many indices under one name: logs-all → logs-2026.09.* . // Atomic swap. All actions succeed or none do. POST _aliases { "actions": [ { "remove": { "index": "products-v1", "alias": "products" } }, { "add": { "index": "products-v2", "alias": "products", "is_write_index": true } } ] } // Filtered alias with routing POST _aliases { "actions": [ { "add": { "index": "orders", "alias": "orders-eu", "filter": { "term": { "region": "EU" } }, "routing": "eu" } } ] } // Remove an index entirely and point the alias elsewhere, atomically POST _aliases { "actions": [ { "remove_index": { "index": "products-old" } }, { "add": { "index": "products-v3", "alias": "products" } } ] } GET _alias/products GET products-v2/_alias GET _cat/aliases?v Rules: An alias pointing to several indices is read-only unless exactly one of them has is_write_index: true . Aliases can point to data streams , but you can't mix data streams and indices in one alias. Filtered aliases apply the filter to searches, counts and delete-by-query run through the alias . Aliases can be defined in index templates ( template.aliases ) so every new index joins automatically. An alias name can't equal an existing index name. 📖 Aliases · Aliases API · Privileges for data streams and aliases 2.7 Exercises: Data Management Create component templates ct-1shard (1 shard, 0 replicas) and ct-app-mappings ( @timestamp date, service keyword, msg text). Create index template app-logs for app-logs-* that creates a data stream , uses both components, and has priority 300. Index a document into app-logs-api and confirm the backing index has 1 shard. Add to (1) a dynamic template: new strings ending in _id become keyword ; all other new strings become text with a keyword sub-field named raw . Prove it with one document (remember to roll over). Create ILM policy app-policy : roll over at 1 GB primary shard size or 1 day; delete 3 days after rollover. Attach it via the template, roll over manually and check _ilm/explain . Index legacy has 5 shards. Produce legacy-1 with 1 shard without reindexing . Indices movies-v1 and movies-v2 exist. Alias movies points to v1. Atomically repoint it to v2 and make v2 the write index. Then create filtered alias movies-recent showing only year >= 2020 . Solution notes composed_of: ["ct-1shard","ct-app-mappings"] , data_stream: {} , priority: 300 . Check with GET _data_stream/app-logs-api , then GET .ds-app-logs-api-*/_settings?filter_path=**.number_of_shards . Put the *_id rule before the catch-all string rule. The catch-all uses "mapping": {"type":"text","fields":{"raw":{"type":"keyword"}}} . Existing backing indices don't change, so POST app-logs-api/_rollover before testing. Hot: rollover: {max_primary_shard_size: "1gb", max_age: "1d"} ; delete: min_age: "3d" . Add index.lifecycle.name: app-policy to the template's settings. Existing backing indices need PUT .ds-app-logs-api-*/_settings , or a rollover. PUT legacy/_settings {"index.blocks.write": true, "index.routing.allocation.require._name": "es01"} , wait for relocation, then POST legacy/_shrink/legacy-1 {"settings":{"index.number_of_shards":1,"index.routing.allocation.require._name":null,"index.blocks.write":null}} . One _aliases call with remove + add ( is_write_index: true ), then add with filter: {range: {year: {gte: 2020}}} . 3. Data Ingestion & Processing Analogy for the whole domain: a restaurant's back office. Deliveries arrive by different routes ( ingest methods ). Each crate is checked against the inventory sheet that says what goes on which shelf ( mapping and field types ). Some produce is washed, chopped and labelled before storage ( ingest pipelines / Streams ). When the shelving layout changes, the stock is moved to new shelves ( Reindex ) or relabelled in place ( Update By Query ). 3.1 Select and configure appropriate ingest methods Method Best for Processing Notes Elastic Agent + Fleet (integrations) Logs, metrics and security data from hosts and cloud services; centrally managed Integration's own ingest pipelines + @custom hooks One agent, many inputs; policies managed in Kibana. Writes to type-dataset-namespace data streams. Standalone Elastic Agent Same, without Fleet Same Configured by YAML Beats (Filebeat, Metricbeat…) Lightweight single-purpose shippers Beat processors + ingest pipelines Legacy but still supported EDOT / OpenTelemetry Collector , OTLP endpoint OpenTelemetry-native logs, metrics and traces Collector processors, Streams Elastic's direction of travel for observability Logstash Heavy transformation, many inputs and outputs, buffering (persistent queue), routing to several clusters Logstash filters (grok, mutate, translate…) Extra infrastructure to run Language clients + _bulk API Application data, product catalogues, custom ETL Optional pipeline= parameter Most control; you manage batching and retries Kibana file upload One-off CSV, NDJSON or log files Auto-generated pipeline and mapping Size-limited; good for quick labs Connectors / crawler Content sources (SharePoint, databases, websites) Connector sync rules + pipelines Search use cases Streams (Kibana) Logs, especially unstructured ones, managed from one UI Streams processors (see 3.6) 9.x feature _reindex from remote Migrating data between clusters Optional pipeline Needs reindex.remote.whitelist How to choose (exam-style reasoning): "Collect system logs from 200 hosts, centrally managed" → Elastic Agent with Fleet and the System integration. "Enrich events from a Kafka topic and send them to two clusters" → Logstash . "Index product data from our application" → client + _bulk , with an ingest pipeline for light cleanup. "Parse free-text logs quickly without writing grok by hand" → Streams . "Transform data already in Elasticsearch" → ingest pipeline + _reindex or _update_by_query . The Bulk API is the workhorse behind almost every method: POST _bulk?pipeline=clean-products { "index": { "_index": "products", "_id": "1" } } { "sku": "A-1", "name": "Café crème", "price": 3.5 } { "create": { "_index": "logs-app-prod" } } { "@timestamp": "2026-09-28T10:00:00Z", "message": "hello" } { "update": { "_index": "products", "_id": "1" } } { "doc": { "price": 3.9 } } { "delete": { "_index": "products", "_id": "2" } } The format is NDJSON: every line ends with a newline, including the last one. The response has a top-level errors: true/false , and each item carries its own status, so always check errors . Aim for batches of about 5–15 MB. 📖 Ingest overview · Ingest tools overview · Ingest architectures · Bulk API · Upload data files · OTLP endpoint · Tune for indexing speed 3.2 Define a mapping that satisfies a given set of requirements A mapping is the schema: which fields exist, their types, and how each is indexed and analysed. Analogy: a customs form. Each box says what the item is ("date", "exact code", "free text"). Customs (Lucene) stores it in the right way. A date written in the "free text" box can't be range-searched later, and you can't change the form once the parcel is shipped. Explicit vs dynamic. Dynamic mapping guesses, for example "2026-09-28" becomes date and "hello" becomes text + .keyword . The exam almost always wants explicit mappings. PUT articles { "mappings": { "dynamic": "strict", "_source": { "excludes": ["internal_notes"] }, "properties": { "title": { "type": "text", "analyzer": "english", "fields": { "raw": { "type": "keyword" }, "suggest": { "type": "search_as_you_type" } } }, "author": { "type": "keyword", "normalizer": "lowercase" }, "body": { "type": "text", "copy_to": "all_text" }, "summary": { "type": "text", "copy_to": "all_text" }, "all_text": { "type": "text" }, "views": { "type": "integer", "null_value": 0 }, "rating": { "type": "half_float", "ignore_malformed": true }, "published":{ "type": "date", "format": "strict_date_optional_time||epoch_millis" }, "status": { "type": "keyword", "doc_values": true, "index": true }, "raw_html": { "type": "text", "index": false }, "comments": { "type": "nested", "properties": { "user": { "type": "keyword" }, "text": { "type": "text" }, "stars": { "type": "byte" } } }, "labels": { "type": "flattened" }, "internal_notes": { "type": "text" } } } } Mapping parameters you must know: Parameter Meaning Typical task wording index: false Not searchable (still in _source ; aggregatable if it has doc values) "stored but not searchable" doc_values: false Can't sort or aggregate; saves disk "never used for aggregations" enabled: false (object only) Whole object kept in _source , not parsed at all "ignore the contents of payload " store: true Stored separately from _source (retrieve with stored_fields ) rare copy_to Copies the value into another field (a custom "all fields" field) "search title and body as one field" fields Multi-fields : same value indexed several ways "full-text and exact-match / sortable" analyzer / search_analyzer Index-time vs query-time analysis "autocomplete", "synonyms at search time" normalizer Analyzer for keyword (lowercase, asciifolding) "case-insensitive exact match" ignore_above Don't index keyword strings longer than N "skip values longer than 256" null_value Index a replacement for explicit null "treat null as 0 / NONE" ignore_malformed Skip bad values instead of rejecting the document "don't fail on bad numbers" coerce Convert "5" to 5 (default true) "reject strings in numeric fields" → coerce: false format Date formats ( yyyy-MM-dd , epoch_second …) "dates like 28/09/2026" → dd/MM/yyyy dynamic (per object) true / false / strict / runtime at any level "strict except in meta " What you can and can't change on an existing index: ✅ Allowed via PUT /_mapping ❌ Needs a new index + reindex Add new fields Change a field's type Add a multi-field to an existing field, then run _update_by_query to populate it Change a field's analyzer Change ignore_above , search_analyzer (with care), mapping _meta Change index , doc_values or nested vs object Add runtime fields Rename a field (use a field alias or reindex) PUT articles/_mapping { "properties": { "author": { "type": "keyword", "fields": { "text": { "type": "text" } } } } } POST articles/_update_by_query?conflicts=proceed // populates author.text for existing docs Text analysis: custom analyzers Analogy: a kitchen processing line. Character filters wash the vegetables (strip HTML, map characters). The tokenizer chops them into pieces (words). Token filters season each piece (lowercase, remove stop words, stem, add synonyms). The same line must be used for indexing and searching , or the pieces won't match. PUT recipes { "settings": { "analysis": { "char_filter": { "amp_to_and": { "type": "mapping", "mappings": ["& => and"] } }, "filter": { "my_stop": { "type": "stop", "stopwords": "_english_" }, "my_syn": { "type": "synonym_graph", "synonyms": ["courgette, zucchini", "aubergine, eggplant"] }, "edge_3_10":{ "type": "edge_ngram", "min_gram": 3, "max_gram": 10 } }, "analyzer": { "recipe_text": { "type": "custom", "char_filter": ["html_strip", "amp_to_and"], "tokenizer": "standard", "filter": ["lowercase", "asciifolding", "my_stop", "porter_stem"] }, "recipe_search": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "asciifolding", "my_syn", "porter_stem"] }, "autocomplete": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "edge_3_10"] } }, "normalizer": { "lower_fold": { "type": "custom", "filter": ["lowercase", "asciifolding"] } } } }, "mappings": { "properties": { "name": { "type": "text", "analyzer": "recipe_text", "search_analyzer": "recipe_search", "fields": { "ac": { "type": "text", "analyzer": "autocomplete", "search_analyzer": "standard" }, "raw": { "type": "keyword", "normalizer": "lower_fold" } } } } } } POST recipes/_analyze { "analyzer": "recipe_text", "text": "Courgette & Crème brûlée recipes" } POST recipes/_analyze { "field": "name.ac", "text": "Courgette" } Analysis facts to remember: The default analyzer is standard (standard tokenizer + lowercase). Built-in analyzers include simple , whitespace , keyword , stop , pattern and language analyzers ( english , french …). Synonyms at search time use synonym_graph in the search_analyzer . To update synonyms without reindexing, use the Synonyms API ( PUT _synonyms/my-set ) with "synonyms_set": "my-set" and "updateable": true on a search-time filter. Edge n-grams go at index time only; search with the standard analyzer, otherwise the query is n-grammed too. keyword fields are not analysed . Use a normalizer for case-insensitivity. Always test with _analyze before creating the real index. 📖 Mapping · Explicit mapping · Update mapping examples · Mapping parameters · Text analysis · Anatomy of an analyzer · Create a custom analyzer · Test an analyzer · Index and search analysis · Synonyms 3.3 Define and use appropriate field data types for indexing, search and aggregation Analogy. Pick the container for what you'll do with the contents. Soup goes in a bowl ( text , for searching within), coins in a counting tray ( numeric , for maths and ranges), keys on labelled hooks ( keyword , for exact lookup, sorting and grouping). Requirement Type Why Full-text search (words, relevance) text Analysed into tokens Exact match, sorting, terms aggregation (IDs, status, tags, emails) keyword Stored as-is, with doc values Both of the above text + fields.keyword ( multi-field ) Default for dynamic strings Same value in every document of an index constant_keyword Nearly free, great for data streams Grep-like *foo* on huge high-cardinality strings wildcard Fast wildcard and regexp Log messages, scoring not needed match_only_text Much smaller than text Integers byte / short / integer / long / unsigned_long Smallest type that fits Decimals float / half_float / double / scaled_float (prices: scaling_factor: 100 ) Dates and times date (ms) / date_nanos Range queries, date histograms True/false boolean IPv4/IPv6 and CIDR queries ip term: {"ip": "10.0.0.0/8"} works Lat/lon, distance, bounding box geo_point Geo queries and aggs Polygons, shapes geo_shape Numeric or date intervals integer_range , date_range , ip_range … "Is 42 inside the doc's range?" Arrays of objects queried together nested Keeps each object's fields bound together Simple JSON objects object (default) Flattened into dotted fields Unknown or unbounded keys (labels, tags maps) flattened One field, avoids mapping explosion Parent/child in the same index join Costly; use sparingly Semantic search (managed embeddings) semantic_text Inference handled for you (see 4.5) Your own embeddings dense_vector (kNN) / sparse_vector Autocomplete as you type search_as_you_type / completion Software versions ( 1.10 > 1.9 ) version Correct semantic ordering Pre-aggregated metrics histogram , aggregate_metric_double Another name for an existing field alias "Rename" without reindex (for search) Arrays. There's no array type. Any field can hold several values of the same type. object vs nested : the most tested pitfall // Document: { "comments": [ { "user": "alice", "stars": 5 }, { "user": "bob", "stars": 1 } ] } With object , Lucene stores comments.user: [alice, bob] and comments.stars: [5, 1] , so the pairing is lost. A search for "alice AND stars 1" wrongly matches. With nested , each comment is a hidden sub-document. You must use the nested query and the nested aggregation: GET articles/_search { "query": { "nested": { "path": "comments", "query": { "bool": { "must": [ { "term": { "comments.user": "alice" } }, { "range": { "comments.stars": { "gte": 4 } } } ] } }, "inner_hits": {} } } } The limits are index.mapping.nested_fields.limit (50) and index.mapping.nested_objects.limit (10,000 per document). Other mapping explosion guards: index.mapping.total_fields.limit ( 1000 ) and index.mapping.depth.limit (20). Runtime fields: schema on read A field computed at query time from _source or other fields. There's nothing to reindex, but it's slower. PUT articles/_mapping { "runtime": { "day_of_week": { "type": "keyword", "script": { "source": "emit(doc['published'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ENGLISH))" } } } } // Or only for one search: GET articles/_search { "runtime_mappings": { "views_k": { "type": "double", "script": "emit(doc['views'].value / 1000.0)" } }, "fields": ["views_k"], "aggs": { "by_day": { "terms": { "field": "day_of_week" } } } } 📖 Field data types · keyword · text · Numeric · date · nested · flattened · Multi-fields · Runtime fields · Mapping limit settings 3.4 Use the Reindex API and Update By Query API Analogy. Reindex is moving house: you pack every item from the old flat and unpack it into a new one that has a different layout (new mapping, new analyzer, new shard count). Update By Query is redecorating in place: same flat, you walk from room to room and change what matches. Reindex POST _reindex?wait_for_completion=false&slices=auto { "source": { "index": "articles", "query": { "range": { "published": { "gte": "2025-01-01" } } }, "_source": ["title", "author", "published", "views"], "size": 1000 }, "dest": { "index": "articles-v2", "pipeline": "normalize-articles", "op_type": "create" }, "script": { "lang": "painless", "source": "ctx._source.views = ctx._source.views == null ? 0 : ctx._source.views; ctx._source.remove('internal_notes')" }, "conflicts": "proceed", "max_docs": 100000 } Option Use source.query Copy only matching documents source._source Copy only some fields dest.pipeline Transform with an ingest pipeline on the way in script Painless on ctx._source (also ctx._id , ctx._index ; set ctx.op = "noop" to skip, or "delete" ) dest.op_type: create Only create missing documents; required when the destination is a data stream dest.version_type: external Keep newer versions already in the destination conflicts: proceed Count version conflicts instead of aborting max_docs Limit the copy slices=auto Parallelise requests_per_second Throttle (change later with _reindex//_rethrottle ) wait_for_completion=false Returns a task ID . Poll GET _tasks/ , cancel with POST _tasks//_cancel . source.remote Reindex from another cluster . The destination node needs reindex.remote.whitelist: "host:9200" in elasticsearch.yml . The destination is NOT created from the source's mapping. Create articles-v2 (or a template for it) first , otherwise dynamic mapping guesses the types. Zero-downtime migration recipe: create v2 with the new mapping → _reindex → verify _count → atomic alias swap (2.6) → delete v1 if asked. Update By Query POST articles/_update_by_query?conflicts=proceed&wait_for_completion=false { "query": { "term": { "status": "draft" } }, "script": { "lang": "painless", "source": "ctx._source.status = params.s; ctx._source.tags = ctx._source.tags ?: []; ctx._source.tags.add('migrated')", "params": { "s": "archived" } } } // Re-run all docs through a pipeline (e.g. to populate a new field) POST articles/_update_by_query?pipeline=add-reading-time { "query": { "bool": { "must_not": { "exists": { "field": "reading_time" } } } } } // Pick up a newly added multi-field for all docs POST articles/_update_by_query?refresh Related APIs: POST /_update/ with doc (partial), script , upsert or doc_as_upsert POST /_delete_by_query ?refresh to make results immediately visible Painless essentials: ctx._source.field , params.x , ctx._source.remove('f') , ctx._source.containsKey('f') , null-safe ?. and elvis ?: , ctx.op = 'noop' | 'delete' . In ingest pipelines, fields are on ctx.field directly (no _source ). 📖 Reindex API · Update by query API · Delete by query API · Update API · Migrate data using reindex · Painless · Update documents with scripts · Tasks API 3.5 Define and use an ingest pipeline that satisfies a given set of requirements An ingest pipeline is an ordered list of processors that runs on ingest nodes before a document is indexed. Analogy: an airport baggage belt. Each station (processor) scans, relabels or reroutes the bag. If a station jams, the bag is either sent to a lost-luggage office ( on_failure ) or the whole belt stops (default: the document is rejected). PUT _ingest/pipeline/web-logs { "description": "Parse Apache-style access logs", "processors": [ { "grok": { "field": "message", "patterns": ["%{IPORHOST:client.ip} %{USER:ident} %{USER:user.name} \\[%{HTTPDATE:ts}\\] \"%{WORD:http.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.status:int} (?:%{NUMBER:bytes:long}|-)"] } }, { "date": { "field": "ts", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"], "target_field": "@timestamp" } }, { "remove": { "field": ["ts", "ident"], "ignore_missing": true } }, { "set": { "field": "event.ingested", "value": "{{{_ingest.timestamp}}}" } }, { "lowercase": { "field": "http.method" } }, { "geoip": { "field": "client.ip", "target_field": "client.geo", "ignore_failure": true } }, { "user_agent": { "field": "user_agent.original", "ignore_missing": true } }, { "set": { "if": "ctx.http?.status != null && ctx.http.status >= 500", "field": "event.outcome", "value": "failure" } }, { "script": { "lang": "painless", "source": "ctx.bytes_kb = ctx.bytes == null ? 0 : ctx.bytes / 1024.0" } }, { "drop": { "if": "ctx.url?.original == '/healthz'" } } ], "on_failure": [ { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } }, { "set": { "field": "_index", "value": "failed-{{ _index }}" } } ] } Frequently used processors: Processor Purpose set , remove , rename , append , convert , lowercase / uppercase , trim , split , join , gsub Field manipulation grok , dissect , kv , csv , json Parse text into fields ( dissect is faster, for fixed delimiters; grok uses regex) date , date_index_name Parse dates; route to date-based index names geoip , user_agent , uri_parts , community_id , network_direction , registered_domain Enrichment enrich Lookup-join against another index inference Run an ML model (e.g. embeddings) script Painless for anything else pipeline Call another pipeline (modular pipelines) reroute Send a document to another data stream (dataset or namespace) foreach Apply a processor to each array element drop , fail Discard a document; raise a custom error fingerprint , redact Deduplication IDs; mask PII Common processor options: if (Painless condition, null-safe with ?. ), ignore_failure , ignore_missing , on_failure (per processor), tag , description . Error handling, in order of scope: Processor-level ignore_failure: true skips silently. Processor-level on_failure: [...] handles the error locally. Pipeline-level on_failure is the catch-all. Metadata: _ingest.on_failure_message , _ingest.on_failure_processor_type , _ingest.on_failure_processor_tag . Nothing catches it → the document is rejected. For data streams with a failure store , it goes to ::failures instead. Testing, which you should always do: POST _ingest/pipeline/web-logs/_simulate?verbose=true { "docs": [ { "_source": { "message": "1.2.3.4 - - [28/Sep/2026:10:00:00 +0000] \"GET /a HTTP/1.1\" 503 12" } } ] } // Simulate with the pipelines that WOULD run for an index (defaults, final, templates) POST _ingest/_simulate { "docs": [ { "_index": "web-2026", "_source": { "message": "…" } } ] } Ways to apply a pipeline: How Scope ?pipeline=name on index, bulk, reindex ( dest.pipeline ) or update-by-query One request index.default_pipeline Default for an index; overridden by ?pipeline= index.final_pipeline Always runs last, even if a request pipeline was given Index template setting Every new index or backing index pipeline processor Nested pipelines Integrations' @custom pipelines (e.g. logs-nginx.access@custom ) Customise Fleet integrations safely PUT web-2026/_settings { "index.default_pipeline": "web-logs" } Enrich processor: a lookup table at ingest time PUT users-ref/_doc/1 { "email": "a@x.com", "dept": "finance", "manager": "zoe" } PUT _enrich/policy/users-policy { "match": { "indices": "users-ref", "match_field": "email", "enrich_fields": ["dept", "manager"] } } POST _enrich/policy/users-policy/_execute // builds the .enrich-* index; re-run after the source changes PUT _ingest/pipeline/add-user-info { "processors": [ { "enrich": { "policy_name": "users-policy", "field": "user.email", "target_field": "user.info", "max_matches": 1, "ignore_missing": true } } ] } Policy types: match (exact term), range (value inside a range), geo_match (point in shape). An enrich policy can't be updated . Delete it and recreate it; you have to remove the processors that use it first. 📖 Ingest pipelines · Processor reference · Error handling · Example: parse logs · Data enrichment · Set up an enrich processor · Simulate pipeline API · Grok debugger · Painless in ingest 3.6 Ingest and process data using Streams Streams is Kibana's UI and API layer for managing log data end to end: getting data in, parsing it, routing it, mapping fields, setting retention and watching data quality. Behind the scenes it creates and manages ordinary Elasticsearch objects (data streams, index templates, ingest pipelines, lifecycle settings). Analogy: a smart sorting office with a control panel. Instead of wiring each conveyor belt (template, pipeline, ILM) yourself, you use one panel: "parse these letters like this , send the ones from Paris to that shelf, keep them for 30 days". The panel builds the belts for you. Don't rewire the belts by hand behind its back. Two kinds of streams (9.2–9.3) Classic streams Wired streams (tech preview) What Your existing data streams (e.g. logs-nginx.access-default ), adopted in place A managed hierarchy rooted at a special endpoint Migration None; data keeps flowing as before Send data to the root stream: logs in 9.2–9.3 ( logs.otel / logs.ecs from 9.4) Processing ✓ (added without touching the underlying index template) ✓ Retention ✓ (inherit from template, custom period, or ILM policy) ✓ (inherit from parent, custom, or ILM) Partitioning (routing into child streams) ✗ ✓, e.g. logs → logs.nginx → logs.nginx.errors Field mapping (Schema) ✓ (add mappings) ✓ (children inherit from the parent) Inheritance n/a Children inherit mappings, lifecycle and processing from their parent A third kind, query streams (virtual, defined by an ES|QL query), appears in newer versions. Typical workflow in the Streams UI Get data in. For wired streams, turn on wired streams in Streams settings, then point Elastic Agent, Filebeat, the OTel Collector or Logstash at the logs index/stream. No template setup is needed. For classic streams, just open an existing data stream in Streams. Partition (wired only). Define routing conditions such as service.name == "nginx" → logs.nginx . Partition by logical groups (team, technology, environment), not by high-cardinality fields. Aim for tens of partitions, not hundreds. Each child is a real data stream. Process. Build a processing pipeline in the Processing tab from processors that mirror ingest processors (grok, dissect, date, set, rename, convert, remove, append, drop, math, json extract, redact, user agent, URI parts, enrich…). Add conditions to steps. The Data preview simulates results on sample documents before you save. With a generative-AI connector configured, Suggest a pipeline can generate grok/dissect rules for you. Map fields ( Schema tab). Promote extracted fields to proper mapped types (keyword, date, long…). Configure retention ( Data lifecycle tab). Choose inherit, a custom retention period (data stream lifecycle), or an existing ILM policy. Monitor data quality ( Data quality tab). Degraded documents are ones with ignored or malformed fields ( _ignored ); failed documents are kept in the failure store so you can fix the processor using the real failing documents. Advanced tab. Inspect the managed index templates, component templates and pipelines Streams created. Rules to remember: Streams-managed components are managed . Change them through the Streams UI or API, not with raw Elasticsearch APIs. Streams is a Kibana feature: its API is under kbn:/api/streams/... (Kibana API), not the Elasticsearch REST API. In Console, prefix with kbn: . Required privileges to manage streams include cluster manage_index_templates , manage_ingest_pipelines , manage_pipeline and read_pipeline , plus data-stream-level read , write , create , manage , monitor , manage_data_stream_lifecycle , read_failure_store , manage_failure_store and manage_ilm . Don't reroute or migrate existing data streams into the wired root; classic streams work in place. Because Streams is new and moving fast, exam tasks are most likely to use the UI flow: add a processor, verify in preview, set retention, check the failure store. Practise it on your lab with sample logs. # Example Kibana API calls from Console (verify paths against the 9.3 Streams API reference) GET kbn:/api/streams GET kbn:/api/streams/logs 📖 Streams overview · Get data into Streams · Wired streams · Partition (organise) data · Process documents · Configure data lifecycle · Map fields · Manage data quality · Advanced settings · Streamlang processors · Streams Kibana API 3.7 Exercises: Ingestion & Processing Create index shop with: name (full-text with the English analyzer, plus a case-insensitive exact sub-field name.exact ), sku (exact), price (two decimals, cheap to store), tags (exact, not aggregatable), specs (arbitrary key/values, no mapping explosion), reviews (array of {user, stars} that must be queried together). Unknown fields must be rejected. Create pipeline shop-clean that trims and upper-cases sku , converts price from string to float, sets ingested_at , drops docs whose status is test , and on any failure writes the error to ingest_error instead of rejecting. Simulate it with 2 docs. Reindex shop into shop-v2 (identical mapping except name uses a custom analyzer with asciifolding ) through shop-clean , only for docs with price > 0 . Then point alias shop-live at shop-v2 . Add a name.ac autocomplete multi-field to shop-v2 and populate it for existing docs without reindexing. Create an enrich policy from index suppliers ( sku → supplier , country ) and add it to shop-clean . In Streams, open the logs-generic-default classic stream (index some raw log lines first), add a dissect processor that extracts level and msg , map level as keyword, set retention to 14 days, and inspect the failure store. Solution notes dynamic: strict ; name: text/english + fields.exact: keyword with a lowercase normalizer; sku: keyword ; price: scaled_float, scaling_factor: 100 ; tags: keyword, doc_values: false ; specs: flattened ; reviews: nested . Processors: trim + uppercase on sku ; convert {type: float} on price ; set ingested_at = {{{_ingest.timestamp}}} ; drop with if: "ctx.status == 'test'" . Pipeline-level on_failure sets ingest_error = {{ _ingest.on_failure_message }} . Verify with _simulate?verbose . Create shop-v2 first . _reindex with source.query: range price gt 0 and dest.pipeline: shop-clean . Compare _count , then use _aliases add. PUT shop-v2/_mapping adding fields.ac with an edge-ngram analyzer. The analyzer must already exist in the index settings; if not, close → add analysis → open. Then POST shop-v2/_update_by_query . PUT _enrich/policy/... (match), _execute , then add an enrich processor to the pipeline. Re- PUT the whole pipeline; the PUT replaces it. The UI flow covered in 3.6. Check the result on the Advanced tab and in GET logs-generic-default/_mapping . 4. Searching Data Analogy for the whole domain: a library. Full-text queries are asking the librarian "books about sailing". She interprets the words and ranks the answers by relevance. Term-level queries are the card catalogue: exact codes, no interpretation. Bool combines requests: "must be about sailing, must be in French, should be recent, must not be a children's book". Aggregations are the statistics desk: "how many books per genre, average page count". ES|QL is a spreadsheet-style pipeline over the same shelves. Semantic search is a librarian who understands meaning ("boat trips" finds "sailing"). Cross-cluster search asks sister libraries in other cities at the same time. 4.1 Search for terms and/or phrases in one or more fields Full-text queries (analysed: the query text goes through the field's analyzer) // Single field, any word (OR by default) GET articles/_search { "query": { "match": { "title": "elastic search guide" } } } // All words required, or a minimum { "query": { "match": { "title": { "query": "elastic search guide", "operator": "and" } } } } { "query": { "match": { "title": { "query": "elastic search guide", "minimum_should_match": "2" } } } } // Typo tolerance { "query": { "match": { "title": { "query": "elastik", "fuzziness": "AUTO" } } } } // Exact phrase, with slop (words may be up to N positions apart) { "query": { "match_phrase": { "body": { "query": "shard allocation", "slop": 2 } } } } // Phrase prefix, "search as you type" style { "query": { "match_phrase_prefix": { "title": "quick bro" } } } // Several fields, with boosting { "query": { "multi_match": { "query": "shard sizing", "fields": ["title^3", "summary^2", "body"], "type": "best_fields", "tie_breaker": 0.3 } } } multi_match types: Type Behaviour Use when best_fields (default) Score of the best-matching field (+ tie_breaker × others) Words expected together in one field most_fields Sum of all field scores Same text indexed several ways (multi-fields) cross_fields Treats the fields as one big field (term-centric) "first_name + last_name" style data phrase / phrase_prefix match_phrase on each field Phrases across fields bool_prefix For search_as_you_type fields Autocomplete Also know combined_fields (like cross_fields for fields sharing an analyzer), query_string (Lucene syntax with AND/OR/NOT , wildcards and field names; throws on bad syntax) and simple_query_string (forgiving, safe for user input: + | - " * ( ) ). Term-level queries (NOT analysed: exact values on keyword, numeric, date, ip, boolean) { "query": { "term": { "status": "published" } } } { "query": { "term": { "status": { "value": "Published", "case_insensitive": true } } } } { "query": { "terms": { "tags": ["elasticsearch", "lucene"] } } } { "query": { "range": { "published": { "gte": "now-30d/d", "lt": "now/d" } } } } { "query": { "range": { "price": { "gt": 10, "lte": 100 } } } } { "query": { "exists": { "field": "author" } } } { "query": { "prefix": { "sku": "ABC-" } } } { "query": { "wildcard": { "sku": { "value": "A*-9?" } } } } { "query": { "regexp": { "sku": "A[0-9]{3}" } } } { "query": { "fuzzy": { "author": { "value": "jonh", "fuzziness": 1 } } } } { "query": { "ids": { "values": ["1", "7"] } } } { "query": { "terms_set": { "skills": { "terms": ["java","go","rust"], "minimum_should_match_field": "required" } } } } The number-one trap: term on a text field. "Published" was indexed as published , so term: "Published" finds nothing. Use match on text, or term on the .keyword sub-field. Date math: now-1d/d (rounded to the day), 2026-09-01||+1M , plus a time_zone parameter on range queries. Shaping results: pagination, sorting, source filtering, highlighting GET articles/_search { "from": 0, "size": 10, "track_total_hits": true, "_source": ["title", "published"], "fields": ["title", "published", "day_of_week"], "sort": [ { "published": "desc" }, { "title.raw": "asc" }, "_score" ], "highlight": { "pre_tags": [""], "post_tags": [""], "fields": { "body": { "fragment_size": 150, "number_of_fragments": 3 } } }, "query": { "match": { "body": "shard allocation" } } } from + size is capped by index.max_result_window ( 10,000 ). For deep paging use search_after with a point in time ( POST /_pit?keep_alive=1m ). Scroll is legacy for this purpose. You can't sort on text fields. Sort on a keyword sub-field, a number or a date. track_total_hits defaults to accurate counting up to 10,000; set true for an exact count. explain: true shows the score calculation. profile: true shows per-shard timings (or use Kibana's Search Profiler ). 📖 Full-text queries · match · match_phrase · multi_match · query_string · Term-level queries · term · range · Paginate results · Sort results · Highlighting · The _search API 4.2 Boolean combinations of queries and filters GET articles/_search { "query": { "bool": { "must": [ { "match": { "body": "elasticsearch" } } ], "should": [ { "match_phrase": { "title": { "query": "certified engineer", "boost": 2 } } }, { "term": { "tags": "exam" } } ], "minimum_should_match": 1, "filter": [ { "term": { "status": "published" } }, { "range": { "published": { "gte": "2025-01-01" } } } ], "must_not": [ { "term": { "author": "spam-bot" } } ] } } } Clause Must match? Affects score? Cached? must Yes Yes No filter Yes No (filter context) Yes should See below Yes (adds to the score) No must_not Must not match No (filter context) Yes should semantics (subtle, and heavily tested): If the bool has no must or filter , at least one should must match ( minimum_should_match defaults to 1). If there is a must or filter , should clauses are optional and only boost the score, unless you set minimum_should_match . Query vs filter context rule of thumb: "how well does it match?" (words, relevance) goes in must / should . "Does it match, yes or no?" (status, dates, ranges, IDs) goes in filter / must_not . Filters are faster and cached. Other compound queries: constant_score : wrap a filter and give every hit the same score. boosting : positive query, with a negative query that demotes (not excludes) matches. dis_max : best of several queries, plus tie_breaker . function_score / script_score : modify scores by recency, popularity ( field_value_factor ), decay functions or scripts. Named queries ( "_name": "q1" ) report which clauses matched each hit. Nesting example: "(title contains X OR summary contains X) AND year ≥ 2020 AND NOT draft": { "query": { "bool": { "must": [ { "bool": { "should": [ { "match": { "title": "X" } }, { "match": { "summary": "X" } } ] } } ], "filter": [ { "range": { "year": { "gte": 2020 } } } ], "must_not": [ { "term": { "status": "draft" } } ] } } } post_filter filters the hits but not the aggregations. Use it for faceted navigation, where facet counts shouldn't shrink when a facet is selected. 📖 Query and filter context · bool query · Compound queries · function_score · Filter search results (post_filter) 4.3 Write and execute metric and bucket aggregations Analogy: sorting sweets. Bucket aggregations tip the sweets into bowls by colour, by size range, or by the day they were bought. Metric aggregations count, weigh or average what's in each bowl. Pipeline aggregations compare the bowls ("which bowl is heaviest?", "running total"). Bowls can hold smaller bowls ( sub-aggregations ). GET kibana_sample_data_ecommerce/_search { "size": 0, "query": { "range": { "order_date": { "gte": "now-30d/d" } } }, "aggs": { "per_day": { "date_histogram": { "field": "order_date", "calendar_interval": "day", "time_zone": "Europe/Paris", "min_doc_count": 0, "extended_bounds": { "min": "now-30d/d", "max": "now/d" } }, "aggs": { "revenue": { "sum": { "field": "taxful_total_price" } }, "unique_buyers": { "cardinality": { "field": "customer_id" } } } }, "top_categories": { "terms": { "field": "category.keyword", "size": 5, "order": { "revenue": "desc" } }, "aggs": { "revenue": { "sum": { "field": "taxful_total_price" } }, "price_stats": { "stats": { "field": "taxful_total_price" } }, "p95": { "percentiles": { "field": "taxful_total_price", "percents": [50, 95, 99] } }, "best_order": { "top_hits": { "size": 1, "sort": [{ "taxful_total_price": "desc" }], "_source": ["order_id", "customer_full_name"] } } } }, "price_bands": { "range": { "field": "taxful_total_price", "ranges": [ { "to": 50 }, { "from": 50, "to": 150 }, { "from": 150 } ] } }, "best_day": { "max_bucket": { "buckets_path": "per_day>revenue" } }, "avg_daily_revenue": { "avg_bucket": { "buckets_path": "per_day>revenue" } } } } Metric aggregations: avg , sum , min , max , value_count , stats , extended_stats (std deviation, variance), cardinality (approximate distinct count, HyperLogLog++, precision_threshold ), percentiles , percentile_ranks , median_absolute_deviation , weighted_avg , top_hits , top_metrics , geo_bounds , geo_centroid , scripted_metric . Bucket aggregations: Aggregation Groups by Key options terms Distinct values (keyword, number, ip…) size (default 10), order ( _count , _key or a sub-agg), min_doc_count , missing , include / exclude , shard_size multi_terms Combinations of several fields terms: [{field}, {field}] rare_terms , significant_terms Rare or unusually frequent terms histogram Numeric intervals interval , min_doc_count , extended_bounds date_histogram Time intervals calendar_interval ( minute , hour , day , week , month , quarter , year , or 1d , 1M ) vs fixed_interval ( 30m , 7d ; multiples allowed); time_zone , format , offset range / date_range / ip_range Custom ranges ( from inclusive, to exclusive) keyed: true filter / filters One or several query-defined buckets other_bucket composite Paginated combinations of sources after key: iterate all buckets nested / reverse_nested Enter or leave nested documents path global Ignores the query scope missing Documents without the field geohash_grid , geotile_grid , auto_date_histogram , variable_width_histogram , sampler Specialised Pipeline aggregations reference other aggregations by buckets_path ( > for nesting, . for a metric's value, _count for doc count): Sibling (a new bucket next to the others): avg_bucket , max_bucket , min_bucket , sum_bucket , stats_bucket , percentiles_bucket . Parent (inside each histogram bucket): derivative , cumulative_sum , moving_fn , bucket_script , bucket_selector (keeps only matching buckets, like HAVING), bucket_sort (sort or truncate buckets), serial_diff . "aggs": { "per_month": { "date_histogram": { "field": "order_date", "calendar_interval": "month" }, "aggs": { "revenue": { "sum": { "field": "taxful_total_price" } }, "cum_rev": { "cumulative_sum": { "buckets_path": "revenue" } }, "big_only": { "bucket_selector": { "buckets_path": { "r": "revenue" }, "script": "params.r > 10000" } } } } } Aggregation facts: Set "size": 0 when you only need aggregations. You can't run terms on a text field (it has no doc values). Use .keyword . terms counts are approximate across shards ( doc_count_error_upper_bound ). Increase shard_size for accuracy. Use composite to list all buckets. Aggregations run on the documents matched by query . Use global to escape that scope, or post_filter to filter hits only. For nested fields, wrap sub-aggs in a nested aggregation. 📖 Aggregations · Aggregations tutorial (eCommerce) · Metric aggregations · Bucket aggregations · Pipeline aggregations · terms · date_histogram · composite 4.4 Write and execute ES|QL queries ES|QL (Elasticsearch Query Language) is a piped language. Each command transforms a table and passes it to the next, like a Unix pipeline or a spreadsheet you keep refining. Analogy: a kitchen assembly line. FROM grabs the ingredients, WHERE throws away what you don't need, EVAL chops new pieces, STATS weighs and groups, SORT arranges the plates, LIMIT serves the first N. POST _query?format=txt { "query": """ FROM kibana_sample_data_logs | WHERE @timestamp >= NOW() - 7 days AND response.keyword != "200" | EVAL kb = bytes / 1024.0, day = DATE_TRUNC(1 day, @timestamp) | STATS errors = COUNT(*), avg_kb = AVG(kb), hosts = COUNT_DISTINCT(host.keyword) BY day, response.keyword | SORT day DESC, errors DESC | LIMIT 20 """ } format can be json (default), txt , csv , tsv , yaml and more. For long queries use POST _query/async and poll with GET _query/async/ . In Kibana, use Discover → ES|QL mode . Commands you must know: Type Command Example Source FROM FROM logs-*, remote:logs-* METADATA _index, _id, _score Source ROW , SHOW INFO ROW a = 1, b = "x" Source TS (time series, newer) TS metrics-* | STATS … Filter WHERE WHERE status IN ("500","503") AND bytes > 1000 Compute EVAL EVAL mb = bytes / 1048576 Aggregate STATS … BY STATS c = COUNT(*), p95 = PERCENTILE(latency, 95) BY service Shape KEEP , DROP , RENAME KEEP @timestamp, host, message Order SORT , LIMIT SORT c DESC NULLS LAST | LIMIT 10 Parse DISSECT , GROK DISSECT message "%{ip} - %{user} [%{ts}]" Enrich ENRICH ENRICH users-policy ON email WITH dept Join LOOKUP JOIN LOOKUP JOIN hosts-lookup ON host.name (the right-hand index must use index.mode: lookup ) Arrays MV_EXPAND Turn a multi-value column into rows Analytics CHANGE_POINT , SAMPLE , FORK , RERANK , COMPLETION Newer commands; know they exist Functions to know: aggregations COUNT , COUNT_DISTINCT , SUM , AVG , MIN , MAX , MEDIAN , PERCENTILE , VALUES , TOP ; grouping BUCKET(@timestamp, 1 hour) ; date DATE_TRUNC , DATE_EXTRACT , DATE_DIFF , DATE_FORMAT , NOW() ; string CONCAT , TO_LOWER , SUBSTRING , LENGTH , SPLIT , STARTS_WITH , LIKE , RLIKE ; conditionals CASE(cond, a, cond2, b, default) , COALESCE ; conversion TO_INTEGER , TO_DATETIME , TO_IP , :: casts; multi-value MV_COUNT , MV_AVG ; full-text MATCH(field, "text") , the : operator ( message:"timeout" ), QSTR("…") , KQL("…") , plus METADATA _score for relevance. ES|QL facts: Default LIMIT is 1000 rows , with a maximum of 10,000 by default. Always add an explicit LIMIT . Case matters for values , not for command names. Use parameters to avoid injection: "query": "FROM x | WHERE a == ?v", "params": [{"v": 5}] . Cross-cluster: FROM cluster_a:logs-*, logs-* (requires remote clusters, see 5.2). ES|QL doesn't return _source documents like _search ; it returns columns and rows . STATS without BY returns one row. Columns not in BY or not aggregated are dropped. Query DSL → ES|QL translation practice: Query DSL ES|QL bool.filter.term status: published WHERE status == "published" range price gte 10 lt 100 WHERE price >= 10 AND price < 100 terms aggregation on category STATS c = COUNT(*) BY category | SORT c DESC | LIMIT 10 date_histogram 1d + sum STATS s = SUM(x) BY d = BUCKET(@timestamp, 1 day) match body: "timeout" WHERE MATCH(body, "timeout") or WHERE body:"timeout" 📖 ES|QL overview · ES|QL reference · ES|QL source commands · ES|QL processing commands · ES|QL functions and operators · LOOKUP JOIN · ES|QL REST API · ES|QL for search · Using ES|QL in Discover 4.5 Implement semantic search Semantic search matches meaning rather than exact words. Text is turned into embeddings (vectors) by an ML model, and queries are compared to documents in that vector space. Analogy: a map of meaning. Each passage becomes a pin on a huge map where similar ideas sit close together. "How do I fix a flat tyre?" lands next to "puncture repair". Keyword search would miss that, because none of the words overlap. Two kinds of embeddings: Sparse (e.g. ELSER , Elastic's learned sparse encoder) Dense (e.g. E5, Jina, OpenAI, Cohere) Representation Weighted tokens (expanded vocabulary) Fixed-length float vector Field type sparse_vector dense_vector Query sparse_vector query knn (HNSW approximate nearest neighbour) Strength Great out-of-domain English, explainable Multilingual, multimodal, many models The easy path: semantic_text (recommended and most exam-friendly) semantic_text hides all the plumbing: chunking, calling the inference endpoint at ingest time, storing embeddings, and embedding the query at search time. Step 1: an inference endpoint. There's a preconfigured default ELSER endpoint , so this step is optional. To create your own: PUT _inference/sparse_embedding/my-elser { "service": "elasticsearch", "service_settings": { "model_id": ".elser_model_2", "adaptive_allocations": { "enabled": true, "min_number_of_allocations": 0, "max_number_of_allocations": 4 }, "num_threads": 1 } } // Dense example with a built-in multilingual model PUT _inference/text_embedding/my-e5 { "service": "elasticsearch", "service_settings": { "model_id": ".multilingual-e5-small", "num_allocations": 1, "num_threads": 1 } } GET _inference/_all POST _inference/sparse_embedding/my-elser { "input": "test sentence" } The cluster needs an ML node (or Elastic Inference Service on Cloud) and a license that allows ML models, so use the trial on your lab. Step 2: a mapping with semantic_text : PUT docs-semantic { "mappings": { "properties": { "title": { "type": "text" }, "content": { "type": "semantic_text", "inference_id": "my-elser" }, "body": { "type": "text", "copy_to": "content" } } } } Omit inference_id to use the default endpoint. copy_to into a semantic_text field lets you keep a normal text field for BM25 alongside. Step 3: index documents normally. Embeddings are generated automatically. Bulk-loading many documents is slow on small ML nodes, so be patient or use a small batch. Step 4: query. // Simplest: the match query works on semantic_text fields GET docs-semantic/_search { "query": { "match": { "content": "how do I keep my cluster available when a zone fails?" } } } // Dedicated semantic query GET docs-semantic/_search { "query": { "semantic": { "field": "content", "query": "zone failure resilience" } }, "highlight": { "fields": { "content": { "type": "semantic", "number_of_fragments": 2 } } } } Hybrid search: BM25 + semantic, fused with RRF GET docs-semantic/_search { "retriever": { "rrf": { "retrievers": [ { "standard": { "query": { "match": { "title": "zone failure" } } } }, { "standard": { "query": { "semantic": { "field": "content", "query": "zone failure" } } } } ], "rank_window_size": 50, "rank_constant": 20 } } } Retrievers are the composable search abstraction: standard , knn , rrf (Reciprocal Rank Fusion), linear (weighted score combination), text_similarity_reranker (semantic reranking), rule . The manual path (know it exists) dense_vector mapping ( dims , similarity: cosine | dot_product | l2_norm | max_inner_product , index_options such as int8_hnsw / bbq_hnsw quantisation) + knn search ( k , num_candidates , filter , query_vector or query_vector_builder ). An inference processor in an ingest pipeline to generate embeddings, or bring your own vectors. sparse_vector field + sparse_vector query with inference_id . GET my-vectors/_search { "knn": { "field": "embedding", "query_vector_builder": { "text_embedding": { "model_id": "my-e5", "model_text": "zone failure" } }, "k": 10, "num_candidates": 100, "filter": { "term": { "lang": "en" } } } } 📖 Semantic search overview · Semantic search with semantic_text · semantic_text field · semantic query · Semantic search with the inference API · Inference API: default endpoints · ELSER · Hybrid search with semantic_text · Retrievers · kNN search · Dense vector · Sparse vector · Semantic reranking 4.6 Write and execute a query that searches across multiple clusters Cross-cluster search (CCS) runs one request from the local cluster against indices on one or more remote clusters. Setting up the remote connection is covered in 5.2; here is the querying side. Analogy: interlibrary loan. You ask your local branch; it forwards your request to the branches in Lyon and Berlin and hands you one merged answer. // Syntax: : GET cluster_b:logs-*/_search { "query": { "match": { "message": "timeout" } } } // Local + several remotes in one request GET logs-*,cluster_b:logs-*,cluster_c:logs-*/_search { "size": 20, "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now-1h" } } } ] } }, "aggs": { "by_cluster": { "terms": { "field": "_index" } } } } // All remotes: wildcard on the alias; exclude with a minus sign GET *:logs-*,-cluster_c:*/_search // ES|QL across clusters POST _query?format=txt { "query": "FROM logs-*, cluster_b:logs-* METADATA _index | STATS c = COUNT(*) BY _index | LIMIT 50" } // Async CCS for long searches POST cluster_b:logs-*/_async_search?wait_for_completion_timeout=2s { "query": { "match_all": {} } } // Check reachability and matching indices before searching GET _resolve/cluster/cluster_b:logs-*,cluster_c:logs-* What to know: Hits show the origin in _index as cluster_b:logs-2026… . The response's _clusters section reports total , successful , skipped , partial and failed . skip_unavailable (per remote cluster setting, default true ): an unreachable remote is skipped instead of failing the whole search. ccs_minimize_roundtrips (default true ): each remote reduces its own results, which means fewer network round trips. Security: the user needs read privileges on the remote indices. With the API-key model, that's the cross-cluster API key's access.search plus the local user's role ( remote_indices ). With the certificate model, it's a role of the same name on the remote cluster. Kibana data views can use cluster_b:logs-* patterns too. CCS works across versions within the compatibility window (see the CCS version compatibility table in the docs). 📖 Cross-cluster search · Resolve cluster before CCS · ES|QL cross-cluster · Async search 4.7 Exercises: Searching Data Use kibana_sample_data_ecommerce , kibana_sample_data_flights and kibana_sample_data_logs . Check field names and types first with GET /_mapping , since some sample fields are text + .keyword and others are plain keyword . Flights: find flights whose OriginCityName or DestCityName contains "New York" (both fields, boost the destination ×2), that are not cancelled, with FlightDelayMin between 30 and 120 and a timestamp in the last 30 days. Return only FlightNum , Carrier , FlightDelayMin , sorted by delay descending. eCommerce: find orders whose products.product_name contains the exact phrase "leather jacket" (slop 1). Highlight the matches. eCommerce: revenue per week ( taxful_total_price ) for the last 8 weeks, with the week-over-week difference and a running total; keep only weeks above 5,000. eCommerce: the top 5 manufacturer.keyword values by number of orders, each with average basket value and the 3 most frequent category.keyword values. Logs, in ES|QL: per host, the count of 4xx and 5xx responses in the last 24 hours, the 95th-percentile bytes , sorted by error count, top 10. Create a semantic_text index faq , index 5 short Q&A documents, and run a hybrid RRF query combining BM25 on question with semantic search on answer . Across clusters: count documents per index on lab and remote for logs-* in one request, and make the search succeed even if remote is down. Solution notes bool.should with match_phrase on both city fields ( boost: 2 on Dest) and minimum_should_match: 1 . filter : term Cancelled: false , range FlightDelayMin gte 30 lte 120 , range timestamp gte now-30d/d . Add _source and sort . match_phrase with slop: 1 on products.product_name , plus highlight.fields . date_histogram with calendar_interval: week and extended_bounds , then sum → derivative → cumulative_sum → bucket_selector (params.r > 5000) . The query uses range order_date gte now-8w/w . terms manufacturer.keyword size 5 → avg taxful_total_price + terms category.keyword size 3 . FROM kibana_sample_data_logs | WHERE @timestamp > NOW() - 24 hours | EVAL code = TO_INTEGER(response.keyword) | WHERE code >= 400 | STATS errors = COUNT(*), p95 = PERCENTILE(bytes, 95) BY host.keyword | SORT errors DESC | LIMIT 10 Mapping answer: semantic_text , question: text . Use an rrf retriever with a match on question and a semantic query on answer. GET logs-*,remote:logs-*/_search {"size":0,"aggs":{"i":{"terms":{"field":"_index"}}}} , with cluster.remote.remote.skip_unavailable: true (the default). 5. Cluster Management 5.1 Monitor and maintain cluster health Analogy: a hospital ward. Green means every patient (shard) has a bed and a backup nurse (replica). Yellow means every patient has a bed, but some backup nurses are missing: no data lost, but less safe. Red means at least one patient has no bed at all: a primary shard is unassigned and its data is unavailable. The allocation explain API is the doctor's chart that says why . The monitoring toolkit GET _cluster/health // status, nodes, unassigned_shards, relocating… GET _cluster/health/orders?level=shards // per index or shard GET _cluster/health?wait_for_status=green&timeout=30s GET _health_report // 9.x health indicators with diagnoses & remediation GET _health_report/shards_availability GET _cat/health?v GET _cat/nodes?v&h=name,node.role,master,heap.percent,ram.percent,cpu,load_1m,disk.used_percent GET _cat/indices?v&health=yellow&s=index GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node&s=state GET _cat/allocation?v // disk per node, shard counts GET _cat/recovery?v&active_only=true GET _cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected GET _cat/pending_tasks?v GET _nodes/stats/jvm,fs,indices,breaker GET _cluster/stats?human GET _tasks?actions=*reindex*&detailed GET _cluster/allocation/explain // first unassigned shard, or specify one 📖 Cluster health API · Health report API · Cluster allocation explain API · cat APIs · Troubleshoot Diagnosing yellow and red: the decision tree GET _cluster/health : which status, and how many unassigned_shards ? GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED (in Console, use &s=state ). GET _cluster/allocation/explain with the index, shard and primary flag. Read allocate_explanation and each node's deciders . Fix according to the cause: Symptom or explanation Cause Fix Replica unassigned, "a copy of this shard is already allocated to this node" Single-node cluster or too few nodes Add nodes or number_of_replicas: 0 (only if the task allows it) awareness decider says NO Forced awareness with a zone missing Expected. Restore the zone, or relax the awareness values. filter decider says NO require / include / exclude rule matches no node Fix or clear the index.routing.allocation.* settings disk_threshold decider says NO Node above the low (85%) or high (90%) watermark Free disk, delete old indices, add nodes, or (temporarily) adjust cluster.routing.allocation.disk.watermark.* Index is read-only, writes fail with FORBIDDEN/12/index read-only / allow delete Flood stage (95%) reached Free disk. The block is removed automatically once usage drops below the high watermark, or clear it: PUT idx/_settings {"index.blocks.read_only_allow_delete": null} max_retry decider: "shard has exceeded the maximum number of retries" Repeated failures (e.g. an earlier disk or mapping problem) Fix the cause, then POST _cluster/reroute?retry_failed=true data_tier decider says NO _tier_preference asks for a tier with no nodes Add a node with that role or change _tier_preference enable decider says NO cluster.routing.allocation.enable set to none or primaries Reset it to null (all) Primary unassigned, "no valid shard copy" ( red ) All copies lost (node or disk gone) Bring the node back, restore from snapshot , or, as a last resort, allocate_stale_primary / allocate_empty_primary via _cluster/reroute with accept_data_loss: true total_shards_per_node limit Too restrictive a per-node cap Raise or remove index.routing.allocation.total_shards_per_node NODE_LEFT + delayed Node left recently; waiting delayed_timeout (1m) Wait, or lower the timeout POST _cluster/reroute?retry_failed=true PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.high": "90%", "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } } // Manual moves / last-resort recovery POST _cluster/reroute { "commands": [ { "move": { "index": "orders", "shard": 0, "from_node": "es01", "to_node": "es02" } }, { "allocate_replica": { "index": "orders", "shard": 1, "node": "es03" } } ] } Cluster settings: persistent survives a full restart; transient is deprecated. Set a setting to null to reset it to the default. GET _cluster/settings?include_defaults=true&flat_settings=true shows everything. 📖 Red or yellow cluster health · Diagnose unassigned shards · Fix watermark errors · Cluster-level shard allocation and routing settings · Cluster reroute API Performance signals to watch Signal Where Worry when… JVM heap _cat/nodes heap.percent , _nodes/stats/jvm Consistently > 85%, long old-GC pauses Circuit breakers _nodes/stats/breaker tripped counts increasing (requests rejected with 429s) Thread-pool rejections _cat/thread_pool rejected > 0 on write or search → back-pressure Disk _cat/allocation Approaching 85/90/95% Slow queries or indexing Slow logs : index.search.slowlog.threshold.query.warn: 5s , index.indexing.slowlog.threshold.index.warn: 10s Repeated slow entries Hot threads GET _nodes/hot_threads CPU-bound nodes Pending tasks _cat/pending_tasks Master overloaded (too many shards or mapping updates) Monitoring solutions: Stack Monitoring : Elastic Agent or Metricbeat collects metrics into a (preferably separate) monitoring cluster; view them in Kibana → Stack Monitoring. Legacy internal collection is deprecated. AutoOps : Elastic's hosted, automated diagnosis with recommendations. Available for ECH, and via Cloud Connect for self-managed clusters. Kibana alerting rules on cluster health, CPU, disk and so on. 📖 Monitoring · Stack monitoring · Collect monitoring data with Elastic Agent · AutoOps · AutoOps vs Stack Monitoring · Slow logs · Update logging levels Maintaining the cluster: snapshots, SLM and routine operations Analogy: snapshots are incremental photo backups of your phone. The first one copies everything; later ones only copy new photos (new segments), yet every snapshot can restore the full album on its own. // 1) Register a repository (shared FS: every node needs path.repo pointing to the same mount) PUT _snapshot/my-repo { "type": "fs", "settings": { "location": "/usr/share/elasticsearch/snapshots/my-repo", "compress": true } } POST _snapshot/my-repo/_verify // 2) Take a snapshot PUT _snapshot/my-repo/snap-2026.09.28?wait_for_completion=true { "indices": "orders,logs-app-*", "include_global_state": false, "metadata": { "taken_by": "exam", "reason": "before migration" } } GET _snapshot/my-repo/_all GET _snapshot/my-repo/snap-2026.09.28/_status GET _cat/snapshots/my-repo?v // 3) Restore (the target index must not exist and be open; rename or close/delete first) POST _snapshot/my-repo/snap-2026.09.28/_restore { "indices": "orders", "rename_pattern": "(.+)", "rename_replacement": "restored-$1", "include_aliases": false, "index_settings": { "index.number_of_replicas": 0 }, "ignore_index_settings": ["index.refresh_interval"] } Snapshot lifecycle management (SLM): PUT _slm/policy/nightly { "schedule": "0 30 1 * * ?", "name": "", "repository": "my-repo", "config": { "indices": ["*"], "include_global_state": true }, "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 } } POST _slm/policy/nightly/_execute // run now GET _slm/policy/nightly // last_success / last_failure GET _slm/stats POST _slm/_execute_retention The SLM schedule uses Quartz cron (seconds first: sec min hour day month weekday ), and the name supports date math . Other routine operations: Searchable snapshots: mount an index straight from a repository. Use POST _snapshot/my-repo/snap/_mount?storage=full_copy (cold) or shared_cache (frozen), or let ILM do it. Repository types: fs , url (read-only), s3 , gcs , azure , source (source-only). Restore a whole cluster or feature states: include_global_state , feature_states (e.g. security , kibana ). Close and open indices to cut resource use: POST idx/_close . Force merge read-only indices: POST idx/_forcemerge?max_num_segments=1 . Clear caches: POST idx/_cache/clear . Refresh/flush: POST idx/_refresh , POST idx/_flush . Rolling restarts and upgrades: disable replica allocation → flush → restart → re-enable (see 1.2). 📖 Snapshot and restore · Manage repositories · Shared file system repository · Create snapshots and SLM · Restore a snapshot · Searchable snapshots · Add and remove nodes 5.2 Perform cross-cluster operations Cross-cluster operations depend on remote clusters : a named connection from the local cluster (which starts the requests) to a remote cluster (which holds the data). Two features use it: Cross-cluster search (CCS): query remote data (queries in 4.6). Cross-cluster replication (CCR): continuously copy leader indices on the remote into read-only follower indices on the local cluster. It needs a Platinum/Enterprise license or the trial. Analogy. CCS is a phone call to another library: you ask and hear the answer, but nothing is copied. CCR is a newspaper subscription : the publisher (leader) keeps sending every new edition, and your copy (follower) is read-only. Step 1: connect a remote cluster API key authentication (recommended) TLS certificate authentication (deprecated) Port on the remote 9443 (dedicated remote cluster server) 9300 (transport) Trust Local trusts the remote's CA; access is limited by a cross-cluster API key Mutual TLS; both clusters trust each other's transport CA Remote privileges Defined in the API key ( access.search , access.replication ) Roles with the same names must exist on the remote Connection modes: sniff (default; connects directly to seed nodes and discovers gateway nodes) or proxy (a single address, e.g. through a load balancer). ECH, ECE and ECK effectively use proxy mode. API key setup, self-managed: # On the REMOTE cluster (elasticsearch.yml): enable the remote cluster server interface remote_cluster_server.enabled: true remote_cluster.port: 9443 xpack.security.remote_cluster_server.ssl.enabled: true xpack.security.remote_cluster_server.ssl.key: certs/remote.key xpack.security.remote_cluster_server.ssl.certificate: certs/remote.crt // On the REMOTE: create a cross-cluster API key that limits what the local cluster may do POST _security/cross_cluster/api_key { "name": "from-lab", "access": { "search": [ { "names": ["logs-*", "orders"] } ], "replication": [ { "names": ["orders", "metrics-*"] } ] } } # On EVERY node of the LOCAL cluster: store the encoded key in the keystore, then reload bin/elasticsearch-keystore add cluster.remote.cluster_b.credentials POST _nodes/reload_secure_settings // On the LOCAL cluster (it also needs to trust the remote's CA: // xpack.security.remote_cluster_client.ssl.certificate_authorities) PUT _cluster/settings { "persistent": { "cluster.remote.cluster_b.seeds": ["remote-es01:9443"], "cluster.remote.cluster_b.skip_unavailable": true } } // Proxy mode alternative PUT _cluster/settings { "persistent": { "cluster.remote.cluster_b.mode": "proxy", "cluster.remote.cluster_b.proxy_address": "lb.example.com:9443" } } GET _remote/info // "connected": true, mode, num_nodes_connected, cluster_credentials The same can be done in Kibana: Stack Management → Remote Clusters . The local nodes that make the connections need the remote_cluster_client role, which is included by default when no roles are set explicitly. Local user privileges for the API key model: add remote_indices to the user's role on the local cluster: PUT _security/role/ccs-reader { "cluster": [], "indices": [ { "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] } ], "remote_indices": [ { "clusters": ["cluster_b"], "names": ["logs-*"], "privileges": ["read", "read_cross_cluster", "view_index_metadata"] } ] } For CCR, the local user's role needs remote_indices with cross_cluster_replication and cross_cluster_replication_internal privileges, plus the local manage_ccr cluster privilege. The docs list the exact privileges per model. 📖 Remote clusters · Security models · Connection modes · Self-managed remote clusters · Add remote clusters with API key auth · Add remote clusters with TLS certificate auth · Migrate from certificate to API key · Create cross-cluster API key Step 2a: cross-cluster search See 4.6 for the queries. Per-remote options: skip_unavailable (default true ); cluster.remote..transport.compress . Check connectivity with GET _remote/info and GET _resolve/cluster/:* . Step 2b: cross-cluster replication Prerequisites: licence (Platinum/Enterprise or trial) on both clusters; the remote connection from the follower (local) cluster to the leader ; soft deletes enabled on the leader index (the default since 7.0); compatible versions (the follower may be the same or newer). // Follow one specific leader index (run on the FOLLOWER / local cluster) PUT orders-copy/_ccr/follow?wait_for_active_shards=1 { "remote_cluster": "cluster_b", "leader_index": "orders", "max_read_request_operation_count": 5120, "settings": { "index.number_of_replicas": 0 } } // Auto-follow: automatically follow NEW leader indices (and data streams) that match patterns PUT _ccr/auto_follow/logs-follower { "remote_cluster": "cluster_b", "leader_index_patterns": ["logs-*", "metrics-*"], "leader_index_exclusion_patterns": ["logs-debug-*"], "follow_index_pattern": "{{leader_index}}-dr" } Managing replication: GET orders-copy/_ccr/stats GET orders-copy/_ccr/info GET _ccr/stats POST orders-copy/_ccr/pause_follow POST orders-copy/_ccr/resume_follow // Turn a follower into a normal writable index (e.g. failover), in order: POST orders-copy/_ccr/pause_follow POST orders-copy/_close POST orders-copy/_ccr/unfollow POST orders-copy/_open GET _ccr/auto_follow POST _ccr/auto_follow/logs-follower/pause DELETE _ccr/auto_follow/logs-follower CCR facts: Followers are read-only . Writing to them fails. Auto-follow only picks up leader indices created after the pattern exists. Follow existing ones explicitly. Replication is pull-based : the follower fetches operations from the leader's shard history (soft deletes, retained by shard history retention leases). Mapping and settings changes on the leader are replicated automatically. Some follower settings can't be changed. Data streams are replicated through auto-follow patterns, which replicate the backing indices and create a matching follower data stream. Uses: disaster recovery (active–passive), bringing data closer to users (geo read replicas), and centralised reporting. Bi-directional replication is possible with two auto-follow setups and distinct write targets (e.g. aliases or data streams per region). The ILM action unfollow converts followers automatically as they age. 📖 Cross-cluster replication · Set up CCR · CCR prerequisites · Configure privileges for CCR · Create a follower index · Create an auto-follow pattern · Manage CCR · Uni-directional disaster recovery · Bi-directional disaster recovery Other cross-cluster operations to recognise: Reindex from remote: _reindex with source.remote (HTTP, port 9200; needs reindex.remote.whitelist ). This one doesn't use remote-cluster connections. Restore a snapshot into another cluster: register the same repository (read-only, "readonly": true , on the second cluster) and restore. Cross-cluster ES|QL and EQL, and Kibana data views with cluster:index patterns. 5.3 Exercises: Cluster Management Your cluster is yellow . Find which shards are unassigned and why, then fix it without adding nodes and without deleting data. Index hot-data is red after node es03 died permanently. A snapshot from last night exists. Recover it as hot-data . Register an fs repository backups , snapshot only indices matching orders* without global state, and create an SLM policy running every day at 02:15 that keeps between 3 and 10 snapshots for 14 days. A node shows disk.used_percent: 96 , and writes to logs-x fail with index read-only / allow delete . Explain what happened and fix it. Connect cluster lab to cluster remote as dr using API-key authentication, permitting search on logs-* and replication of orders . On lab , make orders-dr follow remote 's orders . Then fail over: turn orders-dr into a writable index. Configure auto-follow so every new logs-* index on remote is replicated to lab with the suffix -copy . Solution notes _cat/shards → _cluster/allocation/explain . Typical fixes: remove a bad index.routing.allocation.require.* ; reset cluster.routing.allocation.enable ; _cluster/reroute?retry_failed=true ; or (on a 1-node lab) set number_of_replicas: 0 if the task says data only needs one copy. DELETE hot-data (or close it), then POST _snapshot///_restore {"indices":"hot-data"} . Verify with _cat/indices and _count . PUT _snapshot/backups {"type":"fs","settings":{"location":"…"}} (the path must be under path.repo ). Snapshot with "indices":"orders*","include_global_state":false . SLM: "schedule":"0 15 2 * * ?" , retention: {expire_after:"14d", min_count:3, max_count:10} . The flood-stage watermark (95%) applied index.blocks.read_only_allow_delete . Free disk (delete, move or shrink data, or add capacity). The block auto-releases below the high watermark, or remove it explicitly with null . On remote : enable the remote cluster server and create a cross-cluster API key with search: logs-* , replication: orders . On lab : add the key to the keystore as cluster.remote.dr.credentials on every node, _nodes/reload_secure_settings , set cluster.remote.dr.seeds , then check GET _remote/info . PUT orders-dr/_ccr/follow {"remote_cluster":"dr","leader_index":"orders"} . Failover: pause_follow → _close → _ccr/unfollow → _open . PUT _ccr/auto_follow/logs {"remote_cluster":"dr","leader_index_patterns":["logs-*"],"follow_index_pattern":"{{leader_index}}-copy"} . Only new leader indices are picked up. 6. Security & Access Control Analogy for the whole domain: an office building. Authentication is the badge check at the door: who are you? (native users, API keys, LDAP/AD, SAML, OIDC, PKI, service accounts). Authorization / RBAC is which doors your badge opens: roles bundle privileges on the cluster , on indices (even specific documents and fields inside a room), and in Kibana features and spaces. TLS is the armoured corridor between buildings (nodes) and to visitors (clients). Audit logs are the CCTV. 6.1 Implement role-based access control (RBAC) The model user ──(has)──► roles ──(grant)──► privileges on resources ▲ ├─ cluster: monitor, manage, manage_ilm, manage_index_templates, manage_security… │ ├─ indices: read, write, create_doc, manage, view_index_metadata, all… │ │ + field-level security (FLS) + document-level security (DLS) │ ├─ applications: Kibana features & spaces │ └─ remote_indices / remote_cluster (for CCS/CCR) role mapping (LDAP groups, SAML attributes, PKI DN…) ──► roles Built-in roles worth knowing: superuser , kibana_admin , kibana_system (for Kibana's own service connection), viewer (read-only, including Kibana), editor , monitoring_user , remote_monitoring_agent , ingest_admin , snapshot_user , machine_learning_admin / _user , transform_admin , beats_system , logstash_system . Built-in users: elastic (superuser), kibana_system , logstash_system , beats_system , apm_system , remote_monitoring_user . Privileges, the most-used ones: Cluster privilege Allows monitor Read-only cluster info (health, stats, nodes) manage Everything cluster-level except security manage_index_templates , manage_ilm , manage_slm , manage_pipeline , manage_ingest_pipelines , manage_enrich , manage_ccr , manage_transform , manage_ml The named feature manage_security Users, roles, API keys (effectively admin) manage_own_api_key Create or invalidate your own API keys read_ccr , cross_cluster_search , cross_cluster_replication Cross-cluster operations Index privilege Allows read Search, get, count view_index_metadata Get mappings, settings, aliases (Kibana needs this for data views) write Index, update, delete (includes index , create , delete ) create_doc Only create new documents (ideal for shippers, data streams) create_index , delete_index , manage (settings, aliases, refresh…), manage_ilm , auto_configure , maintenance Index administration monitor Index stats, recovery, segments all Everything on those indices Create roles and users PUT _security/role/sales_analyst { "description": "Read EU sales, hide PII, use Discover & Dashboards in the 'sales' space", "cluster": ["monitor"], "indices": [ { "names": ["sales-*", "orders"], "privileges": ["read", "view_index_metadata"], "field_security": { "grant": ["*"], "except": ["customer.email", "customer.phone", "card.*"] }, "query": { "term": { "region": "EU" } } }, { "names": ["sales-reports"], "privileges": ["read", "write", "create_index"] } ], "applications": [ { "application": "kibana-.kibana", "privileges": ["feature_discover.read", "feature_dashboard.all"], "resources": ["space:sales"] } ], "metadata": { "owner": "exam" } } POST _security/user/jane { "password": "Str0ng-Passw0rd!", "roles": ["sales_analyst"], "full_name": "Jane Doe", "email": "jane@example.com", "metadata": { "team": "sales" } } POST _security/user/jane/_password { "password": "An0ther-Str0ng-One!" } POST _security/user/jane/_disable POST _security/user/jane/_enable GET _security/user/jane GET _security/role/sales_analyst DELETE _security/role/old_role Field-level security (FLS) uses field_security.grant / except . Wildcards are allowed. Metadata fields such as _id are always visible. Document-level security (DLS) uses a query , which can be a templated query based on the user: "query": { "template": { "source": { "term": { "owner": "{{_user.username}}" } } } } When a user has several roles, the permissions are the union of all of them. A role without DLS on an index grants access to all documents of that index, even if another role restricts it. That's a classic exam trap. Kibana privileges: application: "kibana-.kibana" . Use "privileges": ["all"] or ["read"] for base access, or per-feature ones ( feature_discover.read , feature_dashboard.all , feature_visualize.all …) for specific features. resources holds the spaces: space: , or * for all. It's easiest to create these in Kibana → Stack Management → Roles and then check the result with GET _security/role/ . Test what you granted GET _security/_authenticate // who am I, which roles, which realm GET _security/user/_has_privileges { "cluster": ["monitor"], "index": [ { "names": ["sales-2026"], "privileges": ["read", "write"] } ] } // Run a request AS another user (requires the run_as privilege) to verify their view GET sales-*/_search # Header: es-security-runas-user: jane The fastest real-world check is to log into Kibana as the user, or call the API with -u jane:password , and try the allowed and the forbidden actions. API keys POST _security/api_key { "name": "ingest-shipper", "expiration": "30d", "role_descriptors": { "writer": { "cluster": ["monitor"], "indices": [ { "names": ["logs-app-*"], "privileges": ["create_doc", "auto_configure"] } ] } }, "metadata": { "app": "shipper" } } // → returns id, api_key, encoded → header: Authorization: ApiKey GET _security/api_key?name=ingest-shipper DELETE _security/api_key { "name": "ingest-shipper" } // invalidate POST _security/api_key/_query { "query": { "term": { "name": "ingest-shipper" } } } An API key's privileges are the intersection of the role_descriptors and the creator's privileges at creation time. Without role_descriptors , it gets a snapshot of the owner's permissions. Role mappings (external realms) PUT _security/role_mapping/ldap-admins { "enabled": true, "roles": ["superuser"], "rules": { "field": { "groups": "cn=admins,ou=groups,dc=example,dc=com" } } } PUT _security/role_mapping/saml-readers { "enabled": true, "roles": ["viewer"], "rules": { "all": [ { "field": { "realm.name": "saml1" } }, { "field": { "groups": "readers" } } ] } } Rules support field , any , all and except . Fields: username , dn , groups , realm.name , metadata.* . Use role_templates for dynamic role names. Realms, in order of evaluation ( order ): reserved → file → native (both on by default) → ldap , active_directory , pki , saml , oidc , kerberos , jwt . SAML and OIDC work through Kibana. Service accounts (e.g. elastic/kibana , elastic/fleet-server ) use service tokens instead of passwords. 📖 Users and roles · User roles · Defining roles · Role structure · Built-in roles · Security privileges · Kibana privileges · Kibana role management · Document and field level security · Privileges for data streams and aliases · Role mapping · Native realm · Built-in users · Service accounts · Elasticsearch API keys · Submit requests on behalf of other users 6.2 Secure a cluster for production Analogy: moving from a house party to a bank. At a party the door is open and everyone knows each other (development mode). A bank has locked doors (authentication), badges per room (RBAC), armoured transport between branches (transport TLS), bulletproof glass at the counter (HTTP TLS), a vault for secrets (keystore), cameras (audit logs), and a building inspection before opening (bootstrap checks). Layer 1: security on, with TLS everywhere Since 8.0, a fresh self-managed install auto-configures security : it generates certificates, enables TLS on the transport and HTTP layers, prints the elastic password, and gives you an enrollment token for Kibana and additional nodes. Know how to do it manually too: # 1. CA + transport (node-to-node) certificates bin/elasticsearch-certutil ca --out elastic-stack-ca.p12 bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --out elastic-certificates.p12 # 2. HTTP certificates (interactive wizard: CSR or CA-signed, hostnames, IPs) bin/elasticsearch-certutil http # 3. Store keystore passwords bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password # elasticsearch.yml xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.transport.ssl.enabled: true # REQUIRED for multi-node clusters with security xpack.security.transport.ssl.verification_mode: certificate # or full (also checks hostnames) xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 xpack.security.http.ssl.enabled: true # HTTPS for REST clients & Kibana xpack.security.http.ssl.keystore.path: certs/http.p12 # Passwords & tokens bin/elasticsearch-reset-password -u elastic # or -i for interactive bin/elasticsearch-reset-password -u kibana_system bin/elasticsearch-create-enrollment-token -s kibana # or -s node bin/elasticsearch-service-tokens create elastic/kibana my-token # kibana.yml (secured connection to Elasticsearch) elasticsearch.hosts: ["https://es01:9200"] elasticsearch.serviceAccountToken: "AAEAAWVs..." # preferred over username/password elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/http_ca.crt"] server.ssl.enabled: true # HTTPS for browsers → Kibana server.ssl.certificate: /etc/kibana/certs/kibana.crt server.ssl.key: /etc/kibana/certs/kibana.key xpack.encryptedSavedObjects.encryptionKey: "<32+ chars>" # needed for alerting and saved-object secrets xpack.security.encryptionKey: "<32+ chars>" xpack.reporting.encryptionKey: "<32+ chars>" Rules: With security enabled, a multi-node cluster must have transport TLS . Otherwise nodes refuse to form a cluster once they bind to a non-loopback address (production mode). Secrets belong in the keystore ( elasticsearch-keystore ), never in plain elasticsearch.yml . Reloadable secure settings can be refreshed with POST _nodes/reload_secure_settings . Don't use the elastic superuser for daily work or for Kibana. Create personal admin users, and use service accounts or API keys for applications. Anonymous access stays disabled unless explicitly required. Layer 2: production-mode node configuration and bootstrap checks As soon as network.host binds to a non-loopback address, Elasticsearch runs in production mode and enforces bootstrap checks . A failing check stops the node from starting. Check / setting Requirement Heap size -Xms = -Xmx (set in jvm.options.d/*.options or ES_JAVA_OPTS ). By default Elasticsearch sizes the heap automatically; manually, ≤ 50% of RAM and below the compressed-oops threshold (~ 31 GB ). File descriptors ≥ 65,535 ( ulimit -n ) Max threads ≥ 4,096 Virtual memory vm.max_map_count ≥ 262,144 (check the docs for the value recommended for your version) Memory lock If bootstrap.memory_lock: true , the lock must succeed. Also disable swap . Discovery discovery.seed_hosts / cluster.initial_master_nodes (first bootstrap only) or equivalent must be configured Others Max file size, max virtual memory, JVM checks (client JVM, serial GC), system call filter, OnError / OnOutOfMemoryError cluster.name: prod-search node.name: es-prod-01 node.roles: [ master, data_hot, ingest ] network.host: 10.0.1.11 # _site_, _local_, specific IP… http.port: 9200 transport.port: 9300 discovery.seed_hosts: ["10.0.1.11", "10.0.1.12", "10.0.1.13"] cluster.initial_master_nodes: ["es-prod-01", "es-prod-02", "es-prod-03"] # remove after first start path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true action.destructive_requires_name: true # no DELETE on wildcards / _all (the default is true in 8+) Layer 3: least privilege, network restrictions and auditing Least privilege checklist: Personal admin accounts instead of the shared elastic account. Shippers (Agent, Beats, Logstash) get API keys with create_doc + auto_configure on their own data streams only. Applications get dedicated users or API keys limited to their indices. Analysts get read + view_index_metadata , with FLS/DLS where needed, and Kibana privileges limited to specific spaces. action.destructive_requires_name: true prevents DELETE * . IP filtering (self-managed; on Cloud use traffic filters or network security): xpack.security.transport.filter.allow: ["10.0.1.0/24"] xpack.security.transport.filter.deny: _all xpack.security.http.filter.allow: ["10.0.0.0/16"] xpack.security.http.filter.deny: _all Audit logging (needs a Platinum/Enterprise licence or the trial): xpack.security.audit.enabled: true xpack.security.audit.logfile.events.include: ["access_denied", "authentication_failed", "connection_denied", "tampered_request", "run_as_denied", "security_config_change"] xpack.security.audit.logfile.events.emit_request_body: false The output goes to _audit.json in the logs directory. Ignore policies ( xpack.security.audit.logfile.events.ignore_filters.* ) reduce noise. Kibana has its own audit log ( xpack.security.audit.enabled in kibana.yml ). Other production hardening: Encrypt data at rest via the file system or disk (Elasticsearch doesn't encrypt shards itself). On Cloud, use customer-managed keys. Keep Elasticsearch off the public internet . Put it behind a firewall, load balancer or VPN, and expose only Kibana over HTTPS if needed. Set a session timeout in Kibana ( xpack.security.session.idleTimeout , lifespan ). Rotate certificates before they expire. The docs cover rotating with the same CA and with a different CA. Snapshot the security feature state ( feature_states: ["security"] ) so users and roles can be restored. For FIPS environments, use FIPS 140-2 mode with approved algorithms and keystores. 📖 Security overview · Self-managed security setup · Automatic security setup · Minimal security setup · Set up transport TLS · Set up HTTPS · Configure security in Kibana · Manage TLS encryption · Update TLS certificates · Secure settings (keystore) · IP filtering (self-managed) · Audit logging · Enable audit logging · Important system configuration · Bootstrap checks · Important settings · Kibana session management · Tutorial: self-managed secure install 6.3 Exercises: Security Create role logs_reader : read and view metadata on logs-* , no access to the field user.password_hash , only documents where service.environment is production , plus Discover (read) in the Kibana space ops . Create user ana with this role and verify with _has_privileges and by logging in. Create role ingest_writer that can only append documents to data streams matching logs-app-* , and create an API key limited to it that expires in 7 days. User bob has roles A (DLS: team: blue on projects ) and B (plain read on projects ). Which documents can he see, and why? Fix it so he sees only team: blue . Map LDAP group cn=sre,ou=groups,dc=corp to roles monitoring_user and logs_reader . On a three-node lab started without security, enable security with transport TLS (own CA) and HTTPS, reset the elastic password, connect Kibana with a service account token, and turn on audit logging for authentication failures and denied access. List the bootstrap checks that would fail on a VM with vm.max_map_count=65530 , ulimit -n 4096 and swap enabled with bootstrap.memory_lock: true , and fix each one. Solution notes indices: [{names:["logs-*"], privileges:["read","view_index_metadata"], field_security:{grant:["*"], except:["user.password_hash"]}, query:{term:{"service.environment":"production"}}}] , applications: [{application:"kibana-.kibana", privileges:["feature_discover.read"], resources:["space:ops"]}] . Role: indices: [{names:["logs-app-*"], privileges:["create_doc","auto_configure"]}] . API key: POST _security/api_key with expiration: "7d" and the same role_descriptors . He sees all documents: role permissions are a union , and B's unrestricted read wins. Fix by removing B from bob, or adding the same DLS query to B. PUT _security/role_mapping/sre {"roles":["monitoring_user","logs_reader"],"enabled":true,"rules":{"field":{"groups":"cn=sre,ou=groups,dc=corp"}}} . certutil ca → certutil cert (copy to all nodes) → certutil http → transport and HTTP SSL settings + keystore passwords → rolling or full restart → elasticsearch-reset-password -u elastic → elasticsearch-service-tokens create elastic/kibana kibana-token → kibana.yml elasticsearch.serviceAccountToken + CA → xpack.security.audit.enabled: true with the events.include list. Max map count → sysctl -w vm.max_map_count=262144 (persist it in /etc/sysctl.d ). File descriptors → raise the limit to 65535 ( limits.conf or systemd LimitNOFILE ). Memory lock → LimitMEMLOCK=infinity (systemd) or ulimit -l unlimited . Disable swap ( swapoff -a ) as good practice. 7. Final checklist and study plan 7.1 A four-week plan Week Focus Deliverable 1 Domains 1–2: lab build, templates, dynamic templates, data streams, ILM, aliases Every Domain 2 exercise done twice without notes, just the docs 2 Domain 3: mappings, analyzers, field types, reindex/UBQ, pipelines, enrich, Streams A "migration" drill: v1 → v2 with a new mapping, pipeline and alias swap in < 20 min 3 Domain 4: Query DSL, bool, aggregations, ES|QL, semantic search, CCS 20 queries on the sample data; translate 5 of them to ES|QL 4 Domains 5–6 + mock exams: health, snapshots, SLM, remote clusters, CCR, RBAC, TLS Two timed mock exams of 10 mixed tasks in 3 hours 7.2 "Can I do this from a blank Console with only the docs?" ☐ Explain node roles, tiers, primaries vs replicas, and the write and read paths ☐ Configure forced awareness and allocation filtering; explain a yellow cluster ☐ Create an index with custom analyzers, multi-fields, nested fields and strict mapping ☐ Component + index templates with priority; simulate them ☐ Dynamic templates with correct ordering ☐ A data stream from a template, with ILM rollover/warm/delete; _ilm/explain ☐ Alias swap, filtered alias, write index ☐ Choose an ingest method for a scenario ☐ Reindex with query, script and pipeline; update-by-query to populate a new multi-field ☐ A pipeline with grok/dissect, date, set, conditionals, on_failure , enrich; simulate it ☐ Streams: processing, retention and data quality in the UI ☐ match , match_phrase , multi_match , term / range , bool with filter/should logic ☐ Terms/date_histogram/range buckets with metrics and pipeline aggs ☐ ES|QL with WHERE, EVAL, STATS…BY, SORT, LIMIT, DISSECT and LOOKUP JOIN ☐ semantic_text + inference endpoint + semantic/match query + RRF hybrid ☐ Remote cluster (API key) + CCS + CCR follower + auto-follow + failover ☐ Diagnose unassigned shards with allocation explain; fix watermark blocks ☐ Snapshot repository, snapshot, restore with rename, SLM policy ☐ Roles with FLS/DLS and Kibana privileges, users, API keys, role mappings ☐ TLS (transport + HTTP), keystore, bootstrap checks, audit logging 7.3 Further references 📖 Elastic Documentation home : the only resource allowed in the exam 📖 Elasticsearch REST API reference 📖 Kibana API reference 📖 Elasticsearch release notes : skim 9.0 → 9.3 for new features and breaking changes 🎓 Elastic Training: Elasticsearch Engineer course : the official course aligned to the exam 🧪 Elasticsearch Labs : tutorials on semantic search, ES|QL and retrievers 💬 Elastic Discuss forum : exam-experience threads 📝 Community practice exercises by Guido Lena Cota (written for the 8.x exam, still useful for templates, mappings and search): Store data · Deploy and operate a cluster Prefect — Components Deep Dive A study guide to Prefect 3 (Python workflow orchestrator), one component at a time: what it is, the code you will type, when to use it, the traps, and where the official docs live. How to read this guide. Each component has (1) a mental model, usually an analogy, (2) the code, (3) a when to use / when not to verdict, (4) traps and (5) 📖 doc links. The decision tables for “which way do I run this?” (§5-6) and “when is Prefect the wrong tool?” (§13) are the parts to re-read before choosing an orchestrator. The sibling page RustFS — Components Deep Dive covers an S3-compatible store that Prefect can use for results and code. Version note: written 23 Oct 2026; the docs index lists Prefect 3.9 as the latest OSS line. ⚑ marks details I wrote from prior knowledge rather than re-reading the page this session (decorator parameters, exact import paths, CLI flags). Confirm those against the linked docs and your installed version ( prefect version ) before copying. 0. The map: what Prefect is Analogy: an airline operations centre. Your flights are ordinary Python functions. The operations centre does not fly the plane; it tracks every flight, reschedules on delay (retries), reroutes when an airport closes (infrastructure), logs everything, and alerts when something goes wrong. Planes (your code) fly on your own aircraft (your infrastructure). Workflows are plain Python. A flow is a decorated function; a task is a decorated function called inside it. There is no DAG file to declare: dependencies come from data flow and normal control flow ( if , for , try ), so a workflow can branch and loop dynamically. Orchestration is hybrid. Your code runs on your infrastructure; the Prefect API (self-hosted server or Prefect Cloud) stores state, schedules, events and automations. Task runs are orchestrated client-side and their updates are logged in batches, so task state in the UI is eventually consistent. Code is not stored by Prefect. A deployment is server-side metadata (which flow, where its code lives, which infrastructure, schedule, parameters). Deployments reference flows defined in Python files; they do not contain code. Prefect 3 changes (old tutorials differ): workers and work pools replace agents; block-based deployments and storage/infrastructure blocks are the legacy path; assets, transactions and background tasks are first-class. The 2.x to 3.0 upgrade and agents-to-workers guides exist under how-to-guides/migrate . Piece Where it lives Job Flow / task code Your repo or image The actual work Prefect API + database Cloud or self-hosted server (Postgres) State, schedules, events, UI Deployment API (metadata only) Run this flow, from here, on that pool, on this schedule Work pool (+ queues) API A channel between deployments and infrastructure Worker Your infra (or none for push/managed pools) Polls the pool and starts runs 📖 Concepts · Quickstart · Upgrade to 3.0 · Migrate from Airflow 1. Flows Analogy: a recipe card. The card lists the steps; any cook with a kitchen can follow it. The kitchen is infrastructure; the order ticket is a flow run. from prefect import flow, task, get_run_logger @flow(name='daily-etl', retries=1, retry_delay_seconds=60, timeout_seconds=3600, log_prints=True) def daily_etl(day: str, full_refresh: bool = False) -> int: rows = extract(day) return load(rows) if __name__ == '__main__': daily_etl('2026-10-22') # runs locally, recorded in the API if one is configured A flow is callable like a function. Calling it creates a flow run with its own state history. ⚑ Common options: name , retries , retry_delay_seconds , timeout_seconds , log_prints , validate_parameters (type-checks arguments against hints), persist_result , task_runner , on_completion / on_failure / on_cancellation / on_crashed hooks. Subflows: a flow may call another flow; it becomes a child run with its own state. Flows and tasks can call each other with no required nesting pattern. Flow versus task: a flow is the unit you deploy, schedule and parameterise ; a task is a unit you retry, cache and run concurrently . Do not put a deployment boundary inside a task. Flow retries rerun the whole flow ; task retries rerun one step. Prefer task retries plus caching so a flow retry does not redo finished work. Traps. Parameters must be serialisable for deployments (they travel as JSON). Top-level code outside @flow runs on every import, including when a worker loads the file. Keep imports light and put heavy setup inside tasks. 📖 Flows · Write and run a workflow · Pass inputs 2. Tasks Analogy: one station on the line. Small, with a clear input and output, and the supervisor logs how long it took and whether it passed. Tasks are cacheable, retryable units of work with transactional semantics. A task is identified by a task key (a hash of its name and fully qualified function name). The docs recommend small tasks : each should be a logical step or a significant side effect, so the UI narrates your workflow. from datetime import timedelta from prefect import task from prefect.cache_policies import INPUTS, TASK_SOURCE @task(retries=3, retry_delay_seconds=[10, 60, 300], timeout_seconds=120, cache_policy=INPUTS + TASK_SOURCE, cache_expiration=timedelta(hours=1), log_prints=True) def extract(url: str) -> dict: ... 2.1 Three ways to run a task Call Blocks? Returns Use for t(x) Yes The result Simple sequential steps t.submit(x) No PrefectFuture , resolve with .result() or .wait() from the same context Concurrency inside a flow t.delay(x) No Future you need not resolve from the same context Fire-and-forget background work on separate workers (§10) Dependencies are implicit: passing an upstream result or future into a downstream task means it cannot start until the upstream is Completed . Add wait_for=[...] for explicit ordering. t.map(items) fans out over an iterable ⚑. Parameter resolution trap (documented): Prefect walks parameters to resolve nested futures and states. If a dataclass or named tuple field changes, Prefect rebuilds it with dataclasses.replace() , which calls __init__ and __post_init__ again; Pydantic models use model_construct() and skip validators. Use quote(obj) to pass a fully resolved object untouched (no future resolution, no dependency tracking), or opaque(future) to resolve a top-level future without traversing its contents (from prefect.utilities.annotations ). Task run life cycle: Pending → Running → Completed; on error Retrying/AwaitingRetry → Running; retries exhausted → Failed; infrastructure failure → Crashed; a cached result jumps Pending → Completed. Background tasks start as Scheduled . When to make something a task: it has a side effect, may fail transiently, is worth caching, or should run concurrently. When not to: trivial pure helpers, and anything so fine-grained that per-call overhead and UI noise exceed the value. 📖 Tasks · Retries · Run work concurrently 3. Task runners and concurrency control Analogy: how many cooks you hire and whether they share a kitchen. A task runner decides whether submitted tasks run in threads, processes, or on a cluster. Concurrency limits are the 'only two people at the fryer' rules. Task runners control how .submit() / .map() tasks execute: concurrent threads (the default ⚑), parallel processes, or distributed ( prefect-dask , prefect-ray integrations). Set per flow with @flow(task_runner=...) . Choose: I/O-bound work (HTTP, DB, S3) → threads. CPU-bound Python → processes or Dask/Ray. Data too large for one machine → Dask or Ray, or push the work into the warehouse and orchestrate only. Global concurrency limits are named limits you acquire in code and share across flows and deployments, for protecting a database or an API; rate limits use the same mechanism with a refill rate. Tag-based limits cap concurrent task runs carrying a tag. from prefect.concurrency.sync import concurrency, rate_limit # ⚑ import path @task def write_db(rows): with concurrency('warehouse-writes', occupy=1): # blocks if the limit is full ... @task def call_api(x): rate_limit('vendor-api') # waits for a slot at the configured rate ... Traps. A task runner only affects tasks started with .submit() / .map() ; direct calls are sequential. Threads do not speed up CPU-bound code. Forgetting to create the named global limit means the code has nothing to enforce. Worker-level limits (work pool and queue concurrency, §6) limit flow runs , not tasks. 📖 Task runners · Global concurrency limits · Tag-based limits · Per-worker task concurrency example 4. States, results, caching and transactions Analogy: a parcel tracking page, a saved receipt and an undo button. State says where the run is, results are the saved output, caching says 'I already did this exact job', and transactions say 'do these steps all-or-nothing'. States are rich objects (Pending, Running, Completed, Failed, Cancelled, Crashed, Paused, Scheduled, Retrying/AwaitingRetry...). They carry data and messages, drive state-change hooks and are what automations trigger on. Crashed means the infrastructure failed (killed process, out of memory), not that your code raised. Results are the data a flow or task returns. Persisting them is off by default for most runs and is what enables caching and resumption. Choose result_storage (local path, or an S3-compatible bucket via an integration block), persist_result=True , and the serialiser. ⚑ Persisted results need storage reachable by every worker that may read them. Caching skips a task when a cache key matches. Policies combine with + : INPUTS (hash of arguments), TASK_SOURCE (the task's code), FLOW_PARAMETERS , RUN_ID and others ⚑; plus cache_expiration . Use a custom key_fn when arguments are not hashable or should be partly ignored. A cache hit moves the task straight to Completed. Transactions give tasks transactional semantics: group steps so they commit together, and register rollback hooks that run if a later step fails. from prefect.transactions import transaction # ⚑ @task def write_staging(batch): ... @write_staging.on_rollback def drop_staging(txn): ... # runs if the transaction rolls back @flow def load(batch): with transaction(): write_staging(batch) swap_into_prod() # if this raises, drop_staging runs Use caching for expensive, deterministic steps (downloads, parsing, model scoring on fixed input) so retries and reruns are cheap. Do not cache steps whose result depends on time or external state unless the key includes it, or you will happily serve stale data. Idempotency still matters: a retry or a worker crash can run a task twice, so design writes as upserts or partition overwrites. 📖 States · Caching · Customize caching · Persist and retrieve results · Transactional workflows 5. Deployments: three ways to run a flow Analogy: a standing order versus cooking at home. Running the function yourself is cooking at home. A deployment is a standing order registered with the operations centre: it says what, where the recipe lives, which kitchen, and when. Approach How Infrastructure Pick it when Run the function python flow.py Wherever you run it Development and tests flow.serve() A long-lived Python process listens for runs and runs schedules That process (container, VM) Simple, single environment, no pool to manage flow.from_source(...).deploy() or prefect deploy with prefect.yaml Deployment bound to a work pool Dynamically provisioned per run Many deployments, different environments, isolation per run, platform-team guardrails # 1. serve: simplest daily_etl.serve(name='daily-etl', cron='0 6 * * *', parameters={'day': 'today'}) # 2. deploy to a work pool (code pulled from git at run time) from prefect import flow flow.from_source(source='https://github.com/acme/pipelines.git', entrypoint='flows/etl.py:daily_etl').deploy( name='daily-etl-prod', work_pool_name='k8s-pool', cron='0 6 * * *', parameters={'day': 'today'}, job_variables={'image': 'acme/pipelines:1.4'}, tags=['etl']) # ⚑ argument names # trigger on demand from prefect.deployments import run_deployment run_deployment(name='daily-etl/daily-etl-prod', parameters={'day': '2026-10-22'}) Three ways to define a deployment: Python SDK ( .deploy() ), YAML ( prefect.yaml plus prefect deploy ), or Terraform. All populate the same schema: name, flow, parameters, schedules, tags, work pool, job variables, version. Code storage: bake the code into the Docker image (pool types that use images), pull it from git or a cloud bucket at run time, or use a local path with Process pools. Do not rely on a path that exists only on your laptop. Versioning: deployments carry a version and can be rolled back; tag flows and tasks with version too. Override per-run infrastructure with job variables. CI/CD: deploy from CI on merge, one deployment per environment, with the image tag as a job variable. Traps. A deployment does not contain code, so a worker that cannot reach the repo or image fails the run. Changing the flow signature without redeploying breaks scheduled runs that carry old parameters. serve ties execution to one process that must stay up. 📖 Deployments · Create deployments · prefect.yaml · Store flow code · Versioning · CI/CD 6. Work pools, workers and queues Analogy: pub/sub topics. A work pool is a topic. Deployments publish runs to it; workers subscribe and start the runs. Switch a deployment to another pool and a different worker runs it, which is how you promote between environments or debug locally. Pool kind Who starts runs You run a worker? Examples Hybrid A worker in your infrastructure Yes Process, Docker, Kubernetes, AWS ECS, Azure Container Instances, Google Cloud Run (v1, v2), Vertex AI Push (Cloud) Prefect submits to your serverless provider No ECS push, Cloud Run push, ACI push, Modal push Managed (Cloud) Prefect-managed infrastructure No Prefect Managed; Coiled is also available Why pools at all: dynamic infrastructure per run (scale to zero), default job configuration that every deployment inherits and can override, and opinionated, enforced interfaces for platform teams. Workers: prefect worker start --pool my-pool ⚑ polls and submits runs; a hybrid worker needs outbound access to the API and the permissions to create infrastructure. Run it as a supervised service (systemd, a Deployment) and use the worker healthcheck documented under advanced guides. Work queues sit inside a pool. Each has a priority (a unique positive integer; 1 is highest ) and an optional concurrency limit that cannot exceed the pool's global limit. Runs are submitted from the highest priority queue in waterfall order. A queue is READY if a worker polled it in the last 60 seconds, PAUSED when you pause it, otherwise NOT_READY . Example policy: low (priority 10, unlimited), high (priority 5, limit 3), critical (priority 1, limit 1) for urgent backfills. Job variables and the base job template define per-pool defaults (image, resources, env vars, namespace); override them on the deployment or even per run. Customising Kubernetes job templates has its own guide. Pick a pool: laptop or single VM → Process. Containerised isolation on one host → Docker. Production on a cluster → Kubernetes. Serverless without operating a worker → push pool on Cloud. Avoid a Process pool for untrusted or conflicting dependencies. Traps. The worker's own credentials decide what it can launch, so scope them. A pool with no live worker leaves runs Late. Mixing images with different Python or Prefect versions between client and worker causes confusing serialisation errors; keep versions aligned (see the versioning and compatibility page). 📖 Work pools · Workers · Manage work pools · Kubernetes · Docker · Serverless · Job variables · Worker healthchecks 7. Schedules, events and automations Analogy: an alarm clock versus a motion sensor. A schedule fires at times you choose; an event trigger fires when something happens, such as a file arriving or a flow failing. Schedules tell the API to create flow runs automatically: cron, interval or RRule, attached to a deployment, with timezone support. Pause, resume or remove them per deployment. Events record activity in your stack: Prefect emits them for state changes and you can emit custom ones ( emit_event(...) ⚑). They carry a resource, an event name and a payload. Automations pair a trigger (event, metric or compound) with actions: run a deployment, pause a schedule, send a notification, cancel a run. Deployment triggers make event-driven pipelines; chaining deployments with events gives dependencies between separate workflows without coupling the code; you can pass the event payload as flow parameters. Debounce with the within parameter so a burst of events does not create a run per event. Zombie flows (runs that stopped reporting) can be detected with an automation. Choose a schedule for predictable batch windows; choose an event trigger for 'when upstream finished' or 'when data landed'. Prefer events over polling loops inside a flow. Traps. Cron is in UTC unless a timezone is given. A schedule creates runs even if the previous one is still running unless concurrency limits stop it. Triggers that match their own outputs can loop. 📖 Schedules · Events · Automations · Event triggers · Chaining deployments · Debouncing · Zombie flows 8. Configuration: variables, blocks, secrets, settings Thing Holds Use for Not for Variables Dynamically-named, mutable, non-sensitive JSON values Shared config such as a table name or a flag Secrets Blocks Typed, saved configuration with a UI (credentials, buckets, notification targets) Reusable connections and secrets managed centrally Per-run ephemeral data Secret blocks Encrypted sensitive strings API keys, passwords Large documents Settings and profiles PREFECT_* settings per environment API URL, logging, timeouts, server tuning Application config Runtime context Info about the current run ( prefect.runtime ) Flow run ID, parameters, deployment name inside code Persistent state from prefect.variables import Variable from prefect.blocks.system import Secret from prefect.runtime import flow_run limit = Variable.get('batch_size', default='500') token = Secret.load('vendor-api-token').get() print(flow_run.id, flow_run.parameters) # ⚑ runtime attribute names Switch between environments with profiles ( prefect profile use prod ⚑) rather than editing code. Self-hosted servers have security settings (basic auth and others) documented under advanced guides; do not expose an unauthenticated API to a network. 📖 Variables · Blocks · Settings and profiles · Runtime context · Store secrets 9. Observability, artifacts, assets and human input Logging: log_prints=True turns print into run logs; get_run_logger() gives a structured logger. Logging config is customisable. Cloud offers AI summaries of failed-run logs. Artifacts are persisted outputs for humans, rendered in the UI (markdown, tables, links, progress): create_markdown_artifact(...) , create_table_artifact(...) ⚑. Use them for data-quality reports and run summaries, not as a results store. Assets represent objects your workflows produce (a table, a file, a model) and are tracked as lineage across runs. Declare them on tasks with @materialize(...) ⚑ and add metadata and explicit dependencies where inference is not enough. Check your version supports them. State change hooks run code when a flow or task enters a state (on failure, on completion): the place for notifications and cleanup. Human in the loop: a run can pause and wait for input or approval, or suspend and free its infrastructure, then resume from the UI or API. Interactive workflows have a dedicated guide. Cancellation of running flows is a documented workflow too. 📖 Logging · Artifacts · Assets · State hooks · Interactive workflows · Cancel workflows 10. Background tasks Analogy: a drop-box and a back-office team. The web app drops a job in the box and returns to the customer immediately. Back-office workers pick it up whenever they are free. task.delay(...) pushes a task run onto a server-side topic; a pool of task workers executes it. The docs position this as a replacement for Celery plus a broker for queue functionality. Use it when a web request or service must dispatch heavy work without waiting and you want to scale task execution independently. Do not use it for work that needs a flow-level schedule, deployments and per-run infrastructure; use a deployment for that. The caller does not need to resolve the returned future. 📖 Run background tasks · Web app powered by background tasks 11. Hosting: Cloud versus self-hosted Prefect Cloud Self-hosted server You operate Workers and your code only API, database, background services, workers Extras Push and managed pools, workspaces, SSO, service accounts, ACLs, audit logs, webhooks, SLAs, PrivateLink Open source; fewer pool types (hybrid ones) Limits Documented API rate limits and data retention Yours to size Run a server: prefect server start locally; Docker; Docker Compose with Postgres, Redis, server, background services and a worker; Helm chart for Kubernetes. Point clients at it with PREFECT_API_URL . Scale: run multiple server instances for HA (needs Postgres and Redis for messaging); keep the database maintained (monitoring, vacuuming, retention of old runs). Secure: enable authentication via the security settings, restrict network access, and see the page on endpoints and ports for restricted networks. 📖 Server · Docker Compose · Helm · Scale self-hosted · Database maintenance · Security settings 12. Testing and integrations Test: a flow is a function, so unit-test the logic; the docs describe a test harness and pytest fixtures for running flows and tasks against a temporary backend. Test deployment definitions by loading them in CI. Integrations are PyPI packages: prefect-aws (S3, ECS worker, Secrets Manager, Lambda, Glue, Batch), prefect-gcp , prefect-azure , prefect-kubernetes , prefect-docker , prefect-dbt (runner with asset lineage, or per-node orchestration), prefect-snowflake , prefect-sqlalchemy , prefect-databricks , prefect-dask , prefect-ray , prefect-redis , prefect-slack , prefect-email , prefect-github , prefect-gitlab , prefect-bitbucket , prefect-shell . AI assist: a Prefect MCP server lets assistants inspect your workspace. 📖 Test workflows · Integrations · prefect-aws · prefect-dbt · MCP server 13. When is Prefect the right tool? Situation Verdict Python-heavy data or ML pipelines with retries, caching, scheduling, dynamic fan-out Good fit. Flows are ordinary code Event-driven pipelines (file landed, upstream finished) Good fit (events and automations) A team that wants per-run isolated infrastructure without operating a scheduler fleet Good fit (work pools, push or managed) You need a declarative, reviewable DAG with a large existing operator ecosystem and heavy SQL-first teams Airflow may fit better; Prefect has a migration guide, so weigh the rewrite The product is the asset graph with software-defined data assets and a data catalogue Compare with asset-first tools; Prefect's assets are newer, so check maturity for your needs Strict durable execution of business transactions over days with exactly-once semantics A durable-execution engine such as Temporal is a closer match One cron job calling one script Overkill. Use cron or a platform scheduler until you need retries, visibility or dependencies Heavy transformation inside a warehouse Orchestrate it (dbt or SQL tasks) and let the warehouse do the work Challenge before adopting: orchestration adds an API, a database and workers to run, so ask what failure you are actually solving (silent failures, no retries, no visibility) and whether a simpler scheduler plus alerting would do. Also confirm where results and logs may be stored if your data is regulated. 14. Trap checklist and exercises Expecting a deployment to carry code; worker cannot fetch the repo or image. Client, worker and image on different Prefect or Python versions. Non-idempotent writes with retries; caching a time-dependent task with a code-only key. Using threads for CPU-bound work; using .submit() but never resolving futures, so errors go unnoticed. Dataclass fields rebuilt by parameter resolution (use quote or opaque ). Cron in UTC surprises; schedules overlapping long runs; trigger loops. Secrets in variables instead of secret blocks; unauthenticated self-hosted API. A pool with no worker, or worker credentials broader than needed. Persisting results to storage that other workers cannot reach. Exercises. (1) Write a three-task flow with retries=3 on a flaky task and caching on an expensive one; run it twice and read the states. (2) Deploy it with serve , then to a Process work pool, and compare where the code runs. (3) Add a global concurrency limit and submit 20 tasks. (4) Add a deployment trigger that runs a second flow when the first completes. (5) Wrap two writes in a transaction() with a rollback hook and force the second to fail. Solution notes A cached task shows Completed immediately on the second run; flow retries rerun the flow but cached tasks are skipped. serve runs in your process; the pool path needs a worker polling the pool and a reachable code source. Create the limit first (CLI or UI), then acquire it in code with concurrency(name) ; excess tasks wait. Use an automation with a flow-run-completed trigger and a run-deployment action, optionally passing the event payload. The rollback hook of the first write runs, leaving no partial output. Sources: docs.prefect.io (v3): tasks , work pools , deployments , documentation index . Other sections were written from the index summaries plus prior knowledge; items marked ⚑ should be verified. RustFS — Components Deep Dive A study guide to RustFS, an open-source, S3-compatible distributed object store written in Rust: what each component is, the commands and settings you will actually use, when to use it, the traps, and where the docs live. How to read this guide. Each component has (1) a mental model, (2) the code or config, (3) a when to use / when not to verdict, (4) traps and (5) 📖 doc links. The decision tables are in §2 (which deployment mode) and §11 (should I use RustFS at all). The sibling page Prefect — Components Deep Dive covers an orchestrator that can use an S3-compatible store for results and code. Version note: written 23 Oct 2026 from docs.rustfs.com. The S3 compatibility page was snapshotted by RustFS against a commit dated 9 Aug 2026, so treat counts as a moving target. ⚑ marks items from prior knowledge or inference rather than the pages I read (boto3 usage, exact Prefect wiring). Read §11 before adopting: this is a young project, so verify production readiness yourself. 0. The map: what RustFS is Analogy: a warehouse of labelled bins with a front desk that speaks Amazon's language. Customers (applications) hand over boxes (objects) with a bin label (bucket) and a shelf label (key). The front desk (the S3 API) uses the same forms as Amazon's warehouse, so existing tools just work. Behind the desk, each box is split into pieces and spread over many shelves (disks) so a few broken shelves do not lose anything. What: a distributed object storage system in Rust, Apache 2.0 licensed, S3-compatible, aimed at private cloud, data lakes, AI/ML and cloud-native workloads. It runs on Linux, Windows, macOS, FreeBSD and containers across x86, ARM and RISC-V. Positioning from the project: the permissive licence is the headline contrast with AGPL-licensed alternatives; it claims migration and coexistence with S3-compatible platforms such as MinIO and Ceph. The README also claims 2.3x faster than MinIO for 4 KB payloads. That is a vendor benchmark; reproduce it on your hardware and object sizes before it informs a decision. Components: the rustfs server (S3 API on port 9000), the web Console (port 9001), the rc command-line client, optional protocol gateways (WebDAV, FTP/FTPS, SFTP, OpenStack Swift), background services (scanner for bit-rot, healing, lifecycle, replication), Kubernetes Helm chart and Operator, and an MCP server for AI assistants. Object-store vocabulary: an object is a blob plus metadata, addressed by bucket/key . Keys look like paths but the namespace is flat; 'folders' are just key prefixes. Addressing. Path-style by default ( http://host:9000/my-bucket/path/to/hello.txt ). For virtual-host style ( https://my-bucket.s3.example.com/hello.txt ) set RUSTFS_SERVER_DOMAINS=s3.example.com and create wildcard DNS ( *.s3.example.com ) pointing at RustFS. 📖 Docs home · S3 protocol · Virtual-host access · GitHub 1. Installing it Goal Path Try it locally One-command install script, or a container Single-server production SNSD (one disk) or SNMD (several disks), on Linux Multi-server production MNMD after the pre-installation checklists Kubernetes Helm chart (standalone or distributed) or the RustFS Operator (tenants, pools, KMS) Windows or macOS Native install; the Windows build is for development and debugging only 1.1 Docker, the fastest start docker volume create rustfs-data docker run -d --name rustfs --restart unless-stopped \ -p 9000:9000 -p 9001:9001 -v rustfs-data:/data \ -e RUSTFS_ACCESS_KEY='' -e RUSTFS_SECRET_KEY='' \ -e RUSTFS_ADDRESS=':9000' -e RUSTFS_CONSOLE_ADDRESS=':9001' -e RUSTFS_CONSOLE_ENABLE=true \ rustfs/rustfs:latest /data curl --fail http://localhost:9000/health # S3 API health; Console on :9001 Change the default credentials immediately. Never use the well-known rustfsadmin value. On Linux set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY in /etc/default/rustfs and restart; with containers pass them at creation (recreate the container if it started without them; the volume survives). The container runs as non-root user rustfs , UID 10001. A bind-mounted host directory must be owned by 10001 ( chown -R 10001:10001 /path ) or you get permission denied. Named volumes need no setup. Flags win over env vars when both are given ( --access-key , --address , --console-enable ). Compose: the repository ships a docker-compose.yml with optional Grafana, Prometheus, OpenTelemetry collector and Jaeger under --profile observability ; start only the server with docker compose up -d rustfs . An init container fixes volume ownership. Multi-node containers need --network host (the default bridge network does not support it) and name resolution ( /etc/hosts on every node): RUSTFS_VOLUMES='http://node{1...4}:9000/data/rustfs{0...3}' . TLS: mount a cert directory and set RUSTFS_TLS_PATH=/certs . For HTTPS in front, the docs cover Nginx, Traefik, Caddy, Envoy, HAProxy and Apache with separate hostnames for S3 and Console. 📖 Installation · Docker · Linux · Kubernetes · Checklists · TLS · Reverse proxy 2. Deployment modes: which one? Mode Nodes Disks Fault tolerance Use for SNSD 1 1 None. Rely on backups Dev, test, small non-critical SNMD 1 Several Up to M parity disks inside the node; the node itself is a single point of failure Mid-sized non-critical on one server MNMD 4 or more Several per node Disk- and node-level via erasure coding across servers Production Decision rule: if losing a server or its data would hurt, you need MNMD plus a backup or replication strategy (erasure coding protects against disk failures, not against deletion, corruption by software, or a lost site). SNSD is a development tool, not a storage strategy. Disk preparation (Linux). Use dedicated data disks, format, mount and verify them before installing (the docs refer to XFS and a verification step); keep identical service environment configuration on every node. Work through the hardware, network, software and security checklists first. 📖 SNSD · SNMD · MNMD · Disk preparation · Hardware selection 3. Erasure coding and capacity Analogy: a jigsaw with spare pieces. Each object is cut into N-M data pieces plus M extra 'parity' pieces, then spread across N disks. Any N-M of the N pieces rebuild the object, so up to M pieces can be lost. Reed-Solomon erasure coding. Each pool is divided into one or more erasure sets . For a set of N drives with M parity shards: data shards = N-M; tolerable unavailable shards = M; capacity efficiency about (N-M)/N. A 16-drive set with EC:4 stores 12 data and 4 parity shards, about 75% efficiency before file-system, metadata and operational overhead. Failure-domain caveat (from the docs): parity covers unavailable shards within one set . Node-level tolerance depends on how a set's drives are spread over nodes; do not assume EC:4 survives four whole nodes , because one node may hold several drives of the same set. Drives per set (N) Automatic STANDARD parity Approx. efficiency 1 EC:0 100% 2-3 EC:1 50-67% 4-5 EC:2 50-60% 6-7 EC:3 50-57% 8-16 EC:4 50-75% # /etc/default/rustfs RUSTFS_STORAGE_CLASS_STANDARD= # recommended: leave empty for automatic per-pool parity # or pin explicitly (must be valid for the NARROWEST pool): RUSTFS_STORAGE_CLASS_STANDARD=EC:4 RUSTFS_STORAGE_CLASS_RRS=EC:1 # REDUCED_REDUNDANCY writes; default 1 parity on multi-drive sets RUSTFS_ERASURE_SET_DRIVE_COUNT=16 # pin set width only if you verified the whole layout Rules for explicit parity: M <= N/2 in every target set; STANDARD parity >= RRS parity when both non-zero; valid for every pool. An invalid value makes startup fail rather than silently weakening parity. Set widths range from 2 to 16 drives, selected automatically from the topology. Validate against the set width, not the cluster's total drive count. Capacity estimate: usable is roughly raw x (N-M)/N. Sixteen 10 TiB drives as one 16-drive set with EC:4 gives about 120 TiB, and you must still reserve space for metadata, versioning, incomplete multipart uploads, healing and headroom. Verify after start: Console Status page or rc admin info cluster shows layout, parity, drive availability and pool membership. Trap. More parity buys durability at the price of usable capacity and write cost; do not pin high parity on small sets without checking the efficiency. Leaving the default is the documented recommendation unless you have a validated requirement. 📖 EC configuration · Status check · Storage env vars 4. Configuration surface Setting Purpose RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY Root credentials. Set unique values RUSTFS_ADDRESS S3 API bind address (default :9000 ) RUSTFS_CONSOLE_ADDRESS , RUSTFS_CONSOLE_ENABLE Console bind address and switch (port 9001) RUSTFS_VOLUMES The storage endpoints; supports {1...4} range expansion for multi-node RUSTFS_SERVER_DOMAINS Domain for virtual-host style access RUSTFS_TLS_PATH Directory with certificates RUSTFS_STORAGE_CLASS_* , RUSTFS_ERASURE_SET_DRIVE_COUNT Erasure coding (§3) RUSTFS_OBS_* Logging and observability (level, log directory) The reference page also lists variables for KMS, the scanner, healing and event targets. rustfs has subcommands and server flags with environment equivalents (CLI reference). Keep the same environment file on every node. 📖 Environment variables · CLI reference · CORS 5. The S3 API and compatibility Analogy: speaking a dialect. RustFS speaks Amazon S3 well enough for most everyday conversation, but not every idiom. The project says so itself: it is a tested subset , not a claim of full coverage. # AWS CLI against RustFS: just add --endpoint-url aws --endpoint-url "$RUSTFS_ENDPOINT" s3 cp hello.txt s3://my-bucket/path/to/hello.txt aws --endpoint-url "$RUSTFS_ENDPOINT" s3 ls s3://my-bucket/ aws --endpoint-url "$RUSTFS_ENDPOINT" s3api head-object --bucket my-bucket --key path/to/hello.txt aws --endpoint-url "$RUSTFS_ENDPOINT" s3 presign s3://my-bucket/path/to/hello.txt # Python (boto3) ⚑ import boto3 s3 = boto3.client('s3', endpoint_url='http://localhost:9000', aws_access_key_id=KEY, aws_secret_access_key=SECRET, region_name='us-east-1') s3.upload_file('hello.txt', 'my-bucket', 'path/to/hello.txt') Area Status in the compatibility gate Bucket create/delete/list/inspect, tagging, policies, public access block Tested Object put/get/copy/head/delete, listing (prefix, delimiter, marker, max-keys; V1 and V2) Tested Range and conditional reads, user metadata, object tags, presigned GET/PUT Tested Multipart create/upload/complete/abort; selected copy and checksum cases Tested Selected versioning, Object Lock, CORS and lifecycle cases; SSE-C and selected SSE-KMS Tested (only the cases in the implemented lists) Bucket access logging, bucket ownership controls, POST Object form checksum handling Planned (not implemented) ACL authorisation Excluded: intentionally unsupported Multipart upload listing and part-lookup edge cases Excluded from the default gate Gate size (Aug 2026 snapshot): 455 implemented cases, 5 lifecycle cases, 17 planned, 270 excluded, from Ceph s3-tests lists. Read the numbers as 'a lot is covered, but 270 are excluded', not as 'complete'. Encrypted objects are not portable. A passing encryption test means RustFS can read what RustFS encrypted; it does not mean it can read an encrypted object copied raw from another implementation. S3 Select currently accepts only uncompressed input. Other protocols: WebDAV, FTP/FTPS, SFTP and (optional build) OpenStack Swift gateways over the same buckets. Practical advice: if you depend on ACLs, bucket access logging, or exotic multipart listing behaviour, test those exact calls against RustFS before committing. Use rclone or your SDK's integration test suite as a smoke test. 📖 S3 compatibility matrix · SDKs · Python · SFTP · WebDAV 6. Data management features Feature What it does Use it for Gotcha Versioning Keeps prior versions; deletes create markers Recover from overwrites and accidental deletes Storage grows; required by replication Object Lock Retention periods and legal holds on versioned objects WORM and compliance retention Needs versioning; test retention modes before relying on them Lifecycle rules Expire or transition objects by age Cost control, log retention Managed with the rc client; verify rules fire Tiered storage Lifecycle-driven transition to a remote storage tier Move cold data to cheaper storage Reads of tiered objects depend on the remote tier Bucket quota Hard storage quota per bucket Multi-tenant limits Hard limit: writes fail at the cap Bucket replication Asynchronous replication between versioned buckets Copies in another cluster Async, so a failover can lose the last writes; both ends need versioning Site replication Multi-site replication managed with rc Disaster recovery across sites Plan network and consistency expectations Multipart upload Parts for large objects; Console task monitor Big files, resumable uploads Incomplete uploads consume space; clean up Scanner Background object scan with a bit-rot cycle and metrics Detect silent corruption Watch its metrics to confirm scans still run S3 Tables Table buckets with a built-in Iceberg REST catalog Lakehouse without a separate catalog Newer feature; test with your engines (PyIceberg, Spark docs exist) Event notifications Filtered bucket events to external targets (webhook and others) React to uploads, trigger pipelines Targets and queue directories need persistent storage 📖 Versioning · Object Lock · Lifecycle · Tiering · Quota · Bucket replication · Site replication · Scanner · S3 Tables · Event notifications 7. Security Analogy: badges, keys and a locked cabinet. IAM decides who may enter which room (badges). Encryption keeps the contents unreadable even if someone steals a disk (locked cabinet). TLS protects the hallway. IAM: users, groups and policies; policy attachment and evaluation semantics are documented. Service accounts and STS issue scoped, short-lived credentials for applications. Access keys are managed per identity. Prefer service accounts or STS for applications and keep the root key out of code. Bucket policies are applied through the S3 API; public access block exists. ACLs are intentionally unsupported. OIDC: Console single sign-on with providers such as Keycloak and GitLab. Encryption at rest: SSE-S3 (server-managed keys), SSE-C (customer-provided AES-256 keys) and SSE-KMS with a local or HashiCorp Vault backend; the Operator can configure KMS for tenants. Note: the project README marks its KMS as 'under testing' while the docs describe KMS configuration, so validate your key-management path and recovery procedure before putting regulated data on it. TLS with RUSTFS_TLS_PATH ; mTLS and cert-manager are documented for Helm deployments. Audit targets deliver request-level audit records to external systems. Credential management has its own page; rotate the root credentials. Traps. Default credentials left in place. Exposing the Console (9001) to the internet. Putting RustFS behind a reverse proxy that rewrites the Host header without enabling the matching virtual-host settings. Losing the KMS key means losing the data. 📖 Security and compliance · IAM · Service accounts and STS · Encryption · OIDC · Audit · Credentials 8. Operations The rc client: a RustFS command-line client to connect to, inspect and administer a cluster, including lifecycle rules, replication, heal tasks and rc admin info cluster . Health and status: /health on the S3 port; the Console Status page; the status-check workflow covers network access, health endpoints, cluster status and capacity. Observability: export metrics, traces and logs over OTLP ; the docs list recommended alerts and metric names, and integrations with Prometheus, Grafana, Loki, Tempo and Thanos exist. Scaling: expand by adding a storage pool; rebalance to redistribute existing objects over active pools; decommission to drain and retire a pool and verify the data moved. Pools are the unit of growth, so plan pool sizes and symmetry up front. Healing and failures: replace a failed drive and let RustFS heal the missing shards onto it; for a lost node prepare a replacement, update its configuration, restore cluster membership and watch healing. Manual heal tasks can be bounded with rc . Upgrades: binary, container and Kubernetes procedures that preserve availability. Rehearse on a non-production cluster first. Kubernetes: the Helm chart (standalone or distributed, probes, hardening, pool expansion) or the Operator (multi-tenant isolation, pool expansion, KMS). Runbook items to write before go-live: disk replacement, node replacement, credential rotation, certificate renewal, capacity alert thresholds, upgrade order, restore test from backup or replica. 📖 rc client · Observability · Scaling · Node healing · Drive failures · Node failures · Upgrade · Cluster lifecycle 9. Integrations relevant to a data platform RustFS documents step-by-step integrations (mostly Docker Compose based) for tools that speak S3. Grouped by purpose: Purpose Tools with a RustFS guide Orchestration and ingestion Airflow (S3 provider), DolphinScheduler, SeaTunnel, Kafka Connect S3 sink, AutoMQ Lakehouse and query Apache Iceberg (Spark, REST catalog), PyIceberg via S3 Tables, Delta Lake (delta-rs), Hudi, Paimon, Hive, Spark, Flink, Trino, DuckDB (httpfs), ClickHouse (S3 disk), Databend, Doris, lakeFS Vector and ML LanceDB, Milvus, MLflow artifact store, Ray Data, vLLM model weights Observability storage Loki, Tempo, Thanos, Cortex, OpenObserve, GreptimeDB, Fluentd, OpenTelemetry, VictoriaMetrics backups Backup and DR Restic, Kopia, Velero, Longhorn, Vitess DevOps and platform Terraform state with locking, Elasticsearch and OpenSearch snapshots, Gitea LFS, Jenkins artifacts, Harbor and Docker Registry, Flux, Nextcloud Filesystem views s3fs-fuse, rclone, JuiceFS, ZeroFS (NFS), Alluxio cache, SFTPGo, OpenDAL 9.1 Using it with Prefect ⚑ Prefect can keep results and deployment code in S3-compatible storage through the prefect-aws integration. Because RustFS speaks S3, the wiring is the integration's S3 bucket block with a custom endpoint URL pointing at RustFS and a service-account key. I wrote this from how prefect-aws normally works, not from a tested RustFS example; check the prefect-aws docs for the current block and client-parameter names and run a write/read round trip before trusting it. Results: point result storage at a bucket such as prefect-results ; every worker must reach the endpoint and credentials. Code: a pull step can fetch flow code from a bucket at run time. Hygiene: a dedicated service account with a bucket-scoped policy, a lifecycle rule to expire old results, TLS on the endpoint. 📖 Integrations index · Airflow · Iceberg · DuckDB · Spark · lakeFS · Terraform state · rclone · prefect-aws 10. Moving data in and out Migration and coexistence: the project says it supports running alongside other S3 platforms such as MinIO and Ceph. The practical tools are rclone sync , bucket replication (versioned buckets) and application-level dual writes. Validate a migration: compare object counts, sizes and checksums; test presigned URLs, multipart behaviour, listing order and any ACL or logging features you used, because those are the documented gaps. Do not copy encrypted objects raw between implementations (not portable); decrypt and re-encrypt through the S3 API. Exit plan: because the API is S3, leaving RustFS later is an rclone job, which is a point in its favour. Keep that tested. 11. Should you use RustFS? (the part to challenge) Situation Verdict Local dev and CI needing an S3 endpoint Good fit. One container, quick to start Self-hosted S3 for logs, artefacts, backups, ML datasets where a permissive licence matters Reasonable , after your own load and failure tests Data lake or lakehouse storage with Iceberg or Delta Plausible ; documented integrations exist, but pilot with your engines and table sizes Regulated or business-critical primary storage Proceed carefully. Confirm KMS maturity, Object Lock behaviour, support terms and your recovery drills Need full AWS S3 feature parity (ACLs, access logging, every multipart listing edge) Not a fit where those are documented as excluded or planned Cloud-native workload already on a managed S3 Staying on the managed service removes operations; self-host only for cost, sovereignty or latency reasons Questions to answer before adopting: (1) Is the benefit the licence, the cost, performance, or Rust itself, and have you measured it? (2) Who is on call when a node dies at 3 a.m.? (3) Have you rehearsed a disk and node replacement and a restore? (4) What is the backup strategy beyond erasure coding? (5) How mature is the release you pin, and what is the upgrade policy? Pin a version rather than using :latest in production. 12. Trap checklist and exercises Default credentials; Console exposed publicly. Bind-mount directory not owned by UID 10001. Believing EC:4 equals four-node tolerance. Pinning parity that the narrowest pool cannot satisfy (startup fails), or setting a set width that does not divide the topology. SNSD or SNMD treated as production, with no backup. Replication configured without versioning; async replication assumed to be lossless. Relying on ACLs, access logging or encrypted-object portability. Using :latest ; no upgrade rehearsal; no restore test. Windows build used in production. Exercises. (1) Start the Docker command with unique credentials, create a bucket and upload a file with the AWS CLI using --endpoint-url . (2) Enable versioning, overwrite the file, and recover the first version. (3) Compute usable capacity for 8 nodes x 4 x 8 TiB drives if sets are 16 wide with EC:4, and say what a node loss does to a set. (4) Add a lifecycle rule to expire objects under tmp/ after one day and verify. (5) Run a write/read round trip from a Python script with boto3 and a presigned URL. Solution notes Use aws --endpoint-url http://localhost:9000 s3 mb s3://demo , then cp ; configure the CLI with the access key pair you set. Versioning is a per-bucket setting; list versions to see both, then fetch by version ID or remove the delete marker. Raw is 8 x 4 x 8 = 256 TiB; with a 16-wide set and EC:4 efficiency is 75%, so about 192 TiB before overhead. Whether a node loss is survivable depends on how many of a set's 16 drives sit on that node: if a node holds 4 drives of each set, losing it removes 4 shards, which EC:4 can reconstruct, with no margin left. Lifecycle rules are managed with rc ; confirm by checking the object disappears after the scanner or lifecycle cycle runs. boto3 needs endpoint_url ; generate_presigned_url('get_object', ...) gives a URL you can fetch with curl. Sources: docs.rustfs.com: docs home , installation , Docker , EC configuration , S3 compatibility matrix , S3 protocol , docs index ; GitHub README (claims). Sections 6-8 summarise the documentation index descriptions rather than every page body; items marked ⚑ are unverified. Miscs Diagram Types — When to Use Each Structure & relationships Architecture — Shows the components of a system (services, databases, queues, external APIs) and the connections between them. Use it when someone needs to understand how the pieces of a system fit together at a glance — onboarding a new engineer, documenting a service before a migration, or explaining a design in a review. Skip it if you only have 2-3 boxes with one connection; that's a sentence, not a diagram. ER / data model — Entities (tables/objects) with their fields and the relationships (one-to-many, many-to-many) between them. Use it when designing or documenting a database schema, or when you need to reason about how data is normalized and connected before writing migrations. Most useful early in a project, before the schema calcifies. Layer stack — Abstraction layers drawn as horizontal bands, stacked so lower layers support higher ones (e.g. hardware → OS → runtime → application, or OSI network layers). Use it to explain dependency direction between layers of abstraction — what sits on top of what, and what breaks if a lower layer changes. Nested — Hierarchy expressed by literally drawing boxes inside boxes (an org inside a division inside a company, or a VPC inside a region inside a cloud account). Use it when containment itself is the point — when something is physically or logically inside something else, not just related to it. Venn — Two or more overlapping circles showing what's shared vs. unique between sets. Use it for comparing scope or membership — feature overlap between two products, skill overlap between two roles, shared vs. distinct requirements between teams. Gets unreadable past 3 circles; don't force a 4-way Venn. Dependency graph — Nodes and directed edges showing what depends on what, including fan-in (many things depending on one), fan-out (one thing many others depend on), and cycles. Use it to find risky central dependencies, plan safe refactor order, or explain why one library change broke six unrelated services. UML class — Classes with their attributes, methods, and typed relationships (inheritance, composition, association) between them. Use it for object-oriented design docs, before writing code for a non-trivial class hierarchy, or when reviewing someone else's OOP design and you need to see the shape of it. Database schema — Physical tables with actual column names, types, and foreign key lines between them — more concrete and implementation-level than an ER diagram. Use it once the schema is built or close to final, as reference documentation for whoever queries or migrates the database next. Flow & process Flowchart — Decision logic expressed as boxes and diamonds, with branches for different outcomes. Use it for any process with conditional branches — an approval workflow, a troubleshooting procedure, business logic with multiple if/else paths. If there's no branching, it's just a list — use a numbered list instead. Sequence — Actors (services, users, systems) as vertical lifelines, with horizontal arrows showing messages passed between them over time, top to bottom. Use it for anything where order and timing between parties matters — an API request/response cycle, an OAuth token refresh flow, a distributed transaction across microservices. State machine — States as nodes, transitions as labeled edges between them, showing what triggers a move from one state to another. Use it for anything with a finite, well-defined lifecycle — an order's status (pending → paid → shipped → delivered), a connection's states, a UI component's modes. If a system can be in infinitely many configurations, this isn't the right tool. Swimlane — A flowchart split into horizontal or vertical lanes, one per team/role/system, showing who does what and where work crosses a boundary. Use it when a process spans multiple owners and the handoffs between them are the interesting part — e.g. an incident response process moving between on-call, engineering, and comms. Process — A multi-actor sequential workflow, similar to swimlane but framed around ordered steps rather than lane ownership. Use it for step-by-step operational workflows involving more than one participant, where you want the sequence to read left-to-right or top-to-bottom rather than lane-by-lane. Data flow — Pipeline steps scoped to specific roles or systems, showing how data moves and transforms from source to destination. Use it to document an ETL/ELT pipeline, a data ingestion path, or any process where the main story is data being transformed as it passes through stages. Sankey — Flow width is proportional to quantity, and flows can split and merge — the classic "where did the volume go" diagram. Use it when quantity itself carries meaning: budget allocation across categories, user funnel volume splitting into different paths, energy or material flow through a system. Wrong tool if the flows aren't quantitative. Fishbone — A central "spine" (the effect/problem) with angled "bones" grouping contributing causes into categories. Use it for root-cause analysis — postmortems, quality issues, "why did this fail" investigations where causes cluster into categories (people, process, tools, environment, etc.). Deployment — Physical or logical deployment topology: zones, hosts, and the artifacts running on each. Use it to document where things actually run — which services live in which network zone, on which hosts or clusters — useful for infra reviews, security audits, and incident diagnosis. Hierarchy & ranking Tree — Parent-to-children branching, no cross-links. Use it for anything strictly hierarchical with a single path down from any node to the root — file system structure, a decision tree, a taxonomy, a component breakdown. Org chart — People or teams with reporting lines and, optionally, routing/ownership indicators. Use it for team structure, escalation paths, or "who owns what" documentation — anywhere the hierarchy is about people/teams rather than abstract concepts. Pyramid / funnel — A ranked hierarchy (base = most foundational/most volume) or a funnel narrowing toward a smaller output (e.g. conversion funnel). Use a pyramid for priority or maturity models (Maslow-style); use the funnel framing specifically when you're showing drop-off between stages, like a sales or signup funnel. Story map — A horizontal "backbone" of user activities, with vertical slices showing what's included in each release. Use it for product/agile planning — mapping a user journey against release scope, so everyone can see what's in v1 vs. later. Kanban — Columns representing workflow states (to do / in progress / done, or more granular), with cards moving across them. Use it to show current work-in-progress distribution across a pipeline, or as a snapshot of a team's board at a point in time for a status update. Comparison & positioning Quadrant — Two axes crossing to form four quadrants, items placed by where they score on each axis. Use it for prioritization (impact vs. effort), positioning (build vs. buy, cost vs. value), or any comparison that reduces cleanly to two independent dimensions. Radar / spider — Multiple axes radiating from a center, one shape per item being compared, showing relative strength across many dimensions at once. Use it when you're comparing 2-4 options across 4+ criteria simultaneously — vendor comparisons, skill assessments, product feature comparisons. Gets messy with more than a few overlapping shapes. Wardley map — Value chain position (how visible/close to the user) on one axis, evolution stage (genesis → custom-built → product → commodity) on the other. Use it for strategic technology or business planning — deciding what to build vs. buy vs. outsource based on how commoditized a capability is. Niche but powerful in strategy contexts; don't reach for it outside that use case. Polar chart — Similar to radar but encodes a cyclic magnitude with a linear radius — think of periodic/cyclical data (time of day, seasonal patterns) mapped around a circle. Use it specifically for cyclical quantitative data where the "wraparound" nature of the axis matters, unlike a radar chart's fixed set of categorical axes. Time-based Timeline — Discrete events plotted along a single time axis. Use it for historical narratives, project milestones, or any story that's fundamentally "this happened, then this happened." Gantt — Tasks and phases as horizontal bars along a timeline, showing duration and overlap. Use it for project planning and scheduling — when you need to show not just that things happened in sequence but how long each took and where they overlapped. User journey — Stages of an experience, the user's actions at each stage, and their sentiment (frustration, delight) plotted alongside. Use it for UX research and product design — surfacing where in a flow users struggle, to prioritize what to fix. Loop / flywheel — A circular flow representing a reinforcing cycle, often with a shared hub in the middle that all the loop's stages feed back into. Use it for business models or systems that compound over time — growth loops, feedback loops, virtuous cycles — where the point is that the cycle strengthens itself with each pass. Charts (quantitative) Bar chart — Categorical comparison, bar length/height proportional to value. Use it for comparing discrete categories against each other — sales by region, votes by candidate. The default choice for "compare these numbers." Line chart — Trend over a continuous axis, usually time. Use it whenever the trajectory matters more than any single value — revenue over quarters, a metric drifting up or down. Scatter plot — Individual data points plotted on two numeric axes, revealing distribution and correlation. Use it when you have many data points and want to see whether/how two variables relate, or to spot outliers and clusters. Treemap — Rectangles sized by value, nested to show part-of-whole relationships (e.g. storage used by folder, revenue by product line). Use it for hierarchical proportions where area itself needs to communicate scale — this only works if the area-to-value mapping is accurate, so it's less forgiving than a bar chart of small errors. Data-platform specific Medallion — Multi-tier data storage architecture, typically bronze (raw) → silver (cleaned) → gold (aggregated/business-ready). Use it specifically when documenting a data lakehouse or similar layered data architecture — this is a domain-specific variant of the layer stack, tailored to data engineering vocabulary. DP integration — Sources feeding into a core platform, which feeds out to consumers. Use it to document a data platform's ingestion and distribution boundaries — what comes in, what the platform does, who/what consumes the output. DP security matrix — A grid of roles against resources/permissions, showing who can access what. Use it for access-control documentation and audits — a compliance or security review needs exactly this shape: role rows, resource columns, permission cells. When not to diagram at all Quick throwaway visuals for a chat message or terminal output — plain text is faster and just as clear. A list of anything — a table or bullet list beats a diagram every time. Before/after comparisons — a table communicates this more precisely than boxes and arrows. A single labeled box — if there's only one shape, you don't have a diagram, you have a sentence. Write the sentence. The underlying test: would a reader learn more from this than from a well-written paragraph? If not, don't draw it. Decision-Making & Thinking Methods — When to Use Each Elimination-based methods Process of elimination — Systematically ruling out options that fail a hard requirement until only viable ones remain. Use it when you have many options and at least one clear disqualifying criterion (budget cap, hard deadline, legal requirement) — it's fast because you never have to score the options you kill off. Bad fit when no option can be truly disqualified and everything is a matter of degree; then you need scoring, not elimination. MECE exclusion (Mutually Exclusive, Collectively Exhaustive) — Breaking a decision space into buckets that don't overlap and don't leave gaps, then eliminating whole buckets at once instead of individual items. Use it when the option space is large and unstructured (e.g. "how do we reduce costs") — forcing MECE buckets first prevents you from redundantly comparing options that are really the same idea in different words. Veto voting — Each participant can strike out options they find unacceptable, and whatever survives every veto wins. Use it in group decisions where avoiding a bad outcome matters more than optimizing for a great one — picking a restaurant for a group with dietary restrictions, or approving a shortlist where any single stakeholder can kill a candidate. Branching / tree-based methods Decision trees — Mapping out sequential decision points as branches, with the outcomes and probabilities at each leaf. Use it when a decision unfolds in stages and later choices depend on earlier ones — should we invest now or wait for more data, given different ways the market could move. Overkill for one-shot decisions with no sequential dependency. Backward induction — Starting from the desired end state or last decision point and reasoning backward to figure out what has to be true at each earlier stage. Use it for planning problems with a known target and multiple paths to it — game theory, negotiation prep, or working out the last responsible moment to make a call. Only works cleanly when the end state is well-defined; useless for open-ended exploration. Minimax / worst-case branching — At each decision branch, assume the opponent or environment will pick the outcome worst for you, and choose the branch that minimizes that worst case. Use it in adversarial or highly uncertain situations where you can't assign real probabilities — competitive strategy, security threat modeling, negotiating with an unpredictable counterparty. Too conservative for situations where you actually have decent probability estimates; expected-value methods do better there. Comparative / scoring methods Weighted decision matrix — List options as rows, criteria as columns, assign a weight to each criterion, score each option per criterion, and sum. Use it when you have 3+ options, multiple criteria that matter to different degrees, and you want a defensible, auditable comparison — vendor selection, hiring between finalists, picking between architecture options. Don't bother for a 2-option call with one dominant criterion; that's just a comparison sentence. Pairwise comparison — Comparing every option against every other option one-on-one, then tallying wins, instead of scoring all of them on an abstract scale at once. Use it when abstract scoring feels arbitrary or when direct comparison is easier for people than absolute rating — ranking a shortlist of candidates, or prioritizing a backlog where "how much better is A than B" is easier to answer than "rate A from 1-10." Pros/cons list — The simplest form: two columns, upsides and downsides, no weighting. Use it for quick, low-stakes personal decisions where you mainly need to externalize your thinking, not calculate an optimum. Weak for anything with more than a couple of criteria of very different importance — a long pros list can outweigh one severe con it shouldn't. Cost-benefit analysis — Converting the expected costs and benefits of each option into comparable units (usually money or time) and comparing net values. Use it for decisions with quantifiable costs and benefits — capital investment, whether to build vs. buy, whether a project is worth the resourcing. Breaks down when the biggest factors are genuinely unquantifiable (reputation, morale); forcing a number onto those can mislead more than it helps. Root-cause & diagnostic methods Five Whys — Asking "why" repeatedly (usually 5 times) on an observed problem until you reach the root cause rather than a symptom. Use it right after an incident or failure, when the visible problem is probably a downstream effect of something deeper — a production outage, a recurring team conflict, a process that keeps breaking. Weak when a single problem has multiple independent root causes; the format's single-chain structure will make you stop at the first plausible-sounding answer instead of finding all of them. Fishbone / Ishikawa diagram — Mapping many possible contributing causes, grouped into categories (people, process, tools, environment), converging on one effect. Use it when a problem plausibly has several independent contributing factors and you want to explore breadth before committing to one root cause — quality defects, complex incident postmortems. More effort than Five Whys, but better suited to multi-causal problems. First principles thinking — Stripping a problem down to its fundamental truths and reasoning up from there, instead of reasoning by analogy to how it's normally done. Use it when the standard approach to a problem is expensive, slow, or just assumed rather than justified — redesigning a process everyone "always" does a certain way, or challenging an industry convention. Costly to do for routine decisions; reserve it for decisions where the payoff of a genuinely novel answer is high. Framing & structuring methods SWOT analysis (Strengths, Weaknesses, Opportunities, Threats) — A four-quadrant framing that separates internal factors (strengths/weaknesses) from external ones (opportunities/threats). Use it early in strategic planning, when you need a shared, structured starting point for a discussion rather than an actual decision — it's a framing tool, not a decision procedure, so don't expect it to output an answer by itself. Six Thinking Hats — Deliberately switching between six modes of thinking (facts, feelings, caution, optimism, creativity, process) in turn, usually in a group. Use it when a group discussion keeps getting stuck because people default to one mode (usually caution or optimism) and talk past each other — it forces everyone to visit the same mode together instead of arguing across modes. Cynefin framework — Classifying a situation as simple, complicated, complex, or chaotic, and matching your decision approach to that classification (best practice, good practice, emergent practice, or rapid stabilizing action respectively). Use it as a first step before choosing any other method on this list — it tells you whether a decision even has a knowable right answer (simple/complicated) or whether you should be probing and adapting instead (complex/chaotic). Iterative / adaptive methods OODA loop (Observe, Orient, Decide, Act) — A fast repeating cycle for decisions under time pressure and incomplete information, where you act, then immediately re-observe and adjust. Use it in fast-moving, high-uncertainty situations where waiting for full information means missing the window to act — incident response, competitive/adversarial situations, live operational decisions. Wrong tool for decisions where you have time to gather full information up front; the speed advantage isn't worth the lost rigor. PDCA / iterative improvement (Plan, Do, Check, Act) — Running small experiments in a loop, checking results, and adjusting before scaling up. Use it for process improvement where you can test cheaply before committing — a new workflow, a process change, an experimental feature. Not suited to one-shot, high-stakes, irreversible decisions where you don't get a second iteration. Satisficing — Choosing the first option that meets a "good enough" threshold, rather than searching for the optimum. Use it when the cost of continued search or analysis exceeds the expected gain from finding a marginally better option — routine purchasing decisions, hiring under time pressure, anything where "good enough, now" beats "perfect, eventually." Group / consensus methods Delphi method — Anonymously collecting expert opinions in rounds, sharing an aggregated summary between rounds, and repeating until opinions converge. Use it for forecasting or expert judgment on questions with no hard data, where you want to avoid groupthink and status-driven bias — long-range technology forecasts, expert risk assessments. Slow; not for time-pressured decisions. RAPID / RACI — Frameworks that assign explicit decision roles (who Recommends, who Agrees, who Performs, who has Input, who Decides / who's Responsible, Accountable, Consulted, Informed). Use it when a decision keeps stalling because it's unclear who actually has the authority to decide, not because the content itself is unclear — organizational decisions with many stakeholders. Doesn't help at all with the substance of the decision, only the process of making it. Dot voting / brainstorming convergence — After open brainstorming generates many options, each participant places a limited number of votes (dots) on their preferred ones, and the highest-vote options move forward. Use it to converge a large, unstructured set of ideas from a group down to a shortlist quickly, without a full scoring exercise — workshop facilitation, feature prioritization sessions. Heuristics / shortcuts Pareto principle (80/20) — Recognizing that a small fraction of causes/inputs typically drives most of the effect/output, and focusing effort there instead of treating all factors equally. Use it for prioritization when you have many candidate causes or actions and limited time — which bugs to fix first, which customers to focus retention efforts on. Occam's razor — Among competing explanations that fit the evidence equally well, preferring the simplest one. Use it during diagnosis or root-cause analysis when multiple explanations are consistent with what you've observed — don't reach for an exotic explanation when a mundane one accounts for the same facts. Premortem — Before committing to a plan, imagining it has already failed and working backward to figure out why. Use it right before locking in a significant decision, as a check against overconfidence and groupthink — it surfaces risks that "will this work" framing tends to suppress because everyone's already invested in saying yes. Regret minimization — Choosing the option that minimizes your future regret if it turns out badly, rather than the option with the best expected value. Use it for personal, high-stakes, largely irreversible decisions where the emotional cost of "what if" outweighs a purely numerical optimum — career changes, major life decisions. LLMs Large language models: frameworks, orchestration and agent engineering. LangChain — Components Deep Dive A study guide to building agents with LangChain 1.x (Python), one component at a time: what it is, the code you will actually type, when to use it (and when not to), the traps, and where the official docs live. How to read this guide. Every component has (1) a mental model, usually an everyday analogy, (2) the code you will write, (3) a when to use / when not to verdict, (4) the traps, and (5) 📖 links to the official docs. The decision tables for “when do I use tools?” (§3) and “when do I use memory?” (§6) are the parts to re-read before a design review. The companion page is LangGraph — Components Deep Dive . Version note: written 17 Oct 2026 from docs.langchain.com. Features marked ⚑ need a minimum version (table in §14), and model names move fast, so run pip show langchain langgraph and pick current model IDs from your provider before copying anything. 0. The map: what LangChain is in 2026 Analogy: a car. LangChain is the cockpit: steering wheel, pedals and dashboard that feel the same in every car (any model provider). LangGraph is the engine and chassis: it keeps the car running, lets you stop and restart a long trip, and gives you low-level control. Deep Agents is a fully equipped rental: GPS, spare tyre and a trunk (planning, subagents, filesystem) already installed. Runtime Framework Harness Example LangGraph (also Temporal, Inngest) LangChain (also Vercel AI SDK, CrewAI, OpenAI Agents SDK, Google ADK, LlamaIndex) Deep Agents SDK (also Claude Agent SDK) Value add Durable execution, streaming, human-in-the-loop, persistence Abstractions and integrations Predefined tools, prompts, subagents, planning, filesystem Use when Low-level control, long-running stateful workflows, mixing deterministic and agentic steps Getting started quickly, standardising how a team builds agents Autonomous agents on complex, non-deterministic, multi-step tasks Packages. langchain (agents, messages, tools, embeddings), langchain-core (base abstractions), one package per provider (install by extra: pip install 'langchain[anthropic]' ), langgraph (runtime, checkpointers, stores), and langchain-classic (legacy chains, retrievers, hub and community re-exports, moved out of the main namespace in v1). What changed in v1 (the things old tutorials get wrong): create_agent is the way to build an agent. The old prompt parameter is now system_prompt ; dynamic prompts go through middleware. Middleware is the defining feature: hooks around the model call and tool call replace most custom graph surgery. Messages expose standard content blocks ( content_blocks ) that look the same across providers. The langchain namespace is deliberately small. If from langchain.chains import ... or from langchain import hub fails, you want pip install langchain-classic . Agent state schemas must be TypedDict (no Pydantic state in create_agent ). Choosing the layer (quick verdict): single model call such as classify, extract or summarise → a bare model (§1). Model plus tools in a loop → create_agent . Need branching, parallel fan-out, custom loops or durable multi-step flows → LangGraph (companion page), and you can still embed a create_agent agent as a node. Long autonomous work with files and subagents → Deep Agents. 📖 Runtimes, frameworks, harnesses · What's new in v1 · Migration guide · Philosophy 1. Models Analogy: an engine you can swap. The model is the reasoning engine. LangChain gives every engine the same plug (invoke, stream, batch, bind_tools, structured output), so you can change provider without rewriting the car. 1.1 Initialise and call from langchain.chat_models import init_chat_model model = init_chat_model('anthropic:claude-sonnet-5', temperature=0.2, timeout=30, max_tokens=1000, max_retries=6) model.invoke('Why do parrots talk?') # one AIMessage for chunk in model.stream('Tell me a story'): # AIMessageChunk objects, sum them to get the full message print(chunk.text, end='', flush=True) model.batch(['q1', 'q2', 'q3'], config={'max_concurrency': 5}) # client-side parallelism Two ways in: init_chat_model('provider:model') , or the provider class ( ChatAnthropic , ChatOpenAI ...). Model names are passed straight to the provider API, so a new model works without a LangChain update. Resilience is built in: max_retries defaults to 6 with exponential backoff and jitter for network errors, 429 and 5xx. 401 and 404 are not retried. For long agent runs on shaky networks raise it to 10-15 and add a checkpointer. batch() is not the provider's batch API. It parallelises calls client-side; provider batch endpoints are a separate product. Configurable models: init_chat_model(temperature=0) with no model name lets you pick the model per call via config={'configurable': {'model': '...'}} . Use config_prefix when a chain has several models. 1.2 Tool calling and structured output on a bare model model_with_tools = model.bind_tools([get_weather]) # tool_choice='any' forces a call ai = model_with_tools.invoke('Weather in Boston and Tokyo?') # ai.tool_calls may hold several parallel calls from pydantic import BaseModel, Field class Movie(BaseModel): title: str = Field(description='Title') year: int model.with_structured_output(Movie).invoke('Details on Inception') # Movie(...) With a bare model you run the tool and send a ToolMessage back. An agent (§4) does that loop for you. with_structured_output accepts Pydantic, TypedDict or JSON Schema; method is json_schema (provider-native), function_calling (forces a tool call) or json_mode (valid JSON, schema only in the prompt). include_raw=True returns the raw message too, which is how you get token counts. 1.3 Other model features worth knowing Feature What to know Reasoning Read content_blocks of type reasoning . ⚑ A standard reasoning_effort parameter exists for several providers; check model.profile['reasoning_effort_levels'] . Model profile ⚑ model.profile lists capabilities ( max_input_tokens , tool_calling , structured_output , modalities). Summarisation middleware and structured-output auto-selection read it. It is beta and can be overridden with profile= . Multimodal Pass {'type': 'image', 'url': ...} or base64 plus mime_type as content blocks; not every model supports every type. Prompt caching Three levels: implicit (OpenAI, Gemini), explicit provider controls ( cache_control blocks for Anthropic, cachePoint for Bedrock), and middleware that caches stable system prompt and tool content. Usually only engages above a minimum token count. Server-side tools Web search and code interpreters run at the provider. The response carries server_tool_call and server_tool_result blocks in one turn, with no ToolMessage for you to return. Rate limiting rate_limiter=InMemoryRateLimiter(requests_per_second=...) limits request count only, not request size. Exceptions Standard types ( ModelRateLimitError , ModelTimeoutError , ContextOverflowError ...) each carry is_retryable , which retry middleware respects. Token usage msg.usage_metadata , or aggregate across models with get_usage_metadata_callback() . When to use a bare model: one-shot generation, classification, extraction, routing decisions inside your own code, or evaluator calls. When not to: as soon as the model must choose tools and loop, use an agent. Traps. (1) model.invoke returning a string means you built a legacy text-completion LLM, not a chat model. (2) A pre-bound model ( bind_tools already called) cannot be used where create_agent needs structured output. (3) parallel_tool_calls=False exists on some providers if your tools are not safe to run concurrently. 📖 Models · Providers and models · Chat model integrations 2. Messages Analogy: a transcript with stage directions. Each line has a speaker (role), the words (content) and margin notes (metadata such as IDs and token counts). The whole conversation is just a list of these lines. Type Role Notes SystemMessage Instructions that prime behaviour Always a SystemMessage inside ModelRequest.system_message , even when you passed a string. HumanMessage User input, text or multimodal A plain string is shorthand for one HumanMessage. AIMessage Model output .text , .content , .content_blocks , .tool_calls , .usage_metadata , .response_metadata , .id . ToolMessage Result of one tool call tool_call_id must match the AIMessage's call ID. artifact carries data for your app that is not sent to the model. Standard content blocks ( msg.content_blocks ): text (with annotations/citations), reasoning , image , audio , video , file , text-plain , tool_call , tool_call_chunk , invalid_tool_call , server_tool_call , server_tool_result and the escape hatch non_standard . content stays loosely typed (string, provider-native blocks or standard blocks); content_blocks lazily parses it into the standard form. Set output_version='v1' (or LC_OUTPUT_VERSION=v1 ) if another system must read the standard blocks from stored content. Rules that bite. Every tool call in history needs a matching ToolMessage, and most providers want history to start with a user message. Streaming chunks ( AIMessageChunk ) add up with + . Serialisation uses dumpd and load , but never call load() on untrusted data : it instantiates Python objects and can trigger side effects. 📖 Messages 3. Tools Analogy: hands for a brain. The model can only think and talk. A tool lets it reach out: read a database, call an API, write a file. The model decides whether and with what arguments to reach; your code does the reaching. 3.1 Define a tool from langchain.tools import tool @tool def search_orders(user_id: str, status: str, limit: int = 10) -> str: '''Search a user's orders by status. Use when the user asks about order history. Args: user_id: Unique user ID status: pending, shipped or delivered limit: Max results ''' ... Type hints are required (they become the input schema). The docstring is the description the model reads, so write it as a prompt: when to use it, what the arguments mean. Prefer snake_case names; spaces and special characters get rejected by some providers. Override with @tool('web_search') or description=... . Richer inputs: @tool(args_schema=PydanticModel) or a JSON schema. Reserved argument names: config and runtime . Do not use them for your own parameters. 3.2 ToolRuntime: state, context, store, streaming Add runtime: ToolRuntime to the signature. It is injected at call time and hidden from the model's schema . Field What it is Use for runtime.state Short-term memory for this thread (messages, custom fields) Read history, counters, auth flags runtime.context Immutable per-run config passed at invoke time User ID, tenant, API keys, permissions runtime.store Long-term memory, persists across threads Preferences, learned facts runtime.stream_writer Emit custom progress events 'Fetched 10/100 records' runtime.tool_call_id ID of this call Building a matching ToolMessage runtime.execution_info ⚑ thread_id, run_id, attempt number Logging, retry-aware behaviour runtime.server_info ⚑ Assistant/graph ID, authenticated user on LangGraph Server (else None) Per-user scoping on the platform from langchain.tools import tool, ToolRuntime from langchain.messages import ToolMessage from langgraph.types import Command @tool def set_user_name(new_name: str, runtime: ToolRuntime) -> Command: '''Save the user's name.''' return Command(update={ 'user_name': new_name, 'messages': [ToolMessage(content='Name saved.', tool_call_id=runtime.tool_call_id)], }) 3.3 What a tool may return Return Effect Pick it when str Becomes a ToolMessage the model reads Result is naturally text dict /object Serialised and returned; the model reads fields Downstream reasoning needs explicit fields Content-block list Multimodal ToolMessage (text plus image...) Screenshots, charts; model must support the modality Command(update=..., goto=...) Writes to state and can route The tool must change state. Include a ToolMessage with the matching tool_call_id or ToolNode raises ValueError . @tool(return_direct=True) Skips the final model call and returns the tool output Output is already the complete answer Parallel gotchas. Models may call several tools in one step. If two tools write the same state key, define a reducer for it. return_direct only short-circuits the loop when every tool call in that step has it set; a mixed batch goes back to the model. 3.4 Errors, dynamic tools, headless tools, MCP Errors: use ToolErrorMiddleware ⚑ or a @wrap_tool_call that catches the exception and returns a ToolMessage so the model can self-correct. Return the exception type rather than the raw message when details could be sensitive. Put retry inner (later in the list) so exceptions reach the error handler after retries are exhausted. Dynamic tool selection: if all tools are known up front, register them all and filter per call with @wrap_model_call and request.override(tools=...) (by auth state, role, feature flag). If tools are discovered at runtime (MCP, registry), you need two hooks: wrap_model_call to add them and wrap_tool_call to execute them, otherwise the agent cannot run the tool it just advertised. Headless tools: schema-only tools whose implementation lives in the client (browser geolocation, clipboard). The run interrupts with a tool payload and resumes when the client returns a result. MCP ⚑ (beta): from langchain.mcp import MCPAdapter ; async with MCPAdapter(url) as a: tools = await a.list_tools() , then pass to create_agent . Requires langchain[mcp]>=1.4.0 ; the standalone langchain-mcp-adapters package has a migration guide. 3.5 WHEN to use tools (decision table) Situation Use Why Needs fresh or private data (DB, API, files, web) Tool The model cannot know it Needs exact arithmetic, code execution, or a guaranteed-correct lookup Tool Models are unreliable calculators Has a side effect (send, write, delete, pay) Tool plus human approval (§9) Gate the dangerous ones Step is always executed, in a fixed order, with no judgement needed Plain code or a graph node , not a tool Do not pay tokens and latency to have a model 'decide' the obvious You only need the final answer in a fixed shape Structured output (§8) Cheaper and safer than a tool that just returns data Static knowledge that fits in the prompt Prompt or skill No call needed Big knowledge base Retrieval tool (§12) Fetch only relevant chunks More than roughly 10 tools and the model picks wrong Filter dynamically, LLMToolSelector, skills or subagents Too many tools overload context and raise errors Tool output is huge Context editing or filesystem middleware Stops results filling the window Traps. Vague docstrings cause wrong calls. A tool that returns a 50k-token blob poisons every later turn. A tool that mutates state without returning a ToolMessage breaks message history. Tools run again if a node is retried or resumed, so make side effects idempotent . 📖 Tools · MCP · Headless tools · Prebuilt tools 4. Agents (create_agent) Analogy: a new hire with a toolbox and a handbook. Agent = model + harness. The model does the thinking; the harness (prompt, tools, middleware) decides what it sees and may do. The loop is: think, pick a tool, see the result, think again, until it answers. from langchain.agents import create_agent from langgraph.checkpoint.memory import InMemorySaver from langchain_core.utils.uuid import uuid7 agent = create_agent( model='anthropic:claude-sonnet-5', tools=[search_orders], system_prompt='You are a concise support agent.', checkpointer=InMemorySaver(), # dev only name='support_agent', ) cfg = {'configurable': {'thread_id': str(uuid7())}} r1 = agent.invoke({'messages': [{'role': 'user', 'content': 'Where is order 42?'}]}, config=cfg) r2 = agent.invoke({'messages': [{'role': 'user', 'content': 'And the one before?'}]}, config=cfg) # same thread, remembers Parameter Purpose model 'provider:model' string or model instance (not pre-bound if you use structured output) tools Callables, @tool objects or provider tool dicts system_prompt String or SystemMessage (use a SystemMessage for cache control) middleware The customisation primitive (§5) response_format Schema for the final structured answer (§8) state_schema Extra state fields (TypedDict only). Prefer defining them on the middleware that uses them context_schema Shape of per-run immutable context ( runtime.context ) checkpointer Thread persistence (short-term memory, HITL, time travel) store Cross-thread long-term memory name Identifier; vital when the agent is a subgraph, and surfaces as lc_agent_name in stream metadata thread_id versus context. thread_id scopes the conversation (history and checkpoints). context= carries per-run data (user ID, keys). In production pass both on every call. Result shape. result['messages'] is the full history; result['structured_response'] exists when response_format is set. With version='v2' , invoke returns an object with .value and .interrupts (⚑ LangGraph 1.1+). Limits. recursion_limit is a top-level key of config (not inside configurable ); LangGraph 1.0.6+ defaults it to 1000 steps and raises GraphRecursionError beyond it. Add ModelCallLimitMiddleware so a looping agent cannot burn money. When create_agent is enough: a model with tools, memory, guardrails and approvals. When to drop to LangGraph: explicit branching and fan-out you must control, deterministic steps interleaved with agentic ones, or custom loops. 📖 Agents · Quickstart · Runtime 5. Middleware Analogy: airport security checkpoints along the agent's route. Before boarding (before_agent), before each flight (before_model), wrapped around the flight itself (wrap_model_call), after landing (after_model), around each baggage handoff (wrap_tool_call), and at the exit (after_agent). Each checkpoint can inspect, change or turn the passenger around. Style Hook Runs Node before_agent / after_agent Once per invocation Node before_model / after_model Around each model response Wrap wrap_model_call Around each model call (you choose how often to call handler : 0 = short-circuit, 1 = normal, N = retry) Wrap wrap_tool_call Around each tool call Convenience @dynamic_prompt Build the system prompt per call Order. before_* run first to last; after_* run last to first; wrap_* nest like function calls (first middleware is outermost). Put critical guards first. from langchain.agents.middleware import before_model, wrap_model_call, dynamic_prompt, AgentState, ModelRequest, ModelResponse from langgraph.runtime import Runtime from langchain.messages import AIMessage @before_model(can_jump_to=['end']) def stop_long_chats(state: AgentState, runtime: Runtime): if len(state['messages']) >= 50: return {'messages': [AIMessage('Conversation limit reached.')], 'jump_to': 'end'} @dynamic_prompt def role_prompt(request: ModelRequest) -> str: return f'You are a helpful assistant for a {request.runtime.context.user_role}.' @wrap_model_call def route_model(request: ModelRequest, handler) -> ModelResponse: model = big_model if len(request.messages) > 10 else small_model return handler(request.override(model=model)) Jumps: return {'jump_to': 'end' | 'tools' | 'model'} , and declare the target with can_jump_to or @hook_config . State updates: node hooks return a dict applied through the graph's reducers. Wrap hooks return ExtendedModelResponse(model_response=..., command=Command(update=...)) for model calls, or a Command for tool calls. With several layers, commands compose: messages are additive, and for non-reducer keys the outermost value wins. If an outer layer retries, commands from earlier attempts are dropped. Middleware-scoped state: subclass AgentMiddleware , set state_schema = MyState (extra fields live and die with that middleware), and optionally ship tools (for example write_todos ) and stream transformers ⚑. Class versus decorator: decorators for one hook; classes for several hooks, configuration, or when you need both sync and async ( abefore_model ...). Middleware is not a separate runtime. The hooks run inside the compiled LangGraph, so you can embed the whole agent as a node in a larger StateGraph and its HITL, summarisation and retries come with it. 5.1 Built-in middleware catalogue Middleware Use it for Key options ToolErrorMiddleware ⚑ Turn tool exceptions into error ToolMessages the model can recover from on_error (return None to propagate), tools ToolRetryMiddleware Retry flaky tools with backoff max_retries=2 , backoff_factor=2.0 , retry_on , on_failure='continue'|'error' ModelRetryMiddleware Retry model calls Same options; ⚑ default respects is_retryable ModelFallbackMiddleware Fail over to other models/providers Ordered list of fallback models SummarizationMiddleware Compress old history persistently trigger (tokens/messages/fraction, AND within a dict, OR across a list), keep ContextEditingMiddleware Clear old tool outputs when context grows ClearToolUsesEdit(trigger, keep=3, exclude_tools) HumanInTheLoopMiddleware Approve, edit, reject tool calls interrupt_on , interrupt_mode , when ⚑ ModelCallLimitMiddleware Cap model calls (cost, loops) thread_limit (needs checkpointer), run_limit , exit_behavior ToolCallLimitMiddleware Cap total or per-tool calls tool_name , thread_limit , run_limit , exit_behavior PIIMiddleware Detect/handle PII strategy block|redact|mask|hash , custom detector , apply_to_input/output/tool_results TodoListMiddleware Planning via a write_todos tool Custom prompt/description LLMToolSelectorMiddleware Pre-select relevant tools when you have 10+ model , max_tools , always_include ProviderToolSearchMiddleware Defer tools behind provider-side tool search searchable_tools ; Anthropic and recent OpenAI models only ShellToolMiddleware Persistent shell session Execution policy: host, Docker or sandbox. Not compatible with interrupts yet FilesystemFileSearchMiddleware Glob and grep tools over a directory root_path , use_ripgrep FilesystemMiddleware , SubAgentMiddleware , MemoryMiddleware , SkillsMiddleware (deepagents) Files as context, delegation, memory files, on-demand skills Backends: StateBackend , StoreBackend , CompositeBackend RubricMiddleware (beta) Self-grade against a rubric and iterate max_iterations LLMToolEmulator Fake tool results for tests tools , model Provider-specific Anthropic prompt caching, bash, text editor, memory; Bedrock prompt caching; OpenAI moderation See integrations A sensible production stack (outer to inner): call limits → HITL → PII → summarisation or context editing → model retry/fallback → tool error → tool retry. The one ordering rule worth memorising: HITL must wrap retry and error middleware in per-call mode, or a retried call asks the reviewer again. Traps. Keep middleware single-purpose. Hooks that run before a HITL pause run again on resume, so they must be side-effect free. thread_limit on any limiter silently does nothing without a checkpointer. Summarisation compresses text only; images are not shrunk. 📖 Overview · Prebuilt · Custom 6. Memory: when to use which Analogy: a notepad on your desk versus a filing cabinet in the archive. The notepad (short-term memory) holds this conversation and is thrown away or archived with it. The cabinet (long-term memory) holds things worth knowing in any future conversation. The desk is small, so you also need a rule for when to summarise the notepad. 6.1 Decision table You need… Use Mechanism The agent to remember earlier turns of this conversation Short-term memory checkpointer plus a stable thread_id Pause and resume later (approvals, crashes) Short-term memory Same checkpointer; required for HITL The conversation is getting too long for the context window Trim, delete or summarise @before_model trim, RemoveMessage , or SummarizationMiddleware Remember a user across sessions (preferences, profile, learned facts) Long-term memory store with namespace and key; read/write from tools or middleware Find memories by meaning, not exact key Store with semantic index InMemoryStore(index={'embed':..., 'dims':...}) then store.search(query=...) Per-request settings (user ID, tenant, keys) Not memory: runtime context context_schema plus context= on invoke Large intermediate artefacts (search dumps, files) Filesystem , not messages FilesystemMiddleware ; route /memories/ to a StoreBackend to persist Static reference knowledge Not memory: prompt, skills or retrieval §12 6.2 Short-term memory (thread scope) from langgraph.checkpoint.postgres import PostgresSaver # pip install langgraph-checkpoint-postgres 'psycopg[binary]' with PostgresSaver.from_conn_string(DB_URI) as cp: cp.setup() # creates tables agent = create_agent('openai:gpt-5.5', tools=[...], checkpointer=cp) agent.invoke({'messages': 'Hi, I am Bob'}, {'configurable': {'thread_id': 'cust-17'}}) State is read at the start of each step and written when a step completes. Custom fields: subclass AgentState (TypedDict) and pass it through state_schema or, better, a middleware's state_schema . InMemorySaver is for development; use Postgres, SQLite, Mongo or Cosmos-backed savers in production. # Trim (transient is possible via wrap_model_call; this one rewrites state) from langchain.messages import RemoveMessage from langgraph.graph.message import REMOVE_ALL_MESSAGES @before_model def trim(state: AgentState, runtime: Runtime): m = state['messages'] if len(m) <= 3: return None return {'messages': [RemoveMessage(id=REMOVE_ALL_MESSAGES), m[0], *m[-3:]]} # Summarise (persistent) SummarizationMiddleware(model='openai:gpt-5.4-mini', trigger=('tokens', 4000), keep=('messages', 20)) Choosing a strategy. Trim is cheap and lossy: fine for chit-chat. Delete is surgical: remove specific messages. Summarise costs an extra model call but keeps meaning: use for long support or research threads. Context editing clears old tool outputs only: use when tool results are what bloat the window. Always keep history valid : a tool-call AIMessage must be followed by its ToolMessage, and some providers want a user message first. RemoveMessage only works on a key with the add_messages reducer (the default messages key has it). 6.3 Long-term memory (cross-thread) from langgraph.store.memory import InMemoryStore # PostgresStore in production store = InMemoryStore(index={'embed': embeddings, 'dims': 1536, 'fields': ['$']}) store.put(('user_123', 'memories'), 'm1', {'text': 'Prefers short, direct answers'}) hits = store.search(('user_123', 'memories'), query='communication style', limit=3) @tool def save_memory(text: str, runtime: ToolRuntime[Context]) -> str: '''Save a durable fact about the user.''' runtime.store.put((runtime.context.user_id, 'memories'), str(uuid7()), {'text': text}) return 'saved' agent = create_agent(model, tools=[save_memory], store=store, context_schema=Context) Model: JSON documents under a namespace tuple (like a folder, usually including user or org ID) and a key (like a file name). Cross-namespace search is by prefix and content filter. Nothing is injected automatically. You must read memories (a tool the model can call, or a @dynamic_prompt / wrap_model_call that injects them) and write them (a tool, or a background step after the turn). Decide deliberately between 'model decides what to remember' (tool, flexible, sometimes forgets) and 'we extract after every turn' (middleware, consistent, costs a call). Use user-scoped namespaces so one user can never read another's memories. Namespace prefix matching returns sub-namespaces too. On the managed platform the store is provisioned for you; semantic search needs index settings in langgraph.json . Traps. (1) Reusing a thread_id across users leaks history. (2) A fresh thread_id means a blank notepad, not a lost cabinet: long-term memory survives because it is in the store. (3) InMemoryStore and InMemorySaver vanish on restart. (4) Summarisation permanently replaces messages; if you need the originals for audit, write them elsewhere first. 📖 Short-term memory · Long-term memory · Memory concepts · Checkpointer integrations · Store integrations 7. Context engineering Analogy: packing the briefing folder for a consultant. The consultant is capable, but if the folder holds the wrong pages, the advice will be wrong. Most agent failures are the wrong folder, not a weak consultant. Agents usually fail because the right context was not passed to the model, not because the model is too weak. LangChain splits what you control into three kinds: Context type You control Lifetime Model context System prompt, messages, tools, model, response format for one call Transient (does not change saved state) Tool context What tools read and write (state, store, runtime context) Persistent Life-cycle context What happens between steps: summarise, guardrails, logging Persistent Data source Also called Scope Examples Runtime context Static config One run User ID, role, API keys, environment State Short-term memory One conversation Messages, uploaded files, auth status Store Long-term memory Across conversations Preferences, extracted insights Transient versus persistent. wrap_model_call with request.override(...) changes only what this call sees. To change saved history, return a Command from the wrap hook or use before_model , after_model or a tool. Recency matters. Models pay more attention to the last messages, so injected context (compliance rules, style guide) is appended after history rather than before it. Typical moves: dynamic system prompt by role or environment; inject uploaded-file summaries; filter tools by authentication state; route to a bigger model for long threads; switch response_format by stage. Start simple: static prompt and tools first, add one dynamic behaviour at a time, and watch token use and latency. 📖 Context engineering in agents · Context overview 8. Structured output Analogy: a form instead of an essay. You hand the model a form with typed boxes. You get back something your code can use without parsing prose. from pydantic import BaseModel, Field from langchain.agents.structured_output import ToolStrategy, ProviderStrategy class Ticket(BaseModel): category: str = Field(description='billing, technical, account or product') priority: str summary: str agent = create_agent(model='openai:gpt-5.5', tools=[], response_format=Ticket) # auto strategy r = agent.invoke({'messages': [{'role': 'user', 'content': 'My card was charged twice!'}]}) r['structured_response'] # Ticket(...) Option Behaviour Pick it when Pass a schema type Auto: provider-native if the model supports it, else tool-calling Default ProviderStrategy(schema, strict=...) ⚑ Provider enforces the schema; most reliable Provider supports it; need strict adherence ToolStrategy(schema, handle_errors=...) Schema becomes a tool; model must call it; validation errors are fed back for retry Provider lacks native support; union types Schema types: Pydantic (returns validated instance), dataclass or TypedDict (return a dict), JSON Schema (needs top-level title and description , and must be wrapped in an explicit strategy), and Union[...] with ToolStrategy so the model picks one shape. handle_errors : True (default, retry with a template message), a string (always that message), an exception type or tuple (only those), a callable returning a message, or False (raise). If the model returns two structured outputs at once, the agent feeds back an error and asks for one. tool_message_content customises the confirmation line that lands in history. If you pass tools and a schema, the model must support using both together. Structured output versus a tool: use response_format for the final answer's shape ; use a tool when something must happen mid-run. Outside agents, use model.with_structured_output (§1). 📖 Structured output 9. Human-in-the-loop and guardrails Analogy: a second signature on the cheque. The agent may prepare the transfer, but for large amounts a human signs before it goes out. The agent waits, with its place saved, as long as it takes. from langchain.agents.middleware import HumanInTheLoopMiddleware from langgraph.types import Command agent = create_agent('openai:gpt-5.5', tools=[write_file, execute_sql, read_data], middleware=[HumanInTheLoopMiddleware(interrupt_on={ 'write_file': True, # all decisions allowed 'execute_sql': {'allowed_decisions': ['approve', 'reject']}, 'read_data': False, # auto-approve })], checkpointer=InMemorySaver()) # REQUIRED cfg = {'configurable': {'thread_id': 't1'}} r = agent.invoke({'messages': [...]}, config=cfg, version='v2') r.interrupts # what needs review agent.invoke(Command(resume={'decisions': [{'type': 'approve'}]}), config=cfg, version='v2') # same thread_id Decision Effect Note approve Run as proposed edit Run with changed args ( edited_action ) Edit conservatively; big edits make the model re-plan and may repeat the call reject Skip and send feedback message Use this to deny. Say whether the agent may retry respond Your reply becomes the tool result For 'ask user' tools only. The model sees a success , so never use it to deny Batched mode (default): one interrupt per model turn listing all gated calls; resume with an ordered decisions list. interrupt_mode='per_call' ⚑: one interrupt per call, each with a response_schema , resumed with a dict keyed by interrupt ID. The docs recommend per-call for new agents and plan to make it the default next major version. Conditional gating ⚑: 'when': predicate(request) pauses only risky calls (for example SQL that is not a SELECT). The predicate re-runs on resume and must give the same answer, or the reviewer's decision is ignored. In per-call mode an invalid answer is rejected before saving, so send a corrected answer to the same interrupt ID. If several answers are sent together and one is invalid, others may have run, so validate first or answer one at a time. Stream while pausing: stream = agent.stream_events(..., version='v3') , check stream.interrupted and stream.interrupts . Guardrails are policies enforced deterministically regardless of what the model does: PIIMiddleware (redact/mask/hash/block, with custom detectors), call limits, and your own after_agent checks (including model-based safety checks using jump_to='end' ). A system prompt asking the model to behave is not a guardrail. 📖 Human-in-the-loop · HITL concepts · Guardrails 10. Streaming Analogy: a live sports ticker instead of the final score. You can show progress (each step), words as they are typed (tokens), or your own custom events ( 'Fetched 10/100 records' ). Mode Yields updates State update after each agent step (model, tools, model) messages (token, metadata) from any node that calls an LLM (metadata has langgraph_node , lc_agent_name ) custom Anything written with runtime.stream_writer or get_stream_writer() values Full state after each step # Classic modes: v2 format = one dict per chunk with type, ns, data for chunk in agent.stream(inp, stream_mode=['updates', 'messages', 'custom'], version='v2'): if chunk['type'] == 'messages': token, meta = chunk['data'] # Event streaming (recommended for new apps, langchain 1.3+): one typed iterator per projection stream = agent.stream_events(inp, config=cfg, version='v3') for kind, item in stream.interleave('messages', 'tool_calls'): if kind == 'messages': for t in item.text: print(t, end='') final = stream.output Reasoning tokens: iterate message.reasoning in event streaming, or filter content_blocks for reasoning . Subagents and nested graphs: pass subgraphs=True on classic streaming, or read stream.subgraphs in event streaming. Name every agent so you can tell who is talking. Tool streaming: tool calls arrive as tool_call_chunk blocks (partial JSON); complete calls come from updates or by summing chunks until chunk_position == 'last' . Turn it off for a model with streaming=False (or disable_streaming=True ) when you do not want its tokens reaching the client. If a custom tool uses get_stream_writer , it can no longer run outside a LangGraph context. 📖 Streaming · Event streaming · Frontend 11. Multi-agent patterns Analogy: a hospital. A generalist (single agent) can handle most cases. When the case needs a specialist's long reference books, you either call the specialist in ( subagent ), transfer the patient ( handoff ), look up a protocol ( skill ), or triage at the door ( router ). First question: do you need multi-agent at all? A single agent with the right tools and prompt often suffices. Reach for multi-agent when (a) context management: specialised knowledge would overflow one prompt, (b) distributed development: separate teams own capabilities, or (c) parallelisation. Also when one agent has too many tools and picks poorly. Pattern How it works Strengths Subagents Main agent calls subagents as tools; all routing goes through it Isolation, parallel calls, large-context domains; stateless by default Handoffs Tool calls change a state variable; the active agent or its prompt/tools switch Direct user conversation with the specialist; stateful Skills One agent loads specialised prompts on demand Simple, directly conversational; context accumulates Router A step classifies input and sends it to one or more agents, then synthesises Parallel fan-out across knowledge sources; stateless Custom workflow LangGraph graph mixing deterministic and agentic nodes Full control; embed any other pattern as a node Measured trade-offs from the docs: one-shot 'buy coffee' costs 4 model calls with subagents but 3 with handoffs, skills or router. A repeat request costs 8 total with subagents (stateless) versus 5 with handoffs or skills (stateful). Multi-domain comparison (three languages, about 2K tokens of docs each): subagents and router about 9K tokens, skills about 15K (context accumulates), handoffs 14K+ and sequential. Rule of thumb: parallel and big-context → subagents or router; repeat and conversational → handoffs or skills. @tool def ask_fruit_expert(question: str) -> str: '''Ask the fruit expert. Use for ALL fruit questions.''' r = fruit_agent.invoke({'messages': [{'role': 'user', 'content': question}]}) return r['messages'][-1].content supervisor = create_agent(model, tools=[ask_fruit_expert, ask_veggie_expert], checkpointer=cp) Traps. A per-thread stateful subagent ( checkpointer=True ) breaks when the model calls it twice in parallel (checkpoint conflict): add ToolCallLimitMiddleware(tool_name=..., run_limit=1) . The langgraph-supervisor package has a migration guide to this tool-wrapped pattern. Deep Agents ships subagents, skills, planning and filesystem if you would rather not assemble this yourself. 📖 Multi-agent · Subagents · Handoffs · Skills · Router · Custom workflow 12. Retrieval (RAG) Analogy: an open-book exam. Instead of memorising the library, the model gets a librarian (the retriever) who fetches the right pages when asked. Scope note: I did not re-read the RAG tutorials for this page; this section is the stable shape plus pointers. Verify names against the tutorials. The pipeline is: load documents → split into chunks → embed ( init_embeddings ) → store in a vector store → retrieve. Two ways to use it: agentic RAG (expose a retrieval tool ; the model decides when to search and can search several times) and two-step RAG (always retrieve first, then answer; predictable and cheap). Return the passage as the ToolMessage content and put document IDs or page numbers in artifact so your UI can cite without bloating the prompt. Use RAG when knowledge is large, changes often, or must be cited. Do not use it for facts that fit in the prompt, or as a substitute for structured data you can query exactly (use a SQL tool). Memory versus RAG: memory is about this user and these conversations ; RAG is about shared documents . 📖 Build a semantic search engine · Agentic RAG with LangGraph · SQL agent · Vector stores 13. Production: reliability, cost, security, testing Reliability stack: model max_retries plus ModelRetryMiddleware , ModelFallbackMiddleware for provider outages, tool retry and error middleware, a durable checkpointer, and call limits as circuit breakers. Cost: prompt caching, summarisation or context editing, a cheap model for routing and summaries (a wrap_model_call router), tool filtering, and ModelCallLimitMiddleware . Security: treat tool output as untrusted input (prompt injection can arrive inside a web page or document); gate side effects with HITL; run shell tools under Docker or a sandbox policy, never the host policy on untrusted input; redaction rules do not stop exfiltration on the host; use PII middleware; sanitise error text returned to the model; never load() untrusted serialised messages; scope store namespaces per user. Observability: set up LangSmith tracing from day one; it records every turn, tool call and middleware span. Use a trace_policy ⚑ to omit large payloads. Testing: unit-test logic with fake chat models and in-memory persistence (no API calls); integration-test with real models (manage keys, flakiness, cost); evaluate trajectories with deterministic matching or LLM-as-judge; LLMToolEmulator fakes tools when real ones are expensive. Deployment: LangSmith deployment provisions checkpointer and store for you; locally pass them explicitly. 📖 Testing · Observability · Deployment · LangSmith Studio 14. Version gates and the trap list ⚑ Feature Minimum Model profiles, fraction triggers in summarisation langchain 1.1 ProviderStrategy(strict=...) langchain 1.2 Middleware stream transformers; PII output redaction in streams; event streaming langchain 1.3.x (1.3.2 for transformers/PII) HITL when predicate 1.3.3 ToolErrorMiddleware 1.3.14 Middleware trace_policy 1.3.15 Retry middleware respects is_retryable by default 1.3.16 langchain.mcp (beta) langchain[mcp] 1.4.0 HITL interrupt_mode='per_call' 1.4.6 reasoning_effort standard parameter langchain-core 1.5.2 plus provider package versions runtime.execution_info , server_info ; v2 stream format langgraph 1.1.x Trap checklist (read before every review): No checkpointer, yet expecting memory or HITL. Or a reused thread_id across users. InMemory* in production. Tool with a vague docstring, or no ToolMessage after returning a Command. Too many tools; huge tool outputs; no call limits. Retry placed outside HITL; side effects before an interrupt. Structured output with a pre-bound model, or a raw JSON Schema not wrapped in a strategy. Old tutorial imports ( prompt= , langchain.chains , Pydantic agent state) on a v1 install. Trusting the system prompt as a security control. 15. Exercises Build an agent with one @tool that reads runtime.context.user_id , a checkpointer , and a thread_id . Prove it remembers turn 1 on turn 2, then change only the thread_id and show it forgets. Add SummarizationMiddleware with a tiny trigger ( ('messages', 6) ), run 10 turns, and print the history to see the summary message replace the old ones. Add a save_memory tool plus an InMemoryStore . In thread A tell the agent a preference; in thread B (new thread_id , same user_id ) show it can recall it via a search tool. Gate execute_sql with HITL so only non-SELECT queries pause (a when predicate). Approve one, reject one with a message, and observe how the model reacts to the rejection. Write a wrap_model_call router that uses a cheap model for the first 6 messages and a stronger one after. Verify with usage_metadata or LangSmith. Solution notes Same thread_id reloads state from the checkpointer; a new one starts empty. context is separate and must be passed on every call. Summarisation is persistent: the checkpointed state now holds a summary message plus the last keep messages. Trim, by contrast, only changes what is sent if done in wrap_model_call . The store key space is (user_id, 'memories') , independent of thread. The agent will not use the memory unless a tool or prompt injection reads it. when returns True to pause. On resume it is evaluated again and must match. A reject message should tell the agent whether to retry. request.override(model=...) changes only this call; models should be created once outside the hook. Sources: docs.langchain.com (Python): LangChain overview , products , agents , models , messages , tools , short-term memory , long-term memory , middleware , structured output , HITL , streaming , multi-agent , context engineering . RAG section: scope note above. LangGraph — Components Deep Dive A study guide to LangGraph 1.x (Python), taken one component at a time: state, nodes, edges, compile and run, persistence, interrupts, fault tolerance, subgraphs, workflow patterns, the Functional API and streaming. Each part says what it is, shows the code, says when to use it, and lists the traps. How to read this guide. Each component has (1) a mental model, (2) the code you will type, (3) a when to use / when not to verdict, (4) traps, and (5) 📖 doc links. Section 12 is the decision table for “when do I use which kind of memory?” and section 9 for “which workflow pattern?” . The companion page is LangChain — Components Deep Dive (tools, middleware, memory from the agent side). Version note: written 17 Oct 2026 from docs.langchain.com. ⚑ marks features that need a minimum version (table in §14). The Functional API section (§10) draws on the graph-API docs and my prior knowledge more than on a full re-read, so verify it against the linked pages. 0. The map: what LangGraph is Analogy: a factory floor with a conveyor and a clipboard. The clipboard is the shared state. Each station (node) reads the clipboard, does a job, and writes updates. Signs on the floor (edges) say which station comes next. A supervisor with a camera (the checkpointer) photographs the clipboard after every shift, so if the power fails you restart from the last photo, not from the start. Three primitives: State (shared snapshot plus the rules for merging updates), Nodes (functions that do the work and return updates), Edges (functions or fixed links that choose what runs next). Nodes do the work, edges say what to do next. Nodes and edges are plain functions: they may contain an LLM or just ordinary code. Execution model (Pregel-inspired message passing): the program runs in discrete super-steps . Nodes that run in parallel share a super-step; sequential nodes are separate ones. A node becomes active when it receives a message on an incoming edge, runs, and emits updates. The run ends when every node is inactive and nothing is in transit. Checkpoints are written at super-step boundaries , which explains most of the behaviour in §5-7. LangGraph is a low-level runtime. It gives you durable execution, streaming, human-in-the-loop and persistence. LangChain's create_agent is built on it, so you can start high-level and drop down. The docs' guidance: use LangGraph for fine-grained orchestration control, long-running stateful work, workflows mixing deterministic and agentic steps, and production-grade deployment. Two APIs: the Graph API ( StateGraph , explicit nodes and edges; this page's main subject) and the Functional API ( @entrypoint and @task on ordinary Python control flow; §10). You need… Use A model plus tools in a loop, memory, approvals create_agent (LangChain) Branching, fan-out/fan-in, explicit loops, deterministic steps between model calls LangGraph Graph API Mostly linear Python with loops and ifs, but you want checkpointing and interrupts Functional API Planning, files and subagents out of the box for long autonomous tasks Deep Agents (built on LangGraph) 📖 Overview · Thinking in LangGraph · Graph API overview · Choosing between Graph and Functional · LangGraph runtime 1. State Analogy: a shared whiteboard with one marker per column. Each column (state key) has its own rule for what happens when someone writes: overwrite the old value, or append to it. 1.1 Schemas TypedDict is the main documented choice. Use a dataclass if you want default values. A Pydantic BaseModel adds recursive validation but is slower. ( create_agent in LangChain only supports TypedDict state.) Input/output/private schemas. By default input and output equal the graph schema. Pass input_schema= and output_schema= to constrain what the graph accepts and returns; nodes can also write to extra private channels declared by a schema they use. from typing_extensions import TypedDict from langgraph.graph import StateGraph, START, END class InputState(TypedDict): user_input: str class OutputState(TypedDict): graph_output: str class OverallState(TypedDict): foo: str user_input: str graph_output: str class PrivateState(TypedDict): bar: str def node_1(state: InputState) -> OverallState: return {'foo': state['user_input'] + ' name'} def node_2(state: OverallState) -> PrivateState: return {'bar': state['foo'] + ' is'} def node_3(state: PrivateState) -> OutputState: return {'graph_output': state['bar'] + ' Lance'} b = StateGraph(OverallState, input_schema=InputState, output_schema=OutputState) b.add_node('node_1', node_1); b.add_node('node_2', node_2); b.add_node('node_3', node_3) b.add_edge(START, 'node_1'); b.add_edge('node_1', 'node_2'); b.add_edge('node_2', 'node_3'); b.add_edge('node_3', END) b.compile().invoke({'user_input': 'My'}) # {'graph_output': 'My name is Lance'} Trap: private is not secret. Schemas constrain what a node reads and what invoke returns . When streaming with stream_mode='values' the graph emits all channels, private ones included. Pass output_keys=[...] to restrict, or use stream_mode='updates' . 1.2 Reducers Each key has its own reducer: a binary function new = reducer(left=current, right=update) . With no reducer, the update overwrites . from operator import add from typing import Annotated class State(TypedDict): foo: int # overwrite bar: Annotated[list[str], add] # append # state {'foo':1,'bar':['hi']}; a node returns {'foo':2} then {'bar':['bye']} -> {'foo':2,'bar':['hi','bye']} Messages: use Annotated[list[AnyMessage], add_messages] , or subclass MessagesState ( class State(MessagesState): documents: list[str] ). add_messages tracks message IDs , so updating an existing message replaces it and RemoveMessage deletes one; operator.add would just append duplicates. It also deserialises dicts ( {'type': 'human', 'content': ...} ) into message objects, so read attributes with dot notation. Parallel writes: two nodes writing the same key in one super-step need a reducer, or the update is ambiguous. Overwrite: a merging reducer cannot be cleared by returning an empty value ( [] is merged in). Wrap the update: {'errors': Overwrite([])} (from langgraph.types ). This matters for error buffers and retry counters. RemainingSteps: a managed value, remaining_steps: RemainingSteps , tells a node how many super-steps are left before the recursion limit (§4). When to put something in state: if more than one node needs it or it must survive an interrupt or crash. When not to: connections, clients, secrets and per-run config belong in runtime context (§2), because state is checkpointed and must be serialisable. 📖 State · Use the graph API (how-tos) 2. Nodes Analogy: a station on the floor. It receives the clipboard, does one job, writes back only the lines it changed. A node is a sync or async Python function. It can accept state , then optionally config (a RunnableConfig with thread_id , tags) and runtime (a Runtime[Context] ). A node returns a partial update , not the whole state. from dataclasses import dataclass from langgraph.runtime import Runtime @dataclass class Context: user_id: str def my_node(state: State, runtime: Runtime[Context]): print(runtime.context.user_id, runtime.execution_info.thread_id) return {'results': 'Hello, {}!'.format(state['input'])} builder = StateGraph(State, context_schema=Context) builder.add_node('my_node', my_node) # name defaults to the function name if omitted Runtime field Use context Per-run immutable config ( context_schema ; pass context= to invoke) store Long-term memory store stream_writer Emit custom stream events execution_info ⚑ thread_id , run_id , checkpoint_id , task_id , node_attempt (1-indexed), node_first_attempt_time server_info ⚑ Assistant/graph ID and authenticated user when running on LangGraph Server heartbeat() ⚑ Reset an idle timeout manually (§7) control / drain_requested ⚑ Graceful-shutdown signals (§7) 2.1 Re-execution and idempotency (the rule behind many bugs) Checkpoints are saved at super-step boundaries, not mid-function . After an interrupt or a retry the affected node runs again from its first line , so any side effect before the pause runs again. Design nodes so a second run is harmless: upserts instead of inserts, idempotency keys, read-before-write. You may freely add or remove nodes and edges for existing threads; determinism rules apply only to tasks and interrupts inside a node. 2.2 Tasks inside a node If a node does several expensive or non-idempotent operations, wrap each in a @task (from langgraph.func ). Task results are checkpointed, so on resume completed tasks are skipped instead of re-run. from langgraph.func import task @task def fetch(url: str): return requests.get(url).text[:100] def call_api(state): futures = [fetch(u) for u in state['urls']] # run concurrently return {'results': [f.result() for f in futures]} 2.3 Node caching from langgraph.types import CachePolicy from langgraph.cache.memory import InMemoryCache builder.add_node('expensive', expensive, cache_policy=CachePolicy(ttl=3)) # ttl seconds; key_func defaults to a hash of the input graph = builder.compile(cache=InMemoryCache()) Use caching for pure, expensive functions (embedding a fixed text, a slow lookup). Do not cache nodes with side effects or time-dependence. The result carries __metadata__: {'cached': True} on a hit. 📖 Nodes 3. Edges, Send and Command Analogy: signs on the factory floor. Some arrows are painted on (normal edges). Some depend on what is on the clipboard (conditional edges). A foreman can also say 'make one copy of this job per item on the list' (Send) or 'do this update and go there next' (Command). 3.1 Edge types graph.add_edge('a', 'b') # always a -> b graph.add_edge(START, 'a'); graph.add_edge('b', END) # entry / finish graph.add_conditional_edges('a', route_fn) # route_fn(state) returns a node name or list of names graph.add_conditional_edges('a', route_fn, {True: 'b', False: 'c'}) # optional path map graph.add_conditional_edges(START, route_fn) # conditional entry point Fan-out is automatic: a node with several outgoing edges runs all destinations in parallel in the next super-step. A conditional edge that returns a list does the same. One routing mechanism per node. Use normal edges or dynamic routing (conditional edges, Command , tool-driven goto ), never both. Command only adds dynamic edges; static edges still fire, so you would run both destinations. 3.2 Command Command(update=..., goto=..., graph=..., resume=...) . Return it from a node to update state and route in one step. You must annotate the return type with the possible destinations, -> Command[Literal['next_node']] , so rendering and validation know where it can go. def decide(state: State) -> Command[Literal['approve', 'reject']]: return Command(update={'checked': True}, goto='approve' if state['score'] > 5 else 'reject') graph=Command.PARENT navigates from inside a subgraph to a node in the parent (the basis of multi-agent handoffs). If the key is shared by parent and subgraph schemas, the parent needs a reducer for it. Input to invoke : Command(resume=...) is the only Command intended as run input. Never send Command(update=...) to continue a conversation: any Command input resumes from the latest checkpoint, so a finished graph looks stuck. Send a plain input dict ( {'messages': [...]} ) to start a new turn on an existing thread. Use Command versus a conditional edge: Command when you must update state and route; conditional edge when you only route. 3.3 Send (map-reduce) Analogy: a teacher hands each student a different page of the exam to mark, then collects the marks into one pile. from langgraph.types import Send def assign_workers(state: State): return [Send('write_section', {'section': s}) for s in state['sections']] # one worker per item, each with its own input class State(TypedDict): sections: list[Section] completed: Annotated[list, operator.add] # workers write here in parallel, so it needs a reducer graph.add_conditional_edges('orchestrator', assign_workers, ['write_section']) Use Send when the number of parallel branches is unknown until runtime and each branch needs its own input. A Send can carry its own timeout= ⚑ that overrides the target node's default. 📖 Edges · Command · Map-reduce and Send how-to 4. Compile and run You must compile() before use. Compilation checks structure (no orphan nodes) and is where you attach runtime pieces. graph = builder.compile( checkpointer=InMemorySaver(), # thread persistence, HITL, time travel store=my_store, # cross-thread memory cache=InMemoryCache(), # node caching interrupt_before=['node_a'], # static breakpoints (debugging only) ) config = {'configurable': {'thread_id': 't1'}, 'recursion_limit': 50} result = graph.invoke({'messages': [...]}, config, context={'llm_provider': 'anthropic'}) thread_id lives in configurable . recursion_limit is a top-level key of config, not inside configurable . Context is a separate context= argument. Recursion limit = max super-steps in one run. Default 1000 since LangGraph 1.0.6; exceeding it raises GraphRecursionError . The step counter is config['metadata']['langgraph_step'] . Proactive handling (recommended): a RemainingSteps state field plus a router that sends the graph to a fallback node when remaining_steps <= 2 , so the run ends gracefully with partial results and checkpoints. Reactive: catch GraphRecursionError outside; simpler, but the run is terminated. Other metadata in config['metadata'] : langgraph_node , langgraph_triggers , langgraph_path , langgraph_checkpoint_ns . Visualise: graph.get_graph().draw_mermaid_png() (use xray=True to expand subgraphs). v2 output ⚑: invoke(..., version='v2') returns an object with .value (state) and .interrupts , cleanly separating interrupts from state. 📖 Recursion limit · Quickstart · Run a local server 5. Persistence: checkpointers, threads, stores Analogy: save slots in a video game. A thread is one save file. A checkpoint is a save point written at the end of every super-step. Loading a slot continues from the last save; you can also load an older save and branch ( time travel ). The store is the shared hall of fame that every save file can read. Checkpointer: compile with one and LangGraph snapshots state at every super-step, organised in threads. thread_id is the primary key : without it nothing is saved and nothing can resume after an interrupt. Pending writes: if one node fails mid-super-step, successful sibling nodes' writes are stored, so a retry does not redo them. What persistence unlocks: human-in-the-loop, conversational (short-term) memory, time-travel replay and forking, and fault-tolerant execution. Backends: InMemorySaver (dev), SQLite, Postgres ( PostgresSaver , AsyncPostgresSaver ), MongoDB, Azure Cosmos DB. Call .setup() once for SQL backends. On the Agent Server all of this is provisioned for you. Inspect: graph.get_state(config) ; add subgraphs=True to see nested state (only works when the subgraph is statically discoverable: added as a node or called inside a node, not when hidden inside a tool). 5.1 Stores (cross-thread) A store holds JSON documents under a namespace tuple and a key , accessible from any thread. It is the right place for user preferences and learned facts. from langgraph.store.memory import InMemoryStore # PostgresStore / MongoDB / Redis in production store = InMemoryStore(index={'embed': embeddings, 'dims': 1536, 'fields': ['food_preference', '$']}) ns = (user_id, 'memories') store.put(ns, str(uuid7()), {'food_preference': 'I like pizza'}, index=['food_preference']) # index=False: stored, not searchable store.search(ns, query='What does the user like to eat?', limit=3) store.list_namespaces(prefix=(user_id,), max_depth=2) graph = builder.compile(checkpointer=checkpointer, store=store) async def call_model(state: MessagesState, runtime: Runtime[Context]): hits = await runtime.store.asearch((runtime.context.user_id, 'memories'), query=state['messages'][-1].content, limit=3) search traps: namespace_prefix matches by prefix ( ('alice',) also returns ('alice','memories') ); results beyond limit (default 10) are silently dropped , so paginate with offset ; ordering differs by backend (Postgres: most recently updated first; in-memory: insertion order), so sort client-side if order matters. Custom store: subclass BaseStore and implement the five async methods ( aput , aget , adelete , asearch , alist_namespaces ). No conformance suite exists, so test against InMemoryStore . Checkpointer versus store: the checkpointer saves the whole graph state for one thread ; the store saves chosen facts for many threads . You normally want both. Time travel (replay a past execution, or fork from a past checkpoint to explore another path) has its own how-to page; I did not re-read it for this guide. 📖 Persistence · Checkpointers · Stores · Time travel · Add memory 6. Interrupts and human-in-the-loop Analogy: a bookmark with a note in the margin. The graph stops, marks its place, and waits as long as needed. When you hand back an answer, it re-reads the chapter from the start of the page and carries on. Call interrupt(value) inside a node. LangGraph saves state, surfaces value (any JSON-serialisable object), and waits. Resume by invoking again with Command(resume=answer) on the same thread_id ; answer becomes the return value of interrupt() . You need a checkpointer and a thread ID. from langgraph.types import interrupt, Command def approval(state) -> Command[Literal['proceed', 'cancel']]: ok = interrupt({'question': 'Proceed?', 'details': state['action']}) return Command(goto='proceed' if ok else 'cancel') stream = graph.stream_events(inputs, config, version='v3') _ = stream.output # drive the run if stream.interrupted: print(stream.interrupts) # tuple of Interrupt(value=..., id=...) graph.stream_events(Command(resume=True), config, version='v3').output 6.1 The rules (each one is a classic bug) The node restarts from its first line on resume. Everything before interrupt() runs again. Never wrap interrupt() in a bare try/except . It pauses by raising a special exception; swallowing it silently disables the pause. Catch specific exception types only. Do not reorder, skip or loop interrupts non-deterministically. Resume values are matched by index within the node. No conditional skipping, no loops over data that may change. No while True validation loops with interrupt() in one node. Each resume replays earlier iterations. Instead: call interrupt() once, store a pending_question in state on invalid input, and use a conditional edge to loop back. Payloads must be serialisable (strings, numbers, dicts of simple values). No functions or class instances. Side effects before interrupt() must be idempotent : upsert, not insert; or move the side effect after the interrupt or into a separate node. 6.2 Patterns Pattern Shape Approve or reject interrupt returns bool; route with Command(goto=...) Review and edit state edited = interrupt({'content': state['draft']}) ; return {'draft': edited} Interrupt inside a tool The tool calls interrupt({...}) before acting; resume with {'action': 'approve', 'subject': 'new subject'} to approve with edits Validate human input One interrupt per node call plus a conditional-edge loop (rule 4) Several parallel interrupts Resume with Command(resume={interrupt.id: answer, ...}) for all pending IDs at once Typed answers ⚑ interrupt(payload, response_schema=Decision) (Pydantic, TypedDict or dataclass); invalid resume raises ValidationError and you can resend to the same thread. A raw JSON-schema dict is exposed to clients but not validated. Needs langgraph 1.2.12. Static breakpoints ( interrupt_before / interrupt_after at compile or run time, resume with invoke(None, config) ) are for debugging , not for human-in-the-loop. Use interrupt() for that. Subgraph called as a function: both the parent node and the subgraph node restart from their beginnings on resume. Drive a UI loop: call stream_events in a loop, stream tokens from stream.messages , check stream.interrupted , ask the user, then resume with Command(resume=...) , until nothing is interrupted. 📖 Interrupts · HITL concepts 7. Fault tolerance Analogy: a delivery driver with three escalating responses. Knock again (retry). Don't wait forever at one door (timeout). If the house is truly unreachable, follow the 'undeliverable' procedure (error handler): leave a note and return to depot. ⚑ Per-node timeouts, error handlers, set_node_defaults and graceful drain need langgraph 1.2+ . The order is fixed: an attempt raises (including a timeout) → the retry policy decides → only after retries are exhausted does the error handler run. 7.1 Retries from langgraph.types import RetryPolicy, default_retry_on builder.add_node('call_api', call_api, retry_policy=RetryPolicy(max_attempts=3)) # custom: retry_on can be an exception type, a sequence of types, or a callable(exc) -> bool def retry_on(exc): return False if isinstance(exc, MyBug) else default_retry_on(exc) Parameter Default max_attempts (includes the first) 3 initial_interval / backoff_factor / max_interval 0.5 s / 2.0 / 128 s jitter True retry_on default_retry_on Default is conservative: it retries any exception except ValueError , TypeError , ArithmeticError , ImportError , LookupError , NameError , SyntaxError , RuntimeError , ReferenceError , StopIteration , StopAsyncIteration , OSError (programming errors). For requests and httpx only 5xx are retried. NodeTimeoutError is retryable. Inside a node, runtime.execution_info.node_attempt lets you switch to a fallback API on attempt 2+. 7.2 Timeouts from langgraph.types import TimeoutPolicy builder.add_node('call_model', call_model, timeout=60) # seconds, wall clock builder.add_node('call_model', call_model, timeout=TimeoutPolicy(run_timeout=120, idle_timeout=30)) Async nodes only. A sync node with a timeout is rejected at compile time. Wrap blocking I/O with asyncio.to_thread inside an async node. run_timeout : hard cap per attempt, never refreshed. idle_timeout : fires only if the node stops making progress; the clock resets on state writes, stream chunks, child-task scheduling, stream-writer calls and any LangChain callback event (tokens, tool calls). refresh_on='heartbeat' restricts refresh to explicit runtime.heartbeat() calls for long work with no natural progress signal. On timeout: NodeTimeoutError (with node , elapsed , kind ), writes from that attempt are cleared, then the retry policy decides. The clock resets per attempt. 7.3 Error handlers (compensation, the Saga pattern) from langgraph.errors import NodeError def payment_failed(state: State, error: NodeError) -> Command: return Command(update={'status': 'compensated: ' + str(error.error)}, goto='finalize') builder.add_node('charge_payment', charge_payment, retry_policy=RetryPolicy(max_attempts=3, retry_on=ConnectionError), error_handler=payment_failed) The handler gets the state and, if you annotate error: NodeError , the failing node and error . It may also take runtime or config . Return a state update or a Command to route elsewhere. Failure provenance is checkpointed: if the process crashes after the failure but before the handler finishes, the handler sees the same NodeError on resume. interrupt() bypasses retries and handlers (it is control flow, not failure). A failing subgraph surfaces to the parent node's handler. If the handler itself raises, the error propagates as if there were no handler. One handler per node. 7.4 Graph-wide defaults graph = (StateGraph(State) .set_node_defaults(retry_policy=RetryPolicy(max_attempts=3), error_handler=mark_failed, timeout=TimeoutPolicy(run_timeout=30)) .add_node('step_a', step_a) .add_node('charge', charge, error_handler=refund) # per-node value overrides the default .compile()) Default Regular nodes Error-handler nodes retry_policy Yes Yes timeout Yes Yes error_handler Yes No (a handler must not catch itself) cache_policy Yes No (caching a handler is unsafe) Defaults are not inherited by subgraphs . The same timeout= and retry_policy= exist on @task and @entrypoint in the Functional API. 7.5 Graceful shutdown (drain) import signal from langgraph.runtime import RunControl from langgraph.errors import GraphDrained control = RunControl() signal.signal(signal.SIGTERM, lambda *_: control.request_drain('sigterm')) try: graph.invoke(inputs, config, control=control) except GraphDrained as e: pass # checkpoint saved; resume later graph.invoke(None, config) # resume with the same thread_id Drain is cooperative and works between super-steps : a running node (including one mid-retry) finishes first. It does not cancel asyncio tasks or kill threads; pair it with a hard timeout. If the graph finishes on the same tick, it just returns normally (check control.drain_requested ). Inside a node, runtime.drain_requested lets you skip expensive work. 7.6 Changing a deployed graph (migrations) Threads at the end of the graph: any topology change is fine. Threads interrupted : any change except renaming or removing a node the thread is about to enter. State keys: adding or removing is safe both ways; renaming a key loses its saved value in existing threads; incompatible type changes can break old threads. Changes that are technically compatible but alter behaviour (a new tool set, a restructured flow): pin a behavioural version in state so old threads keep the old path while new threads take the new one. Traps. Retrying a non-idempotent node duplicates side effects. A timeout on a sync node fails at compile time. Putting the handler on every node via defaults and then also expecting handler nodes to be covered (they are not). 📖 Fault tolerance · Backward compatibility 8. Subgraphs Analogy: a department inside the company. The company chart (parent graph) treats it as one box. Inside, it has its own stations. Whether the department remembers last week's work depends on its filing policy (checkpointer mode). 8.1 How parent and subgraph talk Pattern Use when How Call inside a node Different state schemas (no shared keys), or you must transform state A wrapper node maps parent state to subgraph input, calls subgraph.invoke(...) , maps the result back Add as a node Shared keys (for example both use messages ) builder.add_node('agent', compiled_subgraph) ; no wrapper # different schemas def call_subgraph(state: Parent): out = subgraph.invoke({'bar': state['foo']}) return {'foo': out['bar']} # shared schema builder.add_node('node_2', compiled_subgraph) 8.2 Persistence mode: the checkpointer argument of compile() Mode checkpointer= Behaviour Pick it for Per-invocation (default) None Each call starts fresh; inherits the parent's checkpointer so interrupts and durable execution work within the call Most apps; subagents handling independent requests Per-thread True State accumulates across calls on the same thread A subagent that needs multi-turn memory Stateless False No checkpointing; runs like a function; no interrupts, no durability Cheap pure helpers; accept that a crash means rerun Per-thread caveats: no parallel calls to the same per-thread subgraph (checkpoint conflict); several different per-thread subgraphs called inside one node get namespaces by call order , so reordering mixes up their states. Give each its own wrapper StateGraph with a unique node name for a stable namespace. Subgraphs added as nodes already get name-based namespaces. The parent must have a checkpointer for interrupts, state inspection and per-thread memory to work in a subgraph. Inspect: graph.get_state(config, subgraphs=True).tasks[0].state . Per-invocation shows only the current call; per-thread shows accumulated state. Stream: stream_events(..., version='v3') and iterate stream.subgraphs (each has graph_name , path , messages , values ); or classic stream(..., subgraphs=True) . Without that flag, stream_mode='messages' on a parent does not emit tokens from an inner agent. Navigate up: Command(goto='other', graph=Command.PARENT) . 📖 Subgraphs 9. Workflow and agent patterns, one by one Workflows follow predetermined code paths; agents choose their own path. All of these are just graphs. Pick the simplest that fits. Pattern Shape Use when Avoid when Prompt chaining A → gate → B → C; each call consumes the previous output Well-defined task splittable into verifiable steps (translate, then verify) Steps are not really sequential Parallelization START fans out to N nodes; all feed an aggregator Independent subtasks for speed, or repeated runs for confidence Branches depend on each other Routing Classifier node (structured output) → conditional edge → specialised node Distinct input types needing different handling Categories overlap heavily Orchestrator-worker Plan → Send one worker per subtask → synthesiser Subtasks cannot be predefined (edit N unknown files) Subtasks are fixed (use parallelization) Evaluator-optimizer Generate → evaluate → loop back with feedback until accepted Clear success criteria that need iteration No reliable evaluator; unbounded cost (add an iteration cap) Agent LLM node ↔ tool node loop until no tool calls Open-ended problems where steps are unpredictable You know the steps in advance # Agent loop in the Graph API def llm_call(state: MessagesState): return {'messages': [llm_with_tools.invoke([SystemMessage('You are a calculator.')] + state['messages'])]} def tool_node(state: MessagesState): out = [] for tc in state['messages'][-1].tool_calls: out.append(ToolMessage(content=tools_by_name[tc['name']].invoke(tc['args']), tool_call_id=tc['id'])) return {'messages': out} def should_continue(state) -> Literal['tool_node', '__end__']: return 'tool_node' if state['messages'][-1].tool_calls else END b = StateGraph(MessagesState) b.add_node('llm_call', llm_call); b.add_node('tool_node', tool_node) b.add_edge(START, 'llm_call') b.add_conditional_edges('llm_call', should_continue, ['tool_node', END]) b.add_edge('tool_node', 'llm_call') agent = b.compile() # Routing: structured output as the router class Route(BaseModel): step: Literal['poem', 'story', 'joke'] router = llm.with_structured_output(Route) def route_node(state): return {'decision': router.invoke(state['input']).step} graph.add_conditional_edges('route_node', lambda s: {'story': 'write_story', 'joke': 'write_joke', 'poem': 'write_poem'}[s['decision']]) 9.1 ToolNode ToolNode (from langgraph.prebuilt ) is the prebuilt node that runs tool calls: it handles parallel tool execution, error handling and state injection . Add it with builder.add_node('tools', ToolNode([search, calculator])) . Tools read graph state and run context through the injected ToolRuntime argument. Tools only see the state passed to the ToolNode: if you call a ToolNode manually from another node, pass the full state ( tool_node.invoke(state) ), not just {'messages': ...} , or custom state fields are invisible. A tool returning Command must include a matching ToolMessage . Hand-written loop versus create_agent : write the loop yourself only when you need control the middleware cannot give (custom routing, extra nodes between model and tools). Otherwise use create_agent and embed it as a node. 📖 Workflows and agents · Agentic RAG · SQL agent 10. The Functional API Analogy: a recipe written as ordinary steps instead of a wiring diagram. You write normal Python ( if , while , for ); the runtime still saves progress after each marked step. from langgraph.func import entrypoint, task @task def generate(topic: str): return llm.invoke('Write a joke about ' + topic).content @task def improve(joke: str): return llm.invoke('Make this funnier: ' + joke).content @entrypoint(checkpointer=InMemorySaver()) def workflow(topic: str): joke = generate(topic).result() # tasks return futures; .result() waits return improve(joke).result() Tasks are the checkpoint unit. Their results are saved, so a resumed run skips completed tasks. Launch several before calling .result() to run in parallel. Determinism and idempotency rules are stricter here, because the whole entrypoint re-runs on resume: keep task and interrupt order stable, and keep side effects inside tasks. @task and @entrypoint accept retry_policy= and timeout= . An entrypoint compiles to a single node. Choose it when the flow reads naturally as code, you do not need a drawn graph, and you want minimal state plumbing. Choose the Graph API when you want explicit visualisation, fine-grained state reducers, parallel branches via edges, or per-node policies. The two can be mixed (a task inside a graph node, §2.2). 📖 Functional API · Use the functional API 11. Streaming Mode Yields values Full state after each step (includes private channels) updates Only what each node changed messages (token, metadata) from any LLM call inside nodes custom Whatever you write with runtime.stream_writer / get_stream_writer() Classic: graph.stream(inp, config, stream_mode=['updates', 'messages'], version='v2') yields dicts with type , ns , data . Interrupts appear under __interrupt__ in updates . Event streaming (recommended for new apps): stream = graph.stream_events(inp, config, version='v3') gives typed projections you consume independently: stream.messages , stream.values , stream.subgraphs , stream.interrupted , stream.interrupts , stream.output . Pass output_keys=[...] to restrict streamed state. Inside a node, get_stream_writer() only works within a LangGraph execution context. Disable token streaming for one model with streaming=False (or disable_streaming=True ); useful on the platform for models whose output must not reach the client. 📖 Streaming · Event streaming · Frontend 12. Memory in LangGraph: when to use which Analogy: the checkpointer is the save file of this playthrough ; the store is the profile that follows the player across playthroughs ; state is the HUD you see right now . Need Use Where it lives Continue the same conversation later Checkpointer + thread_id Thread (all graph state) Pause for a human, survive a crash Checkpointer Thread Keep history from overflowing the context window Trim, RemoveMessage or a summarise node State (rewritten) or model input (transient) Remember facts about a user across threads Store ( runtime.store ) Cross-thread, namespaced Find memories by meaning Store with index config and search(query=...) Cross-thread Per-request configuration or clients Runtime context Not persisted Data that should not be checkpointed (live connections) Keep out of state; use context or closures n/a Pattern for long-term memory in a graph: a call_model node reads relevant memories from the store and adds them to the prompt; a separate update_memory node (or background job) writes new ones. Keep the write path in its own node so retries and interrupts do not duplicate writes, and use user-scoped namespaces. Memory design from the agent side (tools, middleware, summarisation) is on the LangChain page. 13. LangChain agents are LangGraph graphs create_agent returns a compiled graph. Embed it as a node or subgraph, and its middleware (HITL, summarisation, PII, retries) travels with it. As a subgraph, tokens from the inner model only appear in the parent's messages stream with subgraphs=True (or stream.subgraphs ). Give each agent a name to see lc_agent_name in metadata. Multi-agent handoffs use Command(goto=..., graph=Command.PARENT) under the hood; the per-thread subagent rules in §8.2 apply. Deep Agents adds planning, a filesystem and subagents on top of this same runtime. 14. Production, version gates and the trap list Deploy: LangSmith Cloud or the Agent Server provisions checkpointer and store automatically and exposes the graph through an API; locally you pass them yourself. Use LangSmith Studio to inspect state, set breakpoints and replay runs. Trace with LangSmith. Test: unit-test nodes as plain functions, and graphs with in-memory persistence; use get_state to assert on checkpoints. Operate: graceful drain on SIGTERM, per-node timeouts and retries, a default error handler that marks the external job failed, recursion limits with proactive fallback. ⚑ Feature Minimum Default recursion limit 1000 langgraph 1.0.6 v2 stream/invoke format ( GraphOutput ) 1.1 runtime.execution_info , server_info 1.1.5 Per-node timeouts, node error handlers, set_node_defaults , RunControl drain, runtime.heartbeat 1.2 interrupt(..., response_schema=...) 1.2.12 Event streaming ( stream_events(version='v3') ) langchain 1.3-era releases; check your install Trap checklist: No checkpointer yet using interrupt , threads or time travel; or no thread_id . Non-idempotent code before an interrupt() or in a retried node. Bare try/except around interrupt() ; while True prompt loops; reordered interrupts. Parallel writes to one key with no reducer; expecting an empty list to clear a reducer field. Mixing static edges with Command /conditional routing from the same node. A Command return annotation missing; a tool returning Command without a ToolMessage . Sending Command(update=...) as input to continue a chat (the graph looks stuck). recursion_limit placed inside configurable ; no proactive fallback. Timeout on a sync node; relying on defaults being inherited by subgraphs. Renaming a state key or a node while threads are in flight. Treating private state channels as hidden in values streams. store.search silently truncated at the default limit of 10. 15. Exercises Build the agent loop from §9 by hand with MessagesState , compile it with InMemorySaver , and show that two invoke calls on one thread_id share history. Add a node that calls interrupt() before a tool runs. Resume with approval, then repeat and reject. Then deliberately put a counter-incrementing database call before the interrupt and observe the double run. Make a state key errors: Annotated[list[str], add] . Write a node that tries to clear it by returning [] , see that nothing changes, then fix it with Overwrite . Add a flaky async node with RetryPolicy(max_attempts=3) , a TimeoutPolicy(run_timeout=2) , and an error_handler that routes to a finalize node. Make it fail on every attempt and read the final state. Fan out with Send over a list of 5 items into a worker node writing to results: Annotated[list, add] , then synthesise. Add a RemainingSteps guard. Solution notes The checkpointer reloads the whole state for that thread_id ; the messages reducer appends the new turn. On resume the node restarts from its first line, so the pre-interrupt side effect runs twice. Fix by upserting, or by moving the call after the interrupt or into its own node. A merging reducer merges the empty list into the old value; {'errors': Overwrite([])} bypasses the reducer. The order is attempts → retry policy → handler. The handler sees NodeError with the node name and last exception; status is whatever your handler wrote. Remember timeouts need an async node. Each Send gets its own worker input; the reducer on results merges parallel writes. Without the reducer the parallel writes conflict. Sources: docs.langchain.com (Python): LangGraph overview , Graph API , persistence , stores , interrupts , fault tolerance , subgraphs , workflows and agents , runtimes, frameworks, harnesses , LangGraph v1 release notes . Time travel and the Functional API were not fully re-read (see scope notes above).