Data

• Data Platforms — Digest
• Data Platforms — Article
• Data Warehouse — Digest
• Data Warehouse — Article
• Data Warehouse Use Cases — Needed vs. Overkill
• Elasticsearch Engineering – Digest
• Elasticsearch Engineering
• Prefect — Components Deep Dive
• RustFS — Components Deep Dive

Data Platforms — Digest

Definition

A data platform is the infrastructure and set of tools an organization uses to collect, store, process, and serve data so applications and people can reliably use it. It's the plumbing that moves data from where it's generated to where it's needed, in usable form.

Real-life usage

• E-commerce: clickstream, orders, and inventory land in a warehouse (Snowflake/BigQuery/Redshift) and feed both BI dashboards and ML models (recommendations, fraud detection) from the same data.
• Fintech: transaction data flows via streaming pipelines (Kafka) into systems supporting both real-time fraud scoring and end-of-day regulatory reporting.
• Healthcare: patient records, device telemetry, and claims data are unified under strict governance (HIPAA) while remaining queryable for research.
• SaaS internal analytics: app DB → ETL/ELT → warehouse → dashboards for product/finance teams.
• Analogy: a city utility grid — treatment plant (ingestion/cleaning), pipes/reservoirs (storage), pumping stations (processing), taps (serving). Break any layer, everything downstream is affected.

5 Whys — Root Cause

• Why do platforms exist? Raw source systems are wrong for two independent reasons.
• Reason 1 — contention: OLTP DBs are optimized for row-based point lookups. Large analytical scans evict the transactional working set from buffer cache and hold locks longer, causing cache misses and slow transactional reads (e.g., checkout).
• Read replicas solve contention but not query shape — row-store/3NF schemas are structurally bad at large scans/aggregations, hence column-store, star schemas, and pre-aggregation.
• Reason 2 — incoherence: each source system encodes its own implicit business logic (e.g., three different definitions of "active customer" across CRM, billing, and app DB). With N sources and M consumers and no shared contract, definitions drift — it's structurally guaranteed, not a one-off mistake.
• Platforms fix this by centralizing logic into governed, computed-once definitions everyone reads from.

Value vs. Risk

• Value: understanding the "why" lets you choose stack components by the guarantee a workload needs (ordering, replayability, latency, managed vs. self-hosted) instead of by popularity or familiarity — avoiding redundant tools solving the same problem.
• Risk — performance/consistency: black-boxing the platform reproduces the original problems (contention, definitional drift) plus new complexity/understanding debt (unclear why each tool is in the stack).
• Risk — trust erosion: when dashboards disagree and no one can explain why, people stop trusting the platform and quietly bypass it for gut-feel or spreadsheets — a costly, hard-to-reverse adoption failure.

Data Platforms — Article

Definition

A data platform is the infrastructure and set of tools an organization uses to collect, store, process, and serve data so that applications and people can reliably use it. Stripped of jargon: it's the plumbing that gets data from where it's produced to where it's needed, in a form someone can actually use. When you hear "data platform," think less of a single product and more of a layered system — ingestion, storage, processing, and serving — each layer doing a distinct job.

Real-life usage

You've already touched pieces of this: Debezium and NiFi doing change-data-capture and flow-based ingestion, Kafka as the durable transport backbone, Airflow and Azure Data Factory orchestrating batch pipelines, Elasticsearch and MongoDB serving specific access patterns. That's a real, if overlapping, data platform stack.

Beyond your own stack, the pattern repeats everywhere at scale:

• E-commerce platforms capture clickstream, orders, and inventory changes, land them in a warehouse (Snowflake, BigQuery, Redshift), and serve both BI dashboards and ML models (recommendations, fraud detection) from the same underlying data.
• Fintech systems stream transactions through Kafka into pipelines that support both millisecond-latency fraud scoring and end-of-day regulatory batch reporting — same data, wildly different latency contracts.
• Healthcare platforms unify patient records, device telemetry, and claims data while satisfying strict compliance (HIPAA), yet still need to be queryable for research.
• Even a mid-size SaaS company runs a scaled-down version: app DB → ETL/ELT → warehouse → dashboards for product and finance.

A useful mental model: a data platform is like a city's utility grid. A treatment plant cleans and prepares water (ingestion/cleaning), reservoirs and pipes store and move it (storage), pumping stations push it where it's needed (processing), and taps deliver it on demand (serving/access). Nobody turning on a faucet thinks about the treatment plant — but if any layer fails, everything downstream breaks.

5 Whys — Why Data Platforms Exist

The natural question is: why not just query the production database directly? Working through this honestly gets you to two independent root causes.

Root cause 1: resource contention. OLTP databases are optimized for row-based, indexed point lookups — fast reads/writes of single rows, like fetching one customer's cart. Analytical queries instead scan and aggregate across millions of rows. When a large aggregation runs against the same database, it evicts the transactional working set from the buffer cache. A checkout read that used to be an instant in-memory hit becomes a slow disk read. Combine that with longer-held locks from the scan, and the aggregation job and the checkout flow — which have nothing to do with each other logically — end up fighting over the same finite memory, I/O, and lock resources.

The obvious fix, "just add a read replica," solves contention: reads are offloaded to a copy, so the analytical workload stops starving the transactional one. But it doesn't solve the second problem — query shape. A replica has the exact same row-store, normalized (3NF) schema as production, which is structurally bad at scanning and aggregating across huge row counts, replica or not. This is why the industry didn't stop at replicas: it built column-store engines, star/denormalized schemas, and pre-aggregation — physically reshaping the data to match the access pattern of the consumer, not just relocating the same shape to different hardware. Vertical scaling (bigger single machine) eventually hits a ceiling too, which is part of why horizontal separation into distinct, purpose-built systems became the standard architecture rather than just buying a bigger box.

Root cause 2: definitional incoherence. Imagine three teams need "active customer": marketing pulls it from the CRM (logged in within 30 days), finance pulls it from billing (has a paid, non-cancelled subscription), and a data scientist queries the app DB directly (any tracked event in 7 days). All three are querying the same underlying reality, but each source system encodes its own local business logic implicitly — in application code, in how a status field gets set, in team-specific workflow assumptions. There's no shared, enforced contract anywhere. With N source systems and M independent consumers, you get up to N×M different interpretations of the same concept, and nothing structurally prevents that drift — it isn't a one-off mistake someone could have avoided by being careful, it's guaranteed by the absence of a shared definition layer.

Put together: data platforms exist because raw source systems are wrong for two independent reasons — they can't serve analytical access patterns without degrading transactional performance, and they can't produce consistent meaning across consumers without centralizing business logic. Every component of a modern data platform — ingestion tools, warehouses, orchestration, semantic/governance layers — exists to solve one or both of these.

Value vs. Risk

Value. Understanding the "why" rather than just the "how to configure X" changes how you make architectural decisions. Instead of picking a tool because it's popular or because you already know it, you pick based on the guarantee the workload actually needs: does it need ordering and replayability (Kafka), low-latency row-level change capture (Debezium), flexible flow-based routing over heterogeneous data (NiFi), managed Azure-native batch orchestration (ADF), or Python-native complex DAG logic (Airflow)? If two tools in your stack solve the same guarantee — say, two orchestrators — that's a signal of accidental complexity worth questioning directly, not a neutral fact about the stack.

Risk of treating the platform as a black box. The immediate risks are a recurrence of the original two problems — contention and inconsistency — plus a new one: complexity debt, where nobody can explain why a given tool is in the pipeline or what it specifically contributes. But there's a third, more expensive risk that's easy to miss because it isn't a technical failure: trust erosion. When two dashboards disagree and no one can explain why, people stop trusting the platform itself, not just that one number. The failure mode isn't a wrong query — it's leadership quietly reverting to gut feel or personal spreadsheets because they've learned the platform can't be relied on. That's a behavioral, org-wide cost, and by the time it's visible, the damage — bad decisions already made, the platform already abandoned in practice — is largely done. It's far more expensive to reverse than a performance regression, because it's an adoption and credibility problem, not an engineering one.

Data Warehouse — Digest

Definition

A data warehouse is a centralized store of structured, historical data — typically modeled with facts and dimensions (star/snowflake schema) — organized specifically to support analytical querying and reporting, as opposed to transactional operations.

Real-life usage

• Retail/e-commerce: fact_sales (quantity, revenue) joined to dim_product, dim_store, dim_date, dim_customer — slice revenue any way, on demand.
• Banking/fintech: regulatory reporting (risk exposure, AML) needs consistent historical snapshots, not just current state.
• Airlines: fact_bookings/fact_flights joined to dim_route, dim_aircraft, dim_time to analyze load factors and revenue per route over years.
• Marketing: fact_impressions/fact_conversions joined to dim_campaign, dim_channel, dim_audience.
• Analogy: a physical warehouse doesn't manufacture goods — it receives finished goods and organizes them onto labeled shelves by category (dimensions) with clear counts (facts), so anyone can pull exactly what they need without digging through the factory floor.

5 Whys — Root Cause

• Why not just use a regular RDBMS with a star schema? Row-store databases store full rows together on disk — a query needing 2 of 30 columns still has to read all 30 off disk.
• Root cause 1 — columnar storage: warehouse engines store each column contiguously, so aggregation queries read only the columns they need. This also enables better compression and vectorized execution.
• Why also separate facts and dimensions if columnar storage already solves read efficiency? Flattening everything into one wide table causes massive redundancy (a customer's address repeated across every transaction row) and update anomalies (changing one dimension value means rewriting/missing rows across a billion-row fact table).
• Root cause 2 — fact/dimension separation: avoids redundancy and update anomalies by separating high-volume immutable events (facts) from low-volume mutable reference data (dimensions), each managed according to its actual rate of change (see: slowly changing dimensions).

Value vs. Risk

• Value: understanding columnar storage + fact/dimension separation changes how you design any analytical store — not just SQL warehouses. E.g., a medallion architecture built on doc-oriented stores (like Elasticsearch) still needs to factor out dimension keys instead of embedding full dimension attributes in every aggregate document, or it silently rebuilds the flat-table problem.
• Risk — no dedicated warehouse layer: flat/embedded dimension data means a dimension change (e.g., product recategorized) can't be propagated without reindexing every historical record referencing it — leading to silent inconsistency between old and new records with no single source of truth to detect or correct the drift.

Data Warehouse — Article

Definition

A data warehouse is a centralized store of structured, historical data — typically modeled with facts and dimensions using a star or snowflake schema — organized specifically to support analytical querying and reporting, as opposed to the transactional operations a production database handles. It's not just "a database with reporting tables"; the storage engine itself and the data model are shaped around analytical access patterns.

Real-life usage

The clearest way to see this is through fact and dimension tables. A retail warehouse might have fact_sales — one row per line item, carrying measures like quantity and revenue — joined to dim_product, dim_store, dim_date, and dim_customer. Anyone in finance can then slice revenue by region, by month, by category, in combinations nobody had to pre-build.

The same pattern shows up everywhere analytics matters at scale:

• Banking/fintech: regulatory reporting (daily risk exposure, AML) depends on consistent historical snapshots — "what was our exposure on March 31st" — not just current state, which is exactly what a warehouse's historical fact tables are built to answer.
• Airlines: fact_bookings/fact_flights joined to dim_route, dim_aircraft, dim_time let analysts study load factors, delays, and revenue per route across years of history.
• Marketing: fact_impressions/fact_conversions joined to dim_campaign, dim_channel, dim_audience answer "which channel actually drives conversions" across time — something no single operational system tracks end to end.

A useful analogy: a physical warehouse doesn't manufacture goods — it receives finished goods from many sources and organizes them onto labeled shelves by category (dimensions), with clear counts (facts), so anyone can walk in and pull exactly what they need without digging through the factory floor. A data warehouse does the same thing with data instead of goods.

5 Whys — Why Data Warehouses Exist as a Distinct Thing

The natural challenge is: you could build a star schema in any relational database — Postgres, MySQL — so why does the industry build dedicated warehouse engines (Snowflake, BigQuery, Redshift) at all?

Root cause 1: columnar storage matches the analytical access pattern. In a normal row-store database, a table's rows are stored together on disk — all of row 1's columns, then all of row 2's columns, and so on. A typical warehouse query like SELECT SUM(revenue) FROM fact_sales WHERE region = 'EU' only needs 2 of maybe 30 columns, but a row-store still has to pull every column of every matching row off disk into memory before it can discard the ones it doesn't need. That's wasted I/O and wasted cache space, proportional to the columns you don't care about.

Warehouse engines fix this with columnar storage: each column is stored contiguously on disk, so a query reads only the columns it actually references. This also unlocks two side benefits — far better compression (similar values stored together compress much more efficiently than mixed-type rows) and vectorized execution (operating on whole columns of values at once instead of row by row). This is the physical, mechanical reason a warehouse engine behaves differently from a row-store RDBMS running the exact same schema.

Root cause 2: fact/dimension separation avoids redundancy and update anomalies. Given columnar storage already solves read efficiency, it's fair to ask why not just put everything — customer name, product category, store region, revenue, quantity — into one big flat, columnar table. The answer is what happens to descriptive attributes under that design. If "customer address" lives inline in every sales row, and that customer has 500 transactions, the same address string is stored 500 times. At billions of rows, that's significant redundancy even under compression, and it creates an update anomaly: if the customer moves, do you rewrite all 500 (or 5 million) rows referencing them? Miss one, and the data is now silently inconsistent.

The deeper issue is conceptual: a "sale" and a "customer's current address" are different kinds of data that change at fundamentally different rates. Facts are an append-only event log; dimensions are comparatively static reference data that occasionally changes (which is exactly why "slowly changing dimensions" exists as a defined technique — dimensions do change, just rarely, and need an explicit strategy for handling it). Separating facts from dimensions keeps each kind of data managed according to its actual rate of change, instead of conflating an immutable event stream with mutable reference data in one structure.

Put together: data warehouses exist because (1) columnar storage matches the "few columns, many rows" read pattern of analytical queries in a way row-stores structurally cannot, and (2) fact/dimension modeling avoids the redundancy and update anomalies that come from mixing high-volume immutable events with low-volume mutable reference data in a single flat structure.

This root cause generalizes beyond SQL warehouses. A doc-oriented store like Elasticsearch actually does provide columnar-style reads for aggregations, via a structure called doc_values — a separate, purpose-built columnar layer distinct from its inverted index for text search. So root cause 1 can genuinely be satisfied outside a traditional warehouse engine. But root cause 2 is a modeling discipline, not a storage-engine feature — it has to be deliberately designed in. A medallion-style agg_ index built from flat JSON documents that embed both facts and dimension attributes in every document reproduces the exact flat-table problem: dimension values are duplicated across every document that references them, and because a document store has no native join to factor dimensions out, a dimension change either goes unpropagated (silent inconsistency) or requires reindexing every historical document that referenced it.

Value vs. Risk

Value. Understanding these two root causes changes how you evaluate any analytical data store, not just SQL warehouses. It tells you precisely what to check before trusting a design: does the storage layer read only the columns/fields a query needs (columnar or equivalent), and are high-volume facts kept separate from low-volume, referenced dimension data — or are they flattened together for convenience? For a system like a medallion-on-Elasticsearch setup, this means factoring dimension keys/IDs into fact-like documents and maintaining separate dimension indices, resolving them at query or ingest time, rather than embedding full dimension attributes into every aggregate document.

Risk. Keeping facts and dimensions flattened together, whether in a wide SQL table or in JSON documents, defers a cost rather than avoiding it. As data volume grows and dimension values change more often, the concrete failure looks like this: a report segmenting historical sales "by current product category" silently returns wrong numbers, because only some historical records were ever updated to reflect the new category — and there's no single source of truth to detect or correct the drift from, since nothing enforces referential consistency across the duplicated copies. This isn't a performance problem; it's a correctness problem that erodes trust in the numbers exactly the way inconsistent metric definitions do at the platform level.

Data Warehouse Use Cases — Needed vs. Overkill

When a data warehouse is genuinely needed

• BI/reporting — ad hoc slicing across dimensions (region, time, product) at speed; row-stores choke on this at scale.
• Historical/point-in-time analysis — cohort retention, trend analysis, "what did exposure look like on a given date." Needs immutable historical snapshots that production systems don't retain.
• ML feature engineering / training data prep — building wide feature tables by joining large historical fact/dimension sources; needs efficient scans/joins at volume.
• Regulatory & compliance reporting — audits need reproducible, consistent historical figures with lineage, not a moving production DB.
• Cross-domain analytics / data mesh consumption — joining multiple business domains' data together; exactly the "N sources, M consumers, no shared contract" problem centralized modeling solves.
• Forecasting & capacity planning — needs large historical time series at consistent granularity.
• Aggregate-level anomaly/trend detection — e.g. "revenue by region dropped 20% week over week"; inherently an aggregation-over-time problem.

Common thread: all need to scan/aggregate across large historical volumes with consistent, centrally-defined meaning — the two root causes (columnar reads + fact/dimension governance).

When a data warehouse is overkill or the wrong tool

• Single-record, low-latency lookups (get this customer's cart/profile) — the OLTP point-lookup pattern; columnar storage actually hurts here. Use the operational DB.
• Real-time operational decisions inside a live transaction (fraud scoring at checkout, inventory decrement) — needs millisecond latency and current state; warehouses are typically minutes-to-hours behind via batch/CDC.
• Full-text search / relevance ranking — not built for inverted indices or scoring; use Elasticsearch/OpenSearch.
• Unstructured/semi-structured blob storage — raw images, PDFs, logs, un-modeled JSON; forcing this into a star schema is premature structure — use a data lake (or the raw layer of a lakehouse) first.
• High-frequency small writes — IoT sensor ingestion event-by-event; warehouses are read-optimized/batch-oriented. Use a time-series DB or streaming store.
• Simple, low-volume reporting for a small team — a few thousand rows, one monthly summary; the ETL/governance/infra overhead isn't justified. A well-indexed Postgres view or spreadsheet suffices.
• Graph-shaped queries — path-finding, fraud rings, network traversal; star schemas/columnar engines aren't built for relationship traversal. Use a graph database.

Common thread: low-latency point access, unstructured content, full-text relevance, high-frequency writes, or relationship traversal are all problems a warehouse wasn't designed to solve.

Grouped by business domain

Telecom

• Needs it: CDR analysis across billions of records, churn prediction feature tables, network capacity planning, regulatory usage/coverage reporting.
• Overkill/wrong tool: real-time call routing, live network fault detection (streaming/time-series), subscriber lookup during a support call (OLTP).

Retail / e-commerce

• Needs it: sales analysis across store/product/time, inventory trend forecasting, customer segmentation/LTV, marketing attribution.
• Overkill/wrong tool: checking current stock for one SKU at checkout, product search/autocomplete (search engine), real-time cart/session state.

Consulting

• Needs it: multi-client benchmarking and longitudinal engagement analysis, internal utilization/profitability reporting across practices and time.
• Overkill/wrong tool: a single short engagement's ad hoc analysis on a one-off dataset — a spreadsheet or lightweight Postgres/DuckDB is enough.

Industry / manufacturing

• Needs it: production yield analysis across plants/lines/time, supply chain and demand forecasting, quality trend analysis joining defects to equipment/shift/supplier.
• Overkill/wrong tool: real-time sensor/IoT ingestion and shop-floor anomaly alerts (time-series/streaming), machine-level current-status lookups.

Finance / banking

• Needs it: regulatory historical reporting, risk exposure trend analysis, AML pattern detection across large transaction histories.
• Overkill/wrong tool: fraud scoring at the moment of a transaction (real-time), account balance lookup (OLTP).

Healthcare

• Needs it: population health trend analysis, claims/cost analysis across providers and time, research cohort analysis.
• Overkill/wrong tool: pulling a single patient's current chart during a visit (OLTP), clinical decision support needing sub-second current data.

Pattern across every domain: warehouse = historical, cross-dimensional, aggregatable analysis at real volume. Anything current-state, single-record, sub-second, or small-scale is the wrong job for it, regardless of industry.

Elasticsearch Engineering – Digest

The short companion to Elasticsearch Engineering: a glossary, then the facts worth memorising for the Elastic Certified Engineer exam (Elasticsearch 9.3), domain by domain. Use it for revision; go back to the full guide for explanations, examples and doc links.

---

A. Glossary

Term	Meaning (one line)
Alias	A secondary name pointing to one or more indices or data streams; can filter, route and have one write index
Allocation (shard)	The master's decision about which node holds each shard copy, governed by deciders
Allocation awareness	Spreads shard copies across node attributes (zone, rack); forced awareness refuses to overload surviving zones
Allocation explain API	GET _cluster/allocation/explain: says why a shard is (un)assigned
Analyzer	Char filters → tokenizer → token filters; turns text into terms at index and search time
API key	Credential with its own (restricted) privileges; Authorization: ApiKey <encoded>
Auto-follow pattern	CCR rule that automatically follows new leader indices matching patterns
Bool query	Compound query with must, should, filter, must_not
Bootstrap checks	Startup checks enforced in production mode (heap, file descriptors, vm.max_map_count, memory lock…)
Bucket aggregation	Groups documents into buckets (terms, date_histogram, range, filters, composite…)
Bulk API	NDJSON batch of index/create/update/delete actions in one request
CCR	Cross-cluster replication: read-only follower indices pull changes from leader indices on a remote cluster
CCS	Cross-cluster search: query remote_alias:index from the local cluster
Classic stream	A Streams-managed view over an existing data stream (no migration)
Cluster state	Metadata (indices, mappings, settings, routing table) managed by the elected master
Component template	Reusable settings/mappings/aliases block referenced by index templates
Composable index template	_index_template: patterns + priority + composed_of + own template; can create data streams
Coordinating node	The node receiving a request; fans it out and merges results (a node with roles: [] does only this)
copy_to	Copies a field's value into another field at index time
Cross-cluster API key	API key created on the remote (/_security/cross_cluster/api_key) that scopes CCS/CCR access
Data stream	Append-only name over hidden backing indices; needs @timestamp and a template with data_stream: {}
Data stream lifecycle (DSL)	Simpler alternative to ILM for data streams: retention (+ downsampling)
Data tier	Node role groups data_hot/warm/cold/frozen/content; placement via _tier_preference
Dense vector	Fixed-length float embedding; searched with knn (HNSW)
Dissect	Delimiter-based parsing (faster, simpler than grok)
DLS / FLS	Document-level security (role query) / field-level security (field_security.grant/except)
Doc values	On-disk columnar structure used for sorting, aggregations and scripts
Dynamic mapping	Auto-mapping of unknown fields; controlled by dynamic: true/false/strict/runtime
Dynamic template	Rule mapping new fields by detected type, name or path; first match wins
ELSER	Elastic Learned Sparse EncodeR: Elastic's sparse-embedding model for semantic search
Enrich policy / processor	Lookup table built from a source index (_execute) and applied at ingest
ES|QL	Piped query language: FROM … | WHERE … | EVAL … | STATS … BY … | SORT … | LIMIT …
Failure store	Per-data-stream store that keeps documents that failed ingestion (<ds>::failures)
Filter context	Yes/no matching with no scoring, cacheable (filter, must_not)
Flattened	Maps an entire object as one field of keywords (avoids mapping explosion)
Flood-stage watermark	95% disk: indices on that node get read_only_allow_delete
Follower / leader index	CCR copy (read-only) and its source
Grok	Regex-based parsing with named patterns (%{IP:client.ip})
Health report	GET _health_report: indicator-based health with diagnoses
Hybrid search	Combining lexical (BM25) and semantic results, e.g. with an RRF retriever
ILM	Index lifecycle management: hot → warm → cold → frozen → delete phases with actions
Index	Logical collection of documents with one mapping and settings, split into shards
Index mode	standard, logsdb, time_series (TSDS), lookup
Inference endpoint	_inference/<task>/<id>: a configured model/service used to create embeddings
Ingest pipeline	Ordered processors that transform documents before indexing
Keystore	elasticsearch-keystore: encrypted store for secure settings
kNN	k-nearest-neighbour vector search (knn section or query)
LOOKUP JOIN	ES|QL join against an index with index.mode: lookup
Mapping	Schema of an index: fields, types, analyzers, parameters
Master-eligible node	Can be elected master; production uses 3
Metric aggregation	Computes values (avg, sum, cardinality, percentiles, stats, top_hits…)
Multi-field	Same value indexed in several ways (fields: { raw: keyword })
Nested	Field type keeping array-of-object members as separate hidden docs; needs nested query/agg
Normalizer	Analyzer-like chain for keyword fields (no tokenizer): lowercase, asciifolding
Pipeline aggregation	Aggregation over other aggregations' outputs (derivative, bucket_selector, max_bucket…)
Point in time (PIT)	Frozen view of indices for consistent deep pagination with search_after
Primary / replica shard	The original / copy of a shard; never on the same node
Query context	Scored matching ("how well"): must, should
Realm	Authentication source: native, file, LDAP, AD, SAML, OIDC, PKI, Kerberos, JWT
Refresh	Makes recent writes searchable (default 1s)
Reindex	Copies documents from a source to a destination index (optionally with query/script/pipeline)
Remote cluster	Named connection (sniff or proxy mode; API key on port 9443, or certificate) used by CCS/CCR
Reroute processor	Ingest processor sending a doc to another data stream
Retriever	Composable search building block: standard, knn, rrf, linear, text_similarity_reranker
Role / role mapping	Named bundle of privileges / rule assigning roles to externally authenticated users
Rollover	Creating a new write index when conditions (age, size, docs) are met
RRF	Reciprocal Rank Fusion: merges ranked lists without score normalisation
Runtime field	Field computed at query time from a script; no reindex needed
search_after	Cursor-based deep pagination using the last hit's sort values
Searchable snapshot	Index mounted from a snapshot repository (fully mounted = cold, partially = frozen)
Segment	Immutable Lucene sub-index inside a shard; merged over time
semantic_text	Field type that auto-chunks and embeds text through an inference endpoint
Service account	Built-in identity for Elastic services (e.g. elastic/kibana) using service tokens
Shard	A Lucene index holding part of an Elasticsearch index
SLM	Snapshot lifecycle management: scheduled snapshots with retention
Snapshot	Incremental backup of indices or cluster state into a repository
Sparse vector	Token → weight embedding (e.g. ELSER)
Streams	Kibana feature to process, partition, map, retain and monitor log data from one UI/API
Synonyms API	_synonyms/<set>: manage synonym sets usable by updateable search-time analyzers
Term-level query	Exact, non-analysed matching (term, terms, range, exists, prefix…)
Translog	Per-shard write-ahead log for durability between flushes
Update By Query	Updates matching docs in place (script or pipeline)
Watermarks (disk)	Low 85% / high 90% / flood 95%
Wired stream	Streams-managed hierarchy rooted at logs (9.2–9.3) with partitioning and inheritance

---

B. Key things to remember

0. Exam mechanics

• ES 9.3, about 3 hours, about 10–12 hands-on tasks, partial credit, only the Elastic docs allowed.
• Grading is based on final cluster state: exact names matter, don't delete extras, and verify with GET after every task.
• The Console shortcut Ctrl/Cmd+/ opens the docs for the API under the cursor.

1. Architecture

• The stack is Collect (Agent/Fleet, Beats, OTel, Logstash) → Process (pipelines, Logstash, Streams) → Store/Search (Elasticsearch) → Explore (Kibana).
• Routing is hash(_routing) % shards, so the primary count is fixed. Change it with _split, _shrink or reindex.
• 3 master-eligible nodes for quorum; cluster.initial_master_nodes is used only on the first bootstrap.
• Shard size 10–50 GB, < 200M docs. cluster.max_shards_per_node = 1000.
• Watermarks: 85 / 90 / 95%. Flood stage sets index.blocks.read_only_allow_delete.
• Forced awareness keeps replicas unassigned rather than overloading a surviving zone, so a yellow cluster can be correct.
• Allocation filters: include = any, exclude = none, require = all. Attributes: _name, _ip, _host, _tier, and custom node.attr.*.
• index.unassigned.node_left.delayed_timeout = 1m. For rolling restarts, use cluster.routing.allocation.enable: primaries.
• Content indices go to data_content; data stream backing indices go to data_hot.

2. Data Management

• Static settings: shards, codec, index.mode, sort, and analysis (close → update → open). Dynamic: replicas, refresh, allocation, blocks, default/final pipeline.
• dynamic: strict rejects, false keeps in _source but doesn't index, runtime creates runtime fields.
• Template precedence: components (in order) < index template template < create-index request.
• One template per index: the highest priority wins. Equal priority with overlapping patterns is an error.
• Built-in templates at priority 100 (logs-*-*, metrics-*-*, logs.otel*…). Use > 500 to override, or avoid overlapping.
• Test with _index_template/_simulate_index/<name>. Templates affect new indices only; roll over data streams.
• Dynamic templates: first match wins, so put specific rules first and the catch-all last. match = leaf name, path_match = full path. There's no integer detection (long/double).
• Data streams: need a template with data_stream: {} and @timestamp. Append-only (create op). Deleting the stream deletes its backing indices.
• Failure store: data_stream_options.failure_store.enabled: true; read <ds>::failures.
• ILM: min_age counts from rollover. Rollover conditions are OR-ed; empty indices never roll over. Poll interval is 10m.
• Rollover needs a data stream or rollover_alias + a bootstrap -000001 with is_write_index: true.
• migrate is implicit in warm and cold; with no tier nodes, the shards don't move.
• Frozen requires searchable_snapshot. Shrink and forcemerge in hot only with rollover.
• Debug with _ilm/explain, then _ilm/retry.
• DSL vs ILM: when both are set, ILM wins (prefer_ilm). DSL is for simple retention only.
• Aliases: _aliases actions are atomic (add, remove, remove_index). Multi-index aliases need one is_write_index to accept writes. Filtered aliases use filter + routing.

3. Ingestion & Processing

• Choose the ingest method: Agent + Fleet for managed hosts and integrations, Logstash for heavy transforms, buffering or multiple outputs, client + _bulk for applications, Streams for parsing logs from the UI, file upload for one-offs.
• Bulk: NDJSON with a final newline; check errors; 5–15 MB batches.
• Immutable once mapped: type, analyzer, index/doc_values, nested vs object. You can add fields, add multi-fields, and add runtime fields.
• New multi-field → _update_by_query to populate existing docs.
• text for full text, keyword for exact/sort/aggs, both via multi-field. Use a normalizer for case-insensitive keyword.
• nested keeps object pairs together; it needs a nested query/agg. Limits: 50 nested fields, 10k nested objects per doc.
• Limits: total_fields.limit 1000, depth.limit 20. Use flattened for unbounded keys.
• scaled_float for prices, constant_keyword for per-index constants, wildcard for grep-style search, match_only_text for logs, date_nanos for ns precision.
• Analyzer order: char_filter → tokenizer → filter. Test with _analyze. Put edge n-grams at index time only, with search_analyzer: standard.
• Search-time synonyms: synonym_graph + Synonyms API + updateable: true.
• Reindex: create the destination mapping first; op_type: create for data streams; conflicts: proceed; wait_for_completion=false → _tasks; remote reindex needs reindex.remote.whitelist.
• Painless: ctx._source.x in reindex/UBQ scripts vs ctx.x in ingest; ctx.op = 'noop'|'delete'; null-safe ?..
• Pipelines: if conditions, ignore_failure, ignore_missing, and processor or pipeline on_failure (_ingest.on_failure_message). Always _simulate?verbose.
• default_pipeline can be overridden by ?pipeline=; final_pipeline always runs.
• _ingest.timestamp is the ingest time. The reroute processor changes the target data stream. Use @custom pipelines to customise integrations.
• Enrich: create the policy → _execute → enrich processor. Policies are immutable (delete and recreate), and must be re-executed when the source changes.
• Streams: classic = existing data streams in place; wired = hierarchy rooted at logs (9.2–9.3) with partitioning and inheritance. The tabs are Processing (with preview), Schema, Data lifecycle (inherit / custom / ILM), Data quality (degraded vs failed → failure store) and Advanced. Don't edit managed components with raw APIs. The API lives under kbn:/api/streams.

4. Searching Data

• match is analysed (OR by default; operator: and, minimum_should_match, fuzziness: AUTO). term is exact. Never use term on a text field; use .keyword.
• match_phrase + slop; multi_match with fields: ["title^3"], types best_fields (default), most_fields, cross_fields, phrase, bool_prefix.
• Bool: must (scored), filter (unscored, cached), should (scored, optional if a must/filter exists, otherwise at least 1 required), must_not (unscored).
• post_filter filters hits but not aggregations (facets).
• from + size ≤ 10,000 (max_result_window); for deeper pages, use search_after + PIT. You can't sort on text.
• Aggregations: set size: 0; use terms on keyword only; terms counts are approximate (shard_size); use composite to page through all buckets.
• date_histogram: calendar_interval (1d, 1M, week…) vs fixed_interval (30m, 7d); min_doc_count: 0 + extended_bounds for empty buckets; time_zone.
• range agg: from is inclusive, to exclusive.
• buckets_path: > for nesting, . for a metric's value, _count for doc count. bucket_selector works like HAVING; bucket_sort sorts and truncates buckets.
• ES|QL: POST _query (?format=txt). The default LIMIT is 1000 (max 10k). STATS … BY, EVAL, KEEP/DROP/RENAME, DISSECT/GROK, ENRICH, LOOKUP JOIN (the lookup index needs index.mode: lookup), BUCKET(), MATCH() or : for full text, METADATA _index, _score, ? params, and remote:index for CCS.
• Semantic search: an inference endpoint (a default ELSER one exists) + a semantic_text field. Query with match or semantic on it; highlight with type: semantic. Hybrid uses the rrf retriever (rank_window_size, rank_constant). Manual path: dense_vector + knn (k, num_candidates, filter) or sparse_vector query. It needs ML capacity and a licence (trial in the lab).
• CCS: alias:index, *:index, exclusions -alias:*. skip_unavailable defaults to true; ccs_minimize_roundtrips defaults to true. _resolve/cluster checks connectivity. _clusters in the response shows per-cluster status.

5. Cluster Management

• Green / yellow / red = all assigned / replica(s) unassigned / primary unassigned.
• The workflow: _cluster/health → _cat/shards → _cluster/allocation/explain → fix → _cluster/reroute?retry_failed=true if max retries was hit.
• Common causes: too few nodes for the replicas, allocation filters, awareness, disk watermarks, a missing tier, allocation.enable set to none/primaries, max retries.
• Red with all copies lost: restore from a snapshot. allocate_empty_primary / allocate_stale_primary (with accept_data_loss) is the last resort.
• persistent cluster settings survive restarts; transient is deprecated. null resets a setting.
• Monitor heap (> 85% is bad), breakers, thread-pool rejections, disk, slow logs and hot threads. Use Stack Monitoring (Agent/Metricbeat → monitoring cluster) or AutoOps.
• Snapshots: fs repos need path.repo on every node. Snapshots are incremental. Restore needs the target closed or absent; use rename_pattern/rename_replacement otherwise. include_global_state and feature_states control what's restored.
• SLM: Quartz cron (seconds first), date-math names <nightly-{now/d}>, retention (expire_after, min_count, max_count), _execute to run now.
• Remote clusters: the API-key model is recommended (remote port 9443, remote_cluster_server.enabled, cross-cluster API key with access.search/access.replication, local keystore cluster.remote.<alias>.credentials). The certificate model is deprecated (port 9300, same-named roles on the remote). Modes are sniff (default) and proxy. Check with GET _remote/info. Local nodes need remote_cluster_client.
• The local role grants remote access through remote_indices.
• CCR: licence on both clusters; run on the follower cluster: PUT <follower>/_ccr/follow {remote_cluster, leader_index}. Auto-follow uses leader_index_patterns + follow_index_pattern: "{{leader_index}}-x" and covers new indices only. Followers are read-only. Failover (promote a follower): pause_follow → _close → unfollow → _open. Soft deletes must be on (the default).

6. Security & Access Control

• A role has cluster + indices (names, privileges, field_security, query) + applications (Kibana: kibana-.kibana, feature_x.read/all, space:<id>) + remote_indices.
• A user's roles are a union: one role with unrestricted read cancels another role's DLS.
• Key privileges: read + view_index_metadata for Kibana data views; create_doc + auto_configure for shippers writing to data streams; manage_security is effectively admin.
• DLS templating: {{_user.username}}. FLS: grant / except with wildcards.
• API keys get the intersection of their role_descriptors and the creator's privileges; use expiration; invalidate with DELETE _security/api_key.
• Role mappings use rules (field/any/all/except on groups, dn, username, realm.name) for LDAP, AD, SAML and OIDC users.
• Test with _security/_authenticate, _security/user/_has_privileges and the es-security-runas-user header.
• Production security: security on; transport TLS is mandatory for multi-node clusters with security; HTTPS for clients and Kibana; secrets in the keystore; Kibana via a service account token; no daily use of elastic.
• Tools: elasticsearch-certutil ca|cert|http, elasticsearch-reset-password -u <user>, elasticsearch-create-enrollment-token -s kibana|node, elasticsearch-service-tokens create, elasticsearch-keystore add, POST _nodes/reload_secure_settings.
• Bootstrap checks (production mode starts when you bind a non-loopback network.host): heap Xms = Xmx, ≤ 50% RAM, < ~31 GB; file descriptors ≥ 65,535; threads ≥ 4,096; vm.max_map_count ≥ 262,144; memory lock works if enabled; discovery configured; swap disabled.
• Audit logging (xpack.security.audit.enabled, events.include), IP filtering (xpack.security.transport|http.filter.allow/deny), and action.destructive_requires_name: true.
• Kibana needs a 32+ character xpack.encryptedSavedObjects.encryptionKey for alerting and encrypted saved objects.

---

C. Ten-second API recall

Need	API
Health & diagnosis	_cluster/health, _health_report, _cat/shards, _cluster/allocation/explain, _cluster/reroute?retry_failed=true
Templates	_component_template, _index_template, _index_template/_simulate_index/<n>
Data streams	_data_stream/<n>, <ds>/_rollover, <ds>::failures, _data_stream/<n>/_lifecycle
ILM	_ilm/policy/<p>, <idx>/_ilm/explain, <idx>/_ilm/retry
Aliases	POST _aliases {actions:[add/remove/remove_index]}
Mapping & analysis	<idx>/_mapping, <idx>/_analyze, _synonyms/<set>
Move data	_reindex, <idx>/_update_by_query, <idx>/_delete_by_query, _tasks/<id>
Pipelines	_ingest/pipeline/<p>, …/_simulate?verbose, _ingest/_simulate, _enrich/policy/<p>/_execute
Search	<idx>/_search, _async_search, <idx>/_pit, _query (ES|QL), _resolve/cluster/<pattern>
Semantic	_inference/<task>/<id>, semantic_text, semantic query, rrf retriever
Snapshots	_snapshot/<repo>, _snapshot/<repo>/<snap>/_restore, _slm/policy/<p>/_execute
Remote / CCR	_remote/info, _security/cross_cluster/api_key, <f>/_ccr/follow, _ccr/auto_follow/<n>, <f>/_ccr/unfollow
Security	_security/role/<r>, _security/user/<u>, _security/api_key, _security/role_mapping/<m>, _security/_authenticate, _security/user/_has_privileges

Elasticsearch Engineering

A study guide for the Elastic Certified Engineer exam (Elasticsearch 9.3), organised by the official exam objectives.

How to read this guide. Each objective gets (1) a mental model, often an everyday analogy, (2) the APIs you'll actually type, (3) the traps graders like, and (4) links to the exact Elastic documentation pages. The Elastic documentation is the only resource available during the exam, so the 📖 links double as a map of where to go on exam day. Exercises with solution notes close each domain. The companion page, Elasticsearch Engineering – Digest, holds the glossary and the must-remember facts.

---

0. The exam and your preparation

0.1 Format

Item	Detail
Version	Elasticsearch 9.3 (the 8.x exam was retired on 1 Sep 2026)
Type	Hands-on, performance-based, on live Elasticsearch and Kibana clusters, remote-proctored
Duration	About 3 hours
Tasks	About 10–12 practical tasks, often combining several objectives, with partial credit
Allowed resources	Only the Elastic Documentation
Grading	The final state of your clusters: indices, templates, pipelines, roles, and so on

📖 Exam page · Certification FAQ

Analogy. It's a practical driving test, not a written theory test. Nobody asks you to recite the highway code. The examiner checks whether the car ends up correctly parked. The docs are your satnav: allowed, but only useful if you already know roughly where you're going.

0.2 The six domains

#	Domain	Objectives
1	Elasticsearch Architecture	Stack architecture; shard sizing and allocation strategies
2	Data Management	Index definition; dynamic templates; ILM for time series; index template that creates a data stream; aliases
3	Data Ingestion & Processing	Ingest methods; mappings; field data types; Reindex and Update By Query; ingest pipelines; Streams
4	Searching Data	Term and phrase queries; bool combinations; metric and bucket aggregations; ES|QL; semantic search; cross-cluster search
5	Cluster Management	Monitor and maintain cluster health; cross-cluster operations
6	Security & Access Control	RBAC; securing a cluster for production

0.3 Build a lab first

You can't prepare for a hands-on exam by reading. Use 9.3.x so the APIs and UI match the exam.

Option A: single node in one command (good for Domains 2–4)

curl -fsSL https://elastic.co/start-local | sh

📖 Local installation quickstart

Option B: three-node Docker Compose (needed for allocation, awareness, snapshots, and Domain 5)

📖 Multi-node cluster with Docker Compose

Settings to vary per node (as elasticsearch.yml or environment variables):

cluster.name: lab
node.name: es01                                  # es02, es03
node.roles: [ master, data_hot, data_content, ingest, remote_cluster_client ]
node.attr.zone: zone-a                           # zone-b, zone-c → awareness exercises
discovery.seed_hosts: [ es01, es02, es03 ]
cluster.initial_master_nodes: [ es01, es02, es03 ]   # first bootstrap ONLY, then remove
path.repo: [ /usr/share/elasticsearch/snapshots ]    # shared FS repo for snapshots

For Domain 5, add a second cluster (e.g. remote) so you can practise remote clusters, cross-cluster search (CCS) and cross-cluster replication (CCR).

Enable a trial license. CCR, searchable snapshots, some security features and ML-based semantic search need it:

POST _license/start_trial?acknowledge=true

Load sample data. In Kibana: Home → Add data → Sample data (eCommerce, Flights, Web logs). Many exam tasks look exactly like work on these datasets.

0.4 Exam-day habits that earn points

1. Read each task twice and copy names (indices, fields, pipelines, roles) straight from the task text. A typo scores zero.
2. Verify everything. GET what you created. Compare _count before and after a reindex. _simulate a pipeline before using it.
3. Don't destroy state the task didn't mention. Graders inspect the final state.
4. Prototype on scratch indices (tmp-*), then clean up.
5. Time-box. Roughly 15 minutes per task. At 20 minutes, move on and come back later.
6. Know the "immutable" things. You can't change shard count, field types or a field's analyzer in place. The answer is a new index plus _reindex, or _split/_shrink for shards.
7. Use Console shortcuts. Ctrl/Cmd+Enter runs a request, Ctrl/Cmd+/ opens the docs for the API under the cursor, and Ctrl/Cmd+I auto-indents.

📖 Console (Dev Tools)

---

1. Elasticsearch Architecture

1.1 Describe the Elastic Stack architecture

The big picture

Analogy: a postal system. Collectors (Elastic Agent, Beats, OpenTelemetry collectors, Logstash) pick up letters everywhere. Sorting centres (Logstash, ingest pipelines, Streams) open, stamp and classify them. The warehouse (Elasticsearch) stores every letter so it can be found in milliseconds. The front desk (Kibana) is where people ask questions and see dashboards. Fleet is the dispatch office that tells every collector what to pick up.

Layer	Components	Role
Collect	Elastic Agent (+ Fleet Server), Beats (Filebeat, Metricbeat…), EDOT / OpenTelemetry Collector, APM agents, language clients, connectors	Ship logs, metrics, traces and documents
Process	Logstash, Elasticsearch ingest pipelines, Streams, integrations' built-in pipelines	Parse, enrich, route, transform
Store / search / analyse	Elasticsearch	Distributed document store, search engine, analytics and vector database built on Apache Lucene
Explore / manage	Kibana (Discover, Dashboards, Dev Tools, Stack Management, Fleet UI, Streams UI)	Visualise, query, administer
Solutions	Search, Observability, Security	Opinionated apps on top of the same data

Deployment options: Elastic Cloud Hosted (ECH), Elastic Cloud Serverless (no nodes or shards to manage), Elastic Cloud Enterprise (ECE), Elastic Cloud on Kubernetes (ECK), and self-managed. The exam uses self-managed-style clusters, so everything below assumes you can touch nodes and settings.

📖 Elastic fundamentals · Deployment comparison · Ingest tools overview

Inside Elasticsearch: cluster → node → index → shard → segment

Analogy: a library chain. The cluster is the whole chain. Each node is a branch building. An index is a book collection (for example "novels"). Because the collection is too big for one shelf, it's cut into volumes: the primary shards. Each volume has photocopies (replica shards), always kept in a different branch. Inside each volume, pages are glued in batches (segments) that are never edited, only replaced when batches are merged.

Concept	What to know
Cluster	One or more nodes sharing a cluster.name. One elected master manages the cluster state (metadata: indices, mappings, settings, shard routing table).
Node	One Elasticsearch process. Its roles decide what it does (table below).
Index	A logical namespace of JSON documents with one mapping and one set of settings.
Shard	A Lucene index. Primary shards hold the data; replicas are copies that give high availability and extra read throughput. A primary and its replica are never on the same node.
Segment	An immutable, self-contained piece of a shard. Deletes are marked; merges physically drop them.
Document routing	shard = hash(_routing) % number_of_routing_shards → mapped to primaries. _routing defaults to _id. This is why the primary count is fixed at creation.

Node roles (node.roles in elasticsearch.yml):

Role	Purpose
master	Eligible to be elected master. Run 3 dedicated master-eligible nodes in production.
voting_only	Votes in elections but never becomes master (a tie-breaker).
data	Generic data role (holds all tiers).
data_content	Non-time-series content (product catalogues and so on).
data_hot / data_warm / data_cold / data_frozen	Data tiers for time-series data, from fastest and most expensive to cheapest.
ingest	Runs ingest pipelines.
ml	Runs machine-learning jobs and trained models (e.g. ELSER).
remote_cluster_client	Can connect to remote clusters (needed for CCS and CCR).
transform	Runs transforms.
[] (empty)	Coordinating-only node: routes requests and merges results.

Every node is also a coordinating node. Whichever node receives the request fans it out and gathers the results.

Write path: client → coordinating node → routes to the primary shard → primary indexes (in-memory buffer + translog) → forwards to replicas in parallel → acknowledges once the required copies succeed (wait_for_active_shards, default 1 = primary only).

Near real-time search: a refresh (default every 1s, only on indices that have been searched recently) turns the buffer into a searchable segment. A flush makes segments durable on disk and trims the translog. Merges combine small segments in the background.

Read path (query then fetch): the coordinating node sends the query to one copy of each shard. Each shard returns its top IDs and scores (query phase). The coordinator merges them, then fetches the actual documents (fetch phase).

Cluster formation: master-eligible nodes elect a master by quorum (majority of the voting configuration). Three master-eligible nodes tolerate one failure. cluster.initial_master_nodes is used only for the very first bootstrap; remove it afterwards.

📖 Clusters, nodes and shards · Node roles · Reading and writing documents · Near real-time search · Discovery and cluster formation · Quorum-based decisions

Data tiers

Analogy: a wardrobe. Hot is the chair where today's clothes lie: instant access, most expensive space. Warm is the wardrobe. Cold is the suitcase under the bed. Frozen is the storage unit across town: you can still get anything, but it takes a trip. Content is your bookshelf. It's not "time-based", it's just always there.

• New indices go to data_content by default; data stream backing indices go to data_hot.
• Tier placement uses index.routing.allocation.include._tier_preference (e.g. data_warm,data_hot), which ILM's migrate action sets automatically.
• Frozen uses partially mounted searchable snapshots: a local cache plus the object store. Cold can use fully mounted searchable snapshots, which lets it drop replicas.

📖 Data tiers · Searchable snapshots

1.2 Describe shard sizing and allocation strategies

Shard sizing: the rules of thumb

Analogy: moving boxes. If boxes are too big, nobody can lift them: slow recovery, slow rebalancing. If there are too many tiny boxes, the truck spends its time on paperwork: every shard costs heap, file handles and cluster-state overhead. Aim for boxes one person can carry.

Guideline	Value
Target shard size	10 GB – 50 GB (time-series often aims for about 50 GB via rollover max_primary_shard_size)
Documents per shard	Keep under 200 million
Max shards per node (non-frozen)	cluster.max_shards_per_node = 1000 by default. It's a soft limit that blocks index creation.
Heap overhead	Mostly driven by mapped fields and segments. Avoid mapping explosions and thousands of tiny indices.
Searches	Each search runs one thread per shard copy. Many small shards means lots of queuing.
Time series	Use data streams + rollover instead of daily indices to control size, not calendar.
Too many tiny shards?	_shrink, _forcemerge, reindex into fewer indices, raise rollover thresholds.
Shard too big?	_split, or lower rollover thresholds for new data.

GET _cat/shards?v&s=store:desc&h=index,shard,prirep,state,docs,store,node
GET _cat/indices?v&s=pri.store.size:desc
GET _cluster/stats?filter_path=indices.shards,indices.count

📖 Size your shards · Scaling considerations

How shard allocation works

The master's allocator decides where every shard copy lives, filtered by deciders. Each decider can say YES, NO or THROTTLE. The common ones:

Decider	Rule
Same shard	Never two copies of the same shard on one node
Disk threshold	Watermarks: low 85% (no new shards to this node), high 90% (move shards away), flood stage 95% (indices on it become read-only: index.blocks.read_only_allow_delete)
Awareness	Spread copies across attribute values (zone or rack)
Filtering	include / exclude / require rules at index or cluster level
Data tier	_tier_preference must match a node's role
Max retries	After 5 failed allocations (index.allocation.max_retries) the shard gives up until retried
Enable	cluster.routing.allocation.enable: all / primaries / new_primaries / none
Total shards per node	index.routing.allocation.total_shards_per_node hard limit

The single best diagnostic API:

GET _cluster/allocation/explain
{ "index": "orders", "shard": 0, "primary": false }

Strategy 1: allocation awareness (survive losing a zone or rack)

Analogy. Don't keep your passport and its photocopy in the same bag.

# elasticsearch.yml on each node
node.attr.zone: zone-a        # zone-b on others

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone",
    "cluster.routing.allocation.awareness.force.zone.values": "zone-a,zone-b"
  }
}

• Awareness (first setting): Elasticsearch spreads copies across zones where it can.
• Forced awareness (second setting): if zone-b disappears, Elasticsearch won't squeeze all replicas into zone-a (which could overload it). The replicas stay unassigned instead. That's intended behaviour, so a yellow cluster can be the correct answer.

📖 Shard allocation awareness

Strategy 2: allocation filtering (pin or evict shards)

// Index-level: keep "hot-orders" only on nodes tagged box_type=ssd
PUT hot-orders/_settings
{ "index.routing.allocation.require.box_type": "ssd" }

// Index-level by node name (e.g. to prepare a shrink)
PUT legacy/_settings
{ "index.routing.allocation.require._name": "es01" }

// Cluster-level: drain a node before decommissioning it
PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.exclude._name": "es03" } }

include means at least one of the values, exclude means none of them, and require means all of them. Built-in attributes: _name, _host_ip, _publish_ip, _ip, _host, _id, _tier.

📖 Index-level shard allocation · Shard allocation, relocation and recovery

Strategy 3: delayed allocation and rolling restarts

When a node leaves, its replicas are reallocated elsewhere only after index.unassigned.node_left.delayed_timeout (default 1m), which avoids expensive reshuffles when a node is just rebooting. For planned maintenance:

PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": "primaries" } }
POST _flush
// …restart the node, wait for it to join…
PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": null } }

📖 Delaying allocation when a node leaves · Rolling restart

Strategy 4: resilience design

• Small clusters: at least 3 master-eligible nodes, and 1 replica for every index you can't afford to lose. A 2-node cluster isn't resilient: losing the wrong node leaves no voting majority. Add a third node (it can be a small voting_only, master-eligible tie-breaker).
• Larger clusters: dedicated masters, spread across 3 zones, with awareness enabled.

📖 Resilience in small clusters · Resilience in larger clusters

1.3 Exercises: Architecture

1. On your three-node lab, give the nodes attributes zone: a, zone: a, zone: b. Configure forced awareness on zone with values a,b. Create index aware with 1 shard and 1 replica. Stop the zone: b node and explain the resulting health.
2. Create index ssd-only whose shards may only live on nodes with attribute disk: ssd (set it on one node). Prove it with _cat/shards.
3. Decommission node es03: move all its shards off without stopping it.
4. Explain why an index created with 1 primary shard can't be given 3 primaries in place, and name the two ways to get there.

Solution notes

1. Settings as in Strategy 1. With zone b gone, the replica stays unassigned because forced awareness forbids placing both copies in zone a, so health is yellow. _cluster/allocation/explain names the awareness decider.
2. node.attr.disk: ssd on one node, then PUT ssd-only {"settings":{"index.routing.allocation.require.disk":"ssd","number_of_replicas":0}}. With 1 replica, the replica can't allocate because only one node qualifies.
3. PUT _cluster/settings {"persistent":{"cluster.routing.allocation.exclude._name":"es03"}}, then watch _cat/shards until nothing remains on es03.
4. Routing is hash(_routing) % shards, so changing the count would misplace every document. Use _split (by multiples of the routing shards; needs a write block) or _reindex into a new index.

---

2. Data Management

Analogy for the whole domain: a filing cabinet factory. An index is a filing cabinet. Its settings are the hardware: how many drawers (primary shards) and how many photocopied backups (replicas). Its mapping is the labelling scheme for folders. Templates are the factory spec sheet, so every new cabinet named logs-* ships pre-configured. Aliases are nicknames on the door ("current-orders"), so people don't need the serial number. ILM is the records-retention policy: new files on the desk, older ones in the basement, ancient ones in the shredder.

2.1 Define an index that satisfies a given set of requirements

An index creation request has three optional parts: settings, mappings and aliases.

PUT products
{
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "refresh_interval": "30s",
    "analysis": {
      "analyzer": {
        "folded": { "type": "custom", "tokenizer": "standard",
                    "filter": ["lowercase", "asciifolding"] }
      }
    }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "sku":      { "type": "keyword" },
      "name":     { "type": "text", "analyzer": "folded",
                    "fields": { "raw": { "type": "keyword", "ignore_above": 256 } } },
      "price":    { "type": "scaled_float", "scaling_factor": 100 },
      "in_stock": { "type": "boolean" },
      "created":  { "type": "date", "format": "yyyy-MM-dd||epoch_millis" },
      "tags":     { "type": "keyword" }
    }
  },
  "aliases": { "catalog": {} }
}

Static vs dynamic settings. Know which is which:

Static (at creation, or on a closed index)	Dynamic (PUT <index>/_settings any time)
index.number_of_shards	index.number_of_replicas
index.codec (e.g. best_compression)	index.refresh_interval (-1 disables refresh)
index.mode (standard, logsdb, time_series, lookup)	index.routing.allocation.*
index.sort.*	index.blocks.write / read_only / read
analysis definitions (close → update → open)	index.max_result_window, index.default_pipeline, index.final_pipeline

PUT products/_settings
{ "index": { "number_of_replicas": 0, "refresh_interval": "-1" } }   // bulk-load mode

Adding an analyzer to an existing index: POST products/_close → PUT products/_settings with the analysis block → POST products/_open. You still can't change the analyzer of an existing field. Add a new field or multi-field, or reindex.

Changing the shard count:

• _split: more shards, by a multiple. Needs index.blocks.write: true.
• _shrink: fewer shards, to a factor. Needs a write block and a copy of every shard on one node (index.routing.allocation.require._name).
• Otherwise _reindex.

PUT products/_settings { "index.blocks.write": true }
POST products/_split/products-4sh
{ "settings": { "index.number_of_shards": 4, "index.blocks.write": null } }

dynamic mapping modes, a frequent grader trap:

Value	Unknown field in a document…
true (default)	Added to the mapping automatically
runtime	Added as a runtime field (not indexed, computed at query time)
false	Accepted and kept in _source, but not indexed or searchable
strict	Document rejected with an error

Task phrasing to watch for: "unknown fields must be rejected" means strict; "must be stored but not searchable" means false.

Index modes (9.x). logsdb is the storage-optimised mode for logs, with synthetic _source and index sorting. It's the default for logs-*-* data streams in 9.x. time_series (TSDS) is for metrics. lookup is required for the right-hand index of an ES|QL LOOKUP JOIN.

📖 Create index API · Index settings reference · Index fundamentals · Dynamic mapping · Logs data streams (logsdb)

2.2 Index templates and component templates

Composable index templates (_index_template), built from reusable component templates (_component_template), are the standard. Legacy _template templates have been deprecated since 7.8; composable ones always win over them.

Analogy: Lego. Component templates are bricks: a "1 shard" brick, an "HTTP fields" brick. The index template is the instruction sheet saying which bricks to snap together for anything named web-*, plus a few extra pieces of its own.

PUT _component_template/ct-settings-small
{ "template": { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } } }

PUT _component_template/ct-mappings-http
{
  "template": {
    "mappings": {
      "properties": {
        "http.status": { "type": "short" },
        "url":         { "type": "keyword" }
      }
    }
  }
}

PUT _index_template/web-template
{
  "index_patterns": ["web-*"],
  "priority": 200,
  "composed_of": ["ct-settings-small", "ct-mappings-http"],
  "template": {
    "settings": { "number_of_replicas": 2 },
    "aliases":  { "web-all": {} }
  },
  "_meta": { "owner": "exam" }
}

Precedence, from lowest to highest: component templates in composed_of order (later overrides earlier), then the index template's own template block, then settings in the actual create-index request. In the example, replicas end up at 2.

Priority rules:

• Only one index template applies to a new index: the matching one with the highest priority.
• Two templates with equal priority and overlapping patterns are rejected when you create the second.
• Built-in templates use priority 100 for logs-*-*, metrics-*-*, synthetics-*-*, profiling-*, security_solution-*-*, and the Streams patterns logs.otel* and logs.ecs*. Fleet integrations go up to 200. To deliberately override a built-in pattern, the docs recommend priority > 500. Otherwise choose non-overlapping patterns.

Verify before you commit:

POST _index_template/_simulate_index/web-2026.09.28   // what would this index get?
POST _index_template/_simulate/web-template            // resolved template
GET  _index_template/web-template

Templates apply only when an index is created. Changing a template never alters existing indices. For data streams, roll over to pick up changes.

📖 Templates · Simulate multi-component templates · Put index template API · Put component template API

2.3 Define and use a dynamic template that satisfies a given set of requirements

Dynamic templates control how newly detected fields get mapped. You match on the JSON-detected type (match_mapping_type), the field name (match / unmatch, wildcards or "match_pattern": "regex"), or the dotted path (path_match / path_unmatch).

Analogy. A mail-room rulebook: "anything that looks like a parcel and whose label ends in _id goes to shelf K; every other parcel goes to shelf T". The first rule that fits wins.

PUT _index_template/sensor-template
{
  "index_patterns": ["sensor-*"],
  "template": {
    "mappings": {
      "dynamic_templates": [
        { "ids_as_keyword": {
            "match_mapping_type": "string",
            "match": "*_id",
            "mapping": { "type": "keyword" } } },
        { "desc_english": {
            "match_mapping_type": "string",
            "match": "desc_*",
            "mapping": { "type": "text", "analyzer": "english" } } },
        { "counts_as_integer": {
            "match_mapping_type": "long",
            "match": "*_count",
            "mapping": { "type": "integer" } } },
        { "raw_not_indexed": {
            "path_match": "raw.*",
            "mapping": { "type": "keyword", "index": false, "doc_values": false } } },
        { "strings_default": {
            "match_mapping_type": "string",
            "mapping": { "type": "text",
                         "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }
      ]
    }
  }
}

Rules to remember:

• Order matters: first match wins. Put specific rules first and the catch-all (strings_default) last. A catch-all placed first silently shadows the others, which is a classic hidden bug in exam tasks.
• match_mapping_type values are JSON-detected types: string, long, double, boolean, date, object, binary, or *. There's no integer or float detection.
• Placeholders: {name} (field name) and {dynamic_type} (detected type), e.g. "analyzer": "{name}".
• Use "runtime": { "type": "keyword" } instead of "mapping" to create runtime fields dynamically.
• path_match matches the full dotted path (raw.a.b); match matches only the leaf name.
• Dynamic templates apply to new fields only. Fields already mapped are untouched.
• Test it: index one sample document, then GET sensor-1/_mapping.

📖 Dynamic templates · Dynamic field mapping

2.4 Define an index template that creates a new data stream

A data stream is one name (e.g. logs-app-prod) in front of a series of hidden backing indices (.ds-logs-app-prod-2026.09.28-000001, …). It's for append-only, time-stamped data and requires an @timestamp field (date or date_nanos).

Analogy: a notebook with detachable pages. You always write on the current page (the write index). When a page fills up (rollover), a new page is added. You can read all pages at once, and you tear out and throw away old pages individually (ILM delete).

Only a matching index template with "data_stream": {} creates a data stream. Without it, writing to logs-app-prod creates an ordinary index.

PUT _index_template/logs-app-template
{
  "index_patterns": ["logs-app-*"],
  "data_stream": { },
  "priority": 501,
  "composed_of": [],
  "template": {
    "settings": { "index.lifecycle.name": "logs-app-policy" },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message":    { "type": "text" },
        "level":      { "type": "keyword" }
      }
    },
    "data_stream_options": { "failure_store": { "enabled": true } }
  }
}

// Created implicitly on first write…
POST logs-app-prod/_doc
{ "@timestamp": "2026-09-28T10:00:00Z", "message": "started", "level": "INFO" }

// …or explicitly
PUT _data_stream/logs-app-prod

Operations and constraints:

Want to…	Do
Add documents	POST <ds>/_doc, or _bulk with the create action. Data streams are append-only; index only works with op_type=create.
Update or delete documents	_update_by_query / _delete_by_query on the stream, or single-doc ops on the backing index with if_seq_no / if_primary_term
Roll over now	POST logs-app-prod/_rollover
Inspect	GET _data_stream/logs-app-prod, GET _data_stream/logs-app-prod/_stats
Delete	DELETE _data_stream/logs-app-prod (deletes its backing indices too)
Change mappings or settings	Update the template, then roll over. Some changes can also be applied to the stream via the update mapping API.
Reindex into a stream	"dest": { "index": "logs-app-prod", "op_type": "create" }
Convert an alias with a write index into a stream	POST _data_stream/_migrate/<alias>
Capture rejected documents	Failure store (data_stream_options.failure_store.enabled). Query it with FROM logs-app-prod::failures or GET logs-app-prod::failures/_search.

Naming scheme. Elastic's convention is type-dataset-namespace (e.g. logs-nginx.access-prod). Any name works for your own templates, but anything matching logs-*-* hits the built-in template (priority 100) unless yours has higher priority.

📖 Data streams · Set up a data stream · Use a data stream · Modify a data stream · Failure store · Time series data streams

2.5 Define an Index Lifecycle Management policy for a time-series index

ILM moves each index through phases: hot → warm → cold → frozen → delete. Each phase has a min_age (measured from rollover, or from creation if the index never rolled over) and a set of actions.

Analogy: a restaurant kitchen's food rotation. Fresh produce is on the counter (hot). After a few days it moves to the walk-in fridge (warm), then the freezer (cold or frozen), and finally the bin (delete). The chef doesn't move things by hand; the rotation rules do.

PUT _ilm/policy/logs-app-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink":     { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate":   { "number_of_replicas": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": { "searchable_snapshot": { "snapshot_repository": "my-repo" } }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

Where each action is allowed:

Action	Hot	Warm	Cold	Frozen	Delete
rollover	✓
set_priority	✓	✓	✓
unfollow (CCR)	✓	✓	✓	✓
readonly	✓*	✓	✓
shrink, forcemerge	✓*	✓
downsample (TSDS)	✓*	✓	✓
allocate, migrate		✓	✓
searchable_snapshot	✓*		✓	✓ (required)
wait_for_snapshot, delete					✓

* In hot, only when rollover is also configured.

Key facts:

• Rollover conditions are OR-ed (max_age, max_docs, max_size, max_primary_shard_size, max_primary_shard_docs). min_* conditions are AND-ed floors. Elasticsearch never rolls over an empty index.
• Rollover needs a data stream or a write alias: index.lifecycle.rollover_alias plus a bootstrap index name-000001 with "is_write_index": true.
• migrate is implicit in warm and cold. It sets _tier_preference to the phase's tier. With no nodes of that tier, the shards stay where they are, which is a common "why isn't it moving?" moment. Disable it with "migrate": {"enabled": false} when using attribute-based allocate.
• ILM checks every indices.lifecycle.poll_interval (10m default). In a lab, set it to 10s to see transitions quickly.
• Attach a policy to new indices via the template (index.lifecycle.name). For existing indices, use PUT <index>/_settings {"index.lifecycle.name": "…"}.
• Changing a policy: an index already in a phase keeps the cached phase definition and picks up the new one at the next phase.

Bootstrap pattern without data streams (a classic task):

PUT _index_template/orders-tpl
{
  "index_patterns": ["orders-*"],
  "template": { "settings": {
    "index.lifecycle.name": "orders-policy",
    "index.lifecycle.rollover_alias": "orders" } }
}
PUT orders-000001
{ "aliases": { "orders": { "is_write_index": true } } }

Debugging ILM:

GET logs-app-prod/_ilm/explain             // phase, action, step, failed_step, step_info
POST logs-app-prod/_ilm/retry               // after fixing the cause of an ERROR step
POST _ilm/move/<index>                      // force a step (rare)
GET _ilm/status          POST _ilm/start    POST _ilm/stop
PUT _cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "10s" } }

Data stream lifecycle (DSL): the lighter alternative

For data streams you can skip ILM and just declare retention (plus optional downsampling). Elasticsearch handles rollover and deletion itself.

PUT _index_template/metrics-app-template
{
  "index_patterns": ["metrics-app-*"],
  "data_stream": {},
  "template": { "lifecycle": { "data_retention": "30d" } }
}
PUT _data_stream/metrics-app-prod/_lifecycle   { "data_retention": "7d" }
GET _data_stream/metrics-app-prod/_lifecycle/explain

If both are configured, ILM wins (index.lifecycle.prefer_ilm defaults to true). Choose ILM when you need tiers (warm, cold, frozen), shrink, forcemerge or searchable snapshots. Choose DSL when retention is all that matters.

📖 Index lifecycle management · ILM phases and actions · Rollover · Create an ILM policy · Tutorial: time series with data streams · Tutorial: time series without data streams · Check ILM status · Data stream lifecycle

2.6 Define and use index aliases

An alias is a secondary name for one or more indices or data streams.

Analogy: a phone's contact name. You call "Mum", not the number. When Mum changes numbers, you edit the contact once and nobody else has to change anything. Apps query products, and you swap which physical index sits behind it.

Core uses:

1. Zero-downtime reindex: build products-v2, then atomically move the alias.
2. Filtered views: an alias that only shows region: EU documents.
3. Rollover write target: is_write_index: true.
4. Search many indices under one name: logs-all → logs-2026.09.*.

// Atomic swap. All actions succeed or none do.
POST _aliases
{
  "actions": [
    { "remove": { "index": "products-v1", "alias": "products" } },
    { "add":    { "index": "products-v2", "alias": "products", "is_write_index": true } }
  ]
}

// Filtered alias with routing
POST _aliases
{
  "actions": [
    { "add": {
        "index": "orders",
        "alias": "orders-eu",
        "filter": { "term": { "region": "EU" } },
        "routing": "eu"
    } }
  ]
}

// Remove an index entirely and point the alias elsewhere, atomically
POST _aliases
{ "actions": [
  { "remove_index": { "index": "products-old" } },
  { "add": { "index": "products-v3", "alias": "products" } } ] }

GET _alias/products          GET products-v2/_alias          GET _cat/aliases?v

Rules:

• An alias pointing to several indices is read-only unless exactly one of them has is_write_index: true.
• Aliases can point to data streams, but you can't mix data streams and indices in one alias.
• Filtered aliases apply the filter to searches, counts and delete-by-query run through the alias.
• Aliases can be defined in index templates (template.aliases) so every new index joins automatically.
• An alias name can't equal an existing index name.

📖 Aliases · Aliases API · Privileges for data streams and aliases

2.7 Exercises: Data Management

1. Create component templates ct-1shard (1 shard, 0 replicas) and ct-app-mappings (@timestamp date, service keyword, msg text). Create index template app-logs for app-logs-* that creates a data stream, uses both components, and has priority 300. Index a document into app-logs-api and confirm the backing index has 1 shard.
2. Add to (1) a dynamic template: new strings ending in _id become keyword; all other new strings become text with a keyword sub-field named raw. Prove it with one document (remember to roll over).
3. Create ILM policy app-policy: roll over at 1 GB primary shard size or 1 day; delete 3 days after rollover. Attach it via the template, roll over manually and check _ilm/explain.
4. Index legacy has 5 shards. Produce legacy-1 with 1 shard without reindexing.
5. Indices movies-v1 and movies-v2 exist. Alias movies points to v1. Atomically repoint it to v2 and make v2 the write index. Then create filtered alias movies-recent showing only year >= 2020.

Solution notes

1. composed_of: ["ct-1shard","ct-app-mappings"], data_stream: {}, priority: 300. Check with GET _data_stream/app-logs-api, then GET .ds-app-logs-api-*/_settings?filter_path=**.number_of_shards.
2. Put the *_id rule before the catch-all string rule. The catch-all uses "mapping": {"type":"text","fields":{"raw":{"type":"keyword"}}}. Existing backing indices don't change, so POST app-logs-api/_rollover before testing.
3. Hot: rollover: {max_primary_shard_size: "1gb", max_age: "1d"}; delete: min_age: "3d". Add index.lifecycle.name: app-policy to the template's settings. Existing backing indices need PUT .ds-app-logs-api-*/_settings, or a rollover.
4. PUT legacy/_settings {"index.blocks.write": true, "index.routing.allocation.require._name": "es01"}, wait for relocation, then POST legacy/_shrink/legacy-1 {"settings":{"index.number_of_shards":1,"index.routing.allocation.require._name":null,"index.blocks.write":null}}.
5. One _aliases call with remove + add (is_write_index: true), then add with filter: {range: {year: {gte: 2020}}}.

---

3. Data Ingestion & Processing

Analogy for the whole domain: a restaurant's back office. Deliveries arrive by different routes (ingest methods). Each crate is checked against the inventory sheet that says what goes on which shelf (mapping and field types). Some produce is washed, chopped and labelled before storage (ingest pipelines / Streams). When the shelving layout changes, the stock is moved to new shelves (Reindex) or relabelled in place (Update By Query).

3.1 Select and configure appropriate ingest methods

Method	Best for	Processing	Notes
Elastic Agent + Fleet (integrations)	Logs, metrics and security data from hosts and cloud services; centrally managed	Integration's own ingest pipelines + @custom hooks	One agent, many inputs; policies managed in Kibana. Writes to type-dataset-namespace data streams.
Standalone Elastic Agent	Same, without Fleet	Same	Configured by YAML
Beats (Filebeat, Metricbeat…)	Lightweight single-purpose shippers	Beat processors + ingest pipelines	Legacy but still supported
EDOT / OpenTelemetry Collector, OTLP endpoint	OpenTelemetry-native logs, metrics and traces	Collector processors, Streams	Elastic's direction of travel for observability
Logstash	Heavy transformation, many inputs and outputs, buffering (persistent queue), routing to several clusters	Logstash filters (grok, mutate, translate…)	Extra infrastructure to run
Language clients + _bulk API	Application data, product catalogues, custom ETL	Optional pipeline= parameter	Most control; you manage batching and retries
Kibana file upload	One-off CSV, NDJSON or log files	Auto-generated pipeline and mapping	Size-limited; good for quick labs
Connectors / crawler	Content sources (SharePoint, databases, websites)	Connector sync rules + pipelines	Search use cases
Streams (Kibana)	Logs, especially unstructured ones, managed from one UI	Streams processors (see 3.6)	9.x feature
_reindex from remote	Migrating data between clusters	Optional pipeline	Needs reindex.remote.whitelist

How to choose (exam-style reasoning):

• "Collect system logs from 200 hosts, centrally managed" → Elastic Agent with Fleet and the System integration.
• "Enrich events from a Kafka topic and send them to two clusters" → Logstash.
• "Index product data from our application" → client + _bulk, with an ingest pipeline for light cleanup.
• "Parse free-text logs quickly without writing grok by hand" → Streams.
• "Transform data already in Elasticsearch" → ingest pipeline + _reindex or _update_by_query.

The Bulk API is the workhorse behind almost every method:

POST _bulk?pipeline=clean-products
{ "index":  { "_index": "products", "_id": "1" } }
{ "sku": "A-1", "name": "Café crème", "price": 3.5 }
{ "create": { "_index": "logs-app-prod" } }
{ "@timestamp": "2026-09-28T10:00:00Z", "message": "hello" }
{ "update": { "_index": "products", "_id": "1" } }
{ "doc": { "price": 3.9 } }
{ "delete": { "_index": "products", "_id": "2" } }

The format is NDJSON: every line ends with a newline, including the last one. The response has a top-level errors: true/false, and each item carries its own status, so always check errors. Aim for batches of about 5–15 MB.

📖 Ingest overview · Ingest tools overview · Ingest architectures · Bulk API · Upload data files · OTLP endpoint · Tune for indexing speed

3.2 Define a mapping that satisfies a given set of requirements

A mapping is the schema: which fields exist, their types, and how each is indexed and analysed.

Analogy: a customs form. Each box says what the item is ("date", "exact code", "free text"). Customs (Lucene) stores it in the right way. A date written in the "free text" box can't be range-searched later, and you can't change the form once the parcel is shipped.

Explicit vs dynamic. Dynamic mapping guesses, for example "2026-09-28" becomes date and "hello" becomes text + .keyword. The exam almost always wants explicit mappings.

PUT articles
{
  "mappings": {
    "dynamic": "strict",
    "_source": { "excludes": ["internal_notes"] },
    "properties": {
      "title":    { "type": "text", "analyzer": "english",
                    "fields": { "raw": { "type": "keyword" },
                                "suggest": { "type": "search_as_you_type" } } },
      "author":   { "type": "keyword", "normalizer": "lowercase" },
      "body":     { "type": "text", "copy_to": "all_text" },
      "summary":  { "type": "text", "copy_to": "all_text" },
      "all_text": { "type": "text" },
      "views":    { "type": "integer", "null_value": 0 },
      "rating":   { "type": "half_float", "ignore_malformed": true },
      "published":{ "type": "date", "format": "strict_date_optional_time||epoch_millis" },
      "status":   { "type": "keyword", "doc_values": true, "index": true },
      "raw_html": { "type": "text", "index": false },
      "comments": { "type": "nested",
                    "properties": { "user": { "type": "keyword" },
                                    "text": { "type": "text" },
                                    "stars": { "type": "byte" } } },
      "labels":   { "type": "flattened" },
      "internal_notes": { "type": "text" }
    }
  }
}

Mapping parameters you must know:

Parameter	Meaning	Typical task wording
index: false	Not searchable (still in _source; aggregatable if it has doc values)	"stored but not searchable"
doc_values: false	Can't sort or aggregate; saves disk	"never used for aggregations"
enabled: false (object only)	Whole object kept in _source, not parsed at all	"ignore the contents of payload"
store: true	Stored separately from _source (retrieve with stored_fields)	rare
copy_to	Copies the value into another field (a custom "all fields" field)	"search title and body as one field"
fields	Multi-fields: same value indexed several ways	"full-text and exact-match / sortable"
analyzer / search_analyzer	Index-time vs query-time analysis	"autocomplete", "synonyms at search time"
normalizer	Analyzer for keyword (lowercase, asciifolding)	"case-insensitive exact match"
ignore_above	Don't index keyword strings longer than N	"skip values longer than 256"
null_value	Index a replacement for explicit null	"treat null as 0 / NONE"
ignore_malformed	Skip bad values instead of rejecting the document	"don't fail on bad numbers"
coerce	Convert "5" to 5 (default true)	"reject strings in numeric fields" → coerce: false
format	Date formats (yyyy-MM-dd, epoch_second…)	"dates like 28/09/2026" → dd/MM/yyyy
dynamic (per object)	true/false/strict/runtime at any level	"strict except in meta"

What you can and can't change on an existing index:

✅ Allowed via PUT <index>/_mapping	❌ Needs a new index + reindex
Add new fields	Change a field's type
Add a multi-field to an existing field, then run _update_by_query to populate it	Change a field's analyzer
Change ignore_above, search_analyzer (with care), mapping _meta	Change index, doc_values or nested vs object
Add runtime fields	Rename a field (use a field alias or reindex)

PUT articles/_mapping
{ "properties": { "author": { "type": "keyword", "fields": { "text": { "type": "text" } } } } }
POST articles/_update_by_query?conflicts=proceed      // populates author.text for existing docs

Text analysis: custom analyzers

Analogy: a kitchen processing line. Character filters wash the vegetables (strip HTML, map characters). The tokenizer chops them into pieces (words). Token filters season each piece (lowercase, remove stop words, stem, add synonyms). The same line must be used for indexing and searching, or the pieces won't match.

PUT recipes
{
  "settings": {
    "analysis": {
      "char_filter": {
        "amp_to_and": { "type": "mapping", "mappings": ["& => and"] }
      },
      "filter": {
        "my_stop":  { "type": "stop", "stopwords": "_english_" },
        "my_syn":   { "type": "synonym_graph", "synonyms": ["courgette, zucchini", "aubergine, eggplant"] },
        "edge_3_10":{ "type": "edge_ngram", "min_gram": 3, "max_gram": 10 }
      },
      "analyzer": {
        "recipe_text": {
          "type": "custom",
          "char_filter": ["html_strip", "amp_to_and"],
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "my_stop", "porter_stem"]
        },
        "recipe_search": {
          "type": "custom", "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "my_syn", "porter_stem"]
        },
        "autocomplete": {
          "type": "custom", "tokenizer": "standard",
          "filter": ["lowercase", "edge_3_10"]
        }
      },
      "normalizer": {
        "lower_fold": { "type": "custom", "filter": ["lowercase", "asciifolding"] }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text", "analyzer": "recipe_text", "search_analyzer": "recipe_search",
        "fields": {
          "ac":  { "type": "text", "analyzer": "autocomplete", "search_analyzer": "standard" },
          "raw": { "type": "keyword", "normalizer": "lower_fold" }
        }
      }
    }
  }
}

POST recipes/_analyze
{ "analyzer": "recipe_text", "text": "<b>Courgette & Crème</b> brûlée recipes" }

POST recipes/_analyze
{ "field": "name.ac", "text": "Courgette" }

Analysis facts to remember:

• The default analyzer is standard (standard tokenizer + lowercase). Built-in analyzers include simple, whitespace, keyword, stop, pattern and language analyzers (english, french…).
• Synonyms at search time use synonym_graph in the search_analyzer. To update synonyms without reindexing, use the Synonyms API (PUT _synonyms/my-set) with "synonyms_set": "my-set" and "updateable": true on a search-time filter.
• Edge n-grams go at index time only; search with the standard analyzer, otherwise the query is n-grammed too.
• keyword fields are not analysed. Use a normalizer for case-insensitivity.
• Always test with _analyze before creating the real index.

📖 Mapping · Explicit mapping · Update mapping examples · Mapping parameters · Text analysis · Anatomy of an analyzer · Create a custom analyzer · Test an analyzer · Index and search analysis · Synonyms

3.3 Define and use appropriate field data types for indexing, search and aggregation

Analogy. Pick the container for what you'll do with the contents. Soup goes in a bowl (text, for searching within), coins in a counting tray (numeric, for maths and ranges), keys on labelled hooks (keyword, for exact lookup, sorting and grouping).

Requirement	Type	Why
Full-text search (words, relevance)	text	Analysed into tokens
Exact match, sorting, terms aggregation (IDs, status, tags, emails)	keyword	Stored as-is, with doc values
Both of the above	text + fields.keyword (multi-field)	Default for dynamic strings
Same value in every document of an index	constant_keyword	Nearly free, great for data streams
Grep-like *foo* on huge high-cardinality strings	wildcard	Fast wildcard and regexp
Log messages, scoring not needed	match_only_text	Much smaller than text
Integers	byte / short / integer / long / unsigned_long	Smallest type that fits
Decimals	float / half_float / double / scaled_float (prices: scaling_factor: 100)
Dates and times	date (ms) / date_nanos	Range queries, date histograms
True/false	boolean
IPv4/IPv6 and CIDR queries	ip	term: {"ip": "10.0.0.0/8"} works
Lat/lon, distance, bounding box	geo_point	Geo queries and aggs
Polygons, shapes	geo_shape
Numeric or date intervals	integer_range, date_range, ip_range…	"Is 42 inside the doc's range?"
Arrays of objects queried together	nested	Keeps each object's fields bound together
Simple JSON objects	object (default)	Flattened into dotted fields
Unknown or unbounded keys (labels, tags maps)	flattened	One field, avoids mapping explosion
Parent/child in the same index	join	Costly; use sparingly
Semantic search (managed embeddings)	semantic_text	Inference handled for you (see 4.5)
Your own embeddings	dense_vector (kNN) / sparse_vector
Autocomplete as you type	search_as_you_type / completion
Software versions (1.10 > 1.9)	version	Correct semantic ordering
Pre-aggregated metrics	histogram, aggregate_metric_double
Another name for an existing field	alias	"Rename" without reindex (for search)

Arrays. There's no array type. Any field can hold several values of the same type.

object vs nested: the most tested pitfall

// Document:
{ "comments": [ { "user": "alice", "stars": 5 }, { "user": "bob", "stars": 1 } ] }

With object, Lucene stores comments.user: [alice, bob] and comments.stars: [5, 1], so the pairing is lost. A search for "alice AND stars 1" wrongly matches.

With nested, each comment is a hidden sub-document. You must use the nested query and the nested aggregation:

GET articles/_search
{
  "query": {
    "nested": {
      "path": "comments",
      "query": { "bool": { "must": [
        { "term": { "comments.user": "alice" } },
        { "range": { "comments.stars": { "gte": 4 } } } ] } },
      "inner_hits": {}
    }
  }
}

The limits are index.mapping.nested_fields.limit (50) and index.mapping.nested_objects.limit (10,000 per document). Other mapping explosion guards: index.mapping.total_fields.limit (1000) and index.mapping.depth.limit (20).

Runtime fields: schema on read

A field computed at query time from _source or other fields. There's nothing to reindex, but it's slower.

PUT articles/_mapping
{
  "runtime": {
    "day_of_week": {
      "type": "keyword",
      "script": { "source": "emit(doc['published'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ENGLISH))" }
    }
  }
}

// Or only for one search:
GET articles/_search
{
  "runtime_mappings": {
    "views_k": { "type": "double", "script": "emit(doc['views'].value / 1000.0)" }
  },
  "fields": ["views_k"],
  "aggs": { "by_day": { "terms": { "field": "day_of_week" } } }
}

📖 Field data types · keyword · text · Numeric · date · nested · flattened · Multi-fields · Runtime fields · Mapping limit settings

3.4 Use the Reindex API and Update By Query API

Analogy. Reindex is moving house: you pack every item from the old flat and unpack it into a new one that has a different layout (new mapping, new analyzer, new shard count). Update By Query is redecorating in place: same flat, you walk from room to room and change what matches.

Reindex

POST _reindex?wait_for_completion=false&slices=auto
{
  "source": {
    "index": "articles",
    "query": { "range": { "published": { "gte": "2025-01-01" } } },
    "_source": ["title", "author", "published", "views"],
    "size": 1000
  },
  "dest": {
    "index": "articles-v2",
    "pipeline": "normalize-articles",
    "op_type": "create"
  },
  "script": {
    "lang": "painless",
    "source": "ctx._source.views = ctx._source.views == null ? 0 : ctx._source.views; ctx._source.remove('internal_notes')"
  },
  "conflicts": "proceed",
  "max_docs": 100000
}

Option	Use
source.query	Copy only matching documents
source._source	Copy only some fields
dest.pipeline	Transform with an ingest pipeline on the way in
script	Painless on ctx._source (also ctx._id, ctx._index; set ctx.op = "noop" to skip, or "delete")
dest.op_type: create	Only create missing documents; required when the destination is a data stream
dest.version_type: external	Keep newer versions already in the destination
conflicts: proceed	Count version conflicts instead of aborting
max_docs	Limit the copy
slices=auto	Parallelise
requests_per_second	Throttle (change later with _reindex/<task>/_rethrottle)
wait_for_completion=false	Returns a task ID. Poll GET _tasks/<id>, cancel with POST _tasks/<id>/_cancel.
source.remote	Reindex from another cluster. The destination node needs reindex.remote.whitelist: "host:9200" in elasticsearch.yml.

The destination is NOT created from the source's mapping. Create articles-v2 (or a template for it) first, otherwise dynamic mapping guesses the types.

Zero-downtime migration recipe: create v2 with the new mapping → _reindex → verify _count → atomic alias swap (2.6) → delete v1 if asked.

Update By Query

POST articles/_update_by_query?conflicts=proceed&wait_for_completion=false
{
  "query": { "term": { "status": "draft" } },
  "script": {
    "lang": "painless",
    "source": "ctx._source.status = params.s; ctx._source.tags = ctx._source.tags ?: []; ctx._source.tags.add('migrated')",
    "params": { "s": "archived" }
  }
}

// Re-run all docs through a pipeline (e.g. to populate a new field)
POST articles/_update_by_query?pipeline=add-reading-time
{ "query": { "bool": { "must_not": { "exists": { "field": "reading_time" } } } } }

// Pick up a newly added multi-field for all docs
POST articles/_update_by_query?refresh

Related APIs:

• POST <index>/_update/<id> with doc (partial), script, upsert or doc_as_upsert
• POST <index>/_delete_by_query
• ?refresh to make results immediately visible

Painless essentials: ctx._source.field, params.x, ctx._source.remove('f'), ctx._source.containsKey('f'), null-safe ?. and elvis ?:, ctx.op = 'noop' | 'delete'. In ingest pipelines, fields are on ctx.field directly (no _source).

📖 Reindex API · Update by query API · Delete by query API · Update API · Migrate data using reindex · Painless · Update documents with scripts · Tasks API

3.5 Define and use an ingest pipeline that satisfies a given set of requirements

An ingest pipeline is an ordered list of processors that runs on ingest nodes before a document is indexed.

Analogy: an airport baggage belt. Each station (processor) scans, relabels or reroutes the bag. If a station jams, the bag is either sent to a lost-luggage office (on_failure) or the whole belt stops (default: the document is rejected).

PUT _ingest/pipeline/web-logs
{
  "description": "Parse Apache-style access logs",
  "processors": [
    { "grok": {
        "field": "message",
        "patterns": ["%{IPORHOST:client.ip} %{USER:ident} %{USER:user.name} \\[%{HTTPDATE:ts}\\] \"%{WORD:http.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.status:int} (?:%{NUMBER:bytes:long}|-)"]
    } },
    { "date":   { "field": "ts", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"], "target_field": "@timestamp" } },
    { "remove": { "field": ["ts", "ident"], "ignore_missing": true } },
    { "set":    { "field": "event.ingested", "value": "{{{_ingest.timestamp}}}" } },
    { "lowercase": { "field": "http.method" } },
    { "geoip":  { "field": "client.ip", "target_field": "client.geo", "ignore_failure": true } },
    { "user_agent": { "field": "user_agent.original", "ignore_missing": true } },
    { "set": {
        "if": "ctx.http?.status != null && ctx.http.status >= 500",
        "field": "event.outcome", "value": "failure" } },
    { "script": {
        "lang": "painless",
        "source": "ctx.bytes_kb = ctx.bytes == null ? 0 : ctx.bytes / 1024.0"
    } },
    { "drop": { "if": "ctx.url?.original == '/healthz'" } }
  ],
  "on_failure": [
    { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } },
    { "set": { "field": "_index", "value": "failed-{{ _index }}" } }
  ]
}

Frequently used processors:

Processor	Purpose
set, remove, rename, append, convert, lowercase/uppercase, trim, split, join, gsub	Field manipulation
grok, dissect, kv, csv, json	Parse text into fields (dissect is faster, for fixed delimiters; grok uses regex)
date, date_index_name	Parse dates; route to date-based index names
geoip, user_agent, uri_parts, community_id, network_direction, registered_domain	Enrichment
enrich	Lookup-join against another index
inference	Run an ML model (e.g. embeddings)
script	Painless for anything else
pipeline	Call another pipeline (modular pipelines)
reroute	Send a document to another data stream (dataset or namespace)
foreach	Apply a processor to each array element
drop, fail	Discard a document; raise a custom error
fingerprint, redact	Deduplication IDs; mask PII

Common processor options: if (Painless condition, null-safe with ?.), ignore_failure, ignore_missing, on_failure (per processor), tag, description.

Error handling, in order of scope:

1. Processor-level ignore_failure: true skips silently.
2. Processor-level on_failure: [...] handles the error locally.
3. Pipeline-level on_failure is the catch-all. Metadata: _ingest.on_failure_message, _ingest.on_failure_processor_type, _ingest.on_failure_processor_tag.
4. Nothing catches it → the document is rejected. For data streams with a failure store, it goes to ::failures instead.

Testing, which you should always do:

POST _ingest/pipeline/web-logs/_simulate?verbose=true
{ "docs": [ { "_source": { "message": "1.2.3.4 - - [28/Sep/2026:10:00:00 +0000] \"GET /a HTTP/1.1\" 503 12" } } ] }

// Simulate with the pipelines that WOULD run for an index (defaults, final, templates)
POST _ingest/_simulate
{ "docs": [ { "_index": "web-2026", "_source": { "message": "…" } } ] }

Ways to apply a pipeline:

How	Scope
?pipeline=name on index, bulk, reindex (dest.pipeline) or update-by-query	One request
index.default_pipeline	Default for an index; overridden by ?pipeline=
index.final_pipeline	Always runs last, even if a request pipeline was given
Index template setting	Every new index or backing index
pipeline processor	Nested pipelines
Integrations' @custom pipelines (e.g. logs-nginx.access@custom)	Customise Fleet integrations safely

PUT web-2026/_settings { "index.default_pipeline": "web-logs" }

Enrich processor: a lookup table at ingest time

PUT users-ref/_doc/1   { "email": "a@x.com", "dept": "finance", "manager": "zoe" }

PUT _enrich/policy/users-policy
{ "match": { "indices": "users-ref", "match_field": "email", "enrich_fields": ["dept", "manager"] } }

POST _enrich/policy/users-policy/_execute        // builds the .enrich-* index; re-run after the source changes

PUT _ingest/pipeline/add-user-info
{ "processors": [ { "enrich": {
    "policy_name": "users-policy", "field": "user.email",
    "target_field": "user.info", "max_matches": 1, "ignore_missing": true } } ] }

Policy types: match (exact term), range (value inside a range), geo_match (point in shape). An enrich policy can't be updated. Delete it and recreate it; you have to remove the processors that use it first.

📖 Ingest pipelines · Processor reference · Error handling · Example: parse logs · Data enrichment · Set up an enrich processor · Simulate pipeline API · Grok debugger · Painless in ingest

3.6 Ingest and process data using Streams

Streams is Kibana's UI and API layer for managing log data end to end: getting data in, parsing it, routing it, mapping fields, setting retention and watching data quality. Behind the scenes it creates and manages ordinary Elasticsearch objects (data streams, index templates, ingest pipelines, lifecycle settings).

Analogy: a smart sorting office with a control panel. Instead of wiring each conveyor belt (template, pipeline, ILM) yourself, you use one panel: "parse these letters like this, send the ones from Paris to that shelf, keep them for 30 days". The panel builds the belts for you. Don't rewire the belts by hand behind its back.

Two kinds of streams (9.2–9.3)

	Classic streams	Wired streams (tech preview)
What	Your existing data streams (e.g. logs-nginx.access-default), adopted in place	A managed hierarchy rooted at a special endpoint
Migration	None; data keeps flowing as before	Send data to the root stream: logs in 9.2–9.3 (logs.otel / logs.ecs from 9.4)
Processing	✓ (added without touching the underlying index template)	✓
Retention	✓ (inherit from template, custom period, or ILM policy)	✓ (inherit from parent, custom, or ILM)
Partitioning (routing into child streams)	✗	✓, e.g. logs → logs.nginx → logs.nginx.errors
Field mapping (Schema)	✓ (add mappings)	✓ (children inherit from the parent)
Inheritance	n/a	Children inherit mappings, lifecycle and processing from their parent

A third kind, query streams (virtual, defined by an ES|QL query), appears in newer versions.

Typical workflow in the Streams UI

1. Get data in. For wired streams, turn on wired streams in Streams settings, then point Elastic Agent, Filebeat, the OTel Collector or Logstash at the logs index/stream. No template setup is needed. For classic streams, just open an existing data stream in Streams.
2. Partition (wired only). Define routing conditions such as service.name == "nginx" → logs.nginx. Partition by logical groups (team, technology, environment), not by high-cardinality fields. Aim for tens of partitions, not hundreds. Each child is a real data stream.
3. Process. Build a processing pipeline in the Processing tab from processors that mirror ingest processors (grok, dissect, date, set, rename, convert, remove, append, drop, math, json extract, redact, user agent, URI parts, enrich…). Add conditions to steps. The Data preview simulates results on sample documents before you save. With a generative-AI connector configured, Suggest a pipeline can generate grok/dissect rules for you.
4. Map fields (Schema tab). Promote extracted fields to proper mapped types (keyword, date, long…).
5. Configure retention (Data lifecycle tab). Choose inherit, a custom retention period (data stream lifecycle), or an existing ILM policy.
6. Monitor data quality (Data quality tab). Degraded documents are ones with ignored or malformed fields (_ignored); failed documents are kept in the failure store so you can fix the processor using the real failing documents.
7. Advanced tab. Inspect the managed index templates, component templates and pipelines Streams created.

Rules to remember:

• Streams-managed components are managed. Change them through the Streams UI or API, not with raw Elasticsearch APIs.
• Streams is a Kibana feature: its API is under kbn:/api/streams/... (Kibana API), not the Elasticsearch REST API. In Console, prefix with kbn:.
• Required privileges to manage streams include cluster manage_index_templates, manage_ingest_pipelines, manage_pipeline and read_pipeline, plus data-stream-level read, write, create, manage, monitor, manage_data_stream_lifecycle, read_failure_store, manage_failure_store and manage_ilm.
• Don't reroute or migrate existing data streams into the wired root; classic streams work in place.
• Because Streams is new and moving fast, exam tasks are most likely to use the UI flow: add a processor, verify in preview, set retention, check the failure store. Practise it on your lab with sample logs.

# Example Kibana API calls from Console (verify paths against the 9.3 Streams API reference)
GET kbn:/api/streams
GET kbn:/api/streams/logs

📖 Streams overview · Get data into Streams · Wired streams · Partition (organise) data · Process documents · Configure data lifecycle · Map fields · Manage data quality · Advanced settings · Streamlang processors · Streams Kibana API

3.7 Exercises: Ingestion & Processing

1. Create index shop with: name (full-text with the English analyzer, plus a case-insensitive exact sub-field name.exact), sku (exact), price (two decimals, cheap to store), tags (exact, not aggregatable), specs (arbitrary key/values, no mapping explosion), reviews (array of {user, stars} that must be queried together). Unknown fields must be rejected.
2. Create pipeline shop-clean that trims and upper-cases sku, converts price from string to float, sets ingested_at, drops docs whose status is test, and on any failure writes the error to ingest_error instead of rejecting. Simulate it with 2 docs.
3. Reindex shop into shop-v2 (identical mapping except name uses a custom analyzer with asciifolding) through shop-clean, only for docs with price > 0. Then point alias shop-live at shop-v2.
4. Add a name.ac autocomplete multi-field to shop-v2 and populate it for existing docs without reindexing.
5. Create an enrich policy from index suppliers (sku → supplier, country) and add it to shop-clean.
6. In Streams, open the logs-generic-default classic stream (index some raw log lines first), add a dissect processor that extracts level and msg, map level as keyword, set retention to 14 days, and inspect the failure store.

Solution notes

1. dynamic: strict; name: text/english + fields.exact: keyword with a lowercase normalizer; sku: keyword; price: scaled_float, scaling_factor: 100; tags: keyword, doc_values: false; specs: flattened; reviews: nested.
2. Processors: trim + uppercase on sku; convert {type: float} on price; set ingested_at = {{{_ingest.timestamp}}}; drop with if: "ctx.status == 'test'". Pipeline-level on_failure sets ingest_error = {{ _ingest.on_failure_message }}. Verify with _simulate?verbose.
3. Create shop-v2 first. _reindex with source.query: range price gt 0 and dest.pipeline: shop-clean. Compare _count, then use _aliases add.
4. PUT shop-v2/_mapping adding fields.ac with an edge-ngram analyzer. The analyzer must already exist in the index settings; if not, close → add analysis → open. Then POST shop-v2/_update_by_query.
5. PUT _enrich/policy/... (match), _execute, then add an enrich processor to the pipeline. Re-PUT the whole pipeline; the PUT replaces it.
6. The UI flow covered in 3.6. Check the result on the Advanced tab and in GET logs-generic-default/_mapping.

---

4. Searching Data

Analogy for the whole domain: a library. Full-text queries are asking the librarian "books about sailing". She interprets the words and ranks the answers by relevance. Term-level queries are the card catalogue: exact codes, no interpretation. Bool combines requests: "must be about sailing, must be in French, should be recent, must not be a children's book". Aggregations are the statistics desk: "how many books per genre, average page count". ES|QL is a spreadsheet-style pipeline over the same shelves. Semantic search is a librarian who understands meaning ("boat trips" finds "sailing"). Cross-cluster search asks sister libraries in other cities at the same time.

4.1 Search for terms and/or phrases in one or more fields

Full-text queries (analysed: the query text goes through the field's analyzer)

// Single field, any word (OR by default)
GET articles/_search
{ "query": { "match": { "title": "elastic search guide" } } }

// All words required, or a minimum
{ "query": { "match": { "title": { "query": "elastic search guide", "operator": "and" } } } }
{ "query": { "match": { "title": { "query": "elastic search guide", "minimum_should_match": "2" } } } }

// Typo tolerance
{ "query": { "match": { "title": { "query": "elastik", "fuzziness": "AUTO" } } } }

// Exact phrase, with slop (words may be up to N positions apart)
{ "query": { "match_phrase": { "body": { "query": "shard allocation", "slop": 2 } } } }

// Phrase prefix, "search as you type" style
{ "query": { "match_phrase_prefix": { "title": "quick bro" } } }

// Several fields, with boosting
{ "query": { "multi_match": {
    "query": "shard sizing",
    "fields": ["title^3", "summary^2", "body"],
    "type": "best_fields",
    "tie_breaker": 0.3 } } }

multi_match types:

Type	Behaviour	Use when
best_fields (default)	Score of the best-matching field (+ tie_breaker × others)	Words expected together in one field
most_fields	Sum of all field scores	Same text indexed several ways (multi-fields)
cross_fields	Treats the fields as one big field (term-centric)	"first_name + last_name" style data
phrase / phrase_prefix	match_phrase on each field	Phrases across fields
bool_prefix	For search_as_you_type fields	Autocomplete

Also know combined_fields (like cross_fields for fields sharing an analyzer), query_string (Lucene syntax with AND/OR/NOT, wildcards and field names; throws on bad syntax) and simple_query_string (forgiving, safe for user input: + | - " * ( )).

Term-level queries (NOT analysed: exact values on keyword, numeric, date, ip, boolean)

{ "query": { "term":  { "status": "published" } } }
{ "query": { "term":  { "status": { "value": "Published", "case_insensitive": true } } } }
{ "query": { "terms": { "tags": ["elasticsearch", "lucene"] } } }
{ "query": { "range": { "published": { "gte": "now-30d/d", "lt": "now/d" } } } }
{ "query": { "range": { "price": { "gt": 10, "lte": 100 } } } }
{ "query": { "exists": { "field": "author" } } }
{ "query": { "prefix": { "sku": "ABC-" } } }
{ "query": { "wildcard": { "sku": { "value": "A*-9?" } } } }
{ "query": { "regexp": { "sku": "A[0-9]{3}" } } }
{ "query": { "fuzzy": { "author": { "value": "jonh", "fuzziness": 1 } } } }
{ "query": { "ids": { "values": ["1", "7"] } } }
{ "query": { "terms_set": { "skills": { "terms": ["java","go","rust"], "minimum_should_match_field": "required" } } } }

The number-one trap: term on a text field. "Published" was indexed as published, so term: "Published" finds nothing. Use match on text, or term on the .keyword sub-field.

Date math: now-1d/d (rounded to the day), 2026-09-01||+1M, plus a time_zone parameter on range queries.

Shaping results: pagination, sorting, source filtering, highlighting

GET articles/_search
{
  "from": 0, "size": 10,
  "track_total_hits": true,
  "_source": ["title", "published"],
  "fields": ["title", "published", "day_of_week"],
  "sort": [ { "published": "desc" }, { "title.raw": "asc" }, "_score" ],
  "highlight": {
    "pre_tags": ["<mark>"], "post_tags": ["</mark>"],
    "fields": { "body": { "fragment_size": 150, "number_of_fragments": 3 } }
  },
  "query": { "match": { "body": "shard allocation" } }
}

• from + size is capped by index.max_result_window (10,000). For deep paging use search_after with a point in time (POST <index>/_pit?keep_alive=1m). Scroll is legacy for this purpose.
• You can't sort on text fields. Sort on a keyword sub-field, a number or a date.
• track_total_hits defaults to accurate counting up to 10,000; set true for an exact count.
• explain: true shows the score calculation. profile: true shows per-shard timings (or use Kibana's Search Profiler).

📖 Full-text queries · match · match_phrase · multi_match · query_string · Term-level queries · term · range · Paginate results · Sort results · Highlighting · The _search API

4.2 Boolean combinations of queries and filters

GET articles/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "body": "elasticsearch" } }
      ],
      "should": [
        { "match_phrase": { "title": { "query": "certified engineer", "boost": 2 } } },
        { "term": { "tags": "exam" } }
      ],
      "minimum_should_match": 1,
      "filter": [
        { "term":  { "status": "published" } },
        { "range": { "published": { "gte": "2025-01-01" } } }
      ],
      "must_not": [
        { "term": { "author": "spam-bot" } }
      ]
    }
  }
}

Clause	Must match?	Affects score?	Cached?
must	Yes	Yes	No
filter	Yes	No (filter context)	Yes
should	See below	Yes (adds to the score)	No
must_not	Must not match	No (filter context)	Yes

should semantics (subtle, and heavily tested):

• If the bool has no must or filter, at least one should must match (minimum_should_match defaults to 1).
• If there is a must or filter, should clauses are optional and only boost the score, unless you set minimum_should_match.

Query vs filter context rule of thumb: "how well does it match?" (words, relevance) goes in must/should. "Does it match, yes or no?" (status, dates, ranges, IDs) goes in filter/must_not. Filters are faster and cached.

Other compound queries:

• constant_score: wrap a filter and give every hit the same score.
• boosting: positive query, with a negative query that demotes (not excludes) matches.
• dis_max: best of several queries, plus tie_breaker.
• function_score / script_score: modify scores by recency, popularity (field_value_factor), decay functions or scripts.
• Named queries ("_name": "q1") report which clauses matched each hit.

Nesting example: "(title contains X OR summary contains X) AND year ≥ 2020 AND NOT draft":

{ "query": { "bool": {
    "must": [ { "bool": { "should": [
        { "match": { "title": "X" } }, { "match": { "summary": "X" } } ] } } ],
    "filter": [ { "range": { "year": { "gte": 2020 } } } ],
    "must_not": [ { "term": { "status": "draft" } } ] } } }

post_filter filters the hits but not the aggregations. Use it for faceted navigation, where facet counts shouldn't shrink when a facet is selected.

📖 Query and filter context · bool query · Compound queries · function_score · Filter search results (post_filter)

4.3 Write and execute metric and bucket aggregations

Analogy: sorting sweets. Bucket aggregations tip the sweets into bowls by colour, by size range, or by the day they were bought. Metric aggregations count, weigh or average what's in each bowl. Pipeline aggregations compare the bowls ("which bowl is heaviest?", "running total"). Bowls can hold smaller bowls (sub-aggregations).

GET kibana_sample_data_ecommerce/_search
{
  "size": 0,
  "query": { "range": { "order_date": { "gte": "now-30d/d" } } },
  "aggs": {
    "per_day": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day",
        "time_zone": "Europe/Paris",
        "min_doc_count": 0,
        "extended_bounds": { "min": "now-30d/d", "max": "now/d" }
      },
      "aggs": {
        "revenue":     { "sum": { "field": "taxful_total_price" } },
        "unique_buyers": { "cardinality": { "field": "customer_id" } }
      }
    },
    "top_categories": {
      "terms": { "field": "category.keyword", "size": 5, "order": { "revenue": "desc" } },
      "aggs": {
        "revenue":   { "sum": { "field": "taxful_total_price" } },
        "price_stats": { "stats": { "field": "taxful_total_price" } },
        "p95":       { "percentiles": { "field": "taxful_total_price", "percents": [50, 95, 99] } },
        "best_order": { "top_hits": { "size": 1, "sort": [{ "taxful_total_price": "desc" }],
                                      "_source": ["order_id", "customer_full_name"] } }
      }
    },
    "price_bands": {
      "range": { "field": "taxful_total_price",
                 "ranges": [ { "to": 50 }, { "from": 50, "to": 150 }, { "from": 150 } ] }
    },
    "best_day": { "max_bucket": { "buckets_path": "per_day>revenue" } },
    "avg_daily_revenue": { "avg_bucket": { "buckets_path": "per_day>revenue" } }
  }
}

Metric aggregations: avg, sum, min, max, value_count, stats, extended_stats (std deviation, variance), cardinality (approximate distinct count, HyperLogLog++, precision_threshold), percentiles, percentile_ranks, median_absolute_deviation, weighted_avg, top_hits, top_metrics, geo_bounds, geo_centroid, scripted_metric.

Bucket aggregations:

Aggregation	Groups by	Key options
terms	Distinct values (keyword, number, ip…)	size (default 10), order (_count, _key or a sub-agg), min_doc_count, missing, include/exclude, shard_size
multi_terms	Combinations of several fields	terms: [{field}, {field}]
rare_terms, significant_terms	Rare or unusually frequent terms
histogram	Numeric intervals	interval, min_doc_count, extended_bounds
date_histogram	Time intervals	calendar_interval (minute, hour, day, week, month, quarter, year, or 1d, 1M) vs fixed_interval (30m, 7d; multiples allowed); time_zone, format, offset
range / date_range / ip_range	Custom ranges (from inclusive, to exclusive)	keyed: true
filter / filters	One or several query-defined buckets	other_bucket
composite	Paginated combinations of sources	after key: iterate all buckets
nested / reverse_nested	Enter or leave nested documents	path
global	Ignores the query scope
missing	Documents without the field
geohash_grid, geotile_grid, auto_date_histogram, variable_width_histogram, sampler	Specialised

Pipeline aggregations reference other aggregations by buckets_path (> for nesting, . for a metric's value, _count for doc count):

• Sibling (a new bucket next to the others): avg_bucket, max_bucket, min_bucket, sum_bucket, stats_bucket, percentiles_bucket.
• Parent (inside each histogram bucket): derivative, cumulative_sum, moving_fn, bucket_script, bucket_selector (keeps only matching buckets, like HAVING), bucket_sort (sort or truncate buckets), serial_diff.

"aggs": {
  "per_month": {
    "date_histogram": { "field": "order_date", "calendar_interval": "month" },
    "aggs": {
      "revenue":  { "sum": { "field": "taxful_total_price" } },
      "cum_rev":  { "cumulative_sum": { "buckets_path": "revenue" } },
      "big_only": { "bucket_selector": {
          "buckets_path": { "r": "revenue" }, "script": "params.r > 10000" } }
    }
  }
}

Aggregation facts:

• Set "size": 0 when you only need aggregations.
• You can't run terms on a text field (it has no doc values). Use .keyword.
• terms counts are approximate across shards (doc_count_error_upper_bound). Increase shard_size for accuracy. Use composite to list all buckets.
• Aggregations run on the documents matched by query. Use global to escape that scope, or post_filter to filter hits only.
• For nested fields, wrap sub-aggs in a nested aggregation.

📖 Aggregations · Aggregations tutorial (eCommerce) · Metric aggregations · Bucket aggregations · Pipeline aggregations · terms · date_histogram · composite

4.4 Write and execute ES|QL queries

ES|QL (Elasticsearch Query Language) is a piped language. Each command transforms a table and passes it to the next, like a Unix pipeline or a spreadsheet you keep refining.

Analogy: a kitchen assembly line. FROM grabs the ingredients, WHERE throws away what you don't need, EVAL chops new pieces, STATS weighs and groups, SORT arranges the plates, LIMIT serves the first N.

POST _query?format=txt
{
  "query": """
    FROM kibana_sample_data_logs
    | WHERE @timestamp >= NOW() - 7 days AND response.keyword != "200"
    | EVAL kb = bytes / 1024.0, day = DATE_TRUNC(1 day, @timestamp)
    | STATS errors = COUNT(*), avg_kb = AVG(kb), hosts = COUNT_DISTINCT(host.keyword) BY day, response.keyword
    | SORT day DESC, errors DESC
    | LIMIT 20
  """
}

format can be json (default), txt, csv, tsv, yaml and more. For long queries use POST _query/async and poll with GET _query/async/<id>. In Kibana, use Discover → ES|QL mode.

Commands you must know:

Type	Command	Example
Source	FROM	FROM logs-*, remote:logs-* METADATA _index, _id, _score
Source	ROW, SHOW INFO	ROW a = 1, b = "x"
Source	TS (time series, newer)	TS metrics-* | STATS …
Filter	WHERE	WHERE status IN ("500","503") AND bytes > 1000
Compute	EVAL	EVAL mb = bytes / 1048576
Aggregate	STATS … BY	STATS c = COUNT(*), p95 = PERCENTILE(latency, 95) BY service
Shape	KEEP, DROP, RENAME	KEEP @timestamp, host, message
Order	SORT, LIMIT	SORT c DESC NULLS LAST | LIMIT 10
Parse	DISSECT, GROK	DISSECT message "%{ip} - %{user} [%{ts}]"
Enrich	ENRICH	ENRICH users-policy ON email WITH dept
Join	LOOKUP JOIN	LOOKUP JOIN hosts-lookup ON host.name (the right-hand index must use index.mode: lookup)
Arrays	MV_EXPAND	Turn a multi-value column into rows
Analytics	CHANGE_POINT, SAMPLE, FORK, RERANK, COMPLETION	Newer commands; know they exist

Functions to know: aggregations COUNT, COUNT_DISTINCT, SUM, AVG, MIN, MAX, MEDIAN, PERCENTILE, VALUES, TOP; grouping BUCKET(@timestamp, 1 hour); date DATE_TRUNC, DATE_EXTRACT, DATE_DIFF, DATE_FORMAT, NOW(); string CONCAT, TO_LOWER, SUBSTRING, LENGTH, SPLIT, STARTS_WITH, LIKE, RLIKE; conditionals CASE(cond, a, cond2, b, default), COALESCE; conversion TO_INTEGER, TO_DATETIME, TO_IP, :: casts; multi-value MV_COUNT, MV_AVG; full-text MATCH(field, "text"), the : operator (message:"timeout"), QSTR("…"), KQL("…"), plus METADATA _score for relevance.

ES|QL facts:

• Default LIMIT is 1000 rows, with a maximum of 10,000 by default. Always add an explicit LIMIT.
• Case matters for values, not for command names.
• Use parameters to avoid injection: "query": "FROM x | WHERE a == ?v", "params": [{"v": 5}].
• Cross-cluster: FROM cluster_a:logs-*, logs-* (requires remote clusters, see 5.2).
• ES|QL doesn't return _source documents like _search; it returns columns and rows.
• STATS without BY returns one row. Columns not in BY or not aggregated are dropped.

Query DSL → ES|QL translation practice:

Query DSL	ES|QL
bool.filter.term status: published	WHERE status == "published"
range price gte 10 lt 100	WHERE price >= 10 AND price < 100
terms aggregation on category	STATS c = COUNT(*) BY category | SORT c DESC | LIMIT 10
date_histogram 1d + sum	STATS s = SUM(x) BY d = BUCKET(@timestamp, 1 day)
match body: "timeout"	WHERE MATCH(body, "timeout") or WHERE body:"timeout"

📖 ES|QL overview · ES|QL reference · ES|QL source commands · ES|QL processing commands · ES|QL functions and operators · LOOKUP JOIN · ES|QL REST API · ES|QL for search · Using ES|QL in Discover

4.5 Implement semantic search

Semantic search matches meaning rather than exact words. Text is turned into embeddings (vectors) by an ML model, and queries are compared to documents in that vector space.

Analogy: a map of meaning. Each passage becomes a pin on a huge map where similar ideas sit close together. "How do I fix a flat tyre?" lands next to "puncture repair". Keyword search would miss that, because none of the words overlap.

Two kinds of embeddings:

	Sparse (e.g. ELSER, Elastic's learned sparse encoder)	Dense (e.g. E5, Jina, OpenAI, Cohere)
Representation	Weighted tokens (expanded vocabulary)	Fixed-length float vector
Field type	sparse_vector	dense_vector
Query	sparse_vector query	knn (HNSW approximate nearest neighbour)
Strength	Great out-of-domain English, explainable	Multilingual, multimodal, many models

The easy path: semantic_text (recommended and most exam-friendly)

semantic_text hides all the plumbing: chunking, calling the inference endpoint at ingest time, storing embeddings, and embedding the query at search time.

Step 1: an inference endpoint. There's a preconfigured default ELSER endpoint, so this step is optional. To create your own:

PUT _inference/sparse_embedding/my-elser
{
  "service": "elasticsearch",
  "service_settings": {
    "model_id": ".elser_model_2",
    "adaptive_allocations": { "enabled": true, "min_number_of_allocations": 0, "max_number_of_allocations": 4 },
    "num_threads": 1
  }
}

// Dense example with a built-in multilingual model
PUT _inference/text_embedding/my-e5
{ "service": "elasticsearch",
  "service_settings": { "model_id": ".multilingual-e5-small", "num_allocations": 1, "num_threads": 1 } }

GET _inference/_all
POST _inference/sparse_embedding/my-elser { "input": "test sentence" }

The cluster needs an ML node (or Elastic Inference Service on Cloud) and a license that allows ML models, so use the trial on your lab.

Step 2: a mapping with semantic_text:

PUT docs-semantic
{
  "mappings": {
    "properties": {
      "title":   { "type": "text" },
      "content": { "type": "semantic_text", "inference_id": "my-elser" },
      "body":    { "type": "text", "copy_to": "content" }
    }
  }
}

Omit inference_id to use the default endpoint. copy_to into a semantic_text field lets you keep a normal text field for BM25 alongside.

Step 3: index documents normally. Embeddings are generated automatically. Bulk-loading many documents is slow on small ML nodes, so be patient or use a small batch.

Step 4: query.

// Simplest: the match query works on semantic_text fields
GET docs-semantic/_search
{ "query": { "match": { "content": "how do I keep my cluster available when a zone fails?" } } }

// Dedicated semantic query
GET docs-semantic/_search
{ "query": { "semantic": { "field": "content", "query": "zone failure resilience" } },
  "highlight": { "fields": { "content": { "type": "semantic", "number_of_fragments": 2 } } } }

Hybrid search: BM25 + semantic, fused with RRF

GET docs-semantic/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        { "standard": { "query": { "match": { "title": "zone failure" } } } },
        { "standard": { "query": { "semantic": { "field": "content", "query": "zone failure" } } } }
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  }
}

Retrievers are the composable search abstraction: standard, knn, rrf (Reciprocal Rank Fusion), linear (weighted score combination), text_similarity_reranker (semantic reranking), rule.

The manual path (know it exists)

• dense_vector mapping (dims, similarity: cosine | dot_product | l2_norm | max_inner_product, index_options such as int8_hnsw/bbq_hnsw quantisation) + knn search (k, num_candidates, filter, query_vector or query_vector_builder).
• An inference processor in an ingest pipeline to generate embeddings, or bring your own vectors.
• sparse_vector field + sparse_vector query with inference_id.

GET my-vectors/_search
{
  "knn": {
    "field": "embedding",
    "query_vector_builder": { "text_embedding": { "model_id": "my-e5", "model_text": "zone failure" } },
    "k": 10, "num_candidates": 100,
    "filter": { "term": { "lang": "en" } }
  }
}

📖 Semantic search overview · Semantic search with semantic_text · semantic_text field · semantic query · Semantic search with the inference API · Inference API: default endpoints · ELSER · Hybrid search with semantic_text · Retrievers · kNN search · Dense vector · Sparse vector · Semantic reranking

4.6 Write and execute a query that searches across multiple clusters

Cross-cluster search (CCS) runs one request from the local cluster against indices on one or more remote clusters. Setting up the remote connection is covered in 5.2; here is the querying side.

Analogy: interlibrary loan. You ask your local branch; it forwards your request to the branches in Lyon and Berlin and hands you one merged answer.

// Syntax: <remote_alias>:<index_pattern>
GET cluster_b:logs-*/_search
{ "query": { "match": { "message": "timeout" } } }

// Local + several remotes in one request
GET logs-*,cluster_b:logs-*,cluster_c:logs-*/_search
{
  "size": 20,
  "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now-1h" } } } ] } },
  "aggs": { "by_cluster": { "terms": { "field": "_index" } } }
}

// All remotes: wildcard on the alias; exclude with a minus sign
GET *:logs-*,-cluster_c:*/_search

// ES|QL across clusters
POST _query?format=txt
{ "query": "FROM logs-*, cluster_b:logs-* METADATA _index | STATS c = COUNT(*) BY _index | LIMIT 50" }

// Async CCS for long searches
POST cluster_b:logs-*/_async_search?wait_for_completion_timeout=2s
{ "query": { "match_all": {} } }

// Check reachability and matching indices before searching
GET _resolve/cluster/cluster_b:logs-*,cluster_c:logs-*

What to know:

• Hits show the origin in _index as cluster_b:logs-2026…. The response's _clusters section reports total, successful, skipped, partial and failed.
• skip_unavailable (per remote cluster setting, default true): an unreachable remote is skipped instead of failing the whole search.
• ccs_minimize_roundtrips (default true): each remote reduces its own results, which means fewer network round trips.
• Security: the user needs read privileges on the remote indices. With the API-key model, that's the cross-cluster API key's access.search plus the local user's role (remote_indices). With the certificate model, it's a role of the same name on the remote cluster.
• Kibana data views can use cluster_b:logs-* patterns too.
• CCS works across versions within the compatibility window (see the CCS version compatibility table in the docs).

📖 Cross-cluster search · Resolve cluster before CCS · ES|QL cross-cluster · Async search

4.7 Exercises: Searching Data

Use kibana_sample_data_ecommerce, kibana_sample_data_flights and kibana_sample_data_logs. Check field names and types first with GET <index>/_mapping, since some sample fields are text + .keyword and others are plain keyword.

1. Flights: find flights whose OriginCityName or DestCityName contains "New York" (both fields, boost the destination ×2), that are not cancelled, with FlightDelayMin between 30 and 120 and a timestamp in the last 30 days. Return only FlightNum, Carrier, FlightDelayMin, sorted by delay descending.
2. eCommerce: find orders whose products.product_name contains the exact phrase "leather jacket" (slop 1). Highlight the matches.
3. eCommerce: revenue per week (taxful_total_price) for the last 8 weeks, with the week-over-week difference and a running total; keep only weeks above 5,000.
4. eCommerce: the top 5 manufacturer.keyword values by number of orders, each with average basket value and the 3 most frequent category.keyword values.
5. Logs, in ES|QL: per host, the count of 4xx and 5xx responses in the last 24 hours, the 95th-percentile bytes, sorted by error count, top 10.
6. Create a semantic_text index faq, index 5 short Q&A documents, and run a hybrid RRF query combining BM25 on question with semantic search on answer.
7. Across clusters: count documents per index on lab and remote for logs-* in one request, and make the search succeed even if remote is down.

Solution notes

1. bool.should with match_phrase on both city fields (boost: 2 on Dest) and minimum_should_match: 1. filter: term Cancelled: false, range FlightDelayMin gte 30 lte 120, range timestamp gte now-30d/d. Add _source and sort.
2. match_phrase with slop: 1 on products.product_name, plus highlight.fields.
3. date_histogram with calendar_interval: week and extended_bounds, then sum → derivative → cumulative_sum → bucket_selector (params.r > 5000). The query uses range order_date gte now-8w/w.
4. terms manufacturer.keyword size 5 → avg taxful_total_price + terms category.keyword size 3.
5. FROM kibana_sample_data_logs | WHERE @timestamp > NOW() - 24 hours | EVAL code = TO_INTEGER(response.keyword) | WHERE code >= 400 | STATS errors = COUNT(*), p95 = PERCENTILE(bytes, 95) BY host.keyword | SORT errors DESC | LIMIT 10
6. Mapping answer: semantic_text, question: text. Use an rrf retriever with a match on question and a semantic query on answer.
7. GET logs-*,remote:logs-*/_search {"size":0,"aggs":{"i":{"terms":{"field":"_index"}}}}, with cluster.remote.remote.skip_unavailable: true (the default).

---

5. Cluster Management

5.1 Monitor and maintain cluster health

Analogy: a hospital ward. Green means every patient (shard) has a bed and a backup nurse (replica). Yellow means every patient has a bed, but some backup nurses are missing: no data lost, but less safe. Red means at least one patient has no bed at all: a primary shard is unassigned and its data is unavailable. The allocation explain API is the doctor's chart that says why.

The monitoring toolkit

GET _cluster/health                                  // status, nodes, unassigned_shards, relocating…
GET _cluster/health/orders?level=shards              // per index or shard
GET _cluster/health?wait_for_status=green&timeout=30s
GET _health_report                                   // 9.x health indicators with diagnoses & remediation
GET _health_report/shards_availability

GET _cat/health?v
GET _cat/nodes?v&h=name,node.role,master,heap.percent,ram.percent,cpu,load_1m,disk.used_percent
GET _cat/indices?v&health=yellow&s=index
GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node&s=state
GET _cat/allocation?v                               // disk per node, shard counts
GET _cat/recovery?v&active_only=true
GET _cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected
GET _cat/pending_tasks?v
GET _nodes/stats/jvm,fs,indices,breaker
GET _cluster/stats?human
GET _tasks?actions=*reindex*&detailed
GET _cluster/allocation/explain                     // first unassigned shard, or specify one

📖 Cluster health API · Health report API · Cluster allocation explain API · cat APIs · Troubleshoot

Diagnosing yellow and red: the decision tree

1. GET _cluster/health: which status, and how many unassigned_shards?
2. GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED (in Console, use &s=state).
3. GET _cluster/allocation/explain with the index, shard and primary flag. Read allocate_explanation and each node's deciders.
4. Fix according to the cause:

Symptom or explanation	Cause	Fix
Replica unassigned, "a copy of this shard is already allocated to this node"	Single-node cluster or too few nodes	Add nodes or number_of_replicas: 0 (only if the task allows it)
awareness decider says NO	Forced awareness with a zone missing	Expected. Restore the zone, or relax the awareness values.
filter decider says NO	require/include/exclude rule matches no node	Fix or clear the index.routing.allocation.* settings
disk_threshold decider says NO	Node above the low (85%) or high (90%) watermark	Free disk, delete old indices, add nodes, or (temporarily) adjust cluster.routing.allocation.disk.watermark.*
Index is read-only, writes fail with FORBIDDEN/12/index read-only / allow delete	Flood stage (95%) reached	Free disk. The block is removed automatically once usage drops below the high watermark, or clear it: PUT idx/_settings {"index.blocks.read_only_allow_delete": null}
max_retry decider: "shard has exceeded the maximum number of retries"	Repeated failures (e.g. an earlier disk or mapping problem)	Fix the cause, then POST _cluster/reroute?retry_failed=true
data_tier decider says NO	_tier_preference asks for a tier with no nodes	Add a node with that role or change _tier_preference
enable decider says NO	cluster.routing.allocation.enable set to none or primaries	Reset it to null (all)
Primary unassigned, "no valid shard copy" (red)	All copies lost (node or disk gone)	Bring the node back, restore from snapshot, or, as a last resort, allocate_stale_primary / allocate_empty_primary via _cluster/reroute with accept_data_loss: true
total_shards_per_node limit	Too restrictive a per-node cap	Raise or remove index.routing.allocation.total_shards_per_node
NODE_LEFT + delayed	Node left recently; waiting delayed_timeout (1m)	Wait, or lower the timeout

POST _cluster/reroute?retry_failed=true

PUT _cluster/settings
{ "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } }

// Manual moves / last-resort recovery
POST _cluster/reroute
{ "commands": [
    { "move": { "index": "orders", "shard": 0, "from_node": "es01", "to_node": "es02" } },
    { "allocate_replica": { "index": "orders", "shard": 1, "node": "es03" } } ] }

Cluster settings: persistent survives a full restart; transient is deprecated. Set a setting to null to reset it to the default. GET _cluster/settings?include_defaults=true&flat_settings=true shows everything.

📖 Red or yellow cluster health · Diagnose unassigned shards · Fix watermark errors · Cluster-level shard allocation and routing settings · Cluster reroute API

Performance signals to watch

Signal	Where	Worry when…
JVM heap	_cat/nodes heap.percent, _nodes/stats/jvm	Consistently > 85%, long old-GC pauses
Circuit breakers	_nodes/stats/breaker	tripped counts increasing (requests rejected with 429s)
Thread-pool rejections	_cat/thread_pool	rejected > 0 on write or search → back-pressure
Disk	_cat/allocation	Approaching 85/90/95%
Slow queries or indexing	Slow logs: index.search.slowlog.threshold.query.warn: 5s, index.indexing.slowlog.threshold.index.warn: 10s	Repeated slow entries
Hot threads	GET _nodes/hot_threads	CPU-bound nodes
Pending tasks	_cat/pending_tasks	Master overloaded (too many shards or mapping updates)

Monitoring solutions:

• Stack Monitoring: Elastic Agent or Metricbeat collects metrics into a (preferably separate) monitoring cluster; view them in Kibana → Stack Monitoring. Legacy internal collection is deprecated.
• AutoOps: Elastic's hosted, automated diagnosis with recommendations. Available for ECH, and via Cloud Connect for self-managed clusters.
• Kibana alerting rules on cluster health, CPU, disk and so on.

📖 Monitoring · Stack monitoring · Collect monitoring data with Elastic Agent · AutoOps · AutoOps vs Stack Monitoring · Slow logs · Update logging levels

Maintaining the cluster: snapshots, SLM and routine operations

Analogy: snapshots are incremental photo backups of your phone. The first one copies everything; later ones only copy new photos (new segments), yet every snapshot can restore the full album on its own.

// 1) Register a repository (shared FS: every node needs path.repo pointing to the same mount)
PUT _snapshot/my-repo
{ "type": "fs", "settings": { "location": "/usr/share/elasticsearch/snapshots/my-repo", "compress": true } }
POST _snapshot/my-repo/_verify

// 2) Take a snapshot
PUT _snapshot/my-repo/snap-2026.09.28?wait_for_completion=true
{ "indices": "orders,logs-app-*", "include_global_state": false,
  "metadata": { "taken_by": "exam", "reason": "before migration" } }

GET _snapshot/my-repo/_all        GET _snapshot/my-repo/snap-2026.09.28/_status
GET _cat/snapshots/my-repo?v

// 3) Restore (the target index must not exist and be open; rename or close/delete first)
POST _snapshot/my-repo/snap-2026.09.28/_restore
{
  "indices": "orders",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "include_aliases": false,
  "index_settings": { "index.number_of_replicas": 0 },
  "ignore_index_settings": ["index.refresh_interval"]
}

Snapshot lifecycle management (SLM):

PUT _slm/policy/nightly
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "my-repo",
  "config": { "indices": ["*"], "include_global_state": true },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}
POST _slm/policy/nightly/_execute          // run now
GET  _slm/policy/nightly                   // last_success / last_failure
GET  _slm/stats
POST _slm/_execute_retention

The SLM schedule uses Quartz cron (seconds first: sec min hour day month weekday), and the name supports date math.

Other routine operations:

• Searchable snapshots: mount an index straight from a repository. Use POST _snapshot/my-repo/snap/_mount?storage=full_copy (cold) or shared_cache (frozen), or let ILM do it.
• Repository types: fs, url (read-only), s3, gcs, azure, source (source-only).
• Restore a whole cluster or feature states: include_global_state, feature_states (e.g. security, kibana).
• Close and open indices to cut resource use: POST idx/_close.
• Force merge read-only indices: POST idx/_forcemerge?max_num_segments=1.
• Clear caches: POST idx/_cache/clear. Refresh/flush: POST idx/_refresh, POST idx/_flush.
• Rolling restarts and upgrades: disable replica allocation → flush → restart → re-enable (see 1.2).

📖 Snapshot and restore · Manage repositories · Shared file system repository · Create snapshots and SLM · Restore a snapshot · Searchable snapshots · Add and remove nodes

5.2 Perform cross-cluster operations

Cross-cluster operations depend on remote clusters: a named connection from the local cluster (which starts the requests) to a remote cluster (which holds the data). Two features use it:

• Cross-cluster search (CCS): query remote data (queries in 4.6).
• Cross-cluster replication (CCR): continuously copy leader indices on the remote into read-only follower indices on the local cluster. It needs a Platinum/Enterprise license or the trial.

Analogy. CCS is a phone call to another library: you ask and hear the answer, but nothing is copied. CCR is a newspaper subscription: the publisher (leader) keeps sending every new edition, and your copy (follower) is read-only.

Step 1: connect a remote cluster

	API key authentication (recommended)	TLS certificate authentication (deprecated)
Port on the remote	9443 (dedicated remote cluster server)	9300 (transport)
Trust	Local trusts the remote's CA; access is limited by a cross-cluster API key	Mutual TLS; both clusters trust each other's transport CA
Remote privileges	Defined in the API key (access.search, access.replication)	Roles with the same names must exist on the remote

Connection modes: sniff (default; connects directly to seed nodes and discovers gateway nodes) or proxy (a single address, e.g. through a load balancer). ECH, ECE and ECK effectively use proxy mode.

API key setup, self-managed:

# On the REMOTE cluster (elasticsearch.yml): enable the remote cluster server interface
remote_cluster_server.enabled: true
remote_cluster.port: 9443
xpack.security.remote_cluster_server.ssl.enabled: true
xpack.security.remote_cluster_server.ssl.key: certs/remote.key
xpack.security.remote_cluster_server.ssl.certificate: certs/remote.crt

// On the REMOTE: create a cross-cluster API key that limits what the local cluster may do
POST _security/cross_cluster/api_key
{
  "name": "from-lab",
  "access": {
    "search":      [ { "names": ["logs-*", "orders"] } ],
    "replication": [ { "names": ["orders", "metrics-*"] } ]
  }
}

# On EVERY node of the LOCAL cluster: store the encoded key in the keystore, then reload
bin/elasticsearch-keystore add cluster.remote.cluster_b.credentials

POST _nodes/reload_secure_settings

// On the LOCAL cluster (it also needs to trust the remote's CA:
// xpack.security.remote_cluster_client.ssl.certificate_authorities)
PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.cluster_b.seeds": ["remote-es01:9443"],
    "cluster.remote.cluster_b.skip_unavailable": true
  }
}

// Proxy mode alternative
PUT _cluster/settings
{ "persistent": { "cluster.remote.cluster_b.mode": "proxy",
                  "cluster.remote.cluster_b.proxy_address": "lb.example.com:9443" } }

GET _remote/info          // "connected": true, mode, num_nodes_connected, cluster_credentials

The same can be done in Kibana: Stack Management → Remote Clusters. The local nodes that make the connections need the remote_cluster_client role, which is included by default when no roles are set explicitly.

Local user privileges for the API key model: add remote_indices to the user's role on the local cluster:

PUT _security/role/ccs-reader
{
  "cluster": [],
  "indices": [ { "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] } ],
  "remote_indices": [
    { "clusters": ["cluster_b"], "names": ["logs-*"], "privileges": ["read", "read_cross_cluster", "view_index_metadata"] }
  ]
}

For CCR, the local user's role needs remote_indices with cross_cluster_replication and cross_cluster_replication_internal privileges, plus the local manage_ccr cluster privilege. The docs list the exact privileges per model.

📖 Remote clusters · Security models · Connection modes · Self-managed remote clusters · Add remote clusters with API key auth · Add remote clusters with TLS certificate auth · Migrate from certificate to API key · Create cross-cluster API key

Step 2a: cross-cluster search

See 4.6 for the queries. Per-remote options: skip_unavailable (default true); cluster.remote.<alias>.transport.compress. Check connectivity with GET _remote/info and GET _resolve/cluster/<alias>:*.

Step 2b: cross-cluster replication

Prerequisites: licence (Platinum/Enterprise or trial) on both clusters; the remote connection from the follower (local) cluster to the leader; soft deletes enabled on the leader index (the default since 7.0); compatible versions (the follower may be the same or newer).

// Follow one specific leader index (run on the FOLLOWER / local cluster)
PUT orders-copy/_ccr/follow?wait_for_active_shards=1
{
  "remote_cluster": "cluster_b",
  "leader_index": "orders",
  "max_read_request_operation_count": 5120,
  "settings": { "index.number_of_replicas": 0 }
}

// Auto-follow: automatically follow NEW leader indices (and data streams) that match patterns
PUT _ccr/auto_follow/logs-follower
{
  "remote_cluster": "cluster_b",
  "leader_index_patterns": ["logs-*", "metrics-*"],
  "leader_index_exclusion_patterns": ["logs-debug-*"],
  "follow_index_pattern": "{{leader_index}}-dr"
}

Managing replication:

GET orders-copy/_ccr/stats           GET orders-copy/_ccr/info           GET _ccr/stats
POST orders-copy/_ccr/pause_follow
POST orders-copy/_ccr/resume_follow

// Turn a follower into a normal writable index (e.g. failover), in order:
POST orders-copy/_ccr/pause_follow
POST orders-copy/_close
POST orders-copy/_ccr/unfollow
POST orders-copy/_open

GET _ccr/auto_follow            POST _ccr/auto_follow/logs-follower/pause
DELETE _ccr/auto_follow/logs-follower

CCR facts:

• Followers are read-only. Writing to them fails.
• Auto-follow only picks up leader indices created after the pattern exists. Follow existing ones explicitly.
• Replication is pull-based: the follower fetches operations from the leader's shard history (soft deletes, retained by shard history retention leases).
• Mapping and settings changes on the leader are replicated automatically. Some follower settings can't be changed.
• Data streams are replicated through auto-follow patterns, which replicate the backing indices and create a matching follower data stream.
• Uses: disaster recovery (active–passive), bringing data closer to users (geo read replicas), and centralised reporting.
• Bi-directional replication is possible with two auto-follow setups and distinct write targets (e.g. aliases or data streams per region).
• The ILM action unfollow converts followers automatically as they age.

📖 Cross-cluster replication · Set up CCR · CCR prerequisites · Configure privileges for CCR · Create a follower index · Create an auto-follow pattern · Manage CCR · Uni-directional disaster recovery · Bi-directional disaster recovery

Other cross-cluster operations to recognise:

• Reindex from remote: _reindex with source.remote (HTTP, port 9200; needs reindex.remote.whitelist). This one doesn't use remote-cluster connections.
• Restore a snapshot into another cluster: register the same repository (read-only, "readonly": true, on the second cluster) and restore.
• Cross-cluster ES|QL and EQL, and Kibana data views with cluster:index patterns.

5.3 Exercises: Cluster Management

1. Your cluster is yellow. Find which shards are unassigned and why, then fix it without adding nodes and without deleting data.
2. Index hot-data is red after node es03 died permanently. A snapshot from last night exists. Recover it as hot-data.
3. Register an fs repository backups, snapshot only indices matching orders* without global state, and create an SLM policy running every day at 02:15 that keeps between 3 and 10 snapshots for 14 days.
4. A node shows disk.used_percent: 96, and writes to logs-x fail with index read-only / allow delete. Explain what happened and fix it.
5. Connect cluster lab to cluster remote as dr using API-key authentication, permitting search on logs-* and replication of orders.
6. On lab, make orders-dr follow remote's orders. Then fail over: turn orders-dr into a writable index.
7. Configure auto-follow so every new logs-* index on remote is replicated to lab with the suffix -copy.

Solution notes

1. _cat/shards → _cluster/allocation/explain. Typical fixes: remove a bad index.routing.allocation.require.*; reset cluster.routing.allocation.enable; _cluster/reroute?retry_failed=true; or (on a 1-node lab) set number_of_replicas: 0 if the task says data only needs one copy.
2. DELETE hot-data (or close it), then POST _snapshot/<repo>/<snap>/_restore {"indices":"hot-data"}. Verify with _cat/indices and _count.
3. PUT _snapshot/backups {"type":"fs","settings":{"location":"…"}} (the path must be under path.repo). Snapshot with "indices":"orders*","include_global_state":false. SLM: "schedule":"0 15 2 * * ?", retention: {expire_after:"14d", min_count:3, max_count:10}.
4. The flood-stage watermark (95%) applied index.blocks.read_only_allow_delete. Free disk (delete, move or shrink data, or add capacity). The block auto-releases below the high watermark, or remove it explicitly with null.
5. On remote: enable the remote cluster server and create a cross-cluster API key with search: logs-*, replication: orders. On lab: add the key to the keystore as cluster.remote.dr.credentials on every node, _nodes/reload_secure_settings, set cluster.remote.dr.seeds, then check GET _remote/info.
6. PUT orders-dr/_ccr/follow {"remote_cluster":"dr","leader_index":"orders"}. Failover: pause_follow → _close → _ccr/unfollow → _open.
7. PUT _ccr/auto_follow/logs {"remote_cluster":"dr","leader_index_patterns":["logs-*"],"follow_index_pattern":"{{leader_index}}-copy"}. Only new leader indices are picked up.

---

6. Security & Access Control

Analogy for the whole domain: an office building. Authentication is the badge check at the door: who are you? (native users, API keys, LDAP/AD, SAML, OIDC, PKI, service accounts). Authorization / RBAC is which doors your badge opens: roles bundle privileges on the cluster, on indices (even specific documents and fields inside a room), and in Kibana features and spaces. TLS is the armoured corridor between buildings (nodes) and to visitors (clients). Audit logs are the CCTV.

6.1 Implement role-based access control (RBAC)

The model

user ──(has)──► roles ──(grant)──► privileges on resources
   ▲                                   ├─ cluster:   monitor, manage, manage_ilm, manage_index_templates, manage_security…
   │                                   ├─ indices:   read, write, create_doc, manage, view_index_metadata, all…
   │                                   │             + field-level security (FLS) + document-level security (DLS)
   │                                   ├─ applications: Kibana features & spaces
   │                                   └─ remote_indices / remote_cluster (for CCS/CCR)
role mapping (LDAP groups, SAML attributes, PKI DN…) ──► roles

Built-in roles worth knowing: superuser, kibana_admin, kibana_system (for Kibana's own service connection), viewer (read-only, including Kibana), editor, monitoring_user, remote_monitoring_agent, ingest_admin, snapshot_user, machine_learning_admin/_user, transform_admin, beats_system, logstash_system. Built-in users: elastic (superuser), kibana_system, logstash_system, beats_system, apm_system, remote_monitoring_user.

Privileges, the most-used ones:

Cluster privilege	Allows
monitor	Read-only cluster info (health, stats, nodes)
manage	Everything cluster-level except security
manage_index_templates, manage_ilm, manage_slm, manage_pipeline, manage_ingest_pipelines, manage_enrich, manage_ccr, manage_transform, manage_ml	The named feature
manage_security	Users, roles, API keys (effectively admin)
manage_own_api_key	Create or invalidate your own API keys
read_ccr, cross_cluster_search, cross_cluster_replication	Cross-cluster operations

Index privilege	Allows
read	Search, get, count
view_index_metadata	Get mappings, settings, aliases (Kibana needs this for data views)
write	Index, update, delete (includes index, create, delete)
create_doc	Only create new documents (ideal for shippers, data streams)
create_index, delete_index, manage (settings, aliases, refresh…), manage_ilm, auto_configure, maintenance	Index administration
monitor	Index stats, recovery, segments
all	Everything on those indices

Create roles and users

PUT _security/role/sales_analyst
{
  "description": "Read EU sales, hide PII, use Discover & Dashboards in the 'sales' space",
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["sales-*", "orders"],
      "privileges": ["read", "view_index_metadata"],
      "field_security": {
        "grant": ["*"],
        "except": ["customer.email", "customer.phone", "card.*"]
      },
      "query": { "term": { "region": "EU" } }
    },
    {
      "names": ["sales-reports"],
      "privileges": ["read", "write", "create_index"]
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["feature_discover.read", "feature_dashboard.all"],
      "resources": ["space:sales"]
    }
  ],
  "metadata": { "owner": "exam" }
}

POST _security/user/jane
{
  "password": "Str0ng-Passw0rd!",
  "roles": ["sales_analyst"],
  "full_name": "Jane Doe",
  "email": "jane@example.com",
  "metadata": { "team": "sales" }
}

POST _security/user/jane/_password    { "password": "An0ther-Str0ng-One!" }
POST _security/user/jane/_disable     POST _security/user/jane/_enable
GET  _security/user/jane              GET _security/role/sales_analyst
DELETE _security/role/old_role

Field-level security (FLS) uses field_security.grant / except. Wildcards are allowed. Metadata fields such as _id are always visible.

Document-level security (DLS) uses a query, which can be a templated query based on the user:

"query": { "template": { "source": { "term": { "owner": "{{_user.username}}" } } } }

When a user has several roles, the permissions are the union of all of them. A role without DLS on an index grants access to all documents of that index, even if another role restricts it. That's a classic exam trap.

Kibana privileges: application: "kibana-.kibana". Use "privileges": ["all"] or ["read"] for base access, or per-feature ones (feature_discover.read, feature_dashboard.all, feature_visualize.all…) for specific features. resources holds the spaces: space:<id>, or * for all. It's easiest to create these in Kibana → Stack Management → Roles and then check the result with GET _security/role/<name>.

Test what you granted

GET _security/_authenticate                          // who am I, which roles, which realm

GET _security/user/_has_privileges
{ "cluster": ["monitor"], "index": [ { "names": ["sales-2026"], "privileges": ["read", "write"] } ] }

// Run a request AS another user (requires the run_as privilege) to verify their view
GET sales-*/_search
# Header:  es-security-runas-user: jane

The fastest real-world check is to log into Kibana as the user, or call the API with -u jane:password, and try the allowed and the forbidden actions.

API keys

POST _security/api_key
{
  "name": "ingest-shipper",
  "expiration": "30d",
  "role_descriptors": {
    "writer": {
      "cluster": ["monitor"],
      "indices": [ { "names": ["logs-app-*"], "privileges": ["create_doc", "auto_configure"] } ]
    }
  },
  "metadata": { "app": "shipper" }
}
// → returns id, api_key, encoded   → header:  Authorization: ApiKey <encoded>

GET _security/api_key?name=ingest-shipper
DELETE _security/api_key  { "name": "ingest-shipper" }             // invalidate
POST _security/api_key/_query { "query": { "term": { "name": "ingest-shipper" } } }

An API key's privileges are the intersection of the role_descriptors and the creator's privileges at creation time. Without role_descriptors, it gets a snapshot of the owner's permissions.

Role mappings (external realms)

PUT _security/role_mapping/ldap-admins
{
  "enabled": true,
  "roles": ["superuser"],
  "rules": { "field": { "groups": "cn=admins,ou=groups,dc=example,dc=com" } }
}

PUT _security/role_mapping/saml-readers
{
  "enabled": true,
  "roles": ["viewer"],
  "rules": { "all": [ { "field": { "realm.name": "saml1" } },
                      { "field": { "groups": "readers" } } ] }
}

Rules support field, any, all and except. Fields: username, dn, groups, realm.name, metadata.*. Use role_templates for dynamic role names.

Realms, in order of evaluation (order): reserved → file → native (both on by default) → ldap, active_directory, pki, saml, oidc, kerberos, jwt. SAML and OIDC work through Kibana. Service accounts (e.g. elastic/kibana, elastic/fleet-server) use service tokens instead of passwords.

📖 Users and roles · User roles · Defining roles · Role structure · Built-in roles · Security privileges · Kibana privileges · Kibana role management · Document and field level security · Privileges for data streams and aliases · Role mapping · Native realm · Built-in users · Service accounts · Elasticsearch API keys · Submit requests on behalf of other users

6.2 Secure a cluster for production

Analogy: moving from a house party to a bank. At a party the door is open and everyone knows each other (development mode). A bank has locked doors (authentication), badges per room (RBAC), armoured transport between branches (transport TLS), bulletproof glass at the counter (HTTP TLS), a vault for secrets (keystore), cameras (audit logs), and a building inspection before opening (bootstrap checks).

Layer 1: security on, with TLS everywhere

Since 8.0, a fresh self-managed install auto-configures security: it generates certificates, enables TLS on the transport and HTTP layers, prints the elastic password, and gives you an enrollment token for Kibana and additional nodes. Know how to do it manually too:

# 1. CA + transport (node-to-node) certificates
bin/elasticsearch-certutil ca --out elastic-stack-ca.p12
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --out elastic-certificates.p12
# 2. HTTP certificates (interactive wizard: CSR or CA-signed, hostnames, IPs)
bin/elasticsearch-certutil http
# 3. Store keystore passwords
bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

# elasticsearch.yml
xpack.security.enabled: true
xpack.security.enrollment.enabled: true

xpack.security.transport.ssl.enabled: true               # REQUIRED for multi-node clusters with security
xpack.security.transport.ssl.verification_mode: certificate   # or full (also checks hostnames)
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true                    # HTTPS for REST clients & Kibana
xpack.security.http.ssl.keystore.path: certs/http.p12

# Passwords & tokens
bin/elasticsearch-reset-password -u elastic                 # or -i for interactive
bin/elasticsearch-reset-password -u kibana_system
bin/elasticsearch-create-enrollment-token -s kibana         # or -s node
bin/elasticsearch-service-tokens create elastic/kibana my-token

# kibana.yml (secured connection to Elasticsearch)
elasticsearch.hosts: ["https://es01:9200"]
elasticsearch.serviceAccountToken: "AAEAAWVs..."            # preferred over username/password
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/http_ca.crt"]
server.ssl.enabled: true                                     # HTTPS for browsers → Kibana
server.ssl.certificate: /etc/kibana/certs/kibana.crt
server.ssl.key: /etc/kibana/certs/kibana.key
xpack.encryptedSavedObjects.encryptionKey: "<32+ chars>"     # needed for alerting and saved-object secrets
xpack.security.encryptionKey: "<32+ chars>"
xpack.reporting.encryptionKey: "<32+ chars>"

Rules:

• With security enabled, a multi-node cluster must have transport TLS. Otherwise nodes refuse to form a cluster once they bind to a non-loopback address (production mode).
• Secrets belong in the keystore (elasticsearch-keystore), never in plain elasticsearch.yml. Reloadable secure settings can be refreshed with POST _nodes/reload_secure_settings.
• Don't use the elastic superuser for daily work or for Kibana. Create personal admin users, and use service accounts or API keys for applications.
• Anonymous access stays disabled unless explicitly required.

Layer 2: production-mode node configuration and bootstrap checks

As soon as network.host binds to a non-loopback address, Elasticsearch runs in production mode and enforces bootstrap checks. A failing check stops the node from starting.

Check / setting	Requirement
Heap size	-Xms = -Xmx (set in jvm.options.d/*.options or ES_JAVA_OPTS). By default Elasticsearch sizes the heap automatically; manually, ≤ 50% of RAM and below the compressed-oops threshold (~31 GB).
File descriptors	≥ 65,535 (ulimit -n)
Max threads	≥ 4,096
Virtual memory	vm.max_map_count ≥ 262,144 (check the docs for the value recommended for your version)
Memory lock	If bootstrap.memory_lock: true, the lock must succeed. Also disable swap.
Discovery	discovery.seed_hosts / cluster.initial_master_nodes (first bootstrap only) or equivalent must be configured
Others	Max file size, max virtual memory, JVM checks (client JVM, serial GC), system call filter, OnError/OnOutOfMemoryError

cluster.name: prod-search
node.name: es-prod-01
node.roles: [ master, data_hot, ingest ]
network.host: 10.0.1.11            # _site_, _local_, specific IP…
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["10.0.1.11", "10.0.1.12", "10.0.1.13"]
cluster.initial_master_nodes: ["es-prod-01", "es-prod-02", "es-prod-03"]   # remove after first start
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
action.destructive_requires_name: true     # no DELETE on wildcards / _all (the default is true in 8+)

Layer 3: least privilege, network restrictions and auditing

Least privilege checklist:

• Personal admin accounts instead of the shared elastic account.
• Shippers (Agent, Beats, Logstash) get API keys with create_doc + auto_configure on their own data streams only.
• Applications get dedicated users or API keys limited to their indices.
• Analysts get read + view_index_metadata, with FLS/DLS where needed, and Kibana privileges limited to specific spaces.
• action.destructive_requires_name: true prevents DELETE *.

IP filtering (self-managed; on Cloud use traffic filters or network security):

xpack.security.transport.filter.allow: ["10.0.1.0/24"]
xpack.security.transport.filter.deny: _all
xpack.security.http.filter.allow: ["10.0.0.0/16"]
xpack.security.http.filter.deny: _all

Audit logging (needs a Platinum/Enterprise licence or the trial):

xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.include: ["access_denied", "authentication_failed", "connection_denied", "tampered_request", "run_as_denied", "security_config_change"]
xpack.security.audit.logfile.events.emit_request_body: false

The output goes to <cluster>_audit.json in the logs directory. Ignore policies (xpack.security.audit.logfile.events.ignore_filters.*) reduce noise. Kibana has its own audit log (xpack.security.audit.enabled in kibana.yml).

Other production hardening:

• Encrypt data at rest via the file system or disk (Elasticsearch doesn't encrypt shards itself). On Cloud, use customer-managed keys.
• Keep Elasticsearch off the public internet. Put it behind a firewall, load balancer or VPN, and expose only Kibana over HTTPS if needed.
• Set a session timeout in Kibana (xpack.security.session.idleTimeout, lifespan).
• Rotate certificates before they expire. The docs cover rotating with the same CA and with a different CA.
• Snapshot the security feature state (feature_states: ["security"]) so users and roles can be restored.
• For FIPS environments, use FIPS 140-2 mode with approved algorithms and keystores.

📖 Security overview · Self-managed security setup · Automatic security setup · Minimal security setup · Set up transport TLS · Set up HTTPS · Configure security in Kibana · Manage TLS encryption · Update TLS certificates · Secure settings (keystore) · IP filtering (self-managed) · Audit logging · Enable audit logging · Important system configuration · Bootstrap checks · Important settings · Kibana session management · Tutorial: self-managed secure install

6.3 Exercises: Security

1. Create role logs_reader: read and view metadata on logs-*, no access to the field user.password_hash, only documents where service.environment is production, plus Discover (read) in the Kibana space ops. Create user ana with this role and verify with _has_privileges and by logging in.
2. Create role ingest_writer that can only append documents to data streams matching logs-app-*, and create an API key limited to it that expires in 7 days.
3. User bob has roles A (DLS: team: blue on projects) and B (plain read on projects). Which documents can he see, and why? Fix it so he sees only team: blue.
4. Map LDAP group cn=sre,ou=groups,dc=corp to roles monitoring_user and logs_reader.
5. On a three-node lab started without security, enable security with transport TLS (own CA) and HTTPS, reset the elastic password, connect Kibana with a service account token, and turn on audit logging for authentication failures and denied access.
6. List the bootstrap checks that would fail on a VM with vm.max_map_count=65530, ulimit -n 4096 and swap enabled with bootstrap.memory_lock: true, and fix each one.

Solution notes

1. indices: [{names:["logs-*"], privileges:["read","view_index_metadata"], field_security:{grant:["*"], except:["user.password_hash"]}, query:{term:{"service.environment":"production"}}}], applications: [{application:"kibana-.kibana", privileges:["feature_discover.read"], resources:["space:ops"]}].
2. Role: indices: [{names:["logs-app-*"], privileges:["create_doc","auto_configure"]}]. API key: POST _security/api_key with expiration: "7d" and the same role_descriptors.
3. He sees all documents: role permissions are a union, and B's unrestricted read wins. Fix by removing B from bob, or adding the same DLS query to B.
4. PUT _security/role_mapping/sre {"roles":["monitoring_user","logs_reader"],"enabled":true,"rules":{"field":{"groups":"cn=sre,ou=groups,dc=corp"}}}.
5. certutil ca → certutil cert (copy to all nodes) → certutil http → transport and HTTP SSL settings + keystore passwords → rolling or full restart → elasticsearch-reset-password -u elastic → elasticsearch-service-tokens create elastic/kibana kibana-token → kibana.yml elasticsearch.serviceAccountToken + CA → xpack.security.audit.enabled: true with the events.include list.
6. Max map count → sysctl -w vm.max_map_count=262144 (persist it in /etc/sysctl.d). File descriptors → raise the limit to 65535 (limits.conf or systemd LimitNOFILE). Memory lock → LimitMEMLOCK=infinity (systemd) or ulimit -l unlimited. Disable swap (swapoff -a) as good practice.

---

7. Final checklist and study plan

7.1 A four-week plan

Week	Focus	Deliverable
1	Domains 1–2: lab build, templates, dynamic templates, data streams, ILM, aliases	Every Domain 2 exercise done twice without notes, just the docs
2	Domain 3: mappings, analyzers, field types, reindex/UBQ, pipelines, enrich, Streams	A "migration" drill: v1 → v2 with a new mapping, pipeline and alias swap in < 20 min
3	Domain 4: Query DSL, bool, aggregations, ES|QL, semantic search, CCS	20 queries on the sample data; translate 5 of them to ES|QL
4	Domains 5–6 + mock exams: health, snapshots, SLM, remote clusters, CCR, RBAC, TLS	Two timed mock exams of 10 mixed tasks in 3 hours

7.2 "Can I do this from a blank Console with only the docs?"

• ☐ Explain node roles, tiers, primaries vs replicas, and the write and read paths
• ☐ Configure forced awareness and allocation filtering; explain a yellow cluster
• ☐ Create an index with custom analyzers, multi-fields, nested fields and strict mapping
• ☐ Component + index templates with priority; simulate them
• ☐ Dynamic templates with correct ordering
• ☐ A data stream from a template, with ILM rollover/warm/delete; _ilm/explain
• ☐ Alias swap, filtered alias, write index
• ☐ Choose an ingest method for a scenario
• ☐ Reindex with query, script and pipeline; update-by-query to populate a new multi-field
• ☐ A pipeline with grok/dissect, date, set, conditionals, on_failure, enrich; simulate it
• ☐ Streams: processing, retention and data quality in the UI
• ☐ match, match_phrase, multi_match, term/range, bool with filter/should logic
• ☐ Terms/date_histogram/range buckets with metrics and pipeline aggs
• ☐ ES|QL with WHERE, EVAL, STATS…BY, SORT, LIMIT, DISSECT and LOOKUP JOIN
• ☐ semantic_text + inference endpoint + semantic/match query + RRF hybrid
• ☐ Remote cluster (API key) + CCS + CCR follower + auto-follow + failover
• ☐ Diagnose unassigned shards with allocation explain; fix watermark blocks
• ☐ Snapshot repository, snapshot, restore with rename, SLM policy
• ☐ Roles with FLS/DLS and Kibana privileges, users, API keys, role mappings
• ☐ TLS (transport + HTTP), keystore, bootstrap checks, audit logging

7.3 Further references

• 📖 Elastic Documentation home: the only resource allowed in the exam
• 📖 Elasticsearch REST API reference
• 📖 Kibana API reference
• 📖 Elasticsearch release notes: skim 9.0 → 9.3 for new features and breaking changes
• 🎓 Elastic Training: Elasticsearch Engineer course: the official course aligned to the exam
• 🧪 Elasticsearch Labs: tutorials on semantic search, ES|QL and retrievers
• 💬 Elastic Discuss forum: exam-experience threads
• 📝 Community practice exercises by Guido Lena Cota (written for the 8.x exam, still useful for templates, mappings and search): Store data · Deploy and operate a cluster

Prefect — Components Deep Dive

A study guide to Prefect 3 (Python workflow orchestrator), one component at a time: what it is, the code you will type, when to use it, the traps, and where the official docs live.

How to read this guide. Each component has (1) a mental model, usually an analogy, (2) the code, (3) a when to use / when not to verdict, (4) traps and (5) 📖 doc links. The decision tables for “which way do I run this?” (§5-6) and “when is Prefect the wrong tool?” (§13) are the parts to re-read before choosing an orchestrator. The sibling page RustFS — Components Deep Dive covers an S3-compatible store that Prefect can use for results and code. Version note: written 23 Oct 2026; the docs index lists Prefect 3.9 as the latest OSS line. ⚑ marks details I wrote from prior knowledge rather than re-reading the page this session (decorator parameters, exact import paths, CLI flags). Confirm those against the linked docs and your installed version (prefect version) before copying.

---

0. The map: what Prefect is

Analogy: an airline operations centre. Your flights are ordinary Python functions. The operations centre does not fly the plane; it tracks every flight, reschedules on delay (retries), reroutes when an airport closes (infrastructure), logs everything, and alerts when something goes wrong. Planes (your code) fly on your own aircraft (your infrastructure).

• Workflows are plain Python. A flow is a decorated function; a task is a decorated function called inside it. There is no DAG file to declare: dependencies come from data flow and normal control flow (if, for, try), so a workflow can branch and loop dynamically.
• Orchestration is hybrid. Your code runs on your infrastructure; the Prefect API (self-hosted server or Prefect Cloud) stores state, schedules, events and automations. Task runs are orchestrated client-side and their updates are logged in batches, so task state in the UI is eventually consistent.
• Code is not stored by Prefect. A deployment is server-side metadata (which flow, where its code lives, which infrastructure, schedule, parameters). Deployments reference flows defined in Python files; they do not contain code.
• Prefect 3 changes (old tutorials differ): workers and work pools replace agents; block-based deployments and storage/infrastructure blocks are the legacy path; assets, transactions and background tasks are first-class. The 2.x to 3.0 upgrade and agents-to-workers guides exist under how-to-guides/migrate.

Piece	Where it lives	Job
Flow / task code	Your repo or image	The actual work
Prefect API + database	Cloud or self-hosted server (Postgres)	State, schedules, events, UI
Deployment	API (metadata only)	Run this flow, from here, on that pool, on this schedule
Work pool (+ queues)	API	A channel between deployments and infrastructure
Worker	Your infra (or none for push/managed pools)	Polls the pool and starts runs

📖 Concepts · Quickstart · Upgrade to 3.0 · Migrate from Airflow

---

1. Flows

Analogy: a recipe card. The card lists the steps; any cook with a kitchen can follow it. The kitchen is infrastructure; the order ticket is a flow run.

from prefect import flow, task, get_run_logger

@flow(name='daily-etl', retries=1, retry_delay_seconds=60, timeout_seconds=3600, log_prints=True)
def daily_etl(day: str, full_refresh: bool = False) -> int:
    rows = extract(day)
    return load(rows)

if __name__ == '__main__':
    daily_etl('2026-10-22')          # runs locally, recorded in the API if one is configured

• A flow is callable like a function. Calling it creates a flow run with its own state history. ⚑ Common options: name, retries, retry_delay_seconds, timeout_seconds, log_prints, validate_parameters (type-checks arguments against hints), persist_result, task_runner, on_completion/on_failure/on_cancellation/on_crashed hooks.
• Subflows: a flow may call another flow; it becomes a child run with its own state. Flows and tasks can call each other with no required nesting pattern.
• Flow versus task: a flow is the unit you deploy, schedule and parameterise; a task is a unit you retry, cache and run concurrently. Do not put a deployment boundary inside a task.
• Flow retries rerun the whole flow; task retries rerun one step. Prefer task retries plus caching so a flow retry does not redo finished work.

Traps. Parameters must be serialisable for deployments (they travel as JSON). Top-level code outside @flow runs on every import, including when a worker loads the file. Keep imports light and put heavy setup inside tasks.

📖 Flows · Write and run a workflow · Pass inputs

---

2. Tasks

Analogy: one station on the line. Small, with a clear input and output, and the supervisor logs how long it took and whether it passed.

Tasks are cacheable, retryable units of work with transactional semantics. A task is identified by a task key (a hash of its name and fully qualified function name). The docs recommend small tasks: each should be a logical step or a significant side effect, so the UI narrates your workflow.

from datetime import timedelta
from prefect import task
from prefect.cache_policies import INPUTS, TASK_SOURCE

@task(retries=3, retry_delay_seconds=[10, 60, 300], timeout_seconds=120,
      cache_policy=INPUTS + TASK_SOURCE, cache_expiration=timedelta(hours=1), log_prints=True)
def extract(url: str) -> dict:
    ...

2.1 Three ways to run a task

Call	Blocks?	Returns	Use for
t(x)	Yes	The result	Simple sequential steps
t.submit(x)	No	PrefectFuture, resolve with .result() or .wait() from the same context	Concurrency inside a flow
t.delay(x)	No	Future you need not resolve from the same context	Fire-and-forget background work on separate workers (§10)

• Dependencies are implicit: passing an upstream result or future into a downstream task means it cannot start until the upstream is Completed. Add wait_for=[...] for explicit ordering. t.map(items) fans out over an iterable ⚑.
• Parameter resolution trap (documented): Prefect walks parameters to resolve nested futures and states. If a dataclass or named tuple field changes, Prefect rebuilds it with dataclasses.replace(), which calls __init__ and __post_init__ again; Pydantic models use model_construct() and skip validators. Use quote(obj) to pass a fully resolved object untouched (no future resolution, no dependency tracking), or opaque(future) to resolve a top-level future without traversing its contents (from prefect.utilities.annotations).
• Task run life cycle: Pending → Running → Completed; on error Retrying/AwaitingRetry → Running; retries exhausted → Failed; infrastructure failure → Crashed; a cached result jumps Pending → Completed. Background tasks start as Scheduled.

When to make something a task: it has a side effect, may fail transiently, is worth caching, or should run concurrently. When not to: trivial pure helpers, and anything so fine-grained that per-call overhead and UI noise exceed the value.

📖 Tasks · Retries · Run work concurrently

---

3. Task runners and concurrency control

Analogy: how many cooks you hire and whether they share a kitchen. A task runner decides whether submitted tasks run in threads, processes, or on a cluster. Concurrency limits are the 'only two people at the fryer' rules.

• Task runners control how .submit()/.map() tasks execute: concurrent threads (the default ⚑), parallel processes, or distributed (prefect-dask, prefect-ray integrations). Set per flow with @flow(task_runner=...).
• Choose: I/O-bound work (HTTP, DB, S3) → threads. CPU-bound Python → processes or Dask/Ray. Data too large for one machine → Dask or Ray, or push the work into the warehouse and orchestrate only.
• Global concurrency limits are named limits you acquire in code and share across flows and deployments, for protecting a database or an API; rate limits use the same mechanism with a refill rate. Tag-based limits cap concurrent task runs carrying a tag.

from prefect.concurrency.sync import concurrency, rate_limit      # ⚑ import path

@task
def write_db(rows):
    with concurrency('warehouse-writes', occupy=1):   # blocks if the limit is full
        ...
@task
def call_api(x):
    rate_limit('vendor-api')                           # waits for a slot at the configured rate
    ...

Traps. A task runner only affects tasks started with .submit()/.map(); direct calls are sequential. Threads do not speed up CPU-bound code. Forgetting to create the named global limit means the code has nothing to enforce. Worker-level limits (work pool and queue concurrency, §6) limit flow runs, not tasks.

📖 Task runners · Global concurrency limits · Tag-based limits · Per-worker task concurrency example

---

4. States, results, caching and transactions

Analogy: a parcel tracking page, a saved receipt and an undo button. State says where the run is, results are the saved output, caching says 'I already did this exact job', and transactions say 'do these steps all-or-nothing'.

• States are rich objects (Pending, Running, Completed, Failed, Cancelled, Crashed, Paused, Scheduled, Retrying/AwaitingRetry...). They carry data and messages, drive state-change hooks and are what automations trigger on. Crashed means the infrastructure failed (killed process, out of memory), not that your code raised.
• Results are the data a flow or task returns. Persisting them is off by default for most runs and is what enables caching and resumption. Choose result_storage (local path, or an S3-compatible bucket via an integration block), persist_result=True, and the serialiser. ⚑ Persisted results need storage reachable by every worker that may read them.
• Caching skips a task when a cache key matches. Policies combine with +: INPUTS (hash of arguments), TASK_SOURCE (the task's code), FLOW_PARAMETERS, RUN_ID and others ⚑; plus cache_expiration. Use a custom key_fn when arguments are not hashable or should be partly ignored. A cache hit moves the task straight to Completed.
• Transactions give tasks transactional semantics: group steps so they commit together, and register rollback hooks that run if a later step fails.

from prefect.transactions import transaction      # ⚑

@task
def write_staging(batch): ...
@write_staging.on_rollback
def drop_staging(txn): ...                         # runs if the transaction rolls back

@flow
def load(batch):
    with transaction():
        write_staging(batch)
        swap_into_prod()                           # if this raises, drop_staging runs

Use caching for expensive, deterministic steps (downloads, parsing, model scoring on fixed input) so retries and reruns are cheap. Do not cache steps whose result depends on time or external state unless the key includes it, or you will happily serve stale data. Idempotency still matters: a retry or a worker crash can run a task twice, so design writes as upserts or partition overwrites.

📖 States · Caching · Customize caching · Persist and retrieve results · Transactional workflows

---

5. Deployments: three ways to run a flow

Analogy: a standing order versus cooking at home. Running the function yourself is cooking at home. A deployment is a standing order registered with the operations centre: it says what, where the recipe lives, which kitchen, and when.

Approach	How	Infrastructure	Pick it when
Run the function	python flow.py	Wherever you run it	Development and tests
flow.serve()	A long-lived Python process listens for runs and runs schedules	That process (container, VM)	Simple, single environment, no pool to manage
flow.from_source(...).deploy() or prefect deploy with prefect.yaml	Deployment bound to a work pool	Dynamically provisioned per run	Many deployments, different environments, isolation per run, platform-team guardrails

# 1. serve: simplest
daily_etl.serve(name='daily-etl', cron='0 6 * * *', parameters={'day': 'today'})

# 2. deploy to a work pool (code pulled from git at run time)
from prefect import flow
flow.from_source(source='https://github.com/acme/pipelines.git', entrypoint='flows/etl.py:daily_etl').deploy(
    name='daily-etl-prod', work_pool_name='k8s-pool', cron='0 6 * * *',
    parameters={'day': 'today'}, job_variables={'image': 'acme/pipelines:1.4'}, tags=['etl'])     # ⚑ argument names

# trigger on demand
from prefect.deployments import run_deployment
run_deployment(name='daily-etl/daily-etl-prod', parameters={'day': '2026-10-22'})

• Three ways to define a deployment: Python SDK (.deploy()), YAML (prefect.yaml plus prefect deploy), or Terraform. All populate the same schema: name, flow, parameters, schedules, tags, work pool, job variables, version.
• Code storage: bake the code into the Docker image (pool types that use images), pull it from git or a cloud bucket at run time, or use a local path with Process pools. Do not rely on a path that exists only on your laptop.
• Versioning: deployments carry a version and can be rolled back; tag flows and tasks with version too. Override per-run infrastructure with job variables.
• CI/CD: deploy from CI on merge, one deployment per environment, with the image tag as a job variable.

Traps. A deployment does not contain code, so a worker that cannot reach the repo or image fails the run. Changing the flow signature without redeploying breaks scheduled runs that carry old parameters. serve ties execution to one process that must stay up.

📖 Deployments · Create deployments · prefect.yaml · Store flow code · Versioning · CI/CD

---

6. Work pools, workers and queues

Analogy: pub/sub topics. A work pool is a topic. Deployments publish runs to it; workers subscribe and start the runs. Switch a deployment to another pool and a different worker runs it, which is how you promote between environments or debug locally.

Pool kind	Who starts runs	You run a worker?	Examples
Hybrid	A worker in your infrastructure	Yes	Process, Docker, Kubernetes, AWS ECS, Azure Container Instances, Google Cloud Run (v1, v2), Vertex AI
Push (Cloud)	Prefect submits to your serverless provider	No	ECS push, Cloud Run push, ACI push, Modal push
Managed (Cloud)	Prefect-managed infrastructure	No	Prefect Managed; Coiled is also available

• Why pools at all: dynamic infrastructure per run (scale to zero), default job configuration that every deployment inherits and can override, and opinionated, enforced interfaces for platform teams.
• Workers: prefect worker start --pool my-pool ⚑ polls and submits runs; a hybrid worker needs outbound access to the API and the permissions to create infrastructure. Run it as a supervised service (systemd, a Deployment) and use the worker healthcheck documented under advanced guides.
• Work queues sit inside a pool. Each has a priority (a unique positive integer; 1 is highest) and an optional concurrency limit that cannot exceed the pool's global limit. Runs are submitted from the highest priority queue in waterfall order. A queue is READY if a worker polled it in the last 60 seconds, PAUSED when you pause it, otherwise NOT_READY.
• Example policy: low (priority 10, unlimited), high (priority 5, limit 3), critical (priority 1, limit 1) for urgent backfills.
• Job variables and the base job template define per-pool defaults (image, resources, env vars, namespace); override them on the deployment or even per run. Customising Kubernetes job templates has its own guide.

Pick a pool: laptop or single VM → Process. Containerised isolation on one host → Docker. Production on a cluster → Kubernetes. Serverless without operating a worker → push pool on Cloud. Avoid a Process pool for untrusted or conflicting dependencies.

Traps. The worker's own credentials decide what it can launch, so scope them. A pool with no live worker leaves runs Late. Mixing images with different Python or Prefect versions between client and worker causes confusing serialisation errors; keep versions aligned (see the versioning and compatibility page).

📖 Work pools · Workers · Manage work pools · Kubernetes · Docker · Serverless · Job variables · Worker healthchecks

---

7. Schedules, events and automations

Analogy: an alarm clock versus a motion sensor. A schedule fires at times you choose; an event trigger fires when something happens, such as a file arriving or a flow failing.

• Schedules tell the API to create flow runs automatically: cron, interval or RRule, attached to a deployment, with timezone support. Pause, resume or remove them per deployment.
• Events record activity in your stack: Prefect emits them for state changes and you can emit custom ones (emit_event(...) ⚑). They carry a resource, an event name and a payload.
• Automations pair a trigger (event, metric or compound) with actions: run a deployment, pause a schedule, send a notification, cancel a run. Deployment triggers make event-driven pipelines; chaining deployments with events gives dependencies between separate workflows without coupling the code; you can pass the event payload as flow parameters.
• Debounce with the within parameter so a burst of events does not create a run per event. Zombie flows (runs that stopped reporting) can be detected with an automation.

Choose a schedule for predictable batch windows; choose an event trigger for 'when upstream finished' or 'when data landed'. Prefer events over polling loops inside a flow.

Traps. Cron is in UTC unless a timezone is given. A schedule creates runs even if the previous one is still running unless concurrency limits stop it. Triggers that match their own outputs can loop.

📖 Schedules · Events · Automations · Event triggers · Chaining deployments · Debouncing · Zombie flows

---

8. Configuration: variables, blocks, secrets, settings

Thing	Holds	Use for	Not for
Variables	Dynamically-named, mutable, non-sensitive JSON values	Shared config such as a table name or a flag	Secrets
Blocks	Typed, saved configuration with a UI (credentials, buckets, notification targets)	Reusable connections and secrets managed centrally	Per-run ephemeral data
Secret blocks	Encrypted sensitive strings	API keys, passwords	Large documents
Settings and profiles	PREFECT_* settings per environment	API URL, logging, timeouts, server tuning	Application config
Runtime context	Info about the current run (prefect.runtime)	Flow run ID, parameters, deployment name inside code	Persistent state

from prefect.variables import Variable
from prefect.blocks.system import Secret
from prefect.runtime import flow_run

limit = Variable.get('batch_size', default='500')
token = Secret.load('vendor-api-token').get()
print(flow_run.id, flow_run.parameters)          # ⚑ runtime attribute names

Switch between environments with profiles (prefect profile use prod ⚑) rather than editing code. Self-hosted servers have security settings (basic auth and others) documented under advanced guides; do not expose an unauthenticated API to a network.

📖 Variables · Blocks · Settings and profiles · Runtime context · Store secrets

---

9. Observability, artifacts, assets and human input

• Logging: log_prints=True turns print into run logs; get_run_logger() gives a structured logger. Logging config is customisable. Cloud offers AI summaries of failed-run logs.
• Artifacts are persisted outputs for humans, rendered in the UI (markdown, tables, links, progress): create_markdown_artifact(...), create_table_artifact(...) ⚑. Use them for data-quality reports and run summaries, not as a results store.
• Assets represent objects your workflows produce (a table, a file, a model) and are tracked as lineage across runs. Declare them on tasks with @materialize(...) ⚑ and add metadata and explicit dependencies where inference is not enough. Check your version supports them.
• State change hooks run code when a flow or task enters a state (on failure, on completion): the place for notifications and cleanup.
• Human in the loop: a run can pause and wait for input or approval, or suspend and free its infrastructure, then resume from the UI or API. Interactive workflows have a dedicated guide. Cancellation of running flows is a documented workflow too.

📖 Logging · Artifacts · Assets · State hooks · Interactive workflows · Cancel workflows

---

10. Background tasks

Analogy: a drop-box and a back-office team. The web app drops a job in the box and returns to the customer immediately. Back-office workers pick it up whenever they are free.

task.delay(...) pushes a task run onto a server-side topic; a pool of task workers executes it. The docs position this as a replacement for Celery plus a broker for queue functionality. Use it when a web request or service must dispatch heavy work without waiting and you want to scale task execution independently. Do not use it for work that needs a flow-level schedule, deployments and per-run infrastructure; use a deployment for that. The caller does not need to resolve the returned future.

📖 Run background tasks · Web app powered by background tasks

---

11. Hosting: Cloud versus self-hosted

	Prefect Cloud	Self-hosted server
You operate	Workers and your code only	API, database, background services, workers
Extras	Push and managed pools, workspaces, SSO, service accounts, ACLs, audit logs, webhooks, SLAs, PrivateLink	Open source; fewer pool types (hybrid ones)
Limits	Documented API rate limits and data retention	Yours to size

• Run a server: prefect server start locally; Docker; Docker Compose with Postgres, Redis, server, background services and a worker; Helm chart for Kubernetes. Point clients at it with PREFECT_API_URL.
• Scale: run multiple server instances for HA (needs Postgres and Redis for messaging); keep the database maintained (monitoring, vacuuming, retention of old runs).
• Secure: enable authentication via the security settings, restrict network access, and see the page on endpoints and ports for restricted networks.

📖 Server · Docker Compose · Helm · Scale self-hosted · Database maintenance · Security settings

---

12. Testing and integrations

• Test: a flow is a function, so unit-test the logic; the docs describe a test harness and pytest fixtures for running flows and tasks against a temporary backend. Test deployment definitions by loading them in CI.
• Integrations are PyPI packages: prefect-aws (S3, ECS worker, Secrets Manager, Lambda, Glue, Batch), prefect-gcp, prefect-azure, prefect-kubernetes, prefect-docker, prefect-dbt (runner with asset lineage, or per-node orchestration), prefect-snowflake, prefect-sqlalchemy, prefect-databricks, prefect-dask, prefect-ray, prefect-redis, prefect-slack, prefect-email, prefect-github, prefect-gitlab, prefect-bitbucket, prefect-shell.
• AI assist: a Prefect MCP server lets assistants inspect your workspace.

📖 Test workflows · Integrations · prefect-aws · prefect-dbt · MCP server

---

13. When is Prefect the right tool?

Situation	Verdict
Python-heavy data or ML pipelines with retries, caching, scheduling, dynamic fan-out	Good fit. Flows are ordinary code
Event-driven pipelines (file landed, upstream finished)	Good fit (events and automations)
A team that wants per-run isolated infrastructure without operating a scheduler fleet	Good fit (work pools, push or managed)
You need a declarative, reviewable DAG with a large existing operator ecosystem and heavy SQL-first teams	Airflow may fit better; Prefect has a migration guide, so weigh the rewrite
The product is the asset graph with software-defined data assets and a data catalogue	Compare with asset-first tools; Prefect's assets are newer, so check maturity for your needs
Strict durable execution of business transactions over days with exactly-once semantics	A durable-execution engine such as Temporal is a closer match
One cron job calling one script	Overkill. Use cron or a platform scheduler until you need retries, visibility or dependencies
Heavy transformation inside a warehouse	Orchestrate it (dbt or SQL tasks) and let the warehouse do the work

Challenge before adopting: orchestration adds an API, a database and workers to run, so ask what failure you are actually solving (silent failures, no retries, no visibility) and whether a simpler scheduler plus alerting would do. Also confirm where results and logs may be stored if your data is regulated.

---

14. Trap checklist and exercises

1. Expecting a deployment to carry code; worker cannot fetch the repo or image.
2. Client, worker and image on different Prefect or Python versions.
3. Non-idempotent writes with retries; caching a time-dependent task with a code-only key.
4. Using threads for CPU-bound work; using .submit() but never resolving futures, so errors go unnoticed.
5. Dataclass fields rebuilt by parameter resolution (use quote or opaque).
6. Cron in UTC surprises; schedules overlapping long runs; trigger loops.
7. Secrets in variables instead of secret blocks; unauthenticated self-hosted API.
8. A pool with no worker, or worker credentials broader than needed.
9. Persisting results to storage that other workers cannot reach.

Exercises. (1) Write a three-task flow with retries=3 on a flaky task and caching on an expensive one; run it twice and read the states. (2) Deploy it with serve, then to a Process work pool, and compare where the code runs. (3) Add a global concurrency limit and submit 20 tasks. (4) Add a deployment trigger that runs a second flow when the first completes. (5) Wrap two writes in a transaction() with a rollback hook and force the second to fail.

Solution notes

1. A cached task shows Completed immediately on the second run; flow retries rerun the flow but cached tasks are skipped.
2. serve runs in your process; the pool path needs a worker polling the pool and a reachable code source.
3. Create the limit first (CLI or UI), then acquire it in code with concurrency(name); excess tasks wait.
4. Use an automation with a flow-run-completed trigger and a run-deployment action, optionally passing the event payload.
5. The rollback hook of the first write runs, leaving no partial output.

---

Sources: docs.prefect.io (v3): tasks, work pools, deployments, documentation index. Other sections were written from the index summaries plus prior knowledge; items marked ⚑ should be verified.

RustFS — Components Deep Dive

A study guide to RustFS, an open-source, S3-compatible distributed object store written in Rust: what each component is, the commands and settings you will actually use, when to use it, the traps, and where the docs live.

How to read this guide. Each component has (1) a mental model, (2) the code or config, (3) a when to use / when not to verdict, (4) traps and (5) 📖 doc links. The decision tables are in §2 (which deployment mode) and §11 (should I use RustFS at all). The sibling page Prefect — Components Deep Dive covers an orchestrator that can use an S3-compatible store for results and code. Version note: written 23 Oct 2026 from docs.rustfs.com. The S3 compatibility page was snapshotted by RustFS against a commit dated 9 Aug 2026, so treat counts as a moving target. ⚑ marks items from prior knowledge or inference rather than the pages I read (boto3 usage, exact Prefect wiring). Read §11 before adopting: this is a young project, so verify production readiness yourself.

---

0. The map: what RustFS is

Analogy: a warehouse of labelled bins with a front desk that speaks Amazon's language. Customers (applications) hand over boxes (objects) with a bin label (bucket) and a shelf label (key). The front desk (the S3 API) uses the same forms as Amazon's warehouse, so existing tools just work. Behind the desk, each box is split into pieces and spread over many shelves (disks) so a few broken shelves do not lose anything.

• What: a distributed object storage system in Rust, Apache 2.0 licensed, S3-compatible, aimed at private cloud, data lakes, AI/ML and cloud-native workloads. It runs on Linux, Windows, macOS, FreeBSD and containers across x86, ARM and RISC-V.
• Positioning from the project: the permissive licence is the headline contrast with AGPL-licensed alternatives; it claims migration and coexistence with S3-compatible platforms such as MinIO and Ceph. The README also claims 2.3x faster than MinIO for 4 KB payloads. That is a vendor benchmark; reproduce it on your hardware and object sizes before it informs a decision.
• Components: the rustfs server (S3 API on port 9000), the web Console (port 9001), the rc command-line client, optional protocol gateways (WebDAV, FTP/FTPS, SFTP, OpenStack Swift), background services (scanner for bit-rot, healing, lifecycle, replication), Kubernetes Helm chart and Operator, and an MCP server for AI assistants.
• Object-store vocabulary: an object is a blob plus metadata, addressed by bucket/key. Keys look like paths but the namespace is flat; 'folders' are just key prefixes.

Addressing. Path-style by default (http://host:9000/my-bucket/path/to/hello.txt). For virtual-host style (https://my-bucket.s3.example.com/hello.txt) set RUSTFS_SERVER_DOMAINS=s3.example.com and create wildcard DNS (*.s3.example.com) pointing at RustFS.

📖 Docs home · S3 protocol · Virtual-host access · GitHub

---

1. Installing it

Goal	Path
Try it locally	One-command install script, or a container
Single-server production	SNSD (one disk) or SNMD (several disks), on Linux
Multi-server production	MNMD after the pre-installation checklists
Kubernetes	Helm chart (standalone or distributed) or the RustFS Operator (tenants, pools, KMS)
Windows or macOS	Native install; the Windows build is for development and debugging only

1.1 Docker, the fastest start

docker volume create rustfs-data
docker run -d --name rustfs --restart unless-stopped \
  -p 9000:9000 -p 9001:9001 -v rustfs-data:/data \
  -e RUSTFS_ACCESS_KEY='<your-access-key>' -e RUSTFS_SECRET_KEY='<your-secret-key>' \
  -e RUSTFS_ADDRESS=':9000' -e RUSTFS_CONSOLE_ADDRESS=':9001' -e RUSTFS_CONSOLE_ENABLE=true \
  rustfs/rustfs:latest /data
curl --fail http://localhost:9000/health         # S3 API health; Console on :9001

• Change the default credentials immediately. Never use the well-known rustfsadmin value. On Linux set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY in /etc/default/rustfs and restart; with containers pass them at creation (recreate the container if it started without them; the volume survives).
• The container runs as non-root user rustfs, UID 10001. A bind-mounted host directory must be owned by 10001 (chown -R 10001:10001 /path) or you get permission denied. Named volumes need no setup.
• Flags win over env vars when both are given (--access-key, --address, --console-enable).
• Compose: the repository ships a docker-compose.yml with optional Grafana, Prometheus, OpenTelemetry collector and Jaeger under --profile observability; start only the server with docker compose up -d rustfs. An init container fixes volume ownership.
• Multi-node containers need --network host (the default bridge network does not support it) and name resolution (/etc/hosts on every node): RUSTFS_VOLUMES='http://node{1...4}:9000/data/rustfs{0...3}'.
• TLS: mount a cert directory and set RUSTFS_TLS_PATH=/certs. For HTTPS in front, the docs cover Nginx, Traefik, Caddy, Envoy, HAProxy and Apache with separate hostnames for S3 and Console.

📖 Installation · Docker · Linux · Kubernetes · Checklists · TLS · Reverse proxy

---

2. Deployment modes: which one?

Mode	Nodes	Disks	Fault tolerance	Use for
SNSD	1	1	None. Rely on backups	Dev, test, small non-critical
SNMD	1	Several	Up to M parity disks inside the node; the node itself is a single point of failure	Mid-sized non-critical on one server
MNMD	4 or more	Several per node	Disk- and node-level via erasure coding across servers	Production

Decision rule: if losing a server or its data would hurt, you need MNMD plus a backup or replication strategy (erasure coding protects against disk failures, not against deletion, corruption by software, or a lost site). SNSD is a development tool, not a storage strategy.

Disk preparation (Linux). Use dedicated data disks, format, mount and verify them before installing (the docs refer to XFS and a verification step); keep identical service environment configuration on every node. Work through the hardware, network, software and security checklists first.

📖 SNSD · SNMD · MNMD · Disk preparation · Hardware selection

---

3. Erasure coding and capacity

Analogy: a jigsaw with spare pieces. Each object is cut into N-M data pieces plus M extra 'parity' pieces, then spread across N disks. Any N-M of the N pieces rebuild the object, so up to M pieces can be lost.

• Reed-Solomon erasure coding. Each pool is divided into one or more erasure sets. For a set of N drives with M parity shards: data shards = N-M; tolerable unavailable shards = M; capacity efficiency about (N-M)/N. A 16-drive set with EC:4 stores 12 data and 4 parity shards, about 75% efficiency before file-system, metadata and operational overhead.
• Failure-domain caveat (from the docs): parity covers unavailable shards within one set. Node-level tolerance depends on how a set's drives are spread over nodes; do not assume EC:4 survives four whole nodes, because one node may hold several drives of the same set.

Drives per set (N)	Automatic STANDARD parity	Approx. efficiency
1	EC:0	100%
2-3	EC:1	50-67%
4-5	EC:2	50-60%
6-7	EC:3	50-57%
8-16	EC:4	50-75%

# /etc/default/rustfs
RUSTFS_STORAGE_CLASS_STANDARD=          # recommended: leave empty for automatic per-pool parity
# or pin explicitly (must be valid for the NARROWEST pool):
RUSTFS_STORAGE_CLASS_STANDARD=EC:4
RUSTFS_STORAGE_CLASS_RRS=EC:1           # REDUCED_REDUNDANCY writes; default 1 parity on multi-drive sets
RUSTFS_ERASURE_SET_DRIVE_COUNT=16       # pin set width only if you verified the whole layout

• Rules for explicit parity: M <= N/2 in every target set; STANDARD parity >= RRS parity when both non-zero; valid for every pool. An invalid value makes startup fail rather than silently weakening parity.
• Set widths range from 2 to 16 drives, selected automatically from the topology. Validate against the set width, not the cluster's total drive count.
• Capacity estimate: usable is roughly raw x (N-M)/N. Sixteen 10 TiB drives as one 16-drive set with EC:4 gives about 120 TiB, and you must still reserve space for metadata, versioning, incomplete multipart uploads, healing and headroom.
• Verify after start: Console Status page or rc admin info cluster shows layout, parity, drive availability and pool membership.

Trap. More parity buys durability at the price of usable capacity and write cost; do not pin high parity on small sets without checking the efficiency. Leaving the default is the documented recommendation unless you have a validated requirement.

📖 EC configuration · Status check · Storage env vars

---

4. Configuration surface

Setting	Purpose
RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY	Root credentials. Set unique values
RUSTFS_ADDRESS	S3 API bind address (default :9000)
RUSTFS_CONSOLE_ADDRESS, RUSTFS_CONSOLE_ENABLE	Console bind address and switch (port 9001)
RUSTFS_VOLUMES	The storage endpoints; supports {1...4} range expansion for multi-node
RUSTFS_SERVER_DOMAINS	Domain for virtual-host style access
RUSTFS_TLS_PATH	Directory with certificates
RUSTFS_STORAGE_CLASS_*, RUSTFS_ERASURE_SET_DRIVE_COUNT	Erasure coding (§3)
RUSTFS_OBS_*	Logging and observability (level, log directory)

The reference page also lists variables for KMS, the scanner, healing and event targets. rustfs has subcommands and server flags with environment equivalents (CLI reference). Keep the same environment file on every node.

📖 Environment variables · CLI reference · CORS

---

5. The S3 API and compatibility

Analogy: speaking a dialect. RustFS speaks Amazon S3 well enough for most everyday conversation, but not every idiom. The project says so itself: it is a tested subset, not a claim of full coverage.

# AWS CLI against RustFS: just add --endpoint-url
aws --endpoint-url "$RUSTFS_ENDPOINT" s3 cp hello.txt s3://my-bucket/path/to/hello.txt
aws --endpoint-url "$RUSTFS_ENDPOINT" s3 ls s3://my-bucket/
aws --endpoint-url "$RUSTFS_ENDPOINT" s3api head-object --bucket my-bucket --key path/to/hello.txt
aws --endpoint-url "$RUSTFS_ENDPOINT" s3 presign s3://my-bucket/path/to/hello.txt

# Python (boto3)  ⚑
import boto3
s3 = boto3.client('s3', endpoint_url='http://localhost:9000',
                  aws_access_key_id=KEY, aws_secret_access_key=SECRET, region_name='us-east-1')
s3.upload_file('hello.txt', 'my-bucket', 'path/to/hello.txt')

Area	Status in the compatibility gate
Bucket create/delete/list/inspect, tagging, policies, public access block	Tested
Object put/get/copy/head/delete, listing (prefix, delimiter, marker, max-keys; V1 and V2)	Tested
Range and conditional reads, user metadata, object tags, presigned GET/PUT	Tested
Multipart create/upload/complete/abort; selected copy and checksum cases	Tested
Selected versioning, Object Lock, CORS and lifecycle cases; SSE-C and selected SSE-KMS	Tested (only the cases in the implemented lists)
Bucket access logging, bucket ownership controls, POST Object form checksum handling	Planned (not implemented)
ACL authorisation	Excluded: intentionally unsupported
Multipart upload listing and part-lookup edge cases	Excluded from the default gate

• Gate size (Aug 2026 snapshot): 455 implemented cases, 5 lifecycle cases, 17 planned, 270 excluded, from Ceph s3-tests lists. Read the numbers as 'a lot is covered, but 270 are excluded', not as 'complete'.
• Encrypted objects are not portable. A passing encryption test means RustFS can read what RustFS encrypted; it does not mean it can read an encrypted object copied raw from another implementation.
• S3 Select currently accepts only uncompressed input.
• Other protocols: WebDAV, FTP/FTPS, SFTP and (optional build) OpenStack Swift gateways over the same buckets.

Practical advice: if you depend on ACLs, bucket access logging, or exotic multipart listing behaviour, test those exact calls against RustFS before committing. Use rclone or your SDK's integration test suite as a smoke test.

📖 S3 compatibility matrix · SDKs · Python · SFTP · WebDAV

---

6. Data management features

Feature	What it does	Use it for	Gotcha
Versioning	Keeps prior versions; deletes create markers	Recover from overwrites and accidental deletes	Storage grows; required by replication
Object Lock	Retention periods and legal holds on versioned objects	WORM and compliance retention	Needs versioning; test retention modes before relying on them
Lifecycle rules	Expire or transition objects by age	Cost control, log retention	Managed with the rc client; verify rules fire
Tiered storage	Lifecycle-driven transition to a remote storage tier	Move cold data to cheaper storage	Reads of tiered objects depend on the remote tier
Bucket quota	Hard storage quota per bucket	Multi-tenant limits	Hard limit: writes fail at the cap
Bucket replication	Asynchronous replication between versioned buckets	Copies in another cluster	Async, so a failover can lose the last writes; both ends need versioning
Site replication	Multi-site replication managed with rc	Disaster recovery across sites	Plan network and consistency expectations
Multipart upload	Parts for large objects; Console task monitor	Big files, resumable uploads	Incomplete uploads consume space; clean up
Scanner	Background object scan with a bit-rot cycle and metrics	Detect silent corruption	Watch its metrics to confirm scans still run
S3 Tables	Table buckets with a built-in Iceberg REST catalog	Lakehouse without a separate catalog	Newer feature; test with your engines (PyIceberg, Spark docs exist)
Event notifications	Filtered bucket events to external targets (webhook and others)	React to uploads, trigger pipelines	Targets and queue directories need persistent storage

📖 Versioning · Object Lock · Lifecycle · Tiering · Quota · Bucket replication · Site replication · Scanner · S3 Tables · Event notifications

---

7. Security

Analogy: badges, keys and a locked cabinet. IAM decides who may enter which room (badges). Encryption keeps the contents unreadable even if someone steals a disk (locked cabinet). TLS protects the hallway.

• IAM: users, groups and policies; policy attachment and evaluation semantics are documented. Service accounts and STS issue scoped, short-lived credentials for applications. Access keys are managed per identity. Prefer service accounts or STS for applications and keep the root key out of code.
• Bucket policies are applied through the S3 API; public access block exists. ACLs are intentionally unsupported.
• OIDC: Console single sign-on with providers such as Keycloak and GitLab.
• Encryption at rest: SSE-S3 (server-managed keys), SSE-C (customer-provided AES-256 keys) and SSE-KMS with a local or HashiCorp Vault backend; the Operator can configure KMS for tenants. Note: the project README marks its KMS as 'under testing' while the docs describe KMS configuration, so validate your key-management path and recovery procedure before putting regulated data on it.
• TLS with RUSTFS_TLS_PATH; mTLS and cert-manager are documented for Helm deployments.
• Audit targets deliver request-level audit records to external systems. Credential management has its own page; rotate the root credentials.

Traps. Default credentials left in place. Exposing the Console (9001) to the internet. Putting RustFS behind a reverse proxy that rewrites the Host header without enabling the matching virtual-host settings. Losing the KMS key means losing the data.

📖 Security and compliance · IAM · Service accounts and STS · Encryption · OIDC · Audit · Credentials

---

8. Operations

• The rc client: a RustFS command-line client to connect to, inspect and administer a cluster, including lifecycle rules, replication, heal tasks and rc admin info cluster.
• Health and status: /health on the S3 port; the Console Status page; the status-check workflow covers network access, health endpoints, cluster status and capacity.
• Observability: export metrics, traces and logs over OTLP; the docs list recommended alerts and metric names, and integrations with Prometheus, Grafana, Loki, Tempo and Thanos exist.
• Scaling: expand by adding a storage pool; rebalance to redistribute existing objects over active pools; decommission to drain and retire a pool and verify the data moved. Pools are the unit of growth, so plan pool sizes and symmetry up front.
• Healing and failures: replace a failed drive and let RustFS heal the missing shards onto it; for a lost node prepare a replacement, update its configuration, restore cluster membership and watch healing. Manual heal tasks can be bounded with rc.
• Upgrades: binary, container and Kubernetes procedures that preserve availability. Rehearse on a non-production cluster first.
• Kubernetes: the Helm chart (standalone or distributed, probes, hardening, pool expansion) or the Operator (multi-tenant isolation, pool expansion, KMS).

Runbook items to write before go-live: disk replacement, node replacement, credential rotation, certificate renewal, capacity alert thresholds, upgrade order, restore test from backup or replica.

📖 rc client · Observability · Scaling · Node healing · Drive failures · Node failures · Upgrade · Cluster lifecycle

---

9. Integrations relevant to a data platform

RustFS documents step-by-step integrations (mostly Docker Compose based) for tools that speak S3. Grouped by purpose:

Purpose	Tools with a RustFS guide
Orchestration and ingestion	Airflow (S3 provider), DolphinScheduler, SeaTunnel, Kafka Connect S3 sink, AutoMQ
Lakehouse and query	Apache Iceberg (Spark, REST catalog), PyIceberg via S3 Tables, Delta Lake (delta-rs), Hudi, Paimon, Hive, Spark, Flink, Trino, DuckDB (httpfs), ClickHouse (S3 disk), Databend, Doris, lakeFS
Vector and ML	LanceDB, Milvus, MLflow artifact store, Ray Data, vLLM model weights
Observability storage	Loki, Tempo, Thanos, Cortex, OpenObserve, GreptimeDB, Fluentd, OpenTelemetry, VictoriaMetrics backups
Backup and DR	Restic, Kopia, Velero, Longhorn, Vitess
DevOps and platform	Terraform state with locking, Elasticsearch and OpenSearch snapshots, Gitea LFS, Jenkins artifacts, Harbor and Docker Registry, Flux, Nextcloud
Filesystem views	s3fs-fuse, rclone, JuiceFS, ZeroFS (NFS), Alluxio cache, SFTPGo, OpenDAL

9.1 Using it with Prefect ⚑

Prefect can keep results and deployment code in S3-compatible storage through the prefect-aws integration. Because RustFS speaks S3, the wiring is the integration's S3 bucket block with a custom endpoint URL pointing at RustFS and a service-account key. I wrote this from how prefect-aws normally works, not from a tested RustFS example; check the prefect-aws docs for the current block and client-parameter names and run a write/read round trip before trusting it.

• Results: point result storage at a bucket such as prefect-results; every worker must reach the endpoint and credentials.
• Code: a pull step can fetch flow code from a bucket at run time.
• Hygiene: a dedicated service account with a bucket-scoped policy, a lifecycle rule to expire old results, TLS on the endpoint.

📖 Integrations index · Airflow · Iceberg · DuckDB · Spark · lakeFS · Terraform state · rclone · prefect-aws

---

10. Moving data in and out

• Migration and coexistence: the project says it supports running alongside other S3 platforms such as MinIO and Ceph. The practical tools are rclone sync, bucket replication (versioned buckets) and application-level dual writes.
• Validate a migration: compare object counts, sizes and checksums; test presigned URLs, multipart behaviour, listing order and any ACL or logging features you used, because those are the documented gaps.
• Do not copy encrypted objects raw between implementations (not portable); decrypt and re-encrypt through the S3 API.
• Exit plan: because the API is S3, leaving RustFS later is an rclone job, which is a point in its favour. Keep that tested.

---

11. Should you use RustFS? (the part to challenge)

Situation	Verdict
Local dev and CI needing an S3 endpoint	Good fit. One container, quick to start
Self-hosted S3 for logs, artefacts, backups, ML datasets where a permissive licence matters	Reasonable, after your own load and failure tests
Data lake or lakehouse storage with Iceberg or Delta	Plausible; documented integrations exist, but pilot with your engines and table sizes
Regulated or business-critical primary storage	Proceed carefully. Confirm KMS maturity, Object Lock behaviour, support terms and your recovery drills
Need full AWS S3 feature parity (ACLs, access logging, every multipart listing edge)	Not a fit where those are documented as excluded or planned
Cloud-native workload already on a managed S3	Staying on the managed service removes operations; self-host only for cost, sovereignty or latency reasons

Questions to answer before adopting: (1) Is the benefit the licence, the cost, performance, or Rust itself, and have you measured it? (2) Who is on call when a node dies at 3 a.m.? (3) Have you rehearsed a disk and node replacement and a restore? (4) What is the backup strategy beyond erasure coding? (5) How mature is the release you pin, and what is the upgrade policy? Pin a version rather than using :latest in production.

---

12. Trap checklist and exercises

1. Default credentials; Console exposed publicly.
2. Bind-mount directory not owned by UID 10001.
3. Believing EC:4 equals four-node tolerance.
4. Pinning parity that the narrowest pool cannot satisfy (startup fails), or setting a set width that does not divide the topology.
5. SNSD or SNMD treated as production, with no backup.
6. Replication configured without versioning; async replication assumed to be lossless.
7. Relying on ACLs, access logging or encrypted-object portability.
8. Using :latest; no upgrade rehearsal; no restore test.
9. Windows build used in production.

Exercises. (1) Start the Docker command with unique credentials, create a bucket and upload a file with the AWS CLI using --endpoint-url. (2) Enable versioning, overwrite the file, and recover the first version. (3) Compute usable capacity for 8 nodes x 4 x 8 TiB drives if sets are 16 wide with EC:4, and say what a node loss does to a set. (4) Add a lifecycle rule to expire objects under tmp/ after one day and verify. (5) Run a write/read round trip from a Python script with boto3 and a presigned URL.

Solution notes

1. Use aws --endpoint-url http://localhost:9000 s3 mb s3://demo, then cp; configure the CLI with the access key pair you set.
2. Versioning is a per-bucket setting; list versions to see both, then fetch by version ID or remove the delete marker.
3. Raw is 8 x 4 x 8 = 256 TiB; with a 16-wide set and EC:4 efficiency is 75%, so about 192 TiB before overhead. Whether a node loss is survivable depends on how many of a set's 16 drives sit on that node: if a node holds 4 drives of each set, losing it removes 4 shards, which EC:4 can reconstruct, with no margin left.
4. Lifecycle rules are managed with rc; confirm by checking the object disappears after the scanner or lifecycle cycle runs.
5. boto3 needs endpoint_url; generate_presigned_url('get_object', ...) gives a URL you can fetch with curl.

---

Sources: docs.rustfs.com: docs home, installation, Docker, EC configuration, S3 compatibility matrix, S3 protocol, docs index; GitHub README (claims). Sections 6-8 summarise the documentation index descriptions rather than every page body; items marked ⚑ are unverified.