Elasticsearch Engineering

A study guide for the Elastic Certified Engineer exam (Elasticsearch 9.3), organised by the official exam objectives.

How to read this guide. Each objective gets (1) a mental model, often an everyday analogy, (2) the APIs you'll actually type, (3) the traps graders like, and (4) links to the exact Elastic documentation pages. The Elastic documentation is the only resource available during the exam, so the 📖 links double as a map of where to go on exam day. Exercises with solution notes close each domain. The companion page, Elasticsearch Engineering – Digest, holds the glossary and the must-remember facts.

0. The exam and your preparation

0.1 Format

📖 Exam page · Certification FAQ

Analogy. It's a practical driving test, not a written theory test. Nobody asks you to recite the highway code. The examiner checks whether the car ends up correctly parked. The docs are your satnav: allowed, but only useful if you already know roughly where you're going.

0.2 The six domains

0.3 Build a lab first

You can't prepare for a hands-on exam by reading. Use 9.3.x so the APIs and UI match the exam.

Option A: single node in one command (good for Domains 2–4)

curl -fsSL https://elastic.co/start-local | sh

📖 Local installation quickstart

Option B: three-node Docker Compose (needed for allocation, awareness, snapshots, and Domain 5)

📖 Multi-node cluster with Docker Compose

Settings to vary per node (as elasticsearch.yml or environment variables):

cluster.name: lab
node.name: es01                                  # es02, es03
node.roles: [ master, data_hot, data_content, ingest, remote_cluster_client ]
node.attr.zone: zone-a                           # zone-b, zone-c → awareness exercises
discovery.seed_hosts: [ es01, es02, es03 ]
cluster.initial_master_nodes: [ es01, es02, es03 ]   # first bootstrap ONLY, then remove
path.repo: [ /usr/share/elasticsearch/snapshots ]    # shared FS repo for snapshots

For Domain 5, add a second cluster (e.g. remote) so you can practise remote clusters, cross-cluster search (CCS) and cross-cluster replication (CCR).

Enable a trial license. CCR, searchable snapshots, some security features and ML-based semantic search need it:

POST _license/start_trial?acknowledge=true

Load sample data. In Kibana: Home → Add data → Sample data (eCommerce, Flights, Web logs). Many exam tasks look exactly like work on these datasets.

0.4 Exam-day habits that earn points

1. Read each task twice and copy names (indices, fields, pipelines, roles) straight from the task text. A typo scores zero.
2. Verify everything. GET what you created. Compare _count before and after a reindex. _simulate a pipeline before using it.
3. Don't destroy state the task didn't mention. Graders inspect the final state.
4. Prototype on scratch indices (tmp-*), then clean up.
5. Time-box. Roughly 15 minutes per task. At 20 minutes, move on and come back later.
6. Know the "immutable" things. You can't change shard count, field types or a field's analyzer in place. The answer is a new index plus _reindex, or _split/_shrink for shards.
7. Use Console shortcuts. Ctrl/Cmd+Enter runs a request, Ctrl/Cmd+/ opens the docs for the API under the cursor, and Ctrl/Cmd+I auto-indents.

📖 Console (Dev Tools)

1. Elasticsearch Architecture

1.1 Describe the Elastic Stack architecture

The big picture

Analogy: a postal system. Collectors (Elastic Agent, Beats, OpenTelemetry collectors, Logstash) pick up letters everywhere. Sorting centres (Logstash, ingest pipelines, Streams) open, stamp and classify them. The warehouse (Elasticsearch) stores every letter so it can be found in milliseconds. The front desk (Kibana) is where people ask questions and see dashboards. Fleet is the dispatch office that tells every collector what to pick up.

Deployment options: Elastic Cloud Hosted (ECH), Elastic Cloud Serverless (no nodes or shards to manage), Elastic Cloud Enterprise (ECE), Elastic Cloud on Kubernetes (ECK), and self-managed. The exam uses self-managed-style clusters, so everything below assumes you can touch nodes and settings.

📖 Elastic fundamentals · Deployment comparison · Ingest tools overview

Inside Elasticsearch: cluster → node → index → shard → segment

Analogy: a library chain. The cluster is the whole chain. Each node is a branch building. An index is a book collection (for example "novels"). Because the collection is too big for one shelf, it's cut into volumes: the primary shards. Each volume has photocopies (replica shards), always kept in a different branch. Inside each volume, pages are glued in batches (segments) that are never edited, only replaced when batches are merged.

Node roles (node.roles in elasticsearch.yml):

Every node is also a coordinating node. Whichever node receives the request fans it out and gathers the results.

Write path: client → coordinating node → routes to the primary shard → primary indexes (in-memory buffer + translog) → forwards to replicas in parallel → acknowledges once the required copies succeed (wait_for_active_shards, default 1 = primary only).

Near real-time search: a refresh (default every 1s, only on indices that have been searched recently) turns the buffer into a searchable segment. A flush makes segments durable on disk and trims the translog. Merges combine small segments in the background.

Read path (query then fetch): the coordinating node sends the query to one copy of each shard. Each shard returns its top IDs and scores (query phase). The coordinator merges them, then fetches the actual documents (fetch phase).

Cluster formation: master-eligible nodes elect a master by quorum (majority of the voting configuration). Three master-eligible nodes tolerate one failure. cluster.initial_master_nodes is used only for the very first bootstrap; remove it afterwards.

📖 Clusters, nodes and shards · Node roles · Reading and writing documents · Near real-time search · Discovery and cluster formation · Quorum-based decisions

Data tiers

Analogy: a wardrobe. Hot is the chair where today's clothes lie: instant access, most expensive space. Warm is the wardrobe. Cold is the suitcase under the bed. Frozen is the storage unit across town: you can still get anything, but it takes a trip. Content is your bookshelf. It's not "time-based", it's just always there.

• New indices go to data_content by default; data stream backing indices go to data_hot.
• Tier placement uses index.routing.allocation.include._tier_preference (e.g. data_warm,data_hot), which ILM's migrate action sets automatically.
• Frozen uses partially mounted searchable snapshots: a local cache plus the object store. Cold can use fully mounted searchable snapshots, which lets it drop replicas.

📖 Data tiers · Searchable snapshots

1.2 Describe shard sizing and allocation strategies

Shard sizing: the rules of thumb

Analogy: moving boxes. If boxes are too big, nobody can lift them: slow recovery, slow rebalancing. If there are too many tiny boxes, the truck spends its time on paperwork: every shard costs heap, file handles and cluster-state overhead. Aim for boxes one person can carry.

GET _cat/shards?v&s=store:desc&h=index,shard,prirep,state,docs,store,node
GET _cat/indices?v&s=pri.store.size:desc
GET _cluster/stats?filter_path=indices.shards,indices.count

📖 Size your shards · Scaling considerations

How shard allocation works

The master's allocator decides where every shard copy lives, filtered by deciders. Each decider can say YES, NO or THROTTLE. The common ones:

The single best diagnostic API:

GET _cluster/allocation/explain
{ "index": "orders", "shard": 0, "primary": false }

Strategy 1: allocation awareness (survive losing a zone or rack)

Analogy. Don't keep your passport and its photocopy in the same bag.

# elasticsearch.yml on each node
node.attr.zone: zone-a        # zone-b on others

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone",
    "cluster.routing.allocation.awareness.force.zone.values": "zone-a,zone-b"
  }
}

• Awareness (first setting): Elasticsearch spreads copies across zones where it can.
• Forced awareness (second setting): if zone-b disappears, Elasticsearch won't squeeze all replicas into zone-a (which could overload it). The replicas stay unassigned instead. That's intended behaviour, so a yellow cluster can be the correct answer.

📖 Shard allocation awareness

Strategy 2: allocation filtering (pin or evict shards)

// Index-level: keep "hot-orders" only on nodes tagged box_type=ssd
PUT hot-orders/_settings
{ "index.routing.allocation.require.box_type": "ssd" }

// Index-level by node name (e.g. to prepare a shrink)
PUT legacy/_settings
{ "index.routing.allocation.require._name": "es01" }

// Cluster-level: drain a node before decommissioning it
PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.exclude._name": "es03" } }

include means at least one of the values, exclude means none of them, and require means all of them. Built-in attributes: _name, _host_ip, _publish_ip, _ip, _host, _id, _tier.

📖 Index-level shard allocation · Shard allocation, relocation and recovery

Strategy 3: delayed allocation and rolling restarts

When a node leaves, its replicas are reallocated elsewhere only after index.unassigned.node_left.delayed_timeout (default 1m), which avoids expensive reshuffles when a node is just rebooting. For planned maintenance:

PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": "primaries" } }
POST _flush
// …restart the node, wait for it to join…
PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": null } }

📖 Delaying allocation when a node leaves · Rolling restart

Strategy 4: resilience design

• Small clusters: at least 3 master-eligible nodes, and 1 replica for every index you can't afford to lose. A 2-node cluster isn't resilient: losing the wrong node leaves no voting majority. Add a third node (it can be a small voting_only, master-eligible tie-breaker).
• Larger clusters: dedicated masters, spread across 3 zones, with awareness enabled.

📖 Resilience in small clusters · Resilience in larger clusters

1.3 Exercises: Architecture

1. On your three-node lab, give the nodes attributes zone: a, zone: a, zone: b. Configure forced awareness on zone with values a,b. Create index aware with 1 shard and 1 replica. Stop the zone: b node and explain the resulting health.
2. Create index ssd-only whose shards may only live on nodes with attribute disk: ssd (set it on one node). Prove it with _cat/shards.
3. Decommission node es03: move all its shards off without stopping it.
4. Explain why an index created with 1 primary shard can't be given 3 primaries in place, and name the two ways to get there.

2. Data Management

Analogy for the whole domain: a filing cabinet factory. An index is a filing cabinet. Its settings are the hardware: how many drawers (primary shards) and how many photocopied backups (replicas). Its mapping is the labelling scheme for folders. Templates are the factory spec sheet, so every new cabinet named logs-* ships pre-configured. Aliases are nicknames on the door ("current-orders"), so people don't need the serial number. ILM is the records-retention policy: new files on the desk, older ones in the basement, ancient ones in the shredder.

2.1 Define an index that satisfies a given set of requirements

An index creation request has three optional parts: settings, mappings and aliases.

PUT products
{
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "refresh_interval": "30s",
    "analysis": {
      "analyzer": {
        "folded": { "type": "custom", "tokenizer": "standard",
                    "filter": ["lowercase", "asciifolding"] }
      }
    }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "sku":      { "type": "keyword" },
      "name":     { "type": "text", "analyzer": "folded",
                    "fields": { "raw": { "type": "keyword", "ignore_above": 256 } } },
      "price":    { "type": "scaled_float", "scaling_factor": 100 },
      "in_stock": { "type": "boolean" },
      "created":  { "type": "date", "format": "yyyy-MM-dd||epoch_millis" },
      "tags":     { "type": "keyword" }
    }
  },
  "aliases": { "catalog": {} }
}

Static vs dynamic settings. Know which is which:

PUT products/_settings
{ "index": { "number_of_replicas": 0, "refresh_interval": "-1" } }   // bulk-load mode

Adding an analyzer to an existing index: POST products/_close → PUT products/_settings with the analysis block → POST products/_open. You still can't change the analyzer of an existing field. Add a new field or multi-field, or reindex.

Changing the shard count:

• _split: more shards, by a multiple. Needs index.blocks.write: true.
• _shrink: fewer shards, to a factor. Needs a write block and a copy of every shard on one node (index.routing.allocation.require._name).
• Otherwise _reindex.

PUT products/_settings { "index.blocks.write": true }
POST products/_split/products-4sh
{ "settings": { "index.number_of_shards": 4, "index.blocks.write": null } }

dynamic mapping modes, a frequent grader trap:

Task phrasing to watch for: "unknown fields must be rejected" means strict; "must be stored but not searchable" means false.

Index modes (9.x). logsdb is the storage-optimised mode for logs, with synthetic _source and index sorting. It's the default for logs-*-* data streams in 9.x. time_series (TSDS) is for metrics. lookup is required for the right-hand index of an ES|QL LOOKUP JOIN.

📖 Create index API · Index settings reference · Index fundamentals · Dynamic mapping · Logs data streams (logsdb)

2.2 Index templates and component templates

Composable index templates (_index_template), built from reusable component templates (_component_template), are the standard. Legacy _template templates have been deprecated since 7.8; composable ones always win over them.

Analogy: Lego. Component templates are bricks: a "1 shard" brick, an "HTTP fields" brick. The index template is the instruction sheet saying which bricks to snap together for anything named web-*, plus a few extra pieces of its own.

PUT _component_template/ct-settings-small
{ "template": { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } } }

PUT _component_template/ct-mappings-http
{
  "template": {
    "mappings": {
      "properties": {
        "http.status": { "type": "short" },
        "url":         { "type": "keyword" }
      }
    }
  }
}

PUT _index_template/web-template
{
  "index_patterns": ["web-*"],
  "priority": 200,
  "composed_of": ["ct-settings-small", "ct-mappings-http"],
  "template": {
    "settings": { "number_of_replicas": 2 },
    "aliases":  { "web-all": {} }
  },
  "_meta": { "owner": "exam" }
}

Precedence, from lowest to highest: component templates in composed_of order (later overrides earlier), then the index template's own template block, then settings in the actual create-index request. In the example, replicas end up at 2.

Priority rules:

• Only one index template applies to a new index: the matching one with the highest priority.
• Two templates with equal priority and overlapping patterns are rejected when you create the second.
• Built-in templates use priority 100 for logs-*-*, metrics-*-*, synthetics-*-*, profiling-*, security_solution-*-*, and the Streams patterns logs.otel* and logs.ecs*. Fleet integrations go up to 200. To deliberately override a built-in pattern, the docs recommend priority > 500. Otherwise choose non-overlapping patterns.

Verify before you commit:

POST _index_template/_simulate_index/web-2026.09.28   // what would this index get?
POST _index_template/_simulate/web-template            // resolved template
GET  _index_template/web-template

Templates apply only when an index is created. Changing a template never alters existing indices. For data streams, roll over to pick up changes.

📖 Templates · Simulate multi-component templates · Put index template API · Put component template API

2.3 Define and use a dynamic template that satisfies a given set of requirements

Dynamic templates control how newly detected fields get mapped. You match on the JSON-detected type (match_mapping_type), the field name (match / unmatch, wildcards or "match_pattern": "regex"), or the dotted path (path_match / path_unmatch).

Analogy. A mail-room rulebook: "anything that looks like a parcel and whose label ends in _id goes to shelf K; every other parcel goes to shelf T". The first rule that fits wins.

PUT _index_template/sensor-template
{
  "index_patterns": ["sensor-*"],
  "template": {
    "mappings": {
      "dynamic_templates": [
        { "ids_as_keyword": {
            "match_mapping_type": "string",
            "match": "*_id",
            "mapping": { "type": "keyword" } } },
        { "desc_english": {
            "match_mapping_type": "string",
            "match": "desc_*",
            "mapping": { "type": "text", "analyzer": "english" } } },
        { "counts_as_integer": {
            "match_mapping_type": "long",
            "match": "*_count",
            "mapping": { "type": "integer" } } },
        { "raw_not_indexed": {
            "path_match": "raw.*",
            "mapping": { "type": "keyword", "index": false, "doc_values": false } } },
        { "strings_default": {
            "match_mapping_type": "string",
            "mapping": { "type": "text",
                         "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }
      ]
    }
  }
}

Rules to remember:

• Order matters: first match wins. Put specific rules first and the catch-all (strings_default) last. A catch-all placed first silently shadows the others, which is a classic hidden bug in exam tasks.
• match_mapping_type values are JSON-detected types: string, long, double, boolean, date, object, binary, or *. There's no integer or float detection.
• Placeholders: {name} (field name) and {dynamic_type} (detected type), e.g. "analyzer": "{name}".
• Use "runtime": { "type": "keyword" } instead of "mapping" to create runtime fields dynamically.
• path_match matches the full dotted path (raw.a.b); match matches only the leaf name.
• Dynamic templates apply to new fields only. Fields already mapped are untouched.
• Test it: index one sample document, then GET sensor-1/_mapping.

📖 Dynamic templates · Dynamic field mapping

2.4 Define an index template that creates a new data stream

A data stream is one name (e.g. logs-app-prod) in front of a series of hidden backing indices (.ds-logs-app-prod-2026.09.28-000001, …). It's for append-only, time-stamped data and requires an @timestamp field (date or date_nanos).

Analogy: a notebook with detachable pages. You always write on the current page (the write index). When a page fills up (rollover), a new page is added. You can read all pages at once, and you tear out and throw away old pages individually (ILM delete).

Only a matching index template with "data_stream": {} creates a data stream. Without it, writing to logs-app-prod creates an ordinary index.

PUT _index_template/logs-app-template
{
  "index_patterns": ["logs-app-*"],
  "data_stream": { },
  "priority": 501,
  "composed_of": [],
  "template": {
    "settings": { "index.lifecycle.name": "logs-app-policy" },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message":    { "type": "text" },
        "level":      { "type": "keyword" }
      }
    },
    "data_stream_options": { "failure_store": { "enabled": true } }
  }
}

// Created implicitly on first write…
POST logs-app-prod/_doc
{ "@timestamp": "2026-09-28T10:00:00Z", "message": "started", "level": "INFO" }

// …or explicitly
PUT _data_stream/logs-app-prod

Operations and constraints:

Naming scheme. Elastic's convention is type-dataset-namespace (e.g. logs-nginx.access-prod). Any name works for your own templates, but anything matching logs-*-* hits the built-in template (priority 100) unless yours has higher priority.

📖 Data streams · Set up a data stream · Use a data stream · Modify a data stream · Failure store · Time series data streams

2.5 Define an Index Lifecycle Management policy for a time-series index

ILM moves each index through phases: hot → warm → cold → frozen → delete. Each phase has a min_age (measured from rollover, or from creation if the index never rolled over) and a set of actions.

Analogy: a restaurant kitchen's food rotation. Fresh produce is on the counter (hot). After a few days it moves to the walk-in fridge (warm), then the freezer (cold or frozen), and finally the bin (delete). The chef doesn't move things by hand; the rotation rules do.

PUT _ilm/policy/logs-app-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink":     { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate":   { "number_of_replicas": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": { "searchable_snapshot": { "snapshot_repository": "my-repo" } }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

Where each action is allowed:

* In hot, only when rollover is also configured.

Key facts:

• Rollover conditions are OR-ed (max_age, max_docs, max_size, max_primary_shard_size, max_primary_shard_docs). min_* conditions are AND-ed floors. Elasticsearch never rolls over an empty index.
• Rollover needs a data stream or a write alias: index.lifecycle.rollover_alias plus a bootstrap index name-000001 with "is_write_index": true.
• migrate is implicit in warm and cold. It sets _tier_preference to the phase's tier. With no nodes of that tier, the shards stay where they are, which is a common "why isn't it moving?" moment. Disable it with "migrate": {"enabled": false} when using attribute-based allocate.
• ILM checks every indices.lifecycle.poll_interval (10m default). In a lab, set it to 10s to see transitions quickly.
• Attach a policy to new indices via the template (index.lifecycle.name). For existing indices, use PUT <index>/_settings {"index.lifecycle.name": "…"}.
• Changing a policy: an index already in a phase keeps the cached phase definition and picks up the new one at the next phase.

Bootstrap pattern without data streams (a classic task):

PUT _index_template/orders-tpl
{
  "index_patterns": ["orders-*"],
  "template": { "settings": {
    "index.lifecycle.name": "orders-policy",
    "index.lifecycle.rollover_alias": "orders" } }
}
PUT orders-000001
{ "aliases": { "orders": { "is_write_index": true } } }

Debugging ILM:

GET logs-app-prod/_ilm/explain             // phase, action, step, failed_step, step_info
POST logs-app-prod/_ilm/retry               // after fixing the cause of an ERROR step
POST _ilm/move/<index>                      // force a step (rare)
GET _ilm/status          POST _ilm/start    POST _ilm/stop
PUT _cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "10s" } }

Data stream lifecycle (DSL): the lighter alternative

For data streams you can skip ILM and just declare retention (plus optional downsampling). Elasticsearch handles rollover and deletion itself.

PUT _index_template/metrics-app-template
{
  "index_patterns": ["metrics-app-*"],
  "data_stream": {},
  "template": { "lifecycle": { "data_retention": "30d" } }
}
PUT _data_stream/metrics-app-prod/_lifecycle   { "data_retention": "7d" }
GET _data_stream/metrics-app-prod/_lifecycle/explain

If both are configured, ILM wins (index.lifecycle.prefer_ilm defaults to true). Choose ILM when you need tiers (warm, cold, frozen), shrink, forcemerge or searchable snapshots. Choose DSL when retention is all that matters.

📖 Index lifecycle management · ILM phases and actions · Rollover · Create an ILM policy · Tutorial: time series with data streams · Tutorial: time series without data streams · Check ILM status · Data stream lifecycle

2.6 Define and use index aliases

An alias is a secondary name for one or more indices or data streams.

Analogy: a phone's contact name. You call "Mum", not the number. When Mum changes numbers, you edit the contact once and nobody else has to change anything. Apps query products, and you swap which physical index sits behind it.

Core uses:

1. Zero-downtime reindex: build products-v2, then atomically move the alias.
2. Filtered views: an alias that only shows region: EU documents.
3. Rollover write target: is_write_index: true.
4. Search many indices under one name: logs-all → logs-2026.09.*.

// Atomic swap. All actions succeed or none do.
POST _aliases
{
  "actions": [
    { "remove": { "index": "products-v1", "alias": "products" } },
    { "add":    { "index": "products-v2", "alias": "products", "is_write_index": true } }
  ]
}

// Filtered alias with routing
POST _aliases
{
  "actions": [
    { "add": {
        "index": "orders",
        "alias": "orders-eu",
        "filter": { "term": { "region": "EU" } },
        "routing": "eu"
    } }
  ]
}

// Remove an index entirely and point the alias elsewhere, atomically
POST _aliases
{ "actions": [
  { "remove_index": { "index": "products-old" } },
  { "add": { "index": "products-v3", "alias": "products" } } ] }

GET _alias/products          GET products-v2/_alias          GET _cat/aliases?v

Rules:

• An alias pointing to several indices is read-only unless exactly one of them has is_write_index: true.
• Aliases can point to data streams, but you can't mix data streams and indices in one alias.
• Filtered aliases apply the filter to searches, counts and delete-by-query run through the alias.
• Aliases can be defined in index templates (template.aliases) so every new index joins automatically.
• An alias name can't equal an existing index name.

📖 Aliases · Aliases API · Privileges for data streams and aliases

2.7 Exercises: Data Management

1. Create component templates ct-1shard (1 shard, 0 replicas) and ct-app-mappings (@timestamp date, service keyword, msg text). Create index template app-logs for app-logs-* that creates a data stream, uses both components, and has priority 300. Index a document into app-logs-api and confirm the backing index has 1 shard.
2. Add to (1) a dynamic template: new strings ending in _id become keyword; all other new strings become text with a keyword sub-field named raw. Prove it with one document (remember to roll over).
3. Create ILM policy app-policy: roll over at 1 GB primary shard size or 1 day; delete 3 days after rollover. Attach it via the template, roll over manually and check _ilm/explain.
4. Index legacy has 5 shards. Produce legacy-1 with 1 shard without reindexing.
5. Indices movies-v1 and movies-v2 exist. Alias movies points to v1. Atomically repoint it to v2 and make v2 the write index. Then create filtered alias movies-recent showing only year >= 2020.

3. Data Ingestion & Processing

Analogy for the whole domain: a restaurant's back office. Deliveries arrive by different routes (ingest methods). Each crate is checked against the inventory sheet that says what goes on which shelf (mapping and field types). Some produce is washed, chopped and labelled before storage (ingest pipelines / Streams). When the shelving layout changes, the stock is moved to new shelves (Reindex) or relabelled in place (Update By Query).

3.1 Select and configure appropriate ingest methods

How to choose (exam-style reasoning):

• "Collect system logs from 200 hosts, centrally managed" → Elastic Agent with Fleet and the System integration.
• "Enrich events from a Kafka topic and send them to two clusters" → Logstash.
• "Index product data from our application" → client + _bulk, with an ingest pipeline for light cleanup.
• "Parse free-text logs quickly without writing grok by hand" → Streams.
• "Transform data already in Elasticsearch" → ingest pipeline + _reindex or _update_by_query.

The Bulk API is the workhorse behind almost every method:

POST _bulk?pipeline=clean-products
{ "index":  { "_index": "products", "_id": "1" } }
{ "sku": "A-1", "name": "Café crème", "price": 3.5 }
{ "create": { "_index": "logs-app-prod" } }
{ "@timestamp": "2026-09-28T10:00:00Z", "message": "hello" }
{ "update": { "_index": "products", "_id": "1" } }
{ "doc": { "price": 3.9 } }
{ "delete": { "_index": "products", "_id": "2" } }

The format is NDJSON: every line ends with a newline, including the last one. The response has a top-level errors: true/false, and each item carries its own status, so always check errors. Aim for batches of about 5–15 MB.

📖 Ingest overview · Ingest tools overview · Ingest architectures · Bulk API · Upload data files · OTLP endpoint · Tune for indexing speed

3.2 Define a mapping that satisfies a given set of requirements

A mapping is the schema: which fields exist, their types, and how each is indexed and analysed.

Analogy: a customs form. Each box says what the item is ("date", "exact code", "free text"). Customs (Lucene) stores it in the right way. A date written in the "free text" box can't be range-searched later, and you can't change the form once the parcel is shipped.

Explicit vs dynamic. Dynamic mapping guesses, for example "2026-09-28" becomes date and "hello" becomes text + .keyword. The exam almost always wants explicit mappings.

PUT articles
{
  "mappings": {
    "dynamic": "strict",
    "_source": { "excludes": ["internal_notes"] },
    "properties": {
      "title":    { "type": "text", "analyzer": "english",
                    "fields": { "raw": { "type": "keyword" },
                                "suggest": { "type": "search_as_you_type" } } },
      "author":   { "type": "keyword", "normalizer": "lowercase" },
      "body":     { "type": "text", "copy_to": "all_text" },
      "summary":  { "type": "text", "copy_to": "all_text" },
      "all_text": { "type": "text" },
      "views":    { "type": "integer", "null_value": 0 },
      "rating":   { "type": "half_float", "ignore_malformed": true },
      "published":{ "type": "date", "format": "strict_date_optional_time||epoch_millis" },
      "status":   { "type": "keyword", "doc_values": true, "index": true },
      "raw_html": { "type": "text", "index": false },
      "comments": { "type": "nested",
                    "properties": { "user": { "type": "keyword" },
                                    "text": { "type": "text" },
                                    "stars": { "type": "byte" } } },
      "labels":   { "type": "flattened" },
      "internal_notes": { "type": "text" }
    }
  }
}

Mapping parameters you must know:

What you can and can't change on an existing index:

PUT articles/_mapping
{ "properties": { "author": { "type": "keyword", "fields": { "text": { "type": "text" } } } } }
POST articles/_update_by_query?conflicts=proceed      // populates author.text for existing docs

Text analysis: custom analyzers

Analogy: a kitchen processing line. Character filters wash the vegetables (strip HTML, map characters). The tokenizer chops them into pieces (words). Token filters season each piece (lowercase, remove stop words, stem, add synonyms). The same line must be used for indexing and searching, or the pieces won't match.

PUT recipes
{
  "settings": {
    "analysis": {
      "char_filter": {
        "amp_to_and": { "type": "mapping", "mappings": ["& => and"] }
      },
      "filter": {
        "my_stop":  { "type": "stop", "stopwords": "_english_" },
        "my_syn":   { "type": "synonym_graph", "synonyms": ["courgette, zucchini", "aubergine, eggplant"] },
        "edge_3_10":{ "type": "edge_ngram", "min_gram": 3, "max_gram": 10 }
      },
      "analyzer": {
        "recipe_text": {
          "type": "custom",
          "char_filter": ["html_strip", "amp_to_and"],
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "my_stop", "porter_stem"]
        },
        "recipe_search": {
          "type": "custom", "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "my_syn", "porter_stem"]
        },
        "autocomplete": {
          "type": "custom", "tokenizer": "standard",
          "filter": ["lowercase", "edge_3_10"]
        }
      },
      "normalizer": {
        "lower_fold": { "type": "custom", "filter": ["lowercase", "asciifolding"] }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text", "analyzer": "recipe_text", "search_analyzer": "recipe_search",
        "fields": {
          "ac":  { "type": "text", "analyzer": "autocomplete", "search_analyzer": "standard" },
          "raw": { "type": "keyword", "normalizer": "lower_fold" }
        }
      }
    }
  }
}

POST recipes/_analyze
{ "analyzer": "recipe_text", "text": "<b>Courgette & Crème</b> brûlée recipes" }

POST recipes/_analyze
{ "field": "name.ac", "text": "Courgette" }

Analysis facts to remember:

• The default analyzer is standard (standard tokenizer + lowercase). Built-in analyzers include simple, whitespace, keyword, stop, pattern and language analyzers (english, french…).
• Synonyms at search time use synonym_graph in the search_analyzer. To update synonyms without reindexing, use the Synonyms API (PUT _synonyms/my-set) with "synonyms_set": "my-set" and "updateable": true on a search-time filter.
• Edge n-grams go at index time only; search with the standard analyzer, otherwise the query is n-grammed too.
• keyword fields are not analysed. Use a normalizer for case-insensitivity.
• Always test with _analyze before creating the real index.

📖 Mapping · Explicit mapping · Update mapping examples · Mapping parameters · Text analysis · Anatomy of an analyzer · Create a custom analyzer · Test an analyzer · Index and search analysis · Synonyms

3.3 Define and use appropriate field data types for indexing, search and aggregation

Analogy. Pick the container for what you'll do with the contents. Soup goes in a bowl (text, for searching within), coins in a counting tray (numeric, for maths and ranges), keys on labelled hooks (keyword, for exact lookup, sorting and grouping).

Arrays. There's no array type. Any field can hold several values of the same type.

object vs nested: the most tested pitfall

// Document:
{ "comments": [ { "user": "alice", "stars": 5 }, { "user": "bob", "stars": 1 } ] }

With object, Lucene stores comments.user: [alice, bob] and comments.stars: [5, 1], so the pairing is lost. A search for "alice AND stars 1" wrongly matches.

With nested, each comment is a hidden sub-document. You must use the nested query and the nested aggregation:

GET articles/_search
{
  "query": {
    "nested": {
      "path": "comments",
      "query": { "bool": { "must": [
        { "term": { "comments.user": "alice" } },
        { "range": { "comments.stars": { "gte": 4 } } } ] } },
      "inner_hits": {}
    }
  }
}

The limits are index.mapping.nested_fields.limit (50) and index.mapping.nested_objects.limit (10,000 per document). Other mapping explosion guards: index.mapping.total_fields.limit (1000) and index.mapping.depth.limit (20).

Runtime fields: schema on read

A field computed at query time from _source or other fields. There's nothing to reindex, but it's slower.

PUT articles/_mapping
{
  "runtime": {
    "day_of_week": {
      "type": "keyword",
      "script": { "source": "emit(doc['published'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ENGLISH))" }
    }
  }
}

// Or only for one search:
GET articles/_search
{
  "runtime_mappings": {
    "views_k": { "type": "double", "script": "emit(doc['views'].value / 1000.0)" }
  },
  "fields": ["views_k"],
  "aggs": { "by_day": { "terms": { "field": "day_of_week" } } }
}

📖 Field data types · keyword · text · Numeric · date · nested · flattened · Multi-fields · Runtime fields · Mapping limit settings

3.4 Use the Reindex API and Update By Query API

Analogy. Reindex is moving house: you pack every item from the old flat and unpack it into a new one that has a different layout (new mapping, new analyzer, new shard count). Update By Query is redecorating in place: same flat, you walk from room to room and change what matches.

Reindex

POST _reindex?wait_for_completion=false&slices=auto
{
  "source": {
    "index": "articles",
    "query": { "range": { "published": { "gte": "2025-01-01" } } },
    "_source": ["title", "author", "published", "views"],
    "size": 1000
  },
  "dest": {
    "index": "articles-v2",
    "pipeline": "normalize-articles",
    "op_type": "create"
  },
  "script": {
    "lang": "painless",
    "source": "ctx._source.views = ctx._source.views == null ? 0 : ctx._source.views; ctx._source.remove('internal_notes')"
  },
  "conflicts": "proceed",
  "max_docs": 100000
}

The destination is NOT created from the source's mapping. Create articles-v2 (or a template for it) first, otherwise dynamic mapping guesses the types.

Zero-downtime migration recipe: create v2 with the new mapping → _reindex → verify _count → atomic alias swap (2.6) → delete v1 if asked.

Update By Query

POST articles/_update_by_query?conflicts=proceed&wait_for_completion=false
{
  "query": { "term": { "status": "draft" } },
  "script": {
    "lang": "painless",
    "source": "ctx._source.status = params.s; ctx._source.tags = ctx._source.tags ?: []; ctx._source.tags.add('migrated')",
    "params": { "s": "archived" }
  }
}

// Re-run all docs through a pipeline (e.g. to populate a new field)
POST articles/_update_by_query?pipeline=add-reading-time
{ "query": { "bool": { "must_not": { "exists": { "field": "reading_time" } } } } }

// Pick up a newly added multi-field for all docs
POST articles/_update_by_query?refresh

Related APIs:

• POST <index>/_update/<id> with doc (partial), script, upsert or doc_as_upsert
• POST <index>/_delete_by_query
• ?refresh to make results immediately visible

Painless essentials: ctx._source.field, params.x, ctx._source.remove('f'), ctx._source.containsKey('f'), null-safe ?. and elvis ?:, ctx.op = 'noop' | 'delete'. In ingest pipelines, fields are on ctx.field directly (no _source).

📖 Reindex API · Update by query API · Delete by query API · Update API · Migrate data using reindex · Painless · Update documents with scripts · Tasks API

3.5 Define and use an ingest pipeline that satisfies a given set of requirements

An ingest pipeline is an ordered list of processors that runs on ingest nodes before a document is indexed.

Analogy: an airport baggage belt. Each station (processor) scans, relabels or reroutes the bag. If a station jams, the bag is either sent to a lost-luggage office (on_failure) or the whole belt stops (default: the document is rejected).

PUT _ingest/pipeline/web-logs
{
  "description": "Parse Apache-style access logs",
  "processors": [
    { "grok": {
        "field": "message",
        "patterns": ["%{IPORHOST:client.ip} %{USER:ident} %{USER:user.name} \\[%{HTTPDATE:ts}\\] \"%{WORD:http.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.status:int} (?:%{NUMBER:bytes:long}|-)"]
    } },
    { "date":   { "field": "ts", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"], "target_field": "@timestamp" } },
    { "remove": { "field": ["ts", "ident"], "ignore_missing": true } },
    { "set":    { "field": "event.ingested", "value": "{{{_ingest.timestamp}}}" } },
    { "lowercase": { "field": "http.method" } },
    { "geoip":  { "field": "client.ip", "target_field": "client.geo", "ignore_failure": true } },
    { "user_agent": { "field": "user_agent.original", "ignore_missing": true } },
    { "set": {
        "if": "ctx.http?.status != null && ctx.http.status >= 500",
        "field": "event.outcome", "value": "failure" } },
    { "script": {
        "lang": "painless",
        "source": "ctx.bytes_kb = ctx.bytes == null ? 0 : ctx.bytes / 1024.0"
    } },
    { "drop": { "if": "ctx.url?.original == '/healthz'" } }
  ],
  "on_failure": [
    { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } },
    { "set": { "field": "_index", "value": "failed-{{ _index }}" } }
  ]
}

Frequently used processors:

Common processor options: if (Painless condition, null-safe with ?.), ignore_failure, ignore_missing, on_failure (per processor), tag, description.

Error handling, in order of scope:

1. Processor-level ignore_failure: true skips silently.
2. Processor-level on_failure: [...] handles the error locally.
3. Pipeline-level on_failure is the catch-all. Metadata: _ingest.on_failure_message, _ingest.on_failure_processor_type, _ingest.on_failure_processor_tag.
4. Nothing catches it → the document is rejected. For data streams with a failure store, it goes to ::failures instead.

Testing, which you should always do:

POST _ingest/pipeline/web-logs/_simulate?verbose=true
{ "docs": [ { "_source": { "message": "1.2.3.4 - - [28/Sep/2026:10:00:00 +0000] \"GET /a HTTP/1.1\" 503 12" } } ] }

// Simulate with the pipelines that WOULD run for an index (defaults, final, templates)
POST _ingest/_simulate
{ "docs": [ { "_index": "web-2026", "_source": { "message": "…" } } ] }

Ways to apply a pipeline:

PUT web-2026/_settings { "index.default_pipeline": "web-logs" }

Enrich processor: a lookup table at ingest time

PUT users-ref/_doc/1   { "email": "a@x.com", "dept": "finance", "manager": "zoe" }

PUT _enrich/policy/users-policy
{ "match": { "indices": "users-ref", "match_field": "email", "enrich_fields": ["dept", "manager"] } }

POST _enrich/policy/users-policy/_execute        // builds the .enrich-* index; re-run after the source changes

PUT _ingest/pipeline/add-user-info
{ "processors": [ { "enrich": {
    "policy_name": "users-policy", "field": "user.email",
    "target_field": "user.info", "max_matches": 1, "ignore_missing": true } } ] }

Policy types: match (exact term), range (value inside a range), geo_match (point in shape). An enrich policy can't be updated. Delete it and recreate it; you have to remove the processors that use it first.

📖 Ingest pipelines · Processor reference · Error handling · Example: parse logs · Data enrichment · Set up an enrich processor · Simulate pipeline API · Grok debugger · Painless in ingest

3.6 Ingest and process data using Streams

Streams is Kibana's UI and API layer for managing log data end to end: getting data in, parsing it, routing it, mapping fields, setting retention and watching data quality. Behind the scenes it creates and manages ordinary Elasticsearch objects (data streams, index templates, ingest pipelines, lifecycle settings).

Analogy: a smart sorting office with a control panel. Instead of wiring each conveyor belt (template, pipeline, ILM) yourself, you use one panel: "parse these letters like this, send the ones from Paris to that shelf, keep them for 30 days". The panel builds the belts for you. Don't rewire the belts by hand behind its back.

Two kinds of streams (9.2–9.3)

A third kind, query streams (virtual, defined by an ES|QL query), appears in newer versions.

Typical workflow in the Streams UI

1. Get data in. For wired streams, turn on wired streams in Streams settings, then point Elastic Agent, Filebeat, the OTel Collector or Logstash at the logs index/stream. No template setup is needed. For classic streams, just open an existing data stream in Streams.
2. Partition (wired only). Define routing conditions such as service.name == "nginx" → logs.nginx. Partition by logical groups (team, technology, environment), not by high-cardinality fields. Aim for tens of partitions, not hundreds. Each child is a real data stream.
3. Process. Build a processing pipeline in the Processing tab from processors that mirror ingest processors (grok, dissect, date, set, rename, convert, remove, append, drop, math, json extract, redact, user agent, URI parts, enrich…). Add conditions to steps. The Data preview simulates results on sample documents before you save. With a generative-AI connector configured, Suggest a pipeline can generate grok/dissect rules for you.
4. Map fields (Schema tab). Promote extracted fields to proper mapped types (keyword, date, long…).
5. Configure retention (Data lifecycle tab). Choose inherit, a custom retention period (data stream lifecycle), or an existing ILM policy.
6. Monitor data quality (Data quality tab). Degraded documents are ones with ignored or malformed fields (_ignored); failed documents are kept in the failure store so you can fix the processor using the real failing documents.
7. Advanced tab. Inspect the managed index templates, component templates and pipelines Streams created.

Rules to remember:

• Streams-managed components are managed. Change them through the Streams UI or API, not with raw Elasticsearch APIs.
• Streams is a Kibana feature: its API is under kbn:/api/streams/... (Kibana API), not the Elasticsearch REST API. In Console, prefix with kbn:.
• Required privileges to manage streams include cluster manage_index_templates, manage_ingest_pipelines, manage_pipeline and read_pipeline, plus data-stream-level read, write, create, manage, monitor, manage_data_stream_lifecycle, read_failure_store, manage_failure_store and manage_ilm.
• Don't reroute or migrate existing data streams into the wired root; classic streams work in place.
• Because Streams is new and moving fast, exam tasks are most likely to use the UI flow: add a processor, verify in preview, set retention, check the failure store. Practise it on your lab with sample logs.

# Example Kibana API calls from Console (verify paths against the 9.3 Streams API reference)
GET kbn:/api/streams
GET kbn:/api/streams/logs

📖 Streams overview · Get data into Streams · Wired streams · Partition (organise) data · Process documents · Configure data lifecycle · Map fields · Manage data quality · Advanced settings · Streamlang processors · Streams Kibana API

3.7 Exercises: Ingestion & Processing

1. Create index shop with: name (full-text with the English analyzer, plus a case-insensitive exact sub-field name.exact), sku (exact), price (two decimals, cheap to store), tags (exact, not aggregatable), specs (arbitrary key/values, no mapping explosion), reviews (array of {user, stars} that must be queried together). Unknown fields must be rejected.
2. Create pipeline shop-clean that trims and upper-cases sku, converts price from string to float, sets ingested_at, drops docs whose status is test, and on any failure writes the error to ingest_error instead of rejecting. Simulate it with 2 docs.
3. Reindex shop into shop-v2 (identical mapping except name uses a custom analyzer with asciifolding) through shop-clean, only for docs with price > 0. Then point alias shop-live at shop-v2.
4. Add a name.ac autocomplete multi-field to shop-v2 and populate it for existing docs without reindexing.
5. Create an enrich policy from index suppliers (sku → supplier, country) and add it to shop-clean.
6. In Streams, open the logs-generic-default classic stream (index some raw log lines first), add a dissect processor that extracts level and msg, map level as keyword, set retention to 14 days, and inspect the failure store.

4. Searching Data

Analogy for the whole domain: a library. Full-text queries are asking the librarian "books about sailing". She interprets the words and ranks the answers by relevance. Term-level queries are the card catalogue: exact codes, no interpretation. Bool combines requests: "must be about sailing, must be in French, should be recent, must not be a children's book". Aggregations are the statistics desk: "how many books per genre, average page count". ES|QL is a spreadsheet-style pipeline over the same shelves. Semantic search is a librarian who understands meaning ("boat trips" finds "sailing"). Cross-cluster search asks sister libraries in other cities at the same time.

4.1 Search for terms and/or phrases in one or more fields

Full-text queries (analysed: the query text goes through the field's analyzer)

// Single field, any word (OR by default)
GET articles/_search
{ "query": { "match": { "title": "elastic search guide" } } }

// All words required, or a minimum
{ "query": { "match": { "title": { "query": "elastic search guide", "operator": "and" } } } }
{ "query": { "match": { "title": { "query": "elastic search guide", "minimum_should_match": "2" } } } }

// Typo tolerance
{ "query": { "match": { "title": { "query": "elastik", "fuzziness": "AUTO" } } } }

// Exact phrase, with slop (words may be up to N positions apart)
{ "query": { "match_phrase": { "body": { "query": "shard allocation", "slop": 2 } } } }

// Phrase prefix, "search as you type" style
{ "query": { "match_phrase_prefix": { "title": "quick bro" } } }

// Several fields, with boosting
{ "query": { "multi_match": {
    "query": "shard sizing",
    "fields": ["title^3", "summary^2", "body"],
    "type": "best_fields",
    "tie_breaker": 0.3 } } }

multi_match types:

Also know combined_fields (like cross_fields for fields sharing an analyzer), query_string (Lucene syntax with AND/OR/NOT, wildcards and field names; throws on bad syntax) and simple_query_string (forgiving, safe for user input: + | - " * ( )).

Term-level queries (NOT analysed: exact values on keyword, numeric, date, ip, boolean)

{ "query": { "term":  { "status": "published" } } }
{ "query": { "term":  { "status": { "value": "Published", "case_insensitive": true } } } }
{ "query": { "terms": { "tags": ["elasticsearch", "lucene"] } } }
{ "query": { "range": { "published": { "gte": "now-30d/d", "lt": "now/d" } } } }
{ "query": { "range": { "price": { "gt": 10, "lte": 100 } } } }
{ "query": { "exists": { "field": "author" } } }
{ "query": { "prefix": { "sku": "ABC-" } } }
{ "query": { "wildcard": { "sku": { "value": "A*-9?" } } } }
{ "query": { "regexp": { "sku": "A[0-9]{3}" } } }
{ "query": { "fuzzy": { "author": { "value": "jonh", "fuzziness": 1 } } } }
{ "query": { "ids": { "values": ["1", "7"] } } }
{ "query": { "terms_set": { "skills": { "terms": ["java","go","rust"], "minimum_should_match_field": "required" } } } }

The number-one trap: term on a text field. "Published" was indexed as published, so term: "Published" finds nothing. Use match on text, or term on the .keyword sub-field.

Date math: now-1d/d (rounded to the day), 2026-09-01||+1M, plus a time_zone parameter on range queries.

Shaping results: pagination, sorting, source filtering, highlighting

GET articles/_search
{
  "from": 0, "size": 10,
  "track_total_hits": true,
  "_source": ["title", "published"],
  "fields": ["title", "published", "day_of_week"],
  "sort": [ { "published": "desc" }, { "title.raw": "asc" }, "_score" ],
  "highlight": {
    "pre_tags": ["<mark>"], "post_tags": ["</mark>"],
    "fields": { "body": { "fragment_size": 150, "number_of_fragments": 3 } }
  },
  "query": { "match": { "body": "shard allocation" } }
}

• from + size is capped by index.max_result_window (10,000). For deep paging use search_after with a point in time (POST <index>/_pit?keep_alive=1m). Scroll is legacy for this purpose.
• You can't sort on text fields. Sort on a keyword sub-field, a number or a date.
• track_total_hits defaults to accurate counting up to 10,000; set true for an exact count.
• explain: true shows the score calculation. profile: true shows per-shard timings (or use Kibana's Search Profiler).

📖 Full-text queries · match · match_phrase · multi_match · query_string · Term-level queries · term · range · Paginate results · Sort results · Highlighting · The _search API

4.2 Boolean combinations of queries and filters

GET articles/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "body": "elasticsearch" } }
      ],
      "should": [
        { "match_phrase": { "title": { "query": "certified engineer", "boost": 2 } } },
        { "term": { "tags": "exam" } }
      ],
      "minimum_should_match": 1,
      "filter": [
        { "term":  { "status": "published" } },
        { "range": { "published": { "gte": "2025-01-01" } } }
      ],
      "must_not": [
        { "term": { "author": "spam-bot" } }
      ]
    }
  }
}

should semantics (subtle, and heavily tested):

• If the bool has no must or filter, at least one should must match (minimum_should_match defaults to 1).
• If there is a must or filter, should clauses are optional and only boost the score, unless you set minimum_should_match.

Query vs filter context rule of thumb: "how well does it match?" (words, relevance) goes in must/should. "Does it match, yes or no?" (status, dates, ranges, IDs) goes in filter/must_not. Filters are faster and cached.

Other compound queries:

• constant_score: wrap a filter and give every hit the same score.
• boosting: positive query, with a negative query that demotes (not excludes) matches.
• dis_max: best of several queries, plus tie_breaker.
• function_score / script_score: modify scores by recency, popularity (field_value_factor), decay functions or scripts.
• Named queries ("_name": "q1") report which clauses matched each hit.

Nesting example: "(title contains X OR summary contains X) AND year ≥ 2020 AND NOT draft":

{ "query": { "bool": {
    "must": [ { "bool": { "should": [
        { "match": { "title": "X" } }, { "match": { "summary": "X" } } ] } } ],
    "filter": [ { "range": { "year": { "gte": 2020 } } } ],
    "must_not": [ { "term": { "status": "draft" } } ] } } }

post_filter filters the hits but not the aggregations. Use it for faceted navigation, where facet counts shouldn't shrink when a facet is selected.

📖 Query and filter context · bool query · Compound queries · function_score · Filter search results (post_filter)

4.3 Write and execute metric and bucket aggregations

Analogy: sorting sweets. Bucket aggregations tip the sweets into bowls by colour, by size range, or by the day they were bought. Metric aggregations count, weigh or average what's in each bowl. Pipeline aggregations compare the bowls ("which bowl is heaviest?", "running total"). Bowls can hold smaller bowls (sub-aggregations).

GET kibana_sample_data_ecommerce/_search
{
  "size": 0,
  "query": { "range": { "order_date": { "gte": "now-30d/d" } } },
  "aggs": {
    "per_day": {
      "date_histogram": {
        "field": "order_date",
        "calendar_interval": "day",
        "time_zone": "Europe/Paris",
        "min_doc_count": 0,
        "extended_bounds": { "min": "now-30d/d", "max": "now/d" }
      },
      "aggs": {
        "revenue":     { "sum": { "field": "taxful_total_price" } },
        "unique_buyers": { "cardinality": { "field": "customer_id" } }
      }
    },
    "top_categories": {
      "terms": { "field": "category.keyword", "size": 5, "order": { "revenue": "desc" } },
      "aggs": {
        "revenue":   { "sum": { "field": "taxful_total_price" } },
        "price_stats": { "stats": { "field": "taxful_total_price" } },
        "p95":       { "percentiles": { "field": "taxful_total_price", "percents": [50, 95, 99] } },
        "best_order": { "top_hits": { "size": 1, "sort": [{ "taxful_total_price": "desc" }],
                                      "_source": ["order_id", "customer_full_name"] } }
      }
    },
    "price_bands": {
      "range": { "field": "taxful_total_price",
                 "ranges": [ { "to": 50 }, { "from": 50, "to": 150 }, { "from": 150 } ] }
    },
    "best_day": { "max_bucket": { "buckets_path": "per_day>revenue" } },
    "avg_daily_revenue": { "avg_bucket": { "buckets_path": "per_day>revenue" } }
  }
}

Metric aggregations: avg, sum, min, max, value_count, stats, extended_stats (std deviation, variance), cardinality (approximate distinct count, HyperLogLog++, precision_threshold), percentiles, percentile_ranks, median_absolute_deviation, weighted_avg, top_hits, top_metrics, geo_bounds, geo_centroid, scripted_metric.

Bucket aggregations:

Pipeline aggregations reference other aggregations by buckets_path (> for nesting, . for a metric's value, _count for doc count):

• Sibling (a new bucket next to the others): avg_bucket, max_bucket, min_bucket, sum_bucket, stats_bucket, percentiles_bucket.
• Parent (inside each histogram bucket): derivative, cumulative_sum, moving_fn, bucket_script, bucket_selector (keeps only matching buckets, like HAVING), bucket_sort (sort or truncate buckets), serial_diff.

"aggs": {
  "per_month": {
    "date_histogram": { "field": "order_date", "calendar_interval": "month" },
    "aggs": {
      "revenue":  { "sum": { "field": "taxful_total_price" } },
      "cum_rev":  { "cumulative_sum": { "buckets_path": "revenue" } },
      "big_only": { "bucket_selector": {
          "buckets_path": { "r": "revenue" }, "script": "params.r > 10000" } }
    }
  }
}

Aggregation facts:

• Set "size": 0 when you only need aggregations.
• You can't run terms on a text field (it has no doc values). Use .keyword.
• terms counts are approximate across shards (doc_count_error_upper_bound). Increase shard_size for accuracy. Use composite to list all buckets.
• Aggregations run on the documents matched by query. Use global to escape that scope, or post_filter to filter hits only.
• For nested fields, wrap sub-aggs in a nested aggregation.

📖 Aggregations · Aggregations tutorial (eCommerce) · Metric aggregations · Bucket aggregations · Pipeline aggregations · terms · date_histogram · composite

4.4 Write and execute ES|QL queries

ES|QL (Elasticsearch Query Language) is a piped language. Each command transforms a table and passes it to the next, like a Unix pipeline or a spreadsheet you keep refining.

Analogy: a kitchen assembly line. FROM grabs the ingredients, WHERE throws away what you don't need, EVAL chops new pieces, STATS weighs and groups, SORT arranges the plates, LIMIT serves the first N.

POST _query?format=txt
{
  "query": """
    FROM kibana_sample_data_logs
    | WHERE @timestamp >= NOW() - 7 days AND response.keyword != "200"
    | EVAL kb = bytes / 1024.0, day = DATE_TRUNC(1 day, @timestamp)
    | STATS errors = COUNT(*), avg_kb = AVG(kb), hosts = COUNT_DISTINCT(host.keyword) BY day, response.keyword
    | SORT day DESC, errors DESC
    | LIMIT 20
  """
}

format can be json (default), txt, csv, tsv, yaml and more. For long queries use POST _query/async and poll with GET _query/async/<id>. In Kibana, use Discover → ES|QL mode.

Commands you must know:

Functions to know: aggregations COUNT, COUNT_DISTINCT, SUM, AVG, MIN, MAX, MEDIAN, PERCENTILE, VALUES, TOP; grouping BUCKET(@timestamp, 1 hour); date DATE_TRUNC, DATE_EXTRACT, DATE_DIFF, DATE_FORMAT, NOW(); string CONCAT, TO_LOWER, SUBSTRING, LENGTH, SPLIT, STARTS_WITH, LIKE, RLIKE; conditionals CASE(cond, a, cond2, b, default), COALESCE; conversion TO_INTEGER, TO_DATETIME, TO_IP, :: casts; multi-value MV_COUNT, MV_AVG; full-text MATCH(field, "text"), the : operator (message:"timeout"), QSTR("…"), KQL("…"), plus METADATA _score for relevance.

ES|QL facts:

• Default LIMIT is 1000 rows, with a maximum of 10,000 by default. Always add an explicit LIMIT.
• Case matters for values, not for command names.
• Use parameters to avoid injection: "query": "FROM x | WHERE a == ?v", "params": [{"v": 5}].
• Cross-cluster: FROM cluster_a:logs-*, logs-* (requires remote clusters, see 5.2).
• ES|QL doesn't return _source documents like _search; it returns columns and rows.
• STATS without BY returns one row. Columns not in BY or not aggregated are dropped.

Query DSL → ES|QL translation practice:

📖 ES|QL overview · ES|QL reference · ES|QL source commands · ES|QL processing commands · ES|QL functions and operators · LOOKUP JOIN · ES|QL REST API · ES|QL for search · Using ES|QL in Discover

4.5 Implement semantic search

Semantic search matches meaning rather than exact words. Text is turned into embeddings (vectors) by an ML model, and queries are compared to documents in that vector space.

Analogy: a map of meaning. Each passage becomes a pin on a huge map where similar ideas sit close together. "How do I fix a flat tyre?" lands next to "puncture repair". Keyword search would miss that, because none of the words overlap.

Two kinds of embeddings:

The easy path: semantic_text (recommended and most exam-friendly)

semantic_text hides all the plumbing: chunking, calling the inference endpoint at ingest time, storing embeddings, and embedding the query at search time.

Step 1: an inference endpoint. There's a preconfigured default ELSER endpoint, so this step is optional. To create your own:

PUT _inference/sparse_embedding/my-elser
{
  "service": "elasticsearch",
  "service_settings": {
    "model_id": ".elser_model_2",
    "adaptive_allocations": { "enabled": true, "min_number_of_allocations": 0, "max_number_of_allocations": 4 },
    "num_threads": 1
  }
}

// Dense example with a built-in multilingual model
PUT _inference/text_embedding/my-e5
{ "service": "elasticsearch",
  "service_settings": { "model_id": ".multilingual-e5-small", "num_allocations": 1, "num_threads": 1 } }

GET _inference/_all
POST _inference/sparse_embedding/my-elser { "input": "test sentence" }

The cluster needs an ML node (or Elastic Inference Service on Cloud) and a license that allows ML models, so use the trial on your lab.

Step 2: a mapping with semantic_text:

PUT docs-semantic
{
  "mappings": {
    "properties": {
      "title":   { "type": "text" },
      "content": { "type": "semantic_text", "inference_id": "my-elser" },
      "body":    { "type": "text", "copy_to": "content" }
    }
  }
}

Omit inference_id to use the default endpoint. copy_to into a semantic_text field lets you keep a normal text field for BM25 alongside.

Step 3: index documents normally. Embeddings are generated automatically. Bulk-loading many documents is slow on small ML nodes, so be patient or use a small batch.

Step 4: query.

// Simplest: the match query works on semantic_text fields
GET docs-semantic/_search
{ "query": { "match": { "content": "how do I keep my cluster available when a zone fails?" } } }

// Dedicated semantic query
GET docs-semantic/_search
{ "query": { "semantic": { "field": "content", "query": "zone failure resilience" } },
  "highlight": { "fields": { "content": { "type": "semantic", "number_of_fragments": 2 } } } }

Hybrid search: BM25 + semantic, fused with RRF

GET docs-semantic/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        { "standard": { "query": { "match": { "title": "zone failure" } } } },
        { "standard": { "query": { "semantic": { "field": "content", "query": "zone failure" } } } }
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  }
}

Retrievers are the composable search abstraction: standard, knn, rrf (Reciprocal Rank Fusion), linear (weighted score combination), text_similarity_reranker (semantic reranking), rule.

The manual path (know it exists)

• dense_vector mapping (dims, similarity: cosine | dot_product | l2_norm | max_inner_product, index_options such as int8_hnsw/bbq_hnsw quantisation) + knn search (k, num_candidates, filter, query_vector or query_vector_builder).
• An inference processor in an ingest pipeline to generate embeddings, or bring your own vectors.
• sparse_vector field + sparse_vector query with inference_id.

GET my-vectors/_search
{
  "knn": {
    "field": "embedding",
    "query_vector_builder": { "text_embedding": { "model_id": "my-e5", "model_text": "zone failure" } },
    "k": 10, "num_candidates": 100,
    "filter": { "term": { "lang": "en" } }
  }
}

📖 Semantic search overview · Semantic search with semantic_text · semantic_text field · semantic query · Semantic search with the inference API · Inference API: default endpoints · ELSER · Hybrid search with semantic_text · Retrievers · kNN search · Dense vector · Sparse vector · Semantic reranking

4.6 Write and execute a query that searches across multiple clusters

Cross-cluster search (CCS) runs one request from the local cluster against indices on one or more remote clusters. Setting up the remote connection is covered in 5.2; here is the querying side.

Analogy: interlibrary loan. You ask your local branch; it forwards your request to the branches in Lyon and Berlin and hands you one merged answer.

// Syntax: <remote_alias>:<index_pattern>
GET cluster_b:logs-*/_search
{ "query": { "match": { "message": "timeout" } } }

// Local + several remotes in one request
GET logs-*,cluster_b:logs-*,cluster_c:logs-*/_search
{
  "size": 20,
  "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now-1h" } } } ] } },
  "aggs": { "by_cluster": { "terms": { "field": "_index" } } }
}

// All remotes: wildcard on the alias; exclude with a minus sign
GET *:logs-*,-cluster_c:*/_search

// ES|QL across clusters
POST _query?format=txt
{ "query": "FROM logs-*, cluster_b:logs-* METADATA _index | STATS c = COUNT(*) BY _index | LIMIT 50" }

// Async CCS for long searches
POST cluster_b:logs-*/_async_search?wait_for_completion_timeout=2s
{ "query": { "match_all": {} } }

// Check reachability and matching indices before searching
GET _resolve/cluster/cluster_b:logs-*,cluster_c:logs-*

What to know:

• Hits show the origin in _index as cluster_b:logs-2026…. The response's _clusters section reports total, successful, skipped, partial and failed.
• skip_unavailable (per remote cluster setting, default true): an unreachable remote is skipped instead of failing the whole search.
• ccs_minimize_roundtrips (default true): each remote reduces its own results, which means fewer network round trips.
• Security: the user needs read privileges on the remote indices. With the API-key model, that's the cross-cluster API key's access.search plus the local user's role (remote_indices). With the certificate model, it's a role of the same name on the remote cluster.
• Kibana data views can use cluster_b:logs-* patterns too.
• CCS works across versions within the compatibility window (see the CCS version compatibility table in the docs).

📖 Cross-cluster search · Resolve cluster before CCS · ES|QL cross-cluster · Async search

4.7 Exercises: Searching Data

Use kibana_sample_data_ecommerce, kibana_sample_data_flights and kibana_sample_data_logs. Check field names and types first with GET <index>/_mapping, since some sample fields are text + .keyword and others are plain keyword.

1. Flights: find flights whose OriginCityName or DestCityName contains "New York" (both fields, boost the destination ×2), that are not cancelled, with FlightDelayMin between 30 and 120 and a timestamp in the last 30 days. Return only FlightNum, Carrier, FlightDelayMin, sorted by delay descending.
2. eCommerce: find orders whose products.product_name contains the exact phrase "leather jacket" (slop 1). Highlight the matches.
3. eCommerce: revenue per week (taxful_total_price) for the last 8 weeks, with the week-over-week difference and a running total; keep only weeks above 5,000.
4. eCommerce: the top 5 manufacturer.keyword values by number of orders, each with average basket value and the 3 most frequent category.keyword values.
5. Logs, in ES|QL: per host, the count of 4xx and 5xx responses in the last 24 hours, the 95th-percentile bytes, sorted by error count, top 10.
6. Create a semantic_text index faq, index 5 short Q&A documents, and run a hybrid RRF query combining BM25 on question with semantic search on answer.
7. Across clusters: count documents per index on lab and remote for logs-* in one request, and make the search succeed even if remote is down.

5. Cluster Management

5.1 Monitor and maintain cluster health

Analogy: a hospital ward. Green means every patient (shard) has a bed and a backup nurse (replica). Yellow means every patient has a bed, but some backup nurses are missing: no data lost, but less safe. Red means at least one patient has no bed at all: a primary shard is unassigned and its data is unavailable. The allocation explain API is the doctor's chart that says why.

The monitoring toolkit

GET _cluster/health                                  // status, nodes, unassigned_shards, relocating…
GET _cluster/health/orders?level=shards              // per index or shard
GET _cluster/health?wait_for_status=green&timeout=30s
GET _health_report                                   // 9.x health indicators with diagnoses & remediation
GET _health_report/shards_availability

GET _cat/health?v
GET _cat/nodes?v&h=name,node.role,master,heap.percent,ram.percent,cpu,load_1m,disk.used_percent
GET _cat/indices?v&health=yellow&s=index
GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node&s=state
GET _cat/allocation?v                               // disk per node, shard counts
GET _cat/recovery?v&active_only=true
GET _cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected
GET _cat/pending_tasks?v
GET _nodes/stats/jvm,fs,indices,breaker
GET _cluster/stats?human
GET _tasks?actions=*reindex*&detailed
GET _cluster/allocation/explain                     // first unassigned shard, or specify one

📖 Cluster health API · Health report API · Cluster allocation explain API · cat APIs · Troubleshoot

Diagnosing yellow and red: the decision tree

1. GET _cluster/health: which status, and how many unassigned_shards?
2. GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED (in Console, use &s=state).
3. GET _cluster/allocation/explain with the index, shard and primary flag. Read allocate_explanation and each node's deciders.
4. Fix according to the cause:

POST _cluster/reroute?retry_failed=true

PUT _cluster/settings
{ "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } }

// Manual moves / last-resort recovery
POST _cluster/reroute
{ "commands": [
    { "move": { "index": "orders", "shard": 0, "from_node": "es01", "to_node": "es02" } },
    { "allocate_replica": { "index": "orders", "shard": 1, "node": "es03" } } ] }

Cluster settings: persistent survives a full restart; transient is deprecated. Set a setting to null to reset it to the default. GET _cluster/settings?include_defaults=true&flat_settings=true shows everything.

📖 Red or yellow cluster health · Diagnose unassigned shards · Fix watermark errors · Cluster-level shard allocation and routing settings · Cluster reroute API

Performance signals to watch

Monitoring solutions:

• Stack Monitoring: Elastic Agent or Metricbeat collects metrics into a (preferably separate) monitoring cluster; view them in Kibana → Stack Monitoring. Legacy internal collection is deprecated.
• AutoOps: Elastic's hosted, automated diagnosis with recommendations. Available for ECH, and via Cloud Connect for self-managed clusters.
• Kibana alerting rules on cluster health, CPU, disk and so on.

📖 Monitoring · Stack monitoring · Collect monitoring data with Elastic Agent · AutoOps · AutoOps vs Stack Monitoring · Slow logs · Update logging levels

Maintaining the cluster: snapshots, SLM and routine operations

Analogy: snapshots are incremental photo backups of your phone. The first one copies everything; later ones only copy new photos (new segments), yet every snapshot can restore the full album on its own.

// 1) Register a repository (shared FS: every node needs path.repo pointing to the same mount)
PUT _snapshot/my-repo
{ "type": "fs", "settings": { "location": "/usr/share/elasticsearch/snapshots/my-repo", "compress": true } }
POST _snapshot/my-repo/_verify

// 2) Take a snapshot
PUT _snapshot/my-repo/snap-2026.09.28?wait_for_completion=true
{ "indices": "orders,logs-app-*", "include_global_state": false,
  "metadata": { "taken_by": "exam", "reason": "before migration" } }

GET _snapshot/my-repo/_all        GET _snapshot/my-repo/snap-2026.09.28/_status
GET _cat/snapshots/my-repo?v

// 3) Restore (the target index must not exist and be open; rename or close/delete first)
POST _snapshot/my-repo/snap-2026.09.28/_restore
{
  "indices": "orders",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "include_aliases": false,
  "index_settings": { "index.number_of_replicas": 0 },
  "ignore_index_settings": ["index.refresh_interval"]
}

Snapshot lifecycle management (SLM):

PUT _slm/policy/nightly
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "my-repo",
  "config": { "indices": ["*"], "include_global_state": true },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}
POST _slm/policy/nightly/_execute          // run now
GET  _slm/policy/nightly                   // last_success / last_failure
GET  _slm/stats
POST _slm/_execute_retention

The SLM schedule uses Quartz cron (seconds first: sec min hour day month weekday), and the name supports date math.

Other routine operations:

• Searchable snapshots: mount an index straight from a repository. Use POST _snapshot/my-repo/snap/_mount?storage=full_copy (cold) or shared_cache (frozen), or let ILM do it.
• Repository types: fs, url (read-only), s3, gcs, azure, source (source-only).
• Restore a whole cluster or feature states: include_global_state, feature_states (e.g. security, kibana).
• Close and open indices to cut resource use: POST idx/_close.
• Force merge read-only indices: POST idx/_forcemerge?max_num_segments=1.
• Clear caches: POST idx/_cache/clear. Refresh/flush: POST idx/_refresh, POST idx/_flush.
• Rolling restarts and upgrades: disable replica allocation → flush → restart → re-enable (see 1.2).

📖 Snapshot and restore · Manage repositories · Shared file system repository · Create snapshots and SLM · Restore a snapshot · Searchable snapshots · Add and remove nodes

5.2 Perform cross-cluster operations

Cross-cluster operations depend on remote clusters: a named connection from the local cluster (which starts the requests) to a remote cluster (which holds the data). Two features use it:

• Cross-cluster search (CCS): query remote data (queries in 4.6).
• Cross-cluster replication (CCR): continuously copy leader indices on the remote into read-only follower indices on the local cluster. It needs a Platinum/Enterprise license or the trial.

Analogy. CCS is a phone call to another library: you ask and hear the answer, but nothing is copied. CCR is a newspaper subscription: the publisher (leader) keeps sending every new edition, and your copy (follower) is read-only.

Step 1: connect a remote cluster

Connection modes: sniff (default; connects directly to seed nodes and discovers gateway nodes) or proxy (a single address, e.g. through a load balancer). ECH, ECE and ECK effectively use proxy mode.

API key setup, self-managed:

# On the REMOTE cluster (elasticsearch.yml): enable the remote cluster server interface
remote_cluster_server.enabled: true
remote_cluster.port: 9443
xpack.security.remote_cluster_server.ssl.enabled: true
xpack.security.remote_cluster_server.ssl.key: certs/remote.key
xpack.security.remote_cluster_server.ssl.certificate: certs/remote.crt

// On the REMOTE: create a cross-cluster API key that limits what the local cluster may do
POST _security/cross_cluster/api_key
{
  "name": "from-lab",
  "access": {
    "search":      [ { "names": ["logs-*", "orders"] } ],
    "replication": [ { "names": ["orders", "metrics-*"] } ]
  }
}

# On EVERY node of the LOCAL cluster: store the encoded key in the keystore, then reload
bin/elasticsearch-keystore add cluster.remote.cluster_b.credentials

POST _nodes/reload_secure_settings

// On the LOCAL cluster (it also needs to trust the remote's CA:
// xpack.security.remote_cluster_client.ssl.certificate_authorities)
PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.cluster_b.seeds": ["remote-es01:9443"],
    "cluster.remote.cluster_b.skip_unavailable": true
  }
}

// Proxy mode alternative
PUT _cluster/settings
{ "persistent": { "cluster.remote.cluster_b.mode": "proxy",
                  "cluster.remote.cluster_b.proxy_address": "lb.example.com:9443" } }

GET _remote/info          // "connected": true, mode, num_nodes_connected, cluster_credentials

The same can be done in Kibana: Stack Management → Remote Clusters. The local nodes that make the connections need the remote_cluster_client role, which is included by default when no roles are set explicitly.

Local user privileges for the API key model: add remote_indices to the user's role on the local cluster:

PUT _security/role/ccs-reader
{
  "cluster": [],
  "indices": [ { "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] } ],
  "remote_indices": [
    { "clusters": ["cluster_b"], "names": ["logs-*"], "privileges": ["read", "read_cross_cluster", "view_index_metadata"] }
  ]
}

For CCR, the local user's role needs remote_indices with cross_cluster_replication and cross_cluster_replication_internal privileges, plus the local manage_ccr cluster privilege. The docs list the exact privileges per model.

📖 Remote clusters · Security models · Connection modes · Self-managed remote clusters · Add remote clusters with API key auth · Add remote clusters with TLS certificate auth · Migrate from certificate to API key · Create cross-cluster API key

Step 2a: cross-cluster search

See 4.6 for the queries. Per-remote options: skip_unavailable (default true); cluster.remote.<alias>.transport.compress. Check connectivity with GET _remote/info and GET _resolve/cluster/<alias>:*.

Step 2b: cross-cluster replication

Prerequisites: licence (Platinum/Enterprise or trial) on both clusters; the remote connection from the follower (local) cluster to the leader; soft deletes enabled on the leader index (the default since 7.0); compatible versions (the follower may be the same or newer).

// Follow one specific leader index (run on the FOLLOWER / local cluster)
PUT orders-copy/_ccr/follow?wait_for_active_shards=1
{
  "remote_cluster": "cluster_b",
  "leader_index": "orders",
  "max_read_request_operation_count": 5120,
  "settings": { "index.number_of_replicas": 0 }
}

// Auto-follow: automatically follow NEW leader indices (and data streams) that match patterns
PUT _ccr/auto_follow/logs-follower
{
  "remote_cluster": "cluster_b",
  "leader_index_patterns": ["logs-*", "metrics-*"],
  "leader_index_exclusion_patterns": ["logs-debug-*"],
  "follow_index_pattern": "{{leader_index}}-dr"
}

Managing replication:

GET orders-copy/_ccr/stats           GET orders-copy/_ccr/info           GET _ccr/stats
POST orders-copy/_ccr/pause_follow
POST orders-copy/_ccr/resume_follow

// Turn a follower into a normal writable index (e.g. failover), in order:
POST orders-copy/_ccr/pause_follow
POST orders-copy/_close
POST orders-copy/_ccr/unfollow
POST orders-copy/_open

GET _ccr/auto_follow            POST _ccr/auto_follow/logs-follower/pause
DELETE _ccr/auto_follow/logs-follower

CCR facts:

• Followers are read-only. Writing to them fails.
• Auto-follow only picks up leader indices created after the pattern exists. Follow existing ones explicitly.
• Replication is pull-based: the follower fetches operations from the leader's shard history (soft deletes, retained by shard history retention leases).
• Mapping and settings changes on the leader are replicated automatically. Some follower settings can't be changed.
• Data streams are replicated through auto-follow patterns, which replicate the backing indices and create a matching follower data stream.
• Uses: disaster recovery (active–passive), bringing data closer to users (geo read replicas), and centralised reporting.
• Bi-directional replication is possible with two auto-follow setups and distinct write targets (e.g. aliases or data streams per region).
• The ILM action unfollow converts followers automatically as they age.

📖 Cross-cluster replication · Set up CCR · CCR prerequisites · Configure privileges for CCR · Create a follower index · Create an auto-follow pattern · Manage CCR · Uni-directional disaster recovery · Bi-directional disaster recovery

Other cross-cluster operations to recognise:

• Reindex from remote: _reindex with source.remote (HTTP, port 9200; needs reindex.remote.whitelist). This one doesn't use remote-cluster connections.
• Restore a snapshot into another cluster: register the same repository (read-only, "readonly": true, on the second cluster) and restore.
• Cross-cluster ES|QL and EQL, and Kibana data views with cluster:index patterns.

5.3 Exercises: Cluster Management

1. Your cluster is yellow. Find which shards are unassigned and why, then fix it without adding nodes and without deleting data.
2. Index hot-data is red after node es03 died permanently. A snapshot from last night exists. Recover it as hot-data.
3. Register an fs repository backups, snapshot only indices matching orders* without global state, and create an SLM policy running every day at 02:15 that keeps between 3 and 10 snapshots for 14 days.
4. A node shows disk.used_percent: 96, and writes to logs-x fail with index read-only / allow delete. Explain what happened and fix it.
5. Connect cluster lab to cluster remote as dr using API-key authentication, permitting search on logs-* and replication of orders.
6. On lab, make orders-dr follow remote's orders. Then fail over: turn orders-dr into a writable index.
7. Configure auto-follow so every new logs-* index on remote is replicated to lab with the suffix -copy.

6. Security & Access Control

Analogy for the whole domain: an office building. Authentication is the badge check at the door: who are you? (native users, API keys, LDAP/AD, SAML, OIDC, PKI, service accounts). Authorization / RBAC is which doors your badge opens: roles bundle privileges on the cluster, on indices (even specific documents and fields inside a room), and in Kibana features and spaces. TLS is the armoured corridor between buildings (nodes) and to visitors (clients). Audit logs are the CCTV.

6.1 Implement role-based access control (RBAC)

The model

user ──(has)──► roles ──(grant)──► privileges on resources
   ▲                                   ├─ cluster:   monitor, manage, manage_ilm, manage_index_templates, manage_security…
   │                                   ├─ indices:   read, write, create_doc, manage, view_index_metadata, all…
   │                                   │             + field-level security (FLS) + document-level security (DLS)
   │                                   ├─ applications: Kibana features & spaces
   │                                   └─ remote_indices / remote_cluster (for CCS/CCR)
role mapping (LDAP groups, SAML attributes, PKI DN…) ──► roles

Built-in roles worth knowing: superuser, kibana_admin, kibana_system (for Kibana's own service connection), viewer (read-only, including Kibana), editor, monitoring_user, remote_monitoring_agent, ingest_admin, snapshot_user, machine_learning_admin/_user, transform_admin, beats_system, logstash_system. Built-in users: elastic (superuser), kibana_system, logstash_system, beats_system, apm_system, remote_monitoring_user.

Privileges, the most-used ones:

Create roles and users

PUT _security/role/sales_analyst
{
  "description": "Read EU sales, hide PII, use Discover & Dashboards in the 'sales' space",
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["sales-*", "orders"],
      "privileges": ["read", "view_index_metadata"],
      "field_security": {
        "grant": ["*"],
        "except": ["customer.email", "customer.phone", "card.*"]
      },
      "query": { "term": { "region": "EU" } }
    },
    {
      "names": ["sales-reports"],
      "privileges": ["read", "write", "create_index"]
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["feature_discover.read", "feature_dashboard.all"],
      "resources": ["space:sales"]
    }
  ],
  "metadata": { "owner": "exam" }
}

POST _security/user/jane
{
  "password": "Str0ng-Passw0rd!",
  "roles": ["sales_analyst"],
  "full_name": "Jane Doe",
  "email": "jane@example.com",
  "metadata": { "team": "sales" }
}

POST _security/user/jane/_password    { "password": "An0ther-Str0ng-One!" }
POST _security/user/jane/_disable     POST _security/user/jane/_enable
GET  _security/user/jane              GET _security/role/sales_analyst
DELETE _security/role/old_role

Field-level security (FLS) uses field_security.grant / except. Wildcards are allowed. Metadata fields such as _id are always visible.

Document-level security (DLS) uses a query, which can be a templated query based on the user:

"query": { "template": { "source": { "term": { "owner": "{{_user.username}}" } } } }

When a user has several roles, the permissions are the union of all of them. A role without DLS on an index grants access to all documents of that index, even if another role restricts it. That's a classic exam trap.

Kibana privileges: application: "kibana-.kibana". Use "privileges": ["all"] or ["read"] for base access, or per-feature ones (feature_discover.read, feature_dashboard.all, feature_visualize.all…) for specific features. resources holds the spaces: space:<id>, or * for all. It's easiest to create these in Kibana → Stack Management → Roles and then check the result with GET _security/role/<name>.

Test what you granted

GET _security/_authenticate                          // who am I, which roles, which realm

GET _security/user/_has_privileges
{ "cluster": ["monitor"], "index": [ { "names": ["sales-2026"], "privileges": ["read", "write"] } ] }

// Run a request AS another user (requires the run_as privilege) to verify their view
GET sales-*/_search
# Header:  es-security-runas-user: jane

The fastest real-world check is to log into Kibana as the user, or call the API with -u jane:password, and try the allowed and the forbidden actions.

API keys

POST _security/api_key
{
  "name": "ingest-shipper",
  "expiration": "30d",
  "role_descriptors": {
    "writer": {
      "cluster": ["monitor"],
      "indices": [ { "names": ["logs-app-*"], "privileges": ["create_doc", "auto_configure"] } ]
    }
  },
  "metadata": { "app": "shipper" }
}
// → returns id, api_key, encoded   → header:  Authorization: ApiKey <encoded>

GET _security/api_key?name=ingest-shipper
DELETE _security/api_key  { "name": "ingest-shipper" }             // invalidate
POST _security/api_key/_query { "query": { "term": { "name": "ingest-shipper" } } }

An API key's privileges are the intersection of the role_descriptors and the creator's privileges at creation time. Without role_descriptors, it gets a snapshot of the owner's permissions.

Role mappings (external realms)

PUT _security/role_mapping/ldap-admins
{
  "enabled": true,
  "roles": ["superuser"],
  "rules": { "field": { "groups": "cn=admins,ou=groups,dc=example,dc=com" } }
}

PUT _security/role_mapping/saml-readers
{
  "enabled": true,
  "roles": ["viewer"],
  "rules": { "all": [ { "field": { "realm.name": "saml1" } },
                      { "field": { "groups": "readers" } } ] }
}

Rules support field, any, all and except. Fields: username, dn, groups, realm.name, metadata.*. Use role_templates for dynamic role names.

Realms, in order of evaluation (order): reserved → file → native (both on by default) → ldap, active_directory, pki, saml, oidc, kerberos, jwt. SAML and OIDC work through Kibana. Service accounts (e.g. elastic/kibana, elastic/fleet-server) use service tokens instead of passwords.

📖 Users and roles · User roles · Defining roles · Role structure · Built-in roles · Security privileges · Kibana privileges · Kibana role management · Document and field level security · Privileges for data streams and aliases · Role mapping · Native realm · Built-in users · Service accounts · Elasticsearch API keys · Submit requests on behalf of other users

6.2 Secure a cluster for production

Analogy: moving from a house party to a bank. At a party the door is open and everyone knows each other (development mode). A bank has locked doors (authentication), badges per room (RBAC), armoured transport between branches (transport TLS), bulletproof glass at the counter (HTTP TLS), a vault for secrets (keystore), cameras (audit logs), and a building inspection before opening (bootstrap checks).

Layer 1: security on, with TLS everywhere

Since 8.0, a fresh self-managed install auto-configures security: it generates certificates, enables TLS on the transport and HTTP layers, prints the elastic password, and gives you an enrollment token for Kibana and additional nodes. Know how to do it manually too:

# 1. CA + transport (node-to-node) certificates
bin/elasticsearch-certutil ca --out elastic-stack-ca.p12
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --out elastic-certificates.p12
# 2. HTTP certificates (interactive wizard: CSR or CA-signed, hostnames, IPs)
bin/elasticsearch-certutil http
# 3. Store keystore passwords
bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

# elasticsearch.yml
xpack.security.enabled: true
xpack.security.enrollment.enabled: true

xpack.security.transport.ssl.enabled: true               # REQUIRED for multi-node clusters with security
xpack.security.transport.ssl.verification_mode: certificate   # or full (also checks hostnames)
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true                    # HTTPS for REST clients & Kibana
xpack.security.http.ssl.keystore.path: certs/http.p12

# Passwords & tokens
bin/elasticsearch-reset-password -u elastic                 # or -i for interactive
bin/elasticsearch-reset-password -u kibana_system
bin/elasticsearch-create-enrollment-token -s kibana         # or -s node
bin/elasticsearch-service-tokens create elastic/kibana my-token

# kibana.yml (secured connection to Elasticsearch)
elasticsearch.hosts: ["https://es01:9200"]
elasticsearch.serviceAccountToken: "AAEAAWVs..."            # preferred over username/password
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/http_ca.crt"]
server.ssl.enabled: true                                     # HTTPS for browsers → Kibana
server.ssl.certificate: /etc/kibana/certs/kibana.crt
server.ssl.key: /etc/kibana/certs/kibana.key
xpack.encryptedSavedObjects.encryptionKey: "<32+ chars>"     # needed for alerting and saved-object secrets
xpack.security.encryptionKey: "<32+ chars>"
xpack.reporting.encryptionKey: "<32+ chars>"

Rules:

• With security enabled, a multi-node cluster must have transport TLS. Otherwise nodes refuse to form a cluster once they bind to a non-loopback address (production mode).
• Secrets belong in the keystore (elasticsearch-keystore), never in plain elasticsearch.yml. Reloadable secure settings can be refreshed with POST _nodes/reload_secure_settings.
• Don't use the elastic superuser for daily work or for Kibana. Create personal admin users, and use service accounts or API keys for applications.
• Anonymous access stays disabled unless explicitly required.

Layer 2: production-mode node configuration and bootstrap checks

As soon as network.host binds to a non-loopback address, Elasticsearch runs in production mode and enforces bootstrap checks. A failing check stops the node from starting.

cluster.name: prod-search
node.name: es-prod-01
node.roles: [ master, data_hot, ingest ]
network.host: 10.0.1.11            # _site_, _local_, specific IP…
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["10.0.1.11", "10.0.1.12", "10.0.1.13"]
cluster.initial_master_nodes: ["es-prod-01", "es-prod-02", "es-prod-03"]   # remove after first start
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
action.destructive_requires_name: true     # no DELETE on wildcards / _all (the default is true in 8+)

Layer 3: least privilege, network restrictions and auditing

Least privilege checklist:

• Personal admin accounts instead of the shared elastic account.
• Shippers (Agent, Beats, Logstash) get API keys with create_doc + auto_configure on their own data streams only.
• Applications get dedicated users or API keys limited to their indices.
• Analysts get read + view_index_metadata, with FLS/DLS where needed, and Kibana privileges limited to specific spaces.
• action.destructive_requires_name: true prevents DELETE *.

IP filtering (self-managed; on Cloud use traffic filters or network security):

xpack.security.transport.filter.allow: ["10.0.1.0/24"]
xpack.security.transport.filter.deny: _all
xpack.security.http.filter.allow: ["10.0.0.0/16"]
xpack.security.http.filter.deny: _all

Audit logging (needs a Platinum/Enterprise licence or the trial):

xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.include: ["access_denied", "authentication_failed", "connection_denied", "tampered_request", "run_as_denied", "security_config_change"]
xpack.security.audit.logfile.events.emit_request_body: false

The output goes to <cluster>_audit.json in the logs directory. Ignore policies (xpack.security.audit.logfile.events.ignore_filters.*) reduce noise. Kibana has its own audit log (xpack.security.audit.enabled in kibana.yml).

Other production hardening:

• Encrypt data at rest via the file system or disk (Elasticsearch doesn't encrypt shards itself). On Cloud, use customer-managed keys.
• Keep Elasticsearch off the public internet. Put it behind a firewall, load balancer or VPN, and expose only Kibana over HTTPS if needed.
• Set a session timeout in Kibana (xpack.security.session.idleTimeout, lifespan).
• Rotate certificates before they expire. The docs cover rotating with the same CA and with a different CA.
• Snapshot the security feature state (feature_states: ["security"]) so users and roles can be restored.
• For FIPS environments, use FIPS 140-2 mode with approved algorithms and keystores.

📖 Security overview · Self-managed security setup · Automatic security setup · Minimal security setup · Set up transport TLS · Set up HTTPS · Configure security in Kibana · Manage TLS encryption · Update TLS certificates · Secure settings (keystore) · IP filtering (self-managed) · Audit logging · Enable audit logging · Important system configuration · Bootstrap checks · Important settings · Kibana session management · Tutorial: self-managed secure install

6.3 Exercises: Security

1. Create role logs_reader: read and view metadata on logs-*, no access to the field user.password_hash, only documents where service.environment is production, plus Discover (read) in the Kibana space ops. Create user ana with this role and verify with _has_privileges and by logging in.
2. Create role ingest_writer that can only append documents to data streams matching logs-app-*, and create an API key limited to it that expires in 7 days.
3. User bob has roles A (DLS: team: blue on projects) and B (plain read on projects). Which documents can he see, and why? Fix it so he sees only team: blue.
4. Map LDAP group cn=sre,ou=groups,dc=corp to roles monitoring_user and logs_reader.
5. On a three-node lab started without security, enable security with transport TLS (own CA) and HTTPS, reset the elastic password, connect Kibana with a service account token, and turn on audit logging for authentication failures and denied access.
6. List the bootstrap checks that would fail on a VM with vm.max_map_count=65530, ulimit -n 4096 and swap enabled with bootstrap.memory_lock: true, and fix each one.

7. Final checklist and study plan

7.1 A four-week plan

7.2 "Can I do this from a blank Console with only the docs?"

• ☐ Explain node roles, tiers, primaries vs replicas, and the write and read paths
• ☐ Configure forced awareness and allocation filtering; explain a yellow cluster
• ☐ Create an index with custom analyzers, multi-fields, nested fields and strict mapping
• ☐ Component + index templates with priority; simulate them
• ☐ Dynamic templates with correct ordering
• ☐ A data stream from a template, with ILM rollover/warm/delete; _ilm/explain
• ☐ Alias swap, filtered alias, write index
• ☐ Choose an ingest method for a scenario
• ☐ Reindex with query, script and pipeline; update-by-query to populate a new multi-field
• ☐ A pipeline with grok/dissect, date, set, conditionals, on_failure, enrich; simulate it
• ☐ Streams: processing, retention and data quality in the UI
• ☐ match, match_phrase, multi_match, term/range, bool with filter/should logic
• ☐ Terms/date_histogram/range buckets with metrics and pipeline aggs
• ☐ ES|QL with WHERE, EVAL, STATS…BY, SORT, LIMIT, DISSECT and LOOKUP JOIN
• ☐ semantic_text + inference endpoint + semantic/match query + RRF hybrid
• ☐ Remote cluster (API key) + CCS + CCR follower + auto-follow + failover
• ☐ Diagnose unassigned shards with allocation explain; fix watermark blocks
• ☐ Snapshot repository, snapshot, restore with rename, SLM policy
• ☐ Roles with FLS/DLS and Kibana privileges, users, API keys, role mappings
• ☐ TLS (transport + HTTP), keystore, bootstrap checks, audit logging

7.3 Further references

• 📖 Elastic Documentation home: the only resource allowed in the exam
• 📖 Elasticsearch REST API reference
• 📖 Kibana API reference
• 📖 Elasticsearch release notes: skim 9.0 → 9.3 for new features and breaking changes
• 🎓 Elastic Training: Elasticsearch Engineer course: the official course aligned to the exam
• 🧪 Elasticsearch Labs: tutorials on semantic search, ES|QL and retrievers
• 💬 Elastic Discuss forum: exam-experience threads
• 📝 Community practice exercises by Guido Lena Cota (written for the 8.x exam, still useful for templates, mappings and search): Store data · Deploy and operate a cluster