Making Elasticsearch cheaper without deleting data

Every Elasticsearch cost conversation starts the same way. The bill has grown faster than the team expected, someone is asked to bring it down, and the fastest lever within reach is retention. Ninety days becomes thirty. Debug logs get dropped at the pipeline. Tracing gets sampled harder.

It works. The bill goes down the following month, and nobody notices anything is wrong until the first incident that needs data from six weeks ago, or the first auditor who asks for a year of authentication events, or the first investigation where the interesting activity started well before the alert fired.

That is the trade nobody writes down: you paid for the cluster so you could answer questions, and the first thing you cut was its ability to answer them.

There is almost always a cheaper thing to cut first. In rough order of how much money they free up per hour of work, here is what we go through before anyone touches a retention setting.

1. Shard sizing, before anything else

Shard count is the most common source of waste we see, and it is invisible on the storage bill — it shows up as nodes you had to add because heap pressure was killing the cluster.

Every shard costs heap even when nobody is querying it. A cluster with thousands of 200 MB daily indices is spending most of its memory on bookkeeping. The standard guidance still holds: aim for shards in the tens of gigabytes, and keep shard count proportional to heap rather than to how many indices somebody created.

GET _cat/shards?v&h=index,shard,prirep,store,node&s=store:desc
GET _cat/nodes?v&h=name,heap.percent,ram.percent,disk.used_percent

If the average shard is under a gigabyte, fixing that is usually worth more than everything else on this list combined — and it makes queries faster rather than slower, which is the opposite of what retention cuts do.

2. Tiering, so old data stops sitting on expensive disks

Most clusters we look at store a year of data on the same class of hardware, because that is how the cluster was built on day one and nothing forced a change.

Data tiers exist precisely for this. The shape that works:

Tier	What lives there	Storage
Hot	Data still being written and frequently queried	Fast local SSD
Warm	Read-only, still queried regularly	Cheaper disk, fewer replicas
Cold	Rarely queried, must stay searchable	Fully mounted searchable snapshot
Frozen	Compliance and investigation lookback	Partially mounted, object storage

The frozen tier is the one that changes the economics of retention. Data lives in object storage and is fetched into a local cache on demand. Queries are slower — seconds to tens of seconds rather than milliseconds — which is completely acceptable for the queries you actually run against year-old data. You are comparing that latency against the alternative, which is that the data does not exist.

An ILM policy that moves through those phases:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "7d" }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "shrink": { "number_of_shards": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "objectstore" }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "objectstore" }
        }
      },
      "delete": { "min_age": "395d", "actions": { "delete": {} } }
    }
  }
}

Note what the cold and frozen phases do to replicas. Once an index is backed by a snapshot in object storage, the snapshot is the durable copy. You are no longer paying to keep a second full copy on cluster disks for redundancy. On a large estate that alone is a substantial share of the storage bill.

3. Mappings, where the waste is quiet

Dynamic mapping is convenient and expensive. It indexes everything, in every way, whether or not anyone will ever search on it.

Two settings do most of the work:

• "index": false on fields nobody filters or searches — identifiers you only ever display, verbose message bodies you only read after retrieving the document.
• "doc_values": false on fields nobody aggregates, sorts, or scripts on.

{
  "properties": {
    "http.request.body.content": { "type": "text", "index": false },
    "event.original":            { "type": "keyword", "index": false, "doc_values": false },
    "user.name":                 { "type": "keyword", "ignore_above": 256 }
  }
}

ignore_above matters more than it looks. Without it, one malformed log line containing a 40 KB “username” gets indexed as a keyword term, and you pay for that in every segment merge from then on.

For metrics specifically, downsampling on a time series data stream replaces raw points with statistical aggregates at a coarser interval. Dashboards looking at last quarter do not need per-second resolution, and the storage difference is an order of magnitude.

Newer versions also offer synthetic _source, which reconstructs the document from doc values instead of storing a second copy of the original JSON. Check what your version and licence tier support before planning around it — the availability of this one has moved between releases.

4. Only now, retention

After the first three, retention becomes a real decision instead of a panic measure. You are choosing how long to keep data that now costs a fraction of what it did, on a tier priced for the access pattern it actually gets.

That is usually the point where the honest answer is that you can keep more data than before, not less — and still spend less than you were.

How to tell which one applies to you

Three numbers, in this order:

1. Average primary shard size. Under a few gigabytes means problem one.
2. Share of data on the hot tier. Over half means problem two.
3. Index size versus raw ingested bytes. Much above 1:1 means problem three.

None of these require access to your data to answer, which is why the free review we offer starts with exactly these three and takes under an hour.

---

Doing this on your own cluster and want a second opinion? A free SIEM and cluster review is 60–90 minutes with a senior engineer and a written one-page findings sheet.