Search contexts (scrolls, pits) are not cleared from node if an index is relocated to another node and then deleted

[andreidan]

Elasticsearch Version

8.x, 9.x

Installed Plugins

No response

Java Version

bundled

OS Version

darwin

Problem Description

Stale data could be left on nodes after an index is relocated to different nodes, if scrolls/pits were opened on the source node.

Note that the data will remain on the source node even if the index is deleted.

This is because when we clear the contexts here we ignore the NO_LONGER_ASSIGNED reason - but this is a feature, we should NOT be freeing the context for the NO_LONGER_ASSIGNED reason because if we were to remove the contexts in case of NO_LONGER_ASSIGNED the existing opened PITs will not work anymore (searches on those existing PITs will recevei SearchPhaseExecutionException: all shards failed )

The solution here, perhaps something that warrants more discussion, is to transfer the existing contexts to the new nodes where the index relocates. (one could also argue that a possible solution is to enhance the delete index API to free up all context in the cluster - perhaps call something like https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/search/ClearScrollController.java#L150)

Current workaround to free up the space on the source nodes (the original nodes that hosted the index and where scrolls are still open, even after the index was deleted) is to restart the nodes as we clear the contexts on node stop. WARNING - this will clear ALL the contexts on the node that restarts so existing open PITs will start failing.

Steps to Reproduce

Run 2 elasticsearch nodes, node-1 and node-2

PUT /docs
{
    "settings": {
        "index.number_of_shards": 1,
        "index.number_of_replicas": 0,
        "index.routing.allocation.include._name": "node-1"
    }
}


POST /docs/_doc
{
    "hello": "world",
    "number": 1256
}

POST /_sql?format=txt
{
  "query": "SELECT * FROM docs ORDER BY number",
  "fetch_size": 1,
  "page_timeout" :"23h",
  "request_timeout":"23h"
}

GET /_nodes/stats?filter_path=**.search.open_contexts,nodes.*.name

GET /_cat/allocation?v&h=shards,disk.indices,disk.used,disk.avail,disk.total,node

GET /_cat/shards?v

** BEGIN optional and alternative path to SQL **
POST /docs/_pit?keep_alive=24m

POST /_search?pretty&error_trace
{
    "size": 100,
    "query": {
        "match" : {
            "hello" : "world"
        }
    },
    "pit": {
            "id":  " PIT OBTAINED BEFORE",
            "keep_alive": "24m"
    }
}
**END optional step**

// relocate index to the other node 
PUT /docs/_settings
{
    "index.routing.allocation.include._name": "node-2"
}

GET /_cat/shards?v

DELETE /docs

30 minutes after deletion, check logs for node-1 and notice

[2024-11-04T15:52:07,853][WARN ][o.e.i.c.IndicesClusterStateService] [node-1] [[docs/cuXdvCMwQFSeF4nbdn5BCw]] failed to lock all shards for index - timed out after [30m]]org.elasticsearch.env.ShardLockObtainFailedException: [docs][0]: obtaining shard lock for [process pending deletes] timed out after [60000ms], lock already held for [closing shard] with age [249531ms]
        at [email protected]/org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:1023)
        at [email protected]/org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:898)
        at [email protected]/org.elasticsearch.env.NodeEnvironment.lockAllForIndex(NodeEnvironment.java:840)
        at [email protected]/org.elasticsearch.indices.IndicesService.processPendingDeletes(IndicesService.java:1363)
        at [email protected]/org.elasticsearch.indices.cluster.IndicesClusterStateService$1.doRun(IndicesClusterStateService.java:422)

You can also check the data folder for node-1 and notice the index files are still present there. Something along the lines of

> tree data/indices

data/indices
└── aiwh1cDhRGeCcelbhdoQbg
    ├── 0
    │   ├── _state
    │   │   ├── retention-leases-2.st
    │   │   └── state-1.st
    │   ├── index
    │   │   ├── _0.cfe
    │   │   ├── _0.cfs
    │   │   ├── _0.si
    │   │   ├── _1.cfe
    │   │   ├── _1.cfs
    │   │   ├── _1.si
    │   │   ├── segments_3
    │   │   └── write.lock
    │   └── translog
    │       ├── translog-3.tlog
    │       └── translog.ckp
    └── _state
        └── state-3.st

7 directories, 13 files

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)