Cluster state does not always contain correct node roles #71385
Description
Elasticsearch version (bin/elasticsearch --version): 7.11.1, likely others
Plugins installed: []
JVM version (java -version): Bundled
OS version (uname -a if on a Unix-like system): CentOS Linux 7 (Core) 3.10.0-693.21.1.el7.x86_64
Description of the problem including expected versus actual behavior:
A cluster that was fully upgraded to a data-tiers-based 7.11.1 was seen to still have nodes with the data role in the cluster state:
$ cat nodes.json| jq '.nodes[].version' | uniq -c
31 "7.11.1"
$ cat nodes.json| jq '.nodes[].settings.node.roles' -cMr | sort | uniq -c
16 ["data_hot","data_content"]
8 ["data_warm"]
2 ["ingest","transform","remote_cluster_client"]
3 ["master"]
2 ["ml","ingest","transform","remote_cluster_client"]
$ cat cat/cat_nodes.txt | awk 'NR!=1 {print $5}' | sort | uniq -c
12 d
9 hs
2 ilrt
2 irt
3 m
3 w
There should be no d nodes in this cluster any more.
Steps to reproduce:
Unknown at the moment, this is exactly what #64693 is supposed to address.