Since Elasticsearch 6.1.0 environment variable substitutions in lists do not work

[vbohata]

In elasticsearch 6.1.0 and 6.1.1 I can not use variable substitutions for x-pack related configuration in elasticsearch. Also mentioned here: https://discuss.elastic.co/t/x-pack-6-1-0-does-not-handle-variable-substitutions/111953
xpack: security: authc: realms: ssl: certificate_authorities: [ "${ELASTICSEARCH_CONFIG_PATH}/certs/myca.crt" ]
I am receiving:
java.nio.file.NoSuchFileException: /app/volumes/config/${ELASTICSEARCH_CONFIG_PATH}/certs/myca.crt
Last known working version is 6.0.1. I am quite surprised this bug passed release testing cycle so I tried to find if this is a feature but it seems it is not.

[jasontedor]

On its face this looks like an X-Pack issue which we do not handle here as this is for open source Elasticsearch only. However, this is a core feature implemented in open source Elasticsearch.

Have you set ELASTICSEARCH_CONFIG_PATH? What does

$ echo $ELASTICSEARCH_CONFIG_PATH

show? Are you sure that you exported the environment variable? If you start the process (suspend it before it fails) and get its pid, does /proc/<PID>/environ show that environment variable as being exposed to Elasticsearch?

Are you sure that you did not mean ES_CONF_PATH?

I am quite surprised this bug passed release testing cycle so I tried to find if this is a feature but it seems it is not.

I am not sure what this mean. Bugs happen. So does user error. Let's figure it out together.

[vbohata]

I run ES in docker. I use my entrypoint script which exports ELASTICSEARCH_CONFIG_PATH variable before starting elasticsearch. I know the variable is exported (even see it in environ). To fix the issue I just need to build the docker image with older ES - 6.0.1 (not 6.1.0 or 6.1.1). The issue can be easily replicated by very simple installation of elasticsearch + x-pack without the docker. I do not know if this is elasticsearch or x-pack problem or their integration issue, but in pure elasticsearch var substition works (for example for path.data).

I am not sure what this mean. Bugs happen. So does user error. Let's figure it out together.

I wanted to say it looked like a feature for the first time - no one reported it, I expected a lot of people will be affected by this.

[jasontedor]

Would you share a Dockerfile including entrypoint and reproduction of this issue please?

[vbohata]

Dockerfile:

FROM myrepository/elastic/common

ARG APP_VERSION
ARG FILES_REPOSITORY

RUN yum install -y java-1.8.0-openjdk && yum clean all

# install elasticsearch
RUN cd /opt && \
    wget -q "${FILES_REPOSITORY}/elasticsearch/elasticsearch-${APP_VERSION}.tar.gz" && \
    tar -zxf elasticsearch-${APP_VERSION}.tar.gz && \
    rm -f elasticsearch-${APP_VERSION}.tar.gz && \
    mv /opt/elasticsearch* /opt/elasticsearch && \
    chmod 755 -- /opt/elasticsearch/bin/*

# set required env vars
ENV PATH=/opt/elasticsearch/bin:$PATH \
    JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk

# install plugins
RUN for plugin in ingest-geoip ingest-user-agent x-pack; do elasticsearch-plugin install --batch "${FILES_REPOSITORY}/elasticsearch/plugins/${plugin}-${APP_VERSION}.zip"; done

COPY files/app /app
RUN chmod 755 /app/init/entrypoint.sh

CMD [ "/app/init/entrypoint.sh" ]

Part of the entrypoint.sh (without comments, var, ... checking):

#!/bin/bash

. /app/init/libs/bootstrap-all.inc.sh

. /app/init/vars.inc.sh


# check users, set stdin/out perms, set TZ, ...
init_base_environment

es_opts="\
-Enetwork.host=0.0.0.0 \
-Enetwork.bind_host=0.0.0.0 \
-Etransport.bind_host=0.0.0.0 \
-Ehttp.bind_host=0.0.0.0 \
-Epath.data=${_ELASTICSEARCH_DATA_PATH} \
-Epath.logs=${_ELASTICSEARCH_LOGS_PATH} \
-Ehttp.port=${ELASTICSEARCH_CLIENTS_HTTP_PORT} \
-Etransport.tcp.port=${ELASTICSEARCH_NODES_TRANSPORT_PORT} \
-Etransport.profiles.client.port=${ELASTICSEARCH_CLIENTS_TRANSPORT_PORT} \
-Etransport.profiles.client.xpack.security.type=client \
-Ebootstrap.memory_lock=true \
"

[[ -n "$ELASTICSEARCH_PUBLIC_HOST" ]] && es_opts="-Enetwork.publish_host=\"${ELASTICSEARCH_PUBLIC_HOST}\" -Etransport.profiles.client.publish_host=\"${ELASTICSEARCH_PUBLIC_HOST}\" $es_opts"
[[ -n "$ELASTICSEARCH_INTERNODE_HOST" ]] && es_opts="-Etransport.publish_host=\"${ELASTICSEARCH_INTERNODE_HOST}\" -Ehttp.publish_host=\"${ELASTICSEARCH_INTERNODE_HOST}\" $es_opts"
[[ -n "$ELASTICSEARCH_CLUSTER_NAME" ]] && es_opts="-Ecluster.name=${ELASTICSEARCH_CLUSTER_NAME} $es_opts"
[[ -n "$ELASTICSEARCH_NODE_NAME" ]] && es_opts="-Enode.name=${ELASTICSEARCH_NODE_NAME} $es_opts"


# security overrides
if is_true "$ELASTICSEARCH_DISABLE_SECURITY_OVERRIDES"; then
  log_warn "!!! WARNING: BASIC SECURITY OVERRIDES ARE DISABLED !!!"
else
  log_info "Setting basic security overrides"

es_opts="$es_opts \
-Expack.ssl.key=${_ELASTICSEARCH_CLIENTS_SSL_KEY_FILE} \
-Expack.ssl.certificate=${_ELASTICSEARCH_CLIENTS_SSL_CERT_FILE} \
-Expack.ssl.certificate_authorities=${_ELASTICSEARCH_CLIENTS_SSL_CA_FILE} \
-Expack.ssl.supported_protocols=TLSv1.2 \
-Expack.ssl.client_authentication=required \
-Expack.security.enabled=true \
-Expack.security.http.ssl.enabled=true \
-Expack.security.http.ssl.client_authentication=optional \
-Expack.security.http.ssl.supported_protocols=TLSv1.2 \
-Expack.security.transport.ssl.enabled=true \
-Expack.security.transport.ssl.key=${_ELASTICSEARCH_NODES_SSL_KEY_FILE} \
-Expack.security.transport.ssl.certificate=${_ELASTICSEARCH_NODES_SSL_CERT_FILE} \
-Expack.security.transport.ssl.certificate_authorities=${_ELASTICSEARCH_NODES_SSL_CA_FILE} \
-Expack.security.transport.ssl.supported_protocols=TLSv1.2 \
-Expack.security.transport.ssl.client_authentication=required \
-Etransport.profiles.client.xpack.security.ssl.key=${_ELASTICSEARCH_CLIENTS_SSL_KEY_FILE} \
-Etransport.profiles.client.xpack.security.ssl.certificate=${_ELASTICSEARCH_CLIENTS_SSL_CERT_FILE} \
-Etransport.profiles.client.xpack.security.ssl.certificate_authorities=${_ELASTICSEARCH_CLIENTS_SSL_CA_FILE} \
-Etransport.profiles.client.xpack.security.ssl.supported_protocols=TLSv1.2 \
-Etransport.profiles.client.xpack.security.ssl.client_authentication=optional \
-Expack.security.authc.token.enabled=true \
"
fi


# defaults
ELASTICSEARCH_HEAP_SIZE="${ELASTICSEARCH_HEAP_SIZE:-1g}"


ES_JAVA_OPTS="-Xms${ELASTICSEARCH_HEAP_SIZE} -Xmx${ELASTICSEARCH_HEAP_SIZE} $ES_JAVA_OPTS"
export ES_JAVA_OPTS="-Des.cgroups.hierarchy.override=/ -Dlog4j2.disable.jmx=true $ES_JAVA_OPTS"

export ES_HOME="/opt/elasticsearch"

export ES_PATH_CONF="${_ELASTICSEARCH_CONFIG_PATH}"

# useful for path reference in config files, where we can use this variable ... useful - this is independent of ES own ES_PATH_CONF var which could be renamed in future versions again
export ELASTICSEARCH_CONFIG_PATH="$_ELASTICSEARCH_CONFIG_PATH"


cd /opt/elasticsearch
switch_app "./bin/elasticsearch --verbose -p /var/run/elastic/elasticsearch.pid $es_opts"





... where switch_app is:
function switch_app {
  log_info "Switching to application process"
  eval "exec gosu elastic $*" 
}

To reproduce the issue just try to use ${ELASTICSEARCH_CONFIG_PATH} in the arbitrary place under xpack: configuration. For example to set certificate_authorities.

[jasontedor]

I want to help you but this is too far from a useable reproduction. For example, you are trying to switch to the elastic user yet you never created this user. Yes, I could do this, but this is the Nth hurdle in trying to get a working reproduction from what you've provided where now N is too large (I do not know your base image but gosu is not installed, wget is not installed, you have referenced several scripts in the entrypoint that are not provided here, etc. (really, there's more)). I need to put this back on you: please provide a simple reproduction that I can use to debug this. It should not take me more than a few minutes from what you provide to have a working reproduction of the issue that I can iterate on.

[tvernum]

I can reproduce, it seems to be exclusively a problem with array settings, so my guess is that it's a by-product of #26878

In 6.0.1 running with this config file:

discovery.zen.ping.unicast.hosts: [ "${THIS_DOES_NOT_EXIST}" ]

would fail to start with:

Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'THIS_DOES_NOT_EXIST'

In 6.1, it starts, but logs:

[WARN ][o.e.d.z.UnicastZenPing   ] failed to resolve host [${THIS_DOES_NOT_EXIST}]

[tvernum]

It seems to have been a somewhat consious decision in #26878
https://github.com/elastic/elasticsearch/pull/26878/files#diff-ec9e18970c4ce90d89639c46bb07218eR1206

Assigning to @s1monw to comment.

[jasontedor]

Thanks for triaging this one @tvernum.

Thanks for the report @vbohata, no further action is needed from you, we will take it from here.