Snapshot creation with wait_for_completion: response time longer than snapshot duration #65661
Description
Elasticsearch version:
7.10.0
Plugins installed:
repository-s3
JVM version:
openjdk version "15.0.1" 2020-10-20
Description of the problem including expected versus actual behavior:
When executing a snapshot creation command with wait_for_completion=true query parameter, I noticed that the response is returned much later than the effective snapshot duration.
Indeed, the response is returned after 3 minutes when the snapshot duration is 1 or 2 seconds.
The snapshot repository is a S3 repository.
ES cluster is hosted on Kubernetes.
Same behavior with ES 7.9 on AWS, so I guess this is not related to the hosting platform.
Steps to reproduce:
$ ES_HOST=http://localhost:9200
$ INDEX=test
$ printf -v BODY '{"indices":"%s"}' "$INDEX"
$ time curl -XPUT ${ES_HOST}/_snapshot/test/test?wait_for_completion=true \
-H 'Content-Type: application/json' \
-d "$(echo $BODY)"
$ {"snapshot":{"snapshot":"test","uuid":"7CLDN7wSQxGA4QOo-RURkA","version_id":7100099,"version":"7.10.0","indices":["test"],"data_streams":[],"include_global_state":true,"state":"SUCCESS","start_time":"2020-12-01T09:31:44.898Z","start_time_in_millis":1606815104898,"end_time":"2020-12-01T09:31:45.298Z","end_time_in_millis":1606815105298,"duration_in_millis":400,"failures":[],"shards":{"total":2,"failed":0,"successful":2}}}
real 3m1,495s
user 0m0,034s
sys 0m0,008sIs this due to the fact that the snapshot data transfer time to the s3 bucket is actually longer than the snapshot duration ?
I'm currently working on a cronjob to snapshot from cluster A and restore to cluster B, and I need to know if I can rely on wait_for_completion or if I should manage the wait time myself (meaning I would trigger the restore as soon as the snapshot state is SUCCESS).
Thank you for your help