Skip to content

Conversation

@dadoonet
Copy link
Contributor

Probably when we updated Azure SDK, we introduced a regression.
Actually, we are not able to remove files anymore.

For example, if you register a new azure repository, the snapshot service tries to create a temp file and then remove it.
Removing does not work and you can see in logs:

[2016-05-18 11:03:24,914][WARN ][org.elasticsearch.cloud.azure.blobstore] [azure] can not remove [tests-ilmRPJ8URU-sh18yj38O6g/] in container {elasticsearch-snapshots}: The specified blob does not exist.

This fix deals with that. It now list all the files in a flatten mode, remove in the full URL the server and the container name.

As an example, when you are removing a blob which full name is https://dpi24329.blob.core.windows.net/elasticsearch-snapshots/bar/test you need to actually call Azure SDK with bar/test as the path, elasticsearch-snapshots is the container.

Related to #16472.
Related to #18436.

Backport of #18451 in 2.x branch

To test it, I ran some manual tests:

On my laptop, create a file /path/to/azure/config/elasticsearch.yml:

cloud.azure.storage.default.account: ACCOUNT
cloud.azure.storage.default.key: KEY

Run AzureRepositoryF#main() with -Des.cluster.routing.allocation.disk.threshold_enabled=false -Des.path.home=/path/to/azure/ options.

Then run:

curl -XDELETE localhost:9200/foo?pretty
curl -XDELETE localhost:9200/_snapshot/my_backup1?pretty
curl -XPUT localhost:9200/foo/bar/1?pretty -d '{
 "foo": "bar"
}'
curl -XPOST localhost:9200/foo/_refresh?pretty
curl -XGET localhost:9200/foo/_count?pretty
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
   "type": "azure"
}'

curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1?pretty&wait_for_completion=true"
curl -XDELETE localhost:9200/foo?pretty
curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1/_restore?pretty&wait_for_completion=true"
curl -XGET localhost:9200/foo/_count?pretty

Then check files we have on azure platform using the console.
Then run:

curl -XDELETE localhost:9200/_snapshot/my_backup1/snap1?pretty

Then check files we have on azure platform using the console and verify that everything has been cleaned.

Probably when we updated Azure SDK, we introduced a regression.
Actually, we are not able to remove files anymore.

For example, if you register a new azure repository, the snapshot service tries to create a temp file and then remove it.
Removing does not work and you can see in logs:

```
[2016-05-18 11:03:24,914][WARN ][org.elasticsearch.cloud.azure.blobstore] [azure] can not remove [tests-ilmRPJ8URU-sh18yj38O6g/] in container {elasticsearch-snapshots}: The specified blob does not exist.
```

This fix deals with that. It now list all the files in a flatten mode, remove in the full URL the server and the container name.

As an example, when you are removing a blob which full name is `https://dpi24329.blob.core.windows.net/elasticsearch-snapshots/bar/test` you need to actually call Azure SDK with `bar/test` as the path, `elasticsearch-snapshots` is the container.

Related to elastic#16472.
Related to elastic#18436.

Backport of elastic#18451 in 2.x branch

To test it, I ran some manual tests:

On my laptop, create a file `/path/to/azure/config/elasticsearch.yml`:

```yml
cloud.azure.storage.default.account: ACCOUNT
cloud.azure.storage.default.key: KEY
```

Run `AzureRepositoryF#main()` with `-Des.cluster.routing.allocation.disk.threshold_enabled=false -Des.path.home=/path/to/azure/` options.

Then run:

```sh
curl -XDELETE localhost:9200/foo?pretty
curl -XDELETE localhost:9200/_snapshot/my_backup1?pretty
curl -XPUT localhost:9200/foo/bar/1?pretty -d '{
 "foo": "bar"
}'
curl -XPOST localhost:9200/foo/_refresh?pretty
curl -XGET localhost:9200/foo/_count?pretty
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
   "type": "azure"
}'

curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1?pretty&wait_for_completion=true"
curl -XDELETE localhost:9200/foo?pretty
curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1/_restore?pretty&wait_for_completion=true"
curl -XGET localhost:9200/foo/_count?pretty
```

Then check files we have on azure platform using the console.
Then run:

```
curl -XDELETE localhost:9200/_snapshot/my_backup1/snap1?pretty
```

Then check files we have on azure platform using the console and verify that everything has been cleaned.
@dadoonet
Copy link
Contributor Author

@imotov I backported #18451 for 2.x branch. Would you mind giving a quick look?

@imotov
Copy link
Contributor

imotov commented May 27, 2016

LGTM

@imotov imotov removed their assignment May 27, 2016
@imotov imotov removed the review label May 27, 2016
@dadoonet dadoonet merged commit 92382d8 into elastic:2.x May 27, 2016
@clintongormley clintongormley added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Cloud Azure labels Feb 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>bug :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v2.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants