Skip to content

Conversation

@original-brownbear
Copy link
Contributor

@original-brownbear original-brownbear commented Jul 17, 2020

We can't just assume a fixed number for the overall file count.
Depending on how the merging/flushing works out we won't always have
4 files for the index across all versions, systems etc.
Also, we could have x-pack concurrently create some system indices
which could mess up the total numbers here.
Fixed by only snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Closes #59767

Backports:

We can't just assume a fixed number for the overall file count.
Depending on how the merging/flushing works out we won't always have
4 files for the index across all versions, systems etc.
Also, we could have x-pack concurrently create some system indices
which could mess up the total numbers here.
Fixed by only snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Closes elastic#59767
@original-brownbear original-brownbear added >docs General docs changes :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 v7.10.0 labels Jul 17, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-docs (>docs)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@elasticmachine elasticmachine added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Docs Meta label for docs team labels Jul 17, 2020
Copy link
Contributor

@jrodewig jrodewig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 LGTM.

Thanks for fixing. I hadn't considered that this stat may not be predictable.

Copy link
Contributor

@lockewritesdocs lockewritesdocs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@original-brownbear, thank you for fixing this issue! It took some iterating with @jrodewig to fix the original test cases. The tests passed the CI checks, so I'm curious why it's now causing errors. I'll backport this change after it's merged.

@original-brownbear original-brownbear merged commit 3cef123 into elastic:master Jul 17, 2020
@original-brownbear original-brownbear deleted the 59767 branch July 17, 2020 14:34
@original-brownbear
Copy link
Contributor Author

Thanks @jrodewig @lockewritesdocs !

I'll backport this change after it's merged.

Thanks, then I won't backport here and leave it to you :)

lockewritesdocs pushed a commit to lockewritesdocs/elasticsearch that referenced this pull request Jul 17, 2020
We can't just assume a fixed number for the overall file count.
Depending on how the merging/flushing works out we won't always have
4 files for the index across all versions, systems etc.
Also, we could have x-pack concurrently create some system indices
which could mess up the total numbers here.
Fixed by only snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Closes elastic#59767
lockewritesdocs pushed a commit to lockewritesdocs/elasticsearch that referenced this pull request Jul 17, 2020
We can't just assume a fixed number for the overall file count.
Depending on how the merging/flushing works out we won't always have
4 files for the index across all versions, systems etc.
Also, we could have x-pack concurrently create some system indices
which could mess up the total numbers here.
Fixed by only snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Closes elastic#59767
lockewritesdocs pushed a commit to lockewritesdocs/elasticsearch that referenced this pull request Jul 17, 2020
We can't just assume a fixed number for the overall file count.
Depending on how the merging/flushing works out we won't always have
4 files for the index across all versions, systems etc.
Also, we could have x-pack concurrently create some system indices
which could mess up the total numbers here.
Fixed by only snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Closes elastic#59767
lockewritesdocs pushed a commit that referenced this pull request Jul 17, 2020
Introduce a fix to tests by snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Co-authored-by: Armin Braun <[email protected]>
lockewritesdocs pushed a commit that referenced this pull request Jul 17, 2020
Introduce a fix to tests by snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Co-authored-by: Armin Braun <[email protected]>
lockewritesdocs pushed a commit that referenced this pull request Jul 17, 2020
Introduce a fix to tests by snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Co-authored-by: Armin Braun <[email protected]>
@original-brownbear original-brownbear restored the 59767 branch January 6, 2021 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >docs General docs changes Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Docs Meta label for docs team v7.10.0 v8.0.0-alpha1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] Failures in DocsClientYamlTestSuiteIT {yaml=reference/snapshot-restore/apis/get-snapshot-status-api/line_309}

5 participants