Skip to content

Conversation

@edsavage
Copy link
Contributor

@edsavage edsavage commented Jun 19, 2023

The DRA upload pipelines are currently downloading required artifacts from multiple pipeline steps. While this is not an error or causing any harm it is slightly more efficient to only download the artifacts once, from the create_dra_artifacts step.

Logs showing the duplicated downloads:

$ trap 'kill -- $' INT TERM QUIT; buildkite-agent artifact download "build/distributions/*" .
--
  | ./.buildkite/scripts/steps/upload_dra_to_s3.sh
  | 2023-06-18 01:25:31 INFO   Searching for artifacts: "build/distributions/*"
  | 2023-06-18 01:25:31 INFO   Found 22 artifacts. Starting to download to: /buildkite/builds/bk-agent-elastic-k8s-1687051503887884368/elastic/ml-cpp-snapshot-builds
  | 2023-06-18 01:25:32 INFO   Successfully downloaded "build/distributions/dependencies-8.9.0-SNAPSHOT.csv" 2100 bytes
  | 2023-06-18 01:25:34 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-darwin-aarch64.zip" 37342780 bytes
  | 2023-06-18 01:25:35 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-nodeps.zip" 42407945 bytes
  | 2023-06-18 01:25:35 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-darwin-aarch64.zip" 37342780 bytes
  | 2023-06-18 01:25:35 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-windows-x86_64.zip" 41699852 bytes
  | 2023-06-18 01:25:35 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-darwin-x86_64.zip" 56105585 bytes
  | 2023-06-18 01:25:37 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-windows-x86_64.zip" 83354676 bytes
  | 2023-06-18 01:25:38 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-linux-aarch64.zip" 44600895 bytes
  | 2023-06-18 01:25:38 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-linux-aarch64.zip" 44600895 bytes
  | 2023-06-18 01:25:38 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-darwin-x86_64.zip" 101249325 bytes
  | 2023-06-18 01:25:38 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-darwin-x86_64.zip" 101249325 bytes
  | 2023-06-18 01:25:38 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-darwin-aarch64.zip" 108725412 bytes
  | 2023-06-18 01:25:38 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-darwin-aarch64.zip" 108725412 bytes
  | 2023-06-18 01:25:39 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-darwin-x86_64.zip" 56105585 bytes
  | 2023-06-18 01:25:42 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-linux-x86_64.zip" 242501094 bytes
  | 2023-06-18 01:25:44 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-deps.zip" 381987903 bytes
  | 2023-06-18 01:25:52 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-linux-aarch64.zip" 838754478 bytes
  | 2023-06-18 01:25:52 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-linux-aarch64.zip" 838754478 bytes
  | 2023-06-18 01:25:52 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-linux-x86_64.zip" 242501094 bytes
  | 2023-06-18 01:25:53 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-linux-x86_64.zip" 1042714397 bytes
  | 2023-06-18 01:25:55 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT.zip" 424395826 bytes
  | 2023-06-18 01:26:05 INFO   Successfully downloaded "build/distributions/ml-cpp-8.9.0-SNAPSHOT-debug-linux-x86_64.zip" 1042714397 bytes

The DRA upload pipelines are currently downloading required artifacts
from multiple pipeline steps. While this is not an error or causing any
harm it is slightly more efficient to only download the artifacts once,
from the `create_dra_artifacts` step.
Copy link

@droberts195 droberts195 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@edsavage edsavage merged commit 3229cc2 into elastic:main Jun 19, 2023
edsavage added a commit to edsavage/ml-cpp that referenced this pull request Jun 19, 2023
The DRA upload pipelines are currently downloading required artifacts
from multiple pipeline steps. While this is not an error or causing any
harm it is slightly more efficient to only download the artifacts once,
from the `create_dra_artifacts` step.
edsavage added a commit to edsavage/ml-cpp that referenced this pull request Jun 19, 2023
The DRA upload pipelines are currently downloading required artifacts
from multiple pipeline steps. While this is not an error or causing any
harm it is slightly more efficient to only download the artifacts once,
from the `create_dra_artifacts` step.
edsavage added a commit that referenced this pull request Jun 19, 2023
The DRA upload pipelines are currently downloading required artifacts
from multiple pipeline steps. While this is not an error or causing any
harm it is slightly more efficient to only download the artifacts once,
from the `create_dra_artifacts` step.
edsavage added a commit that referenced this pull request Jun 19, 2023
The DRA upload pipelines are currently downloading required artifacts
from multiple pipeline steps. While this is not an error or causing any
harm it is slightly more efficient to only download the artifacts once,
from the `create_dra_artifacts` step.
@edsavage edsavage deleted the buildkite_only_download_artifacts_once branch November 3, 2023 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants