Release `2524` distribution

[jpraynaud]

Why

In order to deploy new features, improvements and bug fixes, we need to release a new distribution.

What

Create and release a new distribution 2524

Important

• Rotate ancillary signer service account keys:
  • release-mainnet
• Add origin tag for Catalyst:
  • pre-release-preview
  • release-preprod
  • release-mainnet
• 🔥 Upgrade to Cardano node 10.4.1 (with upgrade procedure):
  • pre-release-preview
  • release-preprod
  • release-mainnet
• Bug fixes Cardano DB
• Unstable Cardano DB v2 with backends in client CLI
• Unstable Cardano UTxO-HD ledger snasphot conversion in client CLI

Distribution +1:

Note

• TBD
• Rotate ancillary signer service account keys:
  • release-mainnet
• Bump minor version of mithril-common
• Stable Cardano DB v2 with backends
• Update the Introducing the UTXO snapshot converter command in the Mithril client CLI dev blog post with deprecated --cardano-network parameter
• Bump mithril-common minor (update constraint in Cargo.toml)
• Upgrade to Cardano node 10..5.0:
  • pre-release-preview
  • release-preprod
  • release-mainnet

How

Phase 1: Prepare distribution

• Prepare migration process for Cardano node to UTxO-HD
• Run manual successful restoration of Cardano database + restart of the Cardano node:
  • Cardano DB v1
    • with ancillary
    • without ancillary
  • Cardano DB v2
    • with ancillary
    • without ancillary
• Run a backward compatibility test in the CI on the unstable distribution
• Run an all green client multi-platform test in the CI (with the 2524.0 built e2e test)
• Prepare a PR to upgrade repository dependencies (with runbook)
• Reset the CI caches just before merging the PR to upgrade repository dependencies
• Merge the PR to upgrade repository dependencies
• Check that all minor versions of all crates have been bumped for breaking change (if needed)
  • Bump mithril-stm to 0.4
• Check if versions of repository linked crates are correct for publication to crates.io(mithril-common, mithril-client)
• Check if Cardano minimum supported versions in networks.json must be updated following the runbook
• Create a draft PR for rotating the documentation
• Create a PR for rotating the CHANGELOG.md (set the distribution version)
• Merge the PR for rotating the CHANGELOG.md

Phase 2: Rollout pre-release distribution

• Add origin tag for Catalyst on pre-release-preview
• Update Cardano node version to 10.4.1 on pre-release-preview
• Run Cardano node migration phase 1 on pre-release-preview
• Tag and deploy a 2524.0-pre pre-release distribution on the pre-release-preview network
• Fill highlights, distributions compatibility and release notes sections in the pre-release page on GitHub
• Run Cardano node migration phase 2 on pre-release-preview
• Make sure pre-release-preview is working as expected
• Make sure Grafana dashboards are fetching data for pre-release-preview
• Make sure libraries version next have been published on npm registry
• Run an all green client multi-platform test in the CI
• Run an all green aggregator stress test in the CI (not working, but worked earlier wit the unstable release)
• Run a Cardano transaction prover benchmark:
  • Store results in the issue
  • Make sure that there is no performance drop since last release
• Announce the pre-release on discord channel
• Announce any breaking change in:
  • #cardano-scaling slack channel for pre-release
  • #mithril-lace slack channel for pre-release
  • #blockfrost-scaling slack channel for pre-release
  • #mithril-catalyst slack channel for pre-release
  • #mithril-acropolis slack channel for pre-release
  • Cardano Foundation IBC bridge team
• Wait for at least 2 signers running the new version (could be more if era change is tested) over at least 2 epochs

Phase 3: Rollout release distribution

• Tag and deploy a 2524.0 release distribution on the pre-release-preview network
• Fill highlights, distributions compatibility and release notes sections in the release page on GitHub
• Make sure pre-release-preview is working as expected
• Update networks compatibility table in release notes
• Add origin tag for Catalyst on release-mainnet
• Add origin tag for Catalyst on release-preprod
• Update Cardano node version to 10.4.1 on release-mainnet
• Update Cardano node version to 10.4.1 on release-preprod
• Create a new key for the ancillary signer service account of release-mainnet
• Configure release-mainnet environment with new service account credentials
• Run Cardano node migration phase 1 on release-mainnet
• Run Cardano node migration phase 1 on release-preprod
• Release the 2524.0
• Run Cardano node migration phase 2 on release-mainnet
• Run Cardano node migration phase 2 on release-preprod
• Make sure release-mainnet is working as expected
• Make sure release-preprod is working as expected
• Make sure libraries have been published on crates.io
• Make sure libraries version latest have been published on npm registry
• Make sure Grafana dashboards are fetching data for release-mainnet
• Make sure Grafana dashboards are fetching data for release-preprod
• Destroy the previous key for the ancillary signer service account of release-mainnet
• Check that KMS credentials are working with the creation of the next Cardano database snapshot
• Merge the PR for rotating the documentation
• Merge a PR for rotating the CHANGELOG.md:
  • Updating the release date of the distribution
  • Updating the crate versions table
• Announce the release on discord channel (with breaking-change, upcoming features activation, ...)
• Announce any breaking/new feature change in:
  • #cardano-scaling slack channel for release
  • #mithril-lace slack channel for release
  • #blockfrost-scaling slack channel for release
  • #mithril-catalyst slack channel for release
  • #mithril-acropolis slack channel for release
  • Cardano Foundation IBC bridge team
• Update Hydra tutorial if client breaking change for database restoration
• Check if versions of repository linked crates can be bumped after publication to crates.io(mithril-common, mithril-client)
• Dev blog post:
  • Distribution 2524
  • New UTxO-HD snapshot converter command in client CLI
  • Transition to backend system for cardano-db command in client CLI

[jpraynaud]

Script to convert ledger state snapshot when upgrading to 10.4.1:

• Phase 1: prepare the converted ledger state snapshot

----------------- START PHASE 1
## Env vars
export CARDANO_NETWORK=preview
export CARDANO_VERSION=10.4.1
export TEMP_DIR=/home/curry/temp
export CARDANO_DIR=/home/curry/data/${CARDANO_NETWORK}/mithril-aggregator/cardano
export CARDANO_CONFIG_FILE=${TEMP_DIR}/share/${CARDANO_NETWORK}/config.json
export CARDANO_LEDGER_DIR=${CARDANO_DIR}/db/ledger
export CARDANO_LEDGER_SLOT=$(ls $CARDANO_LEDGER_DIR | head -n 1)
echo "LEDGER_SLOT=$CARDANO_LEDGER_SLOT"

## Create temp dir
mkdir -p $TEMP_DIR

## Copy ledger state snapshot to temp
cp ${CARDANO_LEDGER_DIR}/${CARDANO_LEDGER_SLOT} $TEMP_DIR
tree $CARDANO_LEDGER_DIR
tree $TEMP_DIR

## Download snapshot converter
cd $TEMP_DIR
curl --fail -sL -o cardano-node-${CARDANO_VERSION}-linux.tar.gz https://github.com/IntersectMBO/cardano-node/releases/download/${CARDANO_VERSION}/cardano-node-${CARDANO_VERSION}-linux.tar.gz
tar xzf cardano-node-${CARDANO_VERSION}-linux.tar.gz
tree $TEMP_DIR

## Convert snasphot
$TEMP_DIR/bin/snapshot-converter Legacy ${TEMP_DIR}/${CARDANO_LEDGER_SLOT} Mem ${TEMP_DIR}/${CARDANO_LEDGER_SLOT}_mem cardano --config $CARDANO_CONFIG_FILE
tree ${TEMP_DIR}/${CARDANO_LEDGER_SLOT}_mem

----------------- END PHASE 1 

• Phase 2: use the converted ledger state snapshot

----------------- START PHASE 2

## Stop cardano node
docker stop cardano-node-aggregator

# Switch to new ledger state snapshot
rm -rf ${CARDANO_LEDGER_DIR}/*
cp -r ${TEMP_DIR}/${CARDANO_LEDGER_SLOT}_mem ${CARDANO_LEDGER_DIR}/${CARDANO_LEDGER_SLOT}
tree ${CARDANO_LEDGER_DIR}/

## Start cardano node
docker start cardano-node-aggregator

## Cleanup temp dir
cd /home/curry
rm -rf $TEMP_DIR

----------------- END PHASE 2

[jpraynaud]

Cardano transactions prover benchmark

Note

run on GCP with e2-highmem-2 machine with 2 vCPUs and 15GB of RAM
not during Cardano database snapshotting

• Benchmark results:

total_requests	concurrency	transactions/request	requests/s
500	50	1	73.04
500	50	10	52.93
500	50	20	40.49
500	50	30	33.66
500	50	40	26.97
500	50	50	22.95

vs previous results

total_requests	concurrency	transactions/request	requests/s
500	50	1	78.00
500	50	10	53.01
500	50	20	40.84
500	50	30	34.72
500	50	40	29.52
500	50	50	25.98

• Command run:

# Aggregator endpoint
export AGGREGATOR_ENDPOINT=https://aggregator.pre-release-preview.api.mithril.network/aggregator

# Transactions file to prove
export TRANSACTIONS_FILE=transactions-preview.txt

# Transactions proved per request range definition
export TRANSACTIONS_PER_REQUEST_MIN=0
export TRANSACTIONS_PER_REQUEST_MAX=50
export TRANSACTIONS_PER_REQUEST_STEP=10

# Apache benchmark total request sent per benchmark
export AB_TOTAL_REQUESTS=500

# Apache benchmark concurrency level range definition
export AB_CONCURRENCY_MIN=50
export AB_CONCURRENCY_MAX=50
export AB_CONCURRENCY_STEP=50

# Run the benchmark:
./benchmark-aggregator-prover.sh

• Command result:

MITHRIL AGGREGATOR PROVER ROUTE BENCHMARK

>> Aggregator endpoint: https://aggregator.pre-release-preview.api.mithril.network/aggregator
>> Aggregator route: /proof/cardano-transaction
>> Transactions file: transactions-preview.txt
>> Transactions available: [100]
>> Transactions per request range: [0 10 20 30 40 50]
>> AB concurrency range: [50]
>> AB total requests per run: [500]
>> AB total runs: 6
>> Output file: benchmark.csv

>> [#1/6] Running stress test with 500 requests with 1 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (73.04 requests/s)

>> [#2/6] Running stress test with 500 requests with 10 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (52.93 requests/s)

>> [#3/6] Running stress test with 500 requests with 20 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (40.49 requests/s)

>> [#4/6] Running stress test with 500 requests with 30 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (33.66 requests/s)

>> [#5/6] Running stress test with 500 requests with 40 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (26.97 requests/s)

>> [#6/6] Running stress test with 500 requests with 50 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (22.95 requests/s)

>> Benchmark completed:

total_requests,concurrency,transactions/request,requests/s
500,50,1,73.04
500,50,10,52.93
500,50,20,40.49
500,50,30,33.66
500,50,40,26.97
500,50,50,22.95