new proc-macro-based benchmarking syntax (paritytech#12924)

* add stub for new benchmark macro

* benchmark syntax

* add #[extrinsic call] separator

* parse #[benchmark] item as a function

* proper emission of error when #[extrinsic_call] annotation is missing

* clean up

* enclosing module via benchmarks! { } working

* use an attribute macro on the module instead of benchmarks! { }

* cargo fmt

* working component implementation

* WIP

* working

* add syntax for Linear<A, B>

* parsing of param ranges (still need to build tuple though)

* params parsing WIP

* clean up (don't need extrinsic call name)

* use proper Result syntax for BenchmarkDef parsing

* proper parsing of Linear<0, 1> style args

* successfully parse and make use of linear component ranges 💥

* rename support variable => home because eventually will be moved

* compile-time check that param range types implement ParamRange

* switch to using balances as example, failing on instance pallet

* successfully set up __origin and __call with balances 💥

* clean up

* use a module

* don't need a variable for transfer

* rename benchmark_transfer -> transfer because no longer conflicts

* clean up

* working with transfer_increasing_users as well 💥

* re-add BareBlock

* add comments for undocumented structs+functions+traits

* refactor in preparation for removing module requirements

* switch to a block instead of a module

* use the outer macro pattern to to enable #[benchmarks] aggregation

* successfully generate SelectedBenchmark 💥

* implement components for SelectedBenchmark

* implement instance for SelectedBenchmark

* properly track #[extra]

* working impl for fn benchmarks()

* run_benchmarks WIP

* finish run_benchmark! impl 💥

* import balances transfer_best_case benchmark

* import transfer_keep_alive balances pallet benchmark

* import set_balance_creating balances pallet benchmark

* import set_balance_killing balances pallet benchmark

* import force_transfer balances pallet benchmark

* add #[extra] annotation and docs to transfer_increasing_users

* import transfer_all balances pallet benchmark

* import force_unreserve balances pallet benchmark

* prepare to implement impl_benchmark_test_suite!

* ensure tests cover #[extra] before and after #[benchmark] tag

* refactor

* clean up

* fix

* move to outer

* switch to benchmarks/instance_benchmarks

* test impl almost done, strange compiler error

* benchmark test suites working 💥

* clean up

* add stub and basic parsing for where_clause

* working except where clause and extrinsic calls containing method chains

* assume option (2) for now wrt paritytech#12924 (comment)

* clean up

* switch to attribute-style

* properly handle where clauses

* fix subtle missing where clause, now just MessageQueue issues

* fix block formatting in message-queue pallet

* switch to block vs non-block parsing of extrinsic call

* working now but some benchmark tests failing

* message-queue tests working (run order issue fixed) 🎉

* add comments and internal docs for fame_support_procedural::benchmark

* fix license years

* docs for lib.rs

* add docs to new support procedural macros

* don't allow #[benchmark] outside of benchmarking module

* add docs

* use benchmark(extra, skip_meta) style args

* update docs accordingly

* appease clippy

* bump ci

* add notes about `extra` and `skip_meta`

* fix doc tests

* re-run CI

* use `ignore` instead of `no_run` on doc examples

* bump CI

* replace some if-lets with if-elses

* more refactoring of if-let statements

* fix remaining if-lets in BenchmarkDef::from()

* fix if-lets in benchmarks()

* fix remaining if-lets, use nested find_map for extrinsic call

* switch to use #[extrinsic_call] or #[block] situationally

* refactor ExtrinsicCallDef => BenchmarkCallDef

* update docs with info about #[block]

* add macro stub for #[extrinsic_call]

* fix docs and add stub for #[block] as well

* remove unused extern crate line

* fix clippy nits

* Use V2 bench syntax in pallet-example-basic

Just testing the dev-ex...

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* carry over comment

* use curly-brace style for impl_benchmark_test_suite!

* remove unneeded parenthesis

* proper handling of _() extrinsic call style

* add docs for _() syntax

* fix crate access

* simplify keyword access

Co-authored-by: Keith Yeung <[email protected]>

* simplify module content destructuring

Co-authored-by: Keith Yeung <[email protected]>

* fix crate access "frame_benchmarking" => "frame-benchmarking", compiles

* use _() extrinsic call syntax where possible in balances

* simplify attr.path.segments.last()

Co-authored-by: Keith Yeung <[email protected]>

* fix compile error being suppressed

* simplify extrinsic call keyword parsing

Co-authored-by: Keith Yeung <[email protected]>

* use ? operator instead of return None

Co-authored-by: Keith Yeung <[email protected]>

* rename generics => type_use_generics
rename full_generics => type_impl_generics

* simplify extrinsic call extraction with transpose

* bump CI

* nit

* proper handling of too many + too few block/extrinsic call annotations

* change to B >= A

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* remove unneeded ignore

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* remove another ignore

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* add ui tests

* use _() style extrinsic call on accumulate_dummy

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* add range check to ParamRange

* ui test for bad param ranges

* fix failing example

* add ignore back to other failing example

* tweak expr_call span

Co-authored-by: Keith Yeung <[email protected]>

* fix typo

* eliminate a match

Co-authored-by: Keith Yeung <[email protected]>

* change pub fn benchmarks to return Result<TokenStream>

* fix origin error span

* more informative error for invalid benchmark parameter name

* fix spans on a few benchmark errors

* remove unneeded clone

* refactor inner loop of benchmark function parsing

* preserve mod attributes

* refactor outer loop of benchmark def parsing code, greatly simplified

* simplify to use a ? operator when parsing benchmark attr path

* fix another ? operator

* further simplify benchmark function attr parsing with more ? ops

* refactor extrinsic call handling to use if let rather than match

* replace is_ok => is_err

Co-authored-by: Keith Yeung <[email protected]>

* re-use name during expansion of benchmark def

* remove unneeded clone

* fix span for origin missing error

* fix missing semi

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: parity-processbot <>

Author: 3 people

Cargo.lock

@@ -1,6 +1,6 @@
 // This file is part of Substrate.
 
-// Copyright (C) 2020-2022 Parity Technologies (UK) Ltd.
+// Copyright (C) 2020-2023 Parity Technologies (UK) Ltd.
 // SPDX-License-Identifier: Apache-2.0
 
 // Licensed under the Apache License, Version 2.0 (the "License");
@@ -20,22 +20,25 @@
 #![cfg(feature = "runtime-benchmarks")]
 
 use super::*;
+use crate::Pallet as Balances;
 
-use frame_benchmarking::{account, benchmarks_instance_pallet, whitelisted_caller};
+use frame_benchmarking::{account, impl_benchmark_test_suite, whitelisted_caller};
+use frame_support::benchmarking::*;
 use frame_system::RawOrigin;
-use sp_runtime::traits::Bounded;
-
-use crate::Pallet as Balances;
 
 const SEED: u32 = 0;
 // existential deposit multiplier
 const ED_MULTIPLIER: u32 = 10;
 
-benchmarks_instance_pallet! {
+#[instance_benchmarks]
+mod benchmarks {
+	use super::*;
+
 	// Benchmark `transfer` extrinsic with the worst possible conditions:
 	// * Transfer will kill the sender account.
 	// * Transfer will create the recipient account.
-	transfer {
+	#[benchmark]
+	fn transfer() {
 		let existential_deposit = T::ExistentialDeposit::get();
 		let caller = whitelisted_caller();
 
@@ -47,84 +50,103 @@ benchmarks_instance_pallet! {
 		// and reap this user.
 		let recipient: T::AccountId = account("recipient", 0, SEED);
 		let recipient_lookup = T::Lookup::unlookup(recipient.clone());
-		let transfer_amount = existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
-	}: transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount)
-	verify {
+		let transfer_amount =
+			existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
+
+		#[extrinsic_call]
+		_(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount);
+
 		assert_eq!(Balances::<T, I>::free_balance(&caller), Zero::zero());
 		assert_eq!(Balances::<T, I>::free_balance(&recipient), transfer_amount);
 	}
 
 	// Benchmark `transfer` with the best possible condition:
 	// * Both accounts exist and will continue to exist.
-	#[extra]
-	transfer_best_case {
+	#[benchmark(extra)]
+	fn transfer_best_case() {
 		let caller = whitelisted_caller();
 		let recipient: T::AccountId = account("recipient", 0, SEED);
 		let recipient_lookup = T::Lookup::unlookup(recipient.clone());
 
-		// Give the sender account max funds for transfer (their account will never reasonably be killed).
-		let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, T::Balance::max_value());
+		// Give the sender account max funds for transfer (their account will never reasonably be
+		// killed).
+		let _ =
+			<Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, T::Balance::max_value());
 
 		// Give the recipient account existential deposit (thus their account already exists).
 		let existential_deposit = T::ExistentialDeposit::get();
-		let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&recipient, existential_deposit);
+		let _ =
+			<Balances<T, I> as Currency<_>>::make_free_balance_be(&recipient, existential_deposit);
 		let transfer_amount = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
-	}: transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount)
-	verify {
+
+		#[extrinsic_call]
+		transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount);
+
 		assert!(!Balances::<T, I>::free_balance(&caller).is_zero());
 		assert!(!Balances::<T, I>::free_balance(&recipient).is_zero());
 	}
 
 	// Benchmark `transfer_keep_alive` with the worst possible condition:
 	// * The recipient account is created.
-	transfer_keep_alive {
+	#[benchmark]
+	fn transfer_keep_alive() {
 		let caller = whitelisted_caller();
 		let recipient: T::AccountId = account("recipient", 0, SEED);
 		let recipient_lookup = T::Lookup::unlookup(recipient.clone());
 
 		// Give the sender account max funds, thus a transfer will not kill account.
-		let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, T::Balance::max_value());
+		let _ =
+			<Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, T::Balance::max_value());
 		let existential_deposit = T::ExistentialDeposit::get();
 		let transfer_amount = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
-	}: _(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount)
-	verify {
+
+		#[extrinsic_call]
+		_(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount);
+
 		assert!(!Balances::<T, I>::free_balance(&caller).is_zero());
 		assert_eq!(Balances::<T, I>::free_balance(&recipient), transfer_amount);
 	}
 
 	// Benchmark `set_balance` coming from ROOT account. This always creates an account.
-	set_balance_creating {
+	#[benchmark]
+	fn set_balance_creating() {
 		let user: T::AccountId = account("user", 0, SEED);
 		let user_lookup = T::Lookup::unlookup(user.clone());
 
 		// Give the user some initial balance.
 		let existential_deposit = T::ExistentialDeposit::get();
 		let balance_amount = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
 		let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&user, balance_amount);
-	}: set_balance(RawOrigin::Root, user_lookup, balance_amount, balance_amount)
-	verify {
+
+		#[extrinsic_call]
+		set_balance(RawOrigin::Root, user_lookup, balance_amount, balance_amount);
+
 		assert_eq!(Balances::<T, I>::free_balance(&user), balance_amount);
 		assert_eq!(Balances::<T, I>::reserved_balance(&user), balance_amount);
 	}
 
 	// Benchmark `set_balance` coming from ROOT account. This always kills an account.
-	set_balance_killing {
+	#[benchmark]
+	fn set_balance_killing() {
 		let user: T::AccountId = account("user", 0, SEED);
 		let user_lookup = T::Lookup::unlookup(user.clone());
 
 		// Give the user some initial balance.
 		let existential_deposit = T::ExistentialDeposit::get();
 		let balance_amount = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
 		let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&user, balance_amount);
-	}: set_balance(RawOrigin::Root, user_lookup, Zero::zero(), Zero::zero())
-	verify {
+
+		#[extrinsic_call]
+		set_balance(RawOrigin::Root, user_lookup, Zero::zero(), Zero::zero());
+
 		assert!(Balances::<T, I>::free_balance(&user).is_zero());
 	}
 
 	// Benchmark `force_transfer` extrinsic with the worst possible conditions:
 	// * Transfer will kill the sender account.
 	// * Transfer will create the recipient account.
-	force_transfer {
+	#[benchmark]
+	fn force_transfer() {
 		let existential_deposit = T::ExistentialDeposit::get();
 		let source: T::AccountId = account("source", 0, SEED);
 		let source_lookup = T::Lookup::unlookup(source.clone());
@@ -133,23 +155,26 @@ benchmarks_instance_pallet! {
 		let balance = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
 		let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&source, balance);
 
-		// Transfer `e - 1` existential deposits + 1 unit, which guarantees to create one account, and reap this user.
+		// Transfer `e - 1` existential deposits + 1 unit, which guarantees to create one account,
+		// and reap this user.
 		let recipient: T::AccountId = account("recipient", 0, SEED);
 		let recipient_lookup = T::Lookup::unlookup(recipient.clone());
-		let transfer_amount = existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
-	}: force_transfer(RawOrigin::Root, source_lookup, recipient_lookup, transfer_amount)
-	verify {
+		let transfer_amount =
+			existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
+
+		#[extrinsic_call]
+		_(RawOrigin::Root, source_lookup, recipient_lookup, transfer_amount);
+
 		assert_eq!(Balances::<T, I>::free_balance(&source), Zero::zero());
 		assert_eq!(Balances::<T, I>::free_balance(&recipient), transfer_amount);
 	}
 
 	// This benchmark performs the same operation as `transfer` in the worst case scenario,
 	// but additionally introduces many new users into the storage, increasing the the merkle
 	// trie and PoV size.
-	#[extra]
-	transfer_increasing_users {
+	#[benchmark(extra)]
+	fn transfer_increasing_users(u: Linear<0, 1_000>) {
 		// 1_000 is not very much, but this upper bound can be controlled by the CLI.
-		let u in 0 .. 1_000;
 		let existential_deposit = T::ExistentialDeposit::get();
 		let caller = whitelisted_caller();
 
@@ -161,25 +186,29 @@ benchmarks_instance_pallet! {
 		// and reap this user.
 		let recipient: T::AccountId = account("recipient", 0, SEED);
 		let recipient_lookup = T::Lookup::unlookup(recipient.clone());
-		let transfer_amount = existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
+		let transfer_amount =
+			existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
 
 		// Create a bunch of users in storage.
-		for i in 0 .. u {
+		for i in 0..u {
 			// The `account` function uses `blake2_256` to generate unique accounts, so these
 			// should be quite random and evenly distributed in the trie.
 			let new_user: T::AccountId = account("new_user", i, SEED);
 			let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&new_user, balance);
 		}
-	}: transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount)
-	verify {
+
+		#[extrinsic_call]
+		transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount);
+
 		assert_eq!(Balances::<T, I>::free_balance(&caller), Zero::zero());
 		assert_eq!(Balances::<T, I>::free_balance(&recipient), transfer_amount);
 	}
 
 	// Benchmark `transfer_all` with the worst possible condition:
 	// * The recipient account is created
 	// * The sender is killed
-	transfer_all {
+	#[benchmark]
+	fn transfer_all() {
 		let caller = whitelisted_caller();
 		let recipient: T::AccountId = account("recipient", 0, SEED);
 		let recipient_lookup = T::Lookup::unlookup(recipient.clone());
@@ -188,13 +217,16 @@ benchmarks_instance_pallet! {
 		let existential_deposit = T::ExistentialDeposit::get();
 		let balance = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
 		let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, balance);
-	}: _(RawOrigin::Signed(caller.clone()), recipient_lookup, false)
-	verify {
+
+		#[extrinsic_call]
+		_(RawOrigin::Signed(caller.clone()), recipient_lookup, false);
+
 		assert!(Balances::<T, I>::free_balance(&caller).is_zero());
 		assert_eq!(Balances::<T, I>::free_balance(&recipient), balance);
 	}
 
-	force_unreserve {
+	#[benchmark]
+	fn force_unreserve() {
 		let user: T::AccountId = account("user", 0, SEED);
 		let user_lookup = T::Lookup::unlookup(user.clone());
 
@@ -208,15 +240,16 @@ benchmarks_instance_pallet! {
 		assert_eq!(Balances::<T, I>::reserved_balance(&user), balance);
 		assert!(Balances::<T, I>::free_balance(&user).is_zero());
 
-	}: _(RawOrigin::Root, user_lookup, balance)
-	verify {
+		#[extrinsic_call]
+		_(RawOrigin::Root, user_lookup, balance);
+
 		assert!(Balances::<T, I>::reserved_balance(&user).is_zero());
 		assert_eq!(Balances::<T, I>::free_balance(&user), balance);
 	}
 
-	impl_benchmark_test_suite!(
+	impl_benchmark_test_suite! {
 		Balances,
 		crate::tests_composite::ExtBuilder::default().build(),
 		crate::tests_composite::Test,
-	)
+	}
 }