Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Commit d766e22

Browse files
insipxbkchrdvdplm
authored
WASM Local-blob override (#7317)
* Provide WASM overwrite functionality in LocalCallExecutor - add a new module `wasm_overwrite.rs` in client - scrapes given folder for runtimes - add two new CLI Options `wasm-overwrite` and `wasm_overwrite_path` * formatting * Make comment clearer remove sc-runtime-test from dev-dependencies * comments * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * Fix spaces, remove call into backend for 'heap_pages' in 'try_replace' * Error if path is not a directory, Comments, Doc Comment for WasmOverwrite * make WasmOverwrite Option<> * Change to one CLI argument for overwrites - move getting runtime version into LocalCallExecutor * change unwrap() to expect() * comment * Remove `check_overwrites` * Encapsulate checking for overwrites in LocalCallExecutor * move duplicate code into function * Update client/cli/src/params/import_params.rs Co-authored-by: Bastian Köcher <[email protected]> * comma * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * cache hash in WasmBlob * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/service/src/client/client.rs Co-authored-by: Bastian Köcher <[email protected]> * move getting overwrite into its own function * fix error when directory is not a directory * Error on duplicate WASM runtimes * better comment, grammar * docs * Revert StateBackend back to _ * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/service/src/client/call_executor.rs Co-authored-by: Bastian Köcher <[email protected]> * Add two tests, fix doc comments Add a test for the runtime_version method of WasmOverwrite Add a test for check_overwrite method of LocalCallExecutor * remove redundant `Return` from expect msg * Update client/cli/src/params/import_params.rs Co-authored-by: David <[email protected]> * Update client/service/src/client/call_executor.rs Co-authored-by: David <[email protected]> * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: David <[email protected]> * Update client/service/src/config.rs Co-authored-by: David <[email protected]> * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: David <[email protected]> * Add Module Documentation, match on '.wasm' extension * Add test for scraping WASM blob * fix expect * remove creating another block in LocalCallExecutor test * remove unused import * add tests for duplicates and scraping wasm * make tests a bit nicer * add test for ignoring non-.wasm files * check error message in test * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * remove println * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * make tests prettier * Update client/service/src/client/wasm_overwrite.rs Co-authored-by: Bastian Köcher <[email protected]> * comment for seemingly random client * locally-built -> custom * remove unused import * fix comment * rename all references to overwrite with override * fix cli flag in module documentation Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: David <[email protected]>
1 parent 891900a commit d766e22

File tree

15 files changed

+420
-15
lines changed

15 files changed

+420
-15
lines changed

Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

client/cli/src/config.rs

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,15 @@ pub trait CliConfiguration<DCV: DefaultConfigurationValues = ()>: Sized {
278278
.unwrap_or_default())
279279
}
280280

281+
/// Get the path where WASM overrides live.
282+
///
283+
/// By default this is `None`.
284+
fn wasm_runtime_overrides(&self) -> Option<PathBuf> {
285+
self.import_params()
286+
.map(|x| x.wasm_runtime_overrides())
287+
.unwrap_or_default()
288+
}
289+
281290
/// Get the execution strategies.
282291
///
283292
/// By default this is retrieved from `ImportParams` if it is available. Otherwise its
@@ -492,6 +501,7 @@ pub trait CliConfiguration<DCV: DefaultConfigurationValues = ()>: Sized {
492501
state_cache_child_ratio: self.state_cache_child_ratio()?,
493502
pruning: self.pruning(unsafe_pruning, &role)?,
494503
wasm_method: self.wasm_method()?,
504+
wasm_runtime_overrides: self.wasm_runtime_overrides(),
495505
execution_strategies: self.execution_strategies(is_dev, is_validator)?,
496506
rpc_http: self.rpc_http(DCV::rpc_http_listen_port())?,
497507
rpc_ws: self.rpc_ws(DCV::rpc_ws_listen_port())?,

client/cli/src/params/import_params.rs

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ use crate::params::DatabaseParams;
2525
use crate::params::PruningParams;
2626
use sc_client_api::execution_extensions::ExecutionStrategies;
2727
use structopt::StructOpt;
28+
use std::path::PathBuf;
2829

2930
/// Parameters for block import.
3031
#[derive(Debug, StructOpt)]
@@ -55,6 +56,12 @@ pub struct ImportParams {
5556
)]
5657
pub wasm_method: WasmExecutionMethod,
5758

59+
/// Specify the path where local WASM runtimes are stored.
60+
///
61+
/// These runtimes will override on-chain runtimes when the version matches.
62+
#[structopt(long, value_name = "PATH", parse(from_os_str))]
63+
pub wasm_runtime_overrides: Option<PathBuf>,
64+
5865
#[allow(missing_docs)]
5966
#[structopt(flatten)]
6067
pub execution_strategies: ExecutionStrategiesParams,
@@ -103,6 +110,12 @@ impl ImportParams {
103110
self.wasm_method.into()
104111
}
105112

113+
/// Enable overriding on-chain WASM with locally-stored WASM
114+
/// by specifying the path where local WASM is stored.
115+
pub fn wasm_runtime_overrides(&self) -> Option<PathBuf> {
116+
self.wasm_runtime_overrides.clone()
117+
}
118+
106119
/// Get execution strategies for the parameters
107120
pub fn execution_strategies(&self, is_dev: bool, is_validator: bool) -> ExecutionStrategies {
108121
let exec = &self.execution_strategies;

client/service/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ directories = "2.0.2"
8686

8787
[dev-dependencies]
8888
substrate-test-runtime-client = { version = "2.0.0", path = "../../test-utils/runtime/client" }
89+
substrate-test-runtime = { versino = "2.0.0", path = "../../test-utils/runtime/" }
8990
sp-consensus-babe = { version = "0.8.0", path = "../../primitives/consensus/babe" }
9091
grandpa = { version = "0.8.0", package = "sc-finality-grandpa", path = "../finality-grandpa" }
9192
grandpa-primitives = { version = "2.0.0", package = "sp-finality-grandpa", path = "../../primitives/finality-grandpa" }

client/service/src/builder.rs

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -303,8 +303,9 @@ pub fn new_full_parts<TBl, TRtApi, TExecDisp>(
303303
Box::new(task_manager.spawn_handle()),
304304
config.prometheus_config.as_ref().map(|config| config.registry.clone()),
305305
ClientConfig {
306-
offchain_worker_enabled : config.offchain_worker.enabled ,
306+
offchain_worker_enabled : config.offchain_worker.enabled,
307307
offchain_indexing_api: config.offchain_worker.indexing_enabled,
308+
wasm_runtime_overrides: config.wasm_runtime_overrides.clone(),
308309
},
309310
)?
310311
};
@@ -396,7 +397,7 @@ pub fn new_client<E, Block, RA>(
396397
const CANONICALIZATION_DELAY: u64 = 4096;
397398

398399
let backend = Arc::new(Backend::new(settings, CANONICALIZATION_DELAY)?);
399-
let executor = crate::client::LocalCallExecutor::new(backend.clone(), executor, spawn_handle, config.clone());
400+
let executor = crate::client::LocalCallExecutor::new(backend.clone(), executor, spawn_handle, config.clone())?;
400401
Ok((
401402
crate::client::Client::new(
402403
backend.clone(),

client/service/src/client/call_executor.rs

Lines changed: 111 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -28,36 +28,71 @@ use sp_state_machine::{
2828
use sc_executor::{RuntimeVersion, RuntimeInfo, NativeVersion};
2929
use sp_externalities::Extensions;
3030
use sp_core::{
31-
NativeOrEncoded, NeverNativeValue, traits::{CodeExecutor, SpawnNamed},
31+
NativeOrEncoded, NeverNativeValue, traits::{CodeExecutor, SpawnNamed, RuntimeCode},
3232
offchain::storage::OffchainOverlayedChanges,
3333
};
3434
use sp_api::{ProofRecorder, InitializeBlock, StorageTransactionCache};
3535
use sc_client_api::{backend, call_executor::CallExecutor};
36-
use super::client::ClientConfig;
36+
use super::{client::ClientConfig, wasm_override::WasmOverride};
3737

3838
/// Call executor that executes methods locally, querying all required
3939
/// data from local backend.
4040
pub struct LocalCallExecutor<B, E> {
4141
backend: Arc<B>,
4242
executor: E,
43+
wasm_override: Option<WasmOverride<E>>,
4344
spawn_handle: Box<dyn SpawnNamed>,
4445
client_config: ClientConfig,
4546
}
4647

47-
impl<B, E> LocalCallExecutor<B, E> {
48+
impl<B, E> LocalCallExecutor<B, E>
49+
where
50+
E: CodeExecutor + RuntimeInfo + Clone + 'static
51+
{
4852
/// Creates new instance of local call executor.
4953
pub fn new(
5054
backend: Arc<B>,
5155
executor: E,
5256
spawn_handle: Box<dyn SpawnNamed>,
5357
client_config: ClientConfig,
54-
) -> Self {
55-
LocalCallExecutor {
58+
) -> sp_blockchain::Result<Self> {
59+
let wasm_override = client_config.wasm_runtime_overrides
60+
.as_ref()
61+
.map(|p| WasmOverride::new(p.clone(), executor.clone()))
62+
.transpose()?;
63+
64+
Ok(LocalCallExecutor {
5665
backend,
5766
executor,
67+
wasm_override,
5868
spawn_handle,
5969
client_config,
60-
}
70+
})
71+
}
72+
73+
/// Check if local runtime code overrides are enabled and one is available
74+
/// for the given `BlockId`. If yes, return it; otherwise return the same
75+
/// `RuntimeCode` instance that was passed.
76+
fn check_override<'a, Block>(
77+
&'a self,
78+
onchain_code: RuntimeCode<'a>,
79+
id: &BlockId<Block>,
80+
) -> sp_blockchain::Result<RuntimeCode<'a>>
81+
where
82+
Block: BlockT,
83+
B: backend::Backend<Block>,
84+
{
85+
let code = self.wasm_override
86+
.as_ref()
87+
.map::<sp_blockchain::Result<Option<RuntimeCode>>, _>(|o| {
88+
let spec = self.runtime_version(id)?.spec_version;
89+
Ok(o.get(&spec, onchain_code.heap_pages))
90+
})
91+
.transpose()?
92+
.flatten()
93+
.unwrap_or(onchain_code);
94+
95+
Ok(code)
6196
}
6297
}
6398

@@ -66,6 +101,7 @@ impl<B, E> Clone for LocalCallExecutor<B, E> where E: Clone {
66101
LocalCallExecutor {
67102
backend: self.backend.clone(),
68103
executor: self.executor.clone(),
104+
wasm_override: self.wasm_override.clone(),
69105
spawn_handle: self.spawn_handle.clone(),
70106
client_config: self.client_config.clone(),
71107
}
@@ -101,6 +137,8 @@ where
101137
)?;
102138
let state = self.backend.state_at(*id)?;
103139
let state_runtime_code = sp_state_machine::backend::BackendRuntimeCode::new(&state);
140+
let runtime_code = self.check_override(state_runtime_code.runtime_code()?, id)?;
141+
104142
let return_data = StateMachine::new(
105143
&state,
106144
changes_trie,
@@ -110,7 +148,7 @@ where
110148
method,
111149
call_data,
112150
extensions.unwrap_or_default(),
113-
&state_runtime_code.runtime_code()?,
151+
&runtime_code,
114152
self.spawn_handle.clone(),
115153
).execute_using_consensus_failure_handler::<_, NeverNativeValue, fn() -> _>(
116154
strategy.get_manager(),
@@ -173,7 +211,7 @@ where
173211
let state_runtime_code = sp_state_machine::backend::BackendRuntimeCode::new(&trie_state);
174212
// It is important to extract the runtime code here before we create the proof
175213
// recorder.
176-
let runtime_code = state_runtime_code.runtime_code()?;
214+
let runtime_code = self.check_override(state_runtime_code.runtime_code()?, at)?;
177215

178216
let backend = sp_state_machine::ProvingBackend::new_with_recorder(
179217
trie_state,
@@ -198,7 +236,8 @@ where
198236
},
199237
None => {
200238
let state_runtime_code = sp_state_machine::backend::BackendRuntimeCode::new(&state);
201-
let runtime_code = state_runtime_code.runtime_code()?;
239+
let runtime_code = self.check_override(state_runtime_code.runtime_code()?, at)?;
240+
202241
let mut state_machine = StateMachine::new(
203242
&state,
204243
changes_trie_state,
@@ -279,3 +318,66 @@ impl<B, E, Block> sp_version::GetRuntimeVersion<Block> for LocalCallExecutor<B,
279318
CallExecutor::runtime_version(self, at).map_err(|e| format!("{:?}", e))
280319
}
281320
}
321+
322+
#[cfg(test)]
323+
mod tests {
324+
use super::*;
325+
use substrate_test_runtime_client::{LocalExecutor, GenesisInit, runtime};
326+
use sc_executor::{NativeExecutor, WasmExecutionMethod};
327+
use sp_core::{traits::{WrappedRuntimeCode, FetchRuntimeCode}, testing::TaskExecutor};
328+
use sc_client_api::in_mem;
329+
330+
#[test]
331+
fn should_get_override_if_exists() {
332+
let executor =
333+
NativeExecutor::<LocalExecutor>::new(WasmExecutionMethod::Interpreted, Some(128), 1);
334+
335+
let overrides = crate::client::wasm_override::dummy_overrides(&executor);
336+
let onchain_code = WrappedRuntimeCode(substrate_test_runtime::wasm_binary_unwrap().into());
337+
let onchain_code = RuntimeCode {
338+
code_fetcher: &onchain_code,
339+
heap_pages: Some(128),
340+
hash: vec![0, 0, 0, 0],
341+
};
342+
343+
let backend = Arc::new(in_mem::Backend::<runtime::Block>::new());
344+
345+
// wasm_runtime_overrides is `None` here because we construct the
346+
// LocalCallExecutor directly later on
347+
let client_config = ClientConfig {
348+
offchain_worker_enabled: false,
349+
offchain_indexing_api: false,
350+
wasm_runtime_overrides: None,
351+
};
352+
353+
// client is used for the convenience of creating and inserting the genesis block.
354+
let _client = substrate_test_runtime_client::client::new_with_backend::<
355+
_,
356+
_,
357+
runtime::Block,
358+
_,
359+
runtime::RuntimeApi,
360+
>(
361+
backend.clone(),
362+
executor.clone(),
363+
&substrate_test_runtime_client::GenesisParameters::default().genesis_storage(),
364+
None,
365+
Box::new(TaskExecutor::new()),
366+
None,
367+
Default::default(),
368+
).expect("Creates a client");
369+
370+
let call_executor = LocalCallExecutor {
371+
backend: backend.clone(),
372+
executor,
373+
wasm_override: Some(overrides),
374+
spawn_handle: Box::new(TaskExecutor::new()),
375+
client_config,
376+
};
377+
378+
let check = call_executor.check_override(onchain_code, &BlockId::Number(Default::default()))
379+
.expect("RuntimeCode override");
380+
381+
assert_eq!(Some(vec![2, 2, 2, 2, 2, 2, 2, 2]), check.fetch_runtime_code().map(Into::into));
382+
}
383+
}

client/service/src/client/client.rs

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ use std::{
2222
marker::PhantomData,
2323
collections::{HashSet, BTreeMap, HashMap},
2424
sync::Arc, panic::UnwindSafe, result,
25+
path::PathBuf
2526
};
2627
use log::{info, trace, warn};
2728
use parking_lot::{Mutex, RwLock};
@@ -181,6 +182,8 @@ pub struct ClientConfig {
181182
pub offchain_worker_enabled: bool,
182183
/// If true, allows access from the runtime to write into offchain worker db.
183184
pub offchain_indexing_api: bool,
185+
/// Path where WASM files exist to override the on-chain WASM.
186+
pub wasm_runtime_overrides: Option<PathBuf>,
184187
}
185188

186189
/// Create a client with the explicitly provided backend.
@@ -201,7 +204,7 @@ pub fn new_with_backend<B, E, Block, S, RA>(
201204
Block: BlockT,
202205
B: backend::LocalBackend<Block> + 'static,
203206
{
204-
let call_executor = LocalCallExecutor::new(backend.clone(), executor, spawn_handle, config.clone());
207+
let call_executor = LocalCallExecutor::new(backend.clone(), executor, spawn_handle, config.clone())?;
205208
let extensions = ExecutionExtensions::new(Default::default(), keystore);
206209
Client::new(
207210
backend,

client/service/src/client/light.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ pub fn new_light<B, S, RA, E>(
6060
code_executor,
6161
spawn_handle.clone(),
6262
ClientConfig::default()
63-
);
63+
)?;
6464
let executor = GenesisCallExecutor::new(backend.clone(), local_executor);
6565
Client::new(
6666
backend,

client/service/src/client/mod.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ pub mod light;
4949
mod call_executor;
5050
mod client;
5151
mod block_rules;
52+
mod wasm_override;
5253

5354
pub use self::{
5455
call_executor::LocalCallExecutor,

0 commit comments

Comments
 (0)