Tr compiler v2 #342

SarcasticNastik · 2022-04-08T00:57:44Z

This PR builds on top of #291. This aims to introduce an efficient version of the tapscript compiler by using a few heuristics to optimize over the expected average total cost for the TapTree.

Strategy implemented

While merging TapTrees A and B, check whether the compilation of Policy::Or(policy(A), policy(B)) is more efficient than a TapTree with the respective children A and B.

Note: This doesn't include the thresh(k, ...n..) enumeration strategy. Planning on working on it separately.

sanket1729

Some partial review of the first few commits

sanket1729 · 2022-04-11T14:27:01Z

src/policy/concrete.rs

-            },
-        )?;
-        Ok(tree)
+        self.is_valid()?; // Check for validity


We need to re-engineer this to not disallow repeated keys in the taproot policy.

Update: Following discussions in issue #338, we currently disallow repeated keys in taproot policy.

My thinking is as follows: we should allow repeated keys in the policy. While doing individual policy compilations to miniscript, we should check this. Not while compiling the entire tr policy. I want to make sure that we support policy compilations that allow same keys in different tree leaves.

As for this PR, this need not concern us.

src/lib.rs

src/policy/compiler.rs

src/policy/concrete.rs

2b13694 Add non-trivial multi-node example (Aman Rojjha) 1c2a80e Add validity and malleability checks. (Aman Rojjha) 0866807 Add P2Tr compiler (Aman Rojjha) 285207e Internal-key extraction done (Aman Rojjha) 26fc574 Policy to single-leaf TapTree compilation done (Aman Rojjha) Pull request description: This PR builds on top of #278. This is one in a series of PRs aimed to implement a feature-complete *Pay-to-Taproot* **P2Tr** compiler. Specifically, this introduces a basic compilation for a given policy by collecting the leaf nodes of the *tree* generated by considering root-level disjunctive `OR`and using this to generate a `TapTree`. > Assuming that _duplicate keys_ are **NOT** possible even in different branches of the TapTree. # Follow Up PRs - #342 - Uses heuristic for tree-generation/ _merging_ algorithm while buillding `TapTree` to optimize over the *expected average total cost*. - #352 - A future PR implementing enumerative strategies for optimizing `thresh` `k`-of-`n` and similar structures. ACKs for top commit: apoelstra: ACK 2b13694 sanket1729: ACK 2b13694 . Let's push forward on this. In the interest of ACKs and multiple rebases already done here, I am merging this. We recently shifted to rust 2018. And this code has a bunch of warnings because of it. Tree-SHA512: 4ceca51a383f5d52b572a16937bbcc3a9c53f0247e4b6df57a6547fd0b1c7cc33ff04dd9a476914bcf6d0a09e255918b8f7ebfe176c839d6ae31c84613dce67f

SarcasticNastik · 2022-05-24T10:07:38Z

@sanket1729 @apoelstra This PR is ready-for-review, given #291 has been merged now. Kindly take a look at it!
Changes in force-push:

Rebase atop master

sanket1729

approach and algorithm ACK. some code improvments to be done

src/policy/concrete.rs

sanket1729 · 2022-05-26T00:11:25Z

src/policy/concrete.rs

+
+/// [`Miniscript`] -> leaf probability in policy cache
+#[cfg(feature = "compiler")]
+type MsTapCache<Pk> = BTreeMap<TapTree<Pk>, f64>;


What happens when there is a collision? Making sure that we are dealing with those cases in a sane manner.

Sure I'll update this! This was done considering that there were no duplicate keys in the initial policy.

There are no duplicates, but still, there can be collisions in the compilation.

Right, that makes sense! Thanks.

sanket1729 · 2022-05-26T00:13:37Z

src/policy/concrete.rs

-            },
-        )?;
-        Ok(tree)
+        self.is_valid()?; // Check for validity


My thinking is as follows: we should allow repeated keys in the policy. While doing individual policy compilations to miniscript, we should check this. Not while compiling the entire tr policy. I want to make sure that we support policy compilations that allow same keys in different tree leaves.

As for this PR, this need not concern us.

src/policy/concrete.rs

sanket1729 · 2022-05-26T00:31:20Z

examples/taproot.rs

+
+fn main() {
+    let policies_str = [
+        "or(1@pk(A),1@pk(B))",


We should make this example as a tutorial.

Take a example policy,

explain what does

explain the compiled result.

Do this a couple of times for interesting results. Using unspendable internal key, useful internal key being inferred etc..

SarcasticNastik · 2022-06-04T07:08:24Z

Changes in the force-push

Addressing error lifting issues.
Update taproot example as suggested.
Handling collisions in cache and added comments for assumptions.
Rebase on top of master.

sanket1729 · 2022-06-06T05:43:50Z

@SarcasticNastik, we recently merged support for no-std. This would require another rebase

sanket1729 · 2022-06-06T06:44:43Z

Hey @SarcasticNastik, looks are still some issues. Let me know if you are having difficulty in dealing with CI with no-std. The general thumb rules are 1) use core instead of std, 2) Common things should be available via use prelude::*

SarcasticNastik · 2022-06-06T07:14:58Z

@sanket1729 Done :p! Thanks for the advice.

sanket1729

Last iteration(I think :P)

src/policy/concrete.rs

sanket1729 · 2022-06-08T05:14:22Z

src/policy/mod.rs

+                .compile_tr(Some(unspendable_key.clone()))
+                .unwrap();
+            let opt_expected_desc = Descriptor::Tr(
+                Tr::<String>::from_str("tr(A,{{pk(D),pk(C)},{pk(B),and_v(v:pk(E),pk(F))}})")


Did we succeed in finding any policy where these differs?

We did find policies where the TapTree leaf nodes re-arranged, but none yet where we noticed policy-merging. One of those were:
Policy: thresh(1,or(1@pk(A),1@pk(B)),or(1@pk(C),1@or(1@and(pk(E),pk(F)),1@pk(D))))
Private compilation: tr(A,{{and_v(v:pk(E),pk(F)),pk(D)},{pk(C),pk(B)}})#y7cmc3xl
Default compilation: tr(A,{{pk(D),pk(C)},{pk(B),and_v(v:pk(E),pk(F))}})#vv7casf3

examples/taproot.rs

SarcasticNastik · 2022-06-09T13:34:58Z

Changes in the force-push

Fix invalid parent compilation bug.
Use serialized keys for creating pubkeys.

SarcasticNastik · 2022-06-17T12:01:12Z

@sanket1729 I've added the document explaining how the private compilation is efficient too and refactored the example.
I'd also appreciate comments on the doc and address the ambiguities and issues, if any.

sanket1729

Last few comments. Almost there

examples/taproot.rs

sanket1729 · 2022-06-17T19:45:47Z

src/policy/concrete.rs

    /// the respective miniscripts. A Huffman Tree is created from this vector which optimizes over
    /// the probabilitity of satisfaction for the respective branch in the TapTree.
+    ///
+    /// Refer to [doc/Tr compiler.pdf] in the root of the repository to understand why such


Ideally, we also want the source code of this uploaded somewhere where I can copy/edit it. In case, we find some errors in the document, there would be no way to edit it.

Can you create a new repo with the code for this? And also link the source here

I've made the gist here and linked it in the docs.

Cargo.toml

This write-up is aimed to reason about how the private-compilation (as defined in doc) is also cost-efficient.

sanket1729

ACK ef4249d

169d849 Add policy to descriptor target compilation method (Aman Rojjha) Pull request description: This PR works on top of #291 and #342. It introduces a new function `compile_to_descriptor` which requires a `DescriptorCtx` (suggestions for a better enum name are welcome!) allowing directly compilation of a policy into the specified descriptor type. ACKs for top commit: apoelstra: ACK 169d849 sanket1729: ACK 169d849 Tree-SHA512: 0f895f4774ea4f56db76280ac756df616a70e5d7d60cc128f9a050ef86658f786b2d45885d748dba51751146a6e93b3731f4ae193b7d80284ffaea20be1e8f92

ef4249d0989dac009c99fada4c5324c523019fd4 Add taproot compiler example usage (Aman Rojjha) cef2a5b604de584518b3a9574a86fb1c449b65e7 Add Tr-compiler write-up and doc-comment (Aman Rojjha) 5233c6645581e6e5a5afee55946193210f16b7d4 Add Taproot compiler API (Aman Rojjha) Pull request description: This PR builds on top of #291. This aims to introduce an efficient version of the tapscript compiler by using a few heuristics to optimize over the expected average total cost for the TapTree. ## Strategy implemented - While merging TapTrees `A` and `B`, check whether the compilation of `Policy::Or(policy(A), policy(B))` is more efficient than a TapTree with the respective children `A` and `B`. **Note**: This doesn't include the `thresh(k, ...n..)` enumeration strategy. Planning on working on it separately. ACKs for top commit: sanket1729: ACK ef4249d0989dac009c99fada4c5324c523019fd4 Tree-SHA512: ae5949b5170ff4cc8202434655af41b4938e96801209896d117bf4e4bd85c0becc6fd0c7affb100c31655f0085c1ff5e7c19184db3433b2831f73db22d94348d

SarcasticNastik mentioned this pull request Apr 8, 2022

Update to bitcoin 0.28.0 release #337

Closed

sanket1729 reviewed Apr 11, 2022

View reviewed changes

SarcasticNastik force-pushed the tr-compiler-v2 branch 3 times, most recently from 05f2431 to 219119d Compare April 19, 2022 21:17

SarcasticNastik mentioned this pull request Apr 19, 2022

Allow direct compilation of a Policy into a valid Descriptor #352

Merged

SarcasticNastik marked this pull request as ready for review April 19, 2022 23:25

SarcasticNastik changed the title ~~[DO NOT MERGE]Tr compiler v2~~ Tr compiler v2 Apr 19, 2022

SarcasticNastik force-pushed the tr-compiler-v2 branch from 219119d to 989176b Compare May 4, 2022 14:03

SarcasticNastik mentioned this pull request May 9, 2022

Taproot Compiler #291

Merged

SarcasticNastik force-pushed the tr-compiler-v2 branch 3 times, most recently from 01c3f71 to da16d05 Compare May 11, 2022 10:05

SarcasticNastik force-pushed the tr-compiler-v2 branch from da16d05 to 83a685d Compare May 24, 2022 14:05

sanket1729 reviewed May 26, 2022

View reviewed changes

SarcasticNastik force-pushed the tr-compiler-v2 branch from 83a685d to d384b63 Compare June 4, 2022 06:59

SarcasticNastik force-pushed the tr-compiler-v2 branch 4 times, most recently from 5b2f024 to 6ce9fb1 Compare June 4, 2022 13:11

SarcasticNastik force-pushed the tr-compiler-v2 branch 2 times, most recently from 375b94b to 2940416 Compare June 6, 2022 06:07

SarcasticNastik force-pushed the tr-compiler-v2 branch from 2940416 to 7828b38 Compare June 6, 2022 06:59

sanket1729 reviewed Jun 8, 2022

View reviewed changes

SarcasticNastik force-pushed the tr-compiler-v2 branch 2 times, most recently from d4f3245 to 5c456d7 Compare June 9, 2022 13:31

SarcasticNastik force-pushed the tr-compiler-v2 branch from 5c456d7 to 7a5db70 Compare June 9, 2022 14:00

SarcasticNastik force-pushed the tr-compiler-v2 branch 3 times, most recently from 1c34908 to faa4e16 Compare June 17, 2022 11:58

sanket1729 reviewed Jun 17, 2022

View reviewed changes

SarcasticNastik force-pushed the tr-compiler-v2 branch from faa4e16 to 7dc91d9 Compare June 17, 2022 20:59

SarcasticNastik added 3 commits June 18, 2022 02:38

Add Taproot compiler API

5233c66

Add Tr-compiler write-up and doc-comment

cef2a5b

This write-up is aimed to reason about how the private-compilation (as defined in doc) is also cost-efficient.

Add taproot compiler example usage

ef4249d

SarcasticNastik force-pushed the tr-compiler-v2 branch from 7dc91d9 to ef4249d Compare June 17, 2022 21:10

sanket1729 approved these changes Jun 17, 2022

View reviewed changes

sanket1729 merged commit 0662d2e into rust-bitcoin:master Jun 17, 2022

SarcasticNastik deleted the tr-compiler-v2 branch June 18, 2022 06:06

Uh oh!

Tr compiler v2 #342

Tr compiler v2 #342

Uh oh!

Conversation

SarcasticNastik commented Apr 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Strategy implemented

Uh oh!

sanket1729 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SarcasticNastik commented May 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanket1729 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SarcasticNastik Jun 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SarcasticNastik commented Jun 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes in the force-push

Uh oh!

sanket1729 commented Jun 6, 2022

Uh oh!

sanket1729 commented Jun 6, 2022

Uh oh!

SarcasticNastik commented Jun 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanket1729 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

SarcasticNastik commented Jun 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes in the force-push

Uh oh!

SarcasticNastik commented Jun 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanket1729 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SarcasticNastik commented Apr 8, 2022 •

edited

Loading

SarcasticNastik commented May 24, 2022 •

edited

Loading

SarcasticNastik Jun 2, 2022 •

edited

Loading

SarcasticNastik commented Jun 4, 2022 •

edited

Loading

SarcasticNastik commented Jun 6, 2022 •

edited

Loading

SarcasticNastik commented Jun 9, 2022 •

edited

Loading

SarcasticNastik commented Jun 17, 2022 •

edited

Loading

SarcasticNastik Jun 17, 2022 •

edited

Loading