Benchmarking #248
Benchmarking #248
Conversation
devmotion
left a comment
devmotion
left a comment
There was a problem hiding this comment.
I assume this can be very helpful 👍
Maybe put it in /benchmarks/? Since it's for benchmarking DynamicPPL specifically it seems fine to me to put it in this repo.
Good point 👍 I'll also add README.md to the folder specifying how it can be used. |
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: David Widmann <[email protected]>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
@devmotion Could you maybe have another look at this? Found it extremely useful when evaluating whether #269 and #271 will cause issues, so it would be nice to just get it merged. Then we can develop it further once it's there, and potentially separate it out into it's own project if it reaches sufficient complexity/size. |
|
bors r+ |
Changes to DPPL can often have quite significant effects for compilation time and performance of both itself and downstream packages. It's also sometimes difficult to discover these performance regressions. E.g. in #221 we made a small simplification to the compiler and it ended up taking quite a while to figure out what was going wrong and had to test several models to identify the issue. So, this is a WIP PR for including a small set of models which we can `weave` into a document where we can look at the changes. It's unclear to me whether this should go in DPPL itself or in a separate package. I found it useful myself and figured I'd put it here so we can start maybe get some "standard" benchmarks to run for testing purposes. IMO we don't need many of them, as we will add more as we go along. For each model the following will be included in the document: - Benchmarked evaluation of the model on untyped and typed `VarInfo`. - Timing of the compilation of the model in the typed `VarInfo`. - Lowered code for the model. - If `:prefix` is provided to `weave`, the string-representation of `code_typed` for the evaluation of the model will be saved to a file `$(prefix)_(model.name)`. Furthermore, if `:prefix_old` is provided, pointing to `:prefix` used for a previous run (likely using a different version of DPPL), we will `diff` the `code_typed` for the two models by loading the saved files.
|
Build failed: |
|
@torfjelde Some tests are failing in Related #268 |
Seems like there has been some downstream changes that are causing these. Having a look.
Can you elaborate on why #268 is related? Not quite seeing the connection 😕 EDIT: Did you mean to point to #272 ? |
|
Also, this PR makes absolutely no changes to DynamicPPL's functionality; it only touches Still need to figure out what is breaking the tests though! |
Sorry for the confusion - I mean we are experiencing similar falling tests there. |
Changes to DPPL can often have quite significant effects for compilation time and performance of both itself and downstream packages. It's also sometimes difficult to discover these performance regressions.
E.g. in #221 we made a small simplification to the compiler and it ended up taking quite a while to figure out what was going wrong and had to test several models to identify the issue.
So, this is a WIP PR for including a small set of models which we can
weaveinto a document where we can look at the changes. It's unclear to me whether this should go in DPPL itself or in a separate package. I found it useful myself and figured I'd put it here so we can start maybe get some "standard" benchmarks to run for testing purposes. IMO we don't need many of them, as we will add more as we go along.For each model the following will be included in the document:
VarInfo.VarInfo.:prefixis provided toweave, the string-representation ofcode_typedfor the evaluation of the model will be saved to a file$(prefix)_(model.name). Furthermore, if:prefix_oldis provided, pointing to:prefixused for a previous run (likely using a different version of DPPL), we willdiffthecode_typedfor the two models by loading the saved files.