Skip to content

SMT back-end: add support for Bitwuzla and CVC 5 #7519

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 4, 2023

Conversation

tautschnig
Copy link
Collaborator

Adds support for two actively developed SMT solvers, with the caveat that these do not currently pass all regression tests (Bitwuzla fails two tests, CVC 5 times out on one and fails on nine others). Some performance data running (in regression/cbmc, for varying values of <SOLVER>):

/usr/bin/time -v ../test.pl -e -c "../../../build/bin/cbmc <SOLVER>" \
  -X broken-smt-backend -X thorough-smt-backend \
  -X broken-cprover-smt-backend -X thorough-cprover-smt-backend \
  -X broken-z3-smt-backend -X thorough-z3-smt-backend

Note that the following data has bias: it only has the subset of tests that all back-ends are expected to be able to solve. That is, some SMT solver may be faster on this minimum common set of tests, but might be unable to solve tests that another (possibly slower) solver can handle.

  • Bitwuzla (--bitwuzla for <SOLVER>):
  User time (seconds): 90.59
  System time (seconds): 8.68
  Elapsed (wall clock) time (h:mm:ss or m:ss): 1:39.31
  Maximum resident set size (kbytes): 81240
  Exit status: 2
  • CVC 5 (--cvc5 for <SOLVER>):
  User time (seconds): 483.24
  System time (seconds): 10.69
  Elapsed (wall clock) time (h:mm:ss or m:ss): 8:14.18
  Maximum resident set size (kbytes): 473876
  Exit status: 10
  • Z3 (--z3 for <SOLVER>):
  User time (seconds): 202.90
  System time (seconds): 11.29
  Elapsed (wall clock) time (h:mm:ss or m:ss): 3:34.21
  Maximum resident set size (kbytes): 131760
  Exit status: 0
  • In-tree SMT solver (--cprover-smt2 for <SOLVER>):
  User time (seconds): 64.08
  System time (seconds): 9.44
  Elapsed (wall clock) time (h:mm:ss or m:ss): 1:13.51
  Maximum resident set size (kbytes): 81932
  Exit status: 0
  • SAT back-end (omitting <SOLVER>):
  User time (seconds): 33.43
  System time (seconds): 7.94
  Elapsed (wall clock) time (h:mm:ss or m:ss): 0:41.29
  Maximum resident set size (kbytes): 79088
  Exit status: 0
  • Each commit message has a non-empty body, explaining why the change was made.
  • n/a Methods or procedures I have added are documented, following the guidelines provided in CODING_STANDARD.md.
  • The feature or user visible behaviour I have added or modified has been documented in the User Guide in doc/cprover-manual/
  • Regression or unit tests are included, or existing tests cover the modified code (in this case I have detailed which ones those are in the commit message).
  • My commit message includes data points confirming performance improvements (if claimed).
  • My PR is restricted to a single feature or bugfix.
  • n/a White-space or formatting changes outside the feature-related changed lines are in commits of their own.

Adds support for two actively developed SMT solvers, with the caveat
that these do not currently pass all regression tests (Bitwuzla fails
two tests, CVC 5 times out on one and fails on nine others). Some
performance data running (in `regression/cbmc`, for varying values of
`<SOLVER>`):
```
/usr/bin/time -v ../test.pl -e -c "../../../build/bin/cbmc <SOLVER>" \
  -X broken-smt-backend -X thorough-smt-backend \
  -X broken-cprover-smt-backend -X thorough-cprover-smt-backend \
  -X broken-z3-smt-backend -X thorough-z3-smt-backend
```
Note that the following data has bias: it only has the subset of tests
that all back-ends are expected to be able to solve. That is, some SMT
solver may be faster on this minimum common set of tests, but might be
unable to solve tests that another (possibly slower) solver can handle.

* Bitwuzla (`--bitwuzla` for `<SOLVER>`):
```
  User time (seconds): 90.59
  System time (seconds): 8.68
  Elapsed (wall clock) time (h:mm:ss or m:ss): 1:39.31
  Maximum resident set size (kbytes): 81240
  Exit status: 2
```
* CVC 5 (`--cvc5` for `<SOLVER>`):
```
  User time (seconds): 483.24
  System time (seconds): 10.69
  Elapsed (wall clock) time (h:mm:ss or m:ss): 8:14.18
  Maximum resident set size (kbytes): 473876
  Exit status: 10
```
* Z3 (`--z3` for `<SOLVER>`):
```
  User time (seconds): 202.90
  System time (seconds): 11.29
  Elapsed (wall clock) time (h:mm:ss or m:ss): 3:34.21
  Maximum resident set size (kbytes): 131760
  Exit status: 0
```
* In-tree SMT solver (`--cprover-smt2` for `<SOLVER>`):
```
  User time (seconds): 64.08
  System time (seconds): 9.44
  Elapsed (wall clock) time (h:mm:ss or m:ss): 1:13.51
  Maximum resident set size (kbytes): 81932
  Exit status: 0
```
* SAT back-end (omitting `<SOLVER>`):
```
  User time (seconds): 33.43
  System time (seconds): 7.94
  Elapsed (wall clock) time (h:mm:ss or m:ss): 0:41.29
  Maximum resident set size (kbytes): 79088
  Exit status: 0
```
@codecov
Copy link

codecov bot commented Feb 4, 2023

Codecov Report

Base: 78.50% // Head: 78.49% // Decreases project coverage by -0.02% ⚠️

Coverage data is based on head (8fdb8bb) compared to base (3f578e1).
Patch coverage: 20.00% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #7519      +/-   ##
===========================================
- Coverage    78.50%   78.49%   -0.02%     
===========================================
  Files         1663     1663              
  Lines       191297   191331      +34     
===========================================
+ Hits        150174   150180       +6     
- Misses       41123    41151      +28     
Impacted Files Coverage Δ
src/goto-checker/solver_factory.h 100.00% <ø> (ø)
src/solvers/smt2/smt2_conv.cpp 67.26% <0.00%> (-0.38%) ⬇️
src/solvers/smt2/smt2_conv.h 87.50% <ø> (ø)
src/solvers/smt2/smt2_dec.cpp 70.16% <25.00%> (-3.12%) ⬇️
src/goto-checker/solver_factory.cpp 81.23% <55.55%> (-0.79%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@kroening kroening merged commit d6c318e into diffblue:develop Feb 4, 2023
@tautschnig tautschnig deleted the feature/bitwuzla-cvc5 branch February 4, 2023 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants