fix --early-exit #437
fix --early-exit #437
Conversation
|
Things I checked:
Things I still need to check:
|
| @@ -0,0 +1,174 @@ | |||
| import json | |||
There was a problem hiding this comment.
no changes here, just moving build-related functionality out of __main__.py and into its own module
| @@ -0,0 +1,160 @@ | |||
| import io | |||
There was a problem hiding this comment.
no changes here, just moving trace-related functionality out of __main__.py and into its own module
| @@ -0,0 +1,217 @@ | |||
| import concurrent.futures | |||
There was a problem hiding this comment.
new module that provides an executor/futures abstraction layer built on top of subprocess.Popen
src/halmos/solve.py
Outdated
| return SolverOutput.from_result(stdout, stderr, returncode, path_ctx) | ||
|
|
||
|
|
||
| def solve_end_to_end(ctx: PathContext) -> None: |
There was a problem hiding this comment.
used to be gen_model_from_sexpr
| @dataclass(frozen=True) | ||
| class PathContext: | ||
| # id of this path | ||
| path_id: int |
There was a problem hiding this comment.
roughly corresponds to the old iteration idx
|
|
||
| refined_ctx = ctx.refine() | ||
|
|
||
| if refined_ctx.query.smtlib != query.smtlib: |
| @@ -1411,6 +817,8 @@ def _main(_args=None) -> MainResult: | |||
| # | |||
|
|
|||
| def on_exit(exitcode: int) -> MainResult: | |||
| ExecutorRegistry().shutdown_all() | |||
There was a problem hiding this comment.
I don't understand what's going on, this sounds like a safe thing to do (shut down external processes) but when I do this I can somewhat consistently get the following crash when triggering a ctrl-c at the right time:
halmos --root ~/projects/halmos-sandbox --function test_manyCexes --solver-command "yices-smt2 --smt2-model-format" --early-exit ✘ INT took 4s
[⠢] Compiling...
No files changed, compilation skipped
Running 1 tests for test/60_manyCexes.t.sol:Test60
setup: 0.01s (decode: 0.01s)
Generating SMT queries in /tmp/test_manyCexes-2459149073354744aa4b653b5fb94c9f
^C
Shutting down all executors
ASSERTION VIOLATION
File: ../src/ast/ast.cpp
Line: 406
UNEXPECTED CODE WAS REACHED.
Z3 4.12.6.0
Please file an issue with this message and more detail about how you encountered it at https://github.com/Z3Prover/z3/issues/new
(note that the interrupted assertion solver is yices, not z3 -- I don't understand why z3 is involved in this)
0xkarmacoma
force-pushed
the
fix/early-exit
branch
from
625d4df to
bfce538
Compare
(hopefully works on windows too)
0xkarmacoma
force-pushed
the
fix/early-exit
branch
from
1e8c6cd to
5068e59
Compare
|
TODO: fix the case where:
ERROR encountered exception during assertion solving: RuntimeError('Cannot submit to a shutdown executor.')
ERROR exception calling callback for <Future at 0x112aee4e0 state=finished raised RuntimeError>
Traceback (most recent call last):
File "/Users/karma/.pyenv/versions/3.12.1/lib/python3.12/concurrent/futures/_base.py", line 340, in _invoke_callbacks
callback(self)
File "/Users/karma/projects/halmos/src/halmos/__main__.py", line 472, in solve_end_to_end_callback
solver_output = future.result()
^^^^^^^^^^^^^^^
File "/Users/karma/.pyenv/versions/3.12.1/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/Users/karma/.pyenv/versions/3.12.1/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/Users/karma/.pyenv/versions/3.12.1/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/karma/projects/halmos/src/halmos/solve.py", line 438, in solve_end_to_end
solver_output = solve_low_level(ctx)
^^^^^^^^^^^^^^^^^^^^
File "/Users/karma/projects/halmos/src/halmos/solve.py", line 399, in solve_low_level
path_ctx.solving_ctx.executor.submit(future)
File "/Users/karma/projects/halmos/src/halmos/processes.py", line 161, in submit
raise RuntimeError("Cannot submit to a shutdown executor.")
RuntimeError: Cannot submit to a shutdown executor. We need some way to also interrupt path exploration when early exit is triggered. |
✅ in 1fc8649 |
| render_trace(setup_ex.context) | ||
|
|
||
| else: | ||
| setup_exs_no_error.append(setup_ex) |
There was a problem hiding this comment.
this refactoring is incorrect. the smt query needs to be generated at this point. the solver object is shared across paths, and solver.to_smt2() will return a different query if it is called after a different path is explored.
alternatively, we could consider modifying path.to_smt2() to not rely on the shared solver. the idea is to create a temporary new solver with a separate z3 context, add all path constraints to the new solver, and generate a smt query from it. this indeeds already happens when --cache-solver is enabled. so this approach would improve the code maintainability avoiding this potential mistake in the future. but, a downside is the additional performance overhead of creating a new solver and translating all constraints to the new context. wdyt?
| path_ctx = PathContext( | ||
| args=args, | ||
| path_id=path_id, | ||
| query=ex.path.to_smt2(args), |
There was a problem hiding this comment.
as mentioned above, calling path.to_smt2() here will generate an incorrect query unless it's the last path.
the ci failure might be related to this:
https://github.com/a16z/halmos/actions/runs/12958246417/job/36148232775#step:9:27
| libs: dict | ||
|
|
||
| # TODO: check if this is really a contract-level variable | ||
| build_out_map: dict |
There was a problem hiding this comment.
fyi, build_out_map is shared across all contracts compiled using the same compiler version. so in principle, we could consider having another context, say CompileUnitContext, and put build_out_map there, but it might be overkill unless there are other compiler specific items. (the current approach seems reasonable to me.)
There was a problem hiding this comment.
agree, I will leave a comment
| def __str__(self) -> str: | ||
| # expected to be a filename | ||
| if isinstance(self.model, str): | ||
| return f"see {self.model}" |
There was a problem hiding this comment.
do we still need this? we now parse the solver output, and no longer show the filename, right?
There was a problem hiding this comment.
that's right! forgot about this
|
|
||
| if refined_ctx.query.smtlib != query.smtlib: | ||
| # recurse with the refined query | ||
| return solve_end_to_end(refined_ctx) |
There was a problem hiding this comment.
why solve_end_to_end instead of solve_low_level? note that check_unsat_cores cannot return true for the refined query, because it relies on only constraint ids, which don't change after refinement.
There was a problem hiding this comment.
ah good catch, I wasn't sure so solve_end_to_end seemed more general. Can switch it to solve_low_level if you think that works
There was a problem hiding this comment.
yes, solve_low_level() should work here: resulting in the same output, and avoiding the unnecessary unsat_core loop.
and fix off by one error in num_execs
Co-authored-by: Daejun Park <[email protected]>
fixes #243
high-level notes:
thread_pool.shutdown(wait=False)was not what we needed it to be)PopenExecutor.shutdown(wait=False), it does terminate/kill every process that has been submitted to that executor and is still runningsolve_end_to_endfunction calls to a thread pool, but that functions submitssolve_low_levelcalls to the executor and blocks on the result.solve_low_levelcan be interrupted if the underlying process returns, times out, gets killed, etc -- this is the key architectural change--solver-commanddefaults toz3) and we need to parse the model values from the smtlib output. This means we no longer printCounterexample: see xyz.smt2Additionally, to make the transition to this model easier I introduced a hierarchy of context objects: