Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash when using Unigen Python bindings #9

Closed
Eric-Vin opened this issue Dec 14, 2021 · 14 comments
Closed

Crash when using Unigen Python bindings #9

Eric-Vin opened this issue Dec 14, 2021 · 14 comments

Comments

@Eric-Vin
Copy link
Contributor

I've started using the Python bindings in my application and at the same time writing some tests that more vigorously test their usage. I've uncovered a situation where a CryptominiSat assert is failing. It only happens with certain formula/seed combinations as well. The assertion is:

cryptominisat/src/solver.cpp:4370: void CMSat::Solver::detach_xor_clauses(const std::set<unsigned int>&): Assertion seen[v] == 0 failed.

and this is the code block it's referencing (Line 4370 in Solver.cpp):

//Clash on USED xor
for(auto& x: xorclauses) {
x.detached = true;
for(const uint32_t v: x.clash_vars) {
assert(seen[v] == 0);
seen[v] = 2;
}
}

I can't seem to reproduce this on the command line interface, which might be because it does the counting first, but I think it's more likely I've done something wrong in the Python bindings. Do you have any insight as to what this issue is? My instinct is that I'm not setting something up correctly when I bypass the ApproxMC counting process.

I'm also more than happy to provide info on how to reproduce the bug using my tests if you're interested!

@msoos
Copy link
Collaborator

msoos commented Dec 14, 2021

Hi,

Thanks for the bug report. I'd need a full backtrace -- can you please run this under gdb so I can see the backtrace of when it's being called? Just run gdb ./whatever.pyand then when it exits, runbt`. Or, if you code is not super-secret, you can give me the python code that reproduces the issue and debug and fix it, easier for you! Thanks in advance,

Mate

@Eric-Vin
Copy link
Contributor Author

Eric-Vin commented Dec 14, 2021

Hi Mate,

Sure! Here's a link to the repo on the develop branch (https://github.com/UCSCFormalMethods/CIToolkit/tree/develop). I just pushed a commit that exhibits the bug consistently. You can set up the virtual environment by running make, and then you'll have to install the pyapproxmc and pyunigen bindings manually (pip3 install . in the python directory for each tool).

At that point the test in question can be run with the command pytest -s -k test_z3_formula_basic in the project root directory. If you want to run the test manually for easier use with gdb, you can just navigate to the tests directory and run python3 test_z3_formula.py. For that test, it's consistently crashing when sampling with seed 227 (It'll try to run every seed between 1 and 1000). Thanks so much for your help and please let me know what else I can help with!

Eric

@Eric-Vin
Copy link
Contributor Author

Eric-Vin commented Jan 3, 2022

Hi @msoos,

I just wanted to check in and see if I can be of any more help debugging/resolving this.

Thanks!
Eric

@msoos
Copy link
Collaborator

msoos commented Jan 3, 2022

Hey,

Sorry for the late response. It was a bit overwhelming to set up an entire build environment, etc. to debug this. Can you please give me a backtrace? Thanks,

Mate

@Eric-Vin
Copy link
Contributor Author

Eric-Vin commented Jan 4, 2022

No problem! Hopefully this is what you're looking for, but if not please let me know. Thanks again for your help and please let me know how else I can help!

GNU gdb (Ubuntu 10.1-2ubuntu2) 10.1.90.20210411-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...
Reading symbols from /usr/lib/debug/.build-id/64/8f4ad2b5a3e0b29ef3de84e842f34f671e47c8.debug...
Starting program: /home/ericvin/.cache/pypoetry/virtualenvs/citoolkit-dbTxSEV7-py3.9/bin/python3 test_z3_formula.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
227
python3: /home/ericvin/Downloads/cryptominisat/src/solver.cpp:4370: void CMSat::Solver::detach_xor_clauses(const std::set<unsigned int>&): Assertion `seen[v] == 0' failed.

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
49	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  0x00007ffff7c0d864 in __GI_abort () at abort.c:79
#2  0x00007ffff7c0d749 in __assert_fail_base (fmt=0x7ffff7d96f78 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ffff422c9a2 "seen[v] == 0", 
    file=0x7ffff422cef0 "/home/ericvin/Downloads/cryptominisat/src/solver.cpp", line=4370, function=<optimized out>) at assert.c:92
#3  0x00007ffff7c1f3d6 in __GI___assert_fail (assertion=assertion@entry=0x7ffff422c9a2 "seen[v] == 0", 
    file=file@entry=0x7ffff422cef0 "/home/ericvin/Downloads/cryptominisat/src/solver.cpp", line=line@entry=4370, 
    function=function@entry=0x7ffff422ddd8 "void CMSat::Solver::detach_xor_clauses(const std::set<unsigned int>&)") at assert.c:101
#4  0x00007ffff41c4ced in CMSat::Solver::detach_xor_clauses (this=0x10e2c00, clash_vars_unused=std::set with 0 elements) at /home/ericvin/Downloads/cryptominisat/src/solver.cpp:4370
#5  0x00007ffff41cb5b5 in CMSat::Solver::find_and_init_all_matrices (this=0x10e2c00) at /home/ericvin/Downloads/cryptominisat/src/solver.cpp:3941
#6  0x00007ffff41cb891 in CMSat::Solver::iterate_until_solved (this=this@entry=0x10e2c00) at /home/ericvin/Downloads/cryptominisat/src/solver.cpp:1867
#7  0x00007ffff41cbdb4 in CMSat::Solver::solve_with_assumptions (this=0x10e2c00, _assumptions=_assumptions@entry=0x7fffffffca30, only_sampling_solution=only_sampling_solution@entry=true)
    at /home/ericvin/Downloads/cryptominisat/src/solver.cpp:1650
#8  0x00007ffff4220b2b in calc (assumptions=0x7fffffffca30, solve=<optimized out>, data=0xc1fff0, only_sampling_solution=<optimized out>) at /usr/include/c++/10/bits/stl_vector.h:1043
#9  0x00007ffff3d651e6 in Sampler::bounded_sol_count (hm=0x0, out_solutions=0x0, minSolutions=12, hashCount=2, assumps=0x7fffffffca10, maxSolutions=67, this=0x10c9e90)
    at /home/ericvin/Downloads/unigen/src/sampler.cpp:202
#10 Sampler::gen_n_samples (this=0x10c9e90, num_calls=1, lastSuccessfulHashOffset=0x7fffffffcdf0, num_samples_needed=1) at /home/ericvin/Downloads/unigen/src/sampler.cpp:480
#11 0x00007ffff3d66491 in Sampler::generate_samples (this=0x10c9e90, num_samples_needed=1) at /home/ericvin/Downloads/unigen/src/sampler.cpp:403
#12 0x00007ffff3d68648 in UniGen::UniG::sample (this=<optimized out>, sol_count=sol_count@entry=0x7fffffffcf2c, num_samples=num_samples@entry=1)
    at /home/ericvin/Downloads/unigen/src/unigen.cpp:84
#13 0x00007ffff625d7e1 in sample (self=0x7ffff42afe30, args=(64, 2), kwds=0x0) at src/pyunigen.cpp:266
#14 0x0000000000543b6c in cfunction_call (func=<built-in method sample of pyapproxmc.Sampler object at remote 0x7ffff42afe30>, args=<optimized out>, kwargs=<optimized out>)
    at ../Objects/methodobject.c:539
#15 0x0000000000540c31 in _PyObject_Call (kwargs=<optimized out>, args=(64, 2), callable=<built-in method sample of pyapproxmc.Sampler object at remote 0x7ffff42afe30>, tstate=0x96db20)
    at ../Objects/call.c:281
#16 PyObject_Call (callable=<built-in method sample of pyapproxmc.Sampler object at remote 0x7ffff42afe30>, args=(64, 2), kwargs=<optimized out>) at ../Objects/call.c:293
#17 0x000000000051c5cd in do_call_core (kwdict=0x0, callargs=(64, 2), func=<built-in method sample of pyapproxmc.Sampler object at remote 0x7ffff42afe30>, tstate=<optimized out>)
    at ../Python/ceval.c:5092
#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3580
#19 0x0000000000514a95 in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7ffff615ebe0, for file /home/ericvin/Projects/CIToolkit/citoolkit/specifications/bool_formula.py, line 118, in sample (self=<BoolFormula(alphabet=frozenset({0, 1}), clauses=[(28, -29, 30), (28, 29, -30), (-28, -29, -30), (29, -28, 30), (-31, 32), (-33, 32), (32, -28), (-34, 32), (32, -35), (32, -36), (31, 33, 28, 34, 35, 36, -32), (37, -38), (-32, 37, -39), (38, 39, -37), (38, 32, -37), (37, -40), (40, -37), (-29, 41), (41, -42), (41, -43), (41, -44), (41, -45), (41, -46), (29, 42, -41, 43, 44, 45, 46), (-47, 48), (48, -41, -49), (47, 49, -48), (47, 41, -48), (48, -50), (50, -48), (51, -52), (51, -53), (-54, 51), (-55, 51), (51, -30), (51, -56), (52, 53, 54, 55, -51, 30, 56), (57, -58), (57, -59, -51), (-57, 58, 59), (58, 51, -57), (57, -60), (60, -57), (-58,), (28, -1), (1, -28), (33, -2), (2, -33), (36, -3), (3, -36), (31, -4), (4, -31), (34, -5), (5, -34), (35, -6), (6, -35), (39, -7), (7, -39), (38, -8), (8, -38), (29, -9), (9, -29), (46, -10), (10, -46), (43, -11), (11, -43), (42, -12), (12, -42)...(truncated), tstate=0x96db20) at ../Include/internal/pycore_ceval.h:40
#20 _PyEval_EvalCode (tstate=0x96db20, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x7ffff6b5ba18, 
    kwargs=0x7ffff429f5b0, kwcount=<optimized out>, kwstep=1, defs=0x7ffff42840f8, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name='sample', qualname='BoolFormula.sample')
    at ../Python/ceval.c:4327
#21 0x000000000052d572 in _PyFunction_Vectorcall (func=func@entry=<function at remote 0x7ffff428c310>, stack=stack@entry=0x7ffff429f5a0, nargsf=nargsf@entry=2, kwnames=<optimized out>)
    at ../Objects/call.c:396
#22 0x00000000005405cf in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=2, args=0x7ffff429f5a0, callable=<function at remote 0x7ffff428c310>, tstate=0x96db20)
    at ../Include/cpython/abstract.h:118
#23 method_vectorcall (method=<optimized out>, args=0x7ffff429f5a8, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/classobject.c:53
#24 0x00000000005178b5 in _PyObject_VectorcallTstate (kwnames=('seed',), nargsf=<optimized out>, args=<optimized out>, callable=<method at remote 0x7ffff42a4dc0>, tstate=0x96db20)
    at ../Include/cpython/abstract.h:118
#25 PyObject_Vectorcall (kwnames=('seed',), nargsf=<optimized out>, args=<optimized out>, callable=<method at remote 0x7ffff42a4dc0>) at ../Include/cpython/abstract.h:127
#26 call_function (kwnames=('seed',), oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at ../Python/ceval.c:5072
#27 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3535
#28 0x0000000000514a95 in _PyEval_EvalFrame (throwflag=0, 
--Type <RET> for more, q to quit, c to continue without paging--
    f=Frame 0x7ffff429f400, for file /home/ericvin/Projects/CIToolkit/citoolkit/specifications/z3_formula.py, line 279, in sample (self=<Z3Formula(formula=<BoolRef(ast=<Ast at remote 0x7ffff428b640>, ctx=<Context(ctx=<ContextObj at remote 0x7ffff5f20f40>, eh=<CFunctionType at remote 0x7ffff4286280>) at remote 0x7ffff6a60fd0>) at remote 0x7ffff4289c70>, main_variables=[('x', 'BitVec', 8), ('y', 'BitVec', 8), ('z', 'BitVec', 8), ('x_big', 'Bool', 1), ('y_big', 'Bool', 1), ('z_big', 'Bool', 1)], bool_formula_spec=<BoolFormula(alphabet=frozenset({0, 1}), clauses=[(28, -29, 30), (28, 29, -30), (-28, -29, -30), (29, -28, 30), (-31, 32), (-33, 32), (32, -28), (-34, 32), (32, -35), (32, -36), (31, 33, 28, 34, 35, 36, -32), (37, -38), (-32, 37, -39), (38, 39, -37), (38, 32, -37), (37, -40), (40, -37), (-29, 41), (41, -42), (41, -43), (41, -44), (41, -45), (41, -46), (29, 42, -41, 43, 44, 45, 46), (-47, 48), (48, -41, -49), (47, 49, -48), (47, 41, -48), (48, -50), (50, -48), (51, -52), (51, -53), (-54, 51), (-55, 51), (51, -...(truncated), tstate=0x96db20) at ../Include/internal/pycore_ceval.h:40
#29 _PyEval_EvalCode (tstate=0x96db20, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x7ffff6c7c508, 
    kwargs=0xd939c0, kwcount=<optimized out>, kwstep=1, defs=0x7ffff4281468, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name='sample', qualname='Z3Formula.sample')
    at ../Python/ceval.c:4327
#30 0x000000000052d572 in _PyFunction_Vectorcall (func=func@entry=<function at remote 0x7ffff428c8b0>, stack=stack@entry=0xd939b8, nargsf=nargsf@entry=1, kwnames=<optimized out>)
    at ../Objects/call.c:396
#31 0x00000000005405cf in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=1, args=0xd939b8, callable=<function at remote 0x7ffff428c8b0>, tstate=0x96db20)
    at ../Include/cpython/abstract.h:118
#32 method_vectorcall (method=<optimized out>, args=0xd939c0, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/classobject.c:53
#33 0x00000000005178b5 in _PyObject_VectorcallTstate (kwnames=('seed',), nargsf=<optimized out>, args=<optimized out>, callable=<method at remote 0x7ffff42a4f80>, tstate=0x96db20)
    at ../Include/cpython/abstract.h:118
#34 PyObject_Vectorcall (kwnames=('seed',), nargsf=<optimized out>, args=<optimized out>, callable=<method at remote 0x7ffff42a4f80>) at ../Include/cpython/abstract.h:127
#35 call_function (kwnames=('seed',), oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at ../Python/ceval.c:5072
#36 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3535
#37 0x000000000052d3d3 in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0xd937f0, for file /home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py, line 42, in test_z3_formula_basic (x=<BitVecRef(ast=<Ast at remote 0x7ffff428b040>, ctx=<Context(ctx=<ContextObj at remote 0x7ffff5f20f40>, eh=<CFunctionType at remote 0x7ffff4286280>) at remote 0x7ffff6a60fd0>) at remote 0x7ffff6a60f10>, y=<BitVecRef(ast=<Ast at remote 0x7ffff428b240>, ctx=<...>) at remote 0x7ffff427d640>, z=<BitVecRef(ast=<Ast at remote 0x7ffff428b2c0>, ctx=<...>) at remote 0x7ffff427d6a0>, x_big=<BoolRef(ast=<Ast at remote 0x7ffff428b340>, ctx=<...>) at remote 0x7ffff6a60f70>, y_big=<BoolRef(ast=<Ast at remote 0x7ffff428b3c0>, ctx=<...>) at remote 0x7ffff4289460>, z_big=<BoolRef(ast=<Ast at remote 0x7ffff428b440>, ctx=<...>) at remote 0x7ffff4289790>, formula=<BoolRef(ast=<Ast at remote 0x7ffff428b4c0>, ctx=<...>) at remote 0x7ffff42899a0>, main_variables=[('x', 'BitVec', 8), ('y', 'BitVec', 8), ('z', 'BitVec', 8), ('x_big', 'Bool', 1), ('y_big', 'Bool', 1), ('z_big', 'Bool', 1)], spec=<Z3Formula(formula=<B...(truncated), tstate=0x96db20) at ../Include/internal/pycore_ceval.h:40
#38 function_code_fastcall (globals=<optimized out>, nargs=0, args=<optimized out>, co=<optimized out>, tstate=0x96db20) at ../Objects/call.c:330
#39 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:367
#40 0x000000000051638d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x9c9060, callable=<function at remote 0x7ffff6c58040>, tstate=0x96db20)
    at ../Include/cpython/abstract.h:118
#41 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x9c9060, callable=<function at remote 0x7ffff6c58040>) at ../Include/cpython/abstract.h:127
#42 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x96db20) at ../Python/ceval.c:5072
#43 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3518
#44 0x0000000000514a95 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x9c8ef0, for file /home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py, line 51, in <module> (), tstate=0x96db20)
    at ../Include/internal/pycore_ceval.h:40
#45 _PyEval_EvalCode (tstate=0x96db20, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, 
    kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at ../Python/ceval.c:4327
#46 0x000000000051482b in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=args@entry=0x0, argcount=argcount@entry=0, 
    kwnames=kwnames@entry=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at ../Python/ceval.c:4359
#47 0x00000000005fa5f7 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<optimized out>, globals=<optimized out>, 
    _co=<optimized out>) at ../Python/ceval.c:4375
#48 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:826
#49 0x00000000006201eb in run_eval_code_obj (tstate=0x96db20, co=0x7ffff6bab450, 
    globals={'__name__': '__main__', '__doc__': ' Tests for the Z3Formula class', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py') at remote 0x7ffff6c8ac10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff6c73900>, '__file__': '/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py', '__cached__': None, 'pytest': <module at remote 0x7ffff6a585e0>, 'z3': <module at remote 0x7ffff6a5eb30>, 'Z3Formula': <ABCMeta(__module__='citoolkit.specifications.z3_formula', __doc__=' The Z3Formula class encodes a Z3 formula specification. The Z3Formula must support\n    the bitblasting tactic as all the language_size() and sample() computations are performed\n    by transforming the Z3Formula spec into an equivalent BoolFormula Spec. All main_vars must\n    also be either Bool or BitVec variables.\n\n    As an additional restriction, the specification reserves all variable names of the form\n    "CI_Internal_(*)_(...(truncated), 
--Type <RET> for more, q to quit, c to continue without paging--
    locals={'__name__': '__main__', '__doc__': ' Tests for the Z3Formula class', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py') at remote 0x7ffff6c8ac10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff6c73900>, '__file__': '/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py', '__cached__': None, 'pytest': <module at remote 0x7ffff6a585e0>, 'z3': <module at remote 0x7ffff6a5eb30>, 'Z3Formula': <ABCMeta(__module__='citoolkit.specifications.z3_formula', __doc__=' The Z3Formula class encodes a Z3 formula specification. The Z3Formula must support\n    the bitblasting tactic as all the language_size() and sample() computations are performed\n    by transforming the Z3Formula spec into an equivalent BoolFormula Spec. All main_vars must\n    also be either Bool or BitVec variables.\n\n    As an additional restriction, the specification reserves all variable names of the form\n    "CI_Internal_(*)_(...(truncated)) at ../Python/pythonrun.c:1219
#50 0x000000000061b314 in run_mod (mod=<optimized out>, filename=<optimized out>, 
    globals={'__name__': '__main__', '__doc__': ' Tests for the Z3Formula class', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py') at remote 0x7ffff6c8ac10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff6c73900>, '__file__': '/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py', '__cached__': None, 'pytest': <module at remote 0x7ffff6a585e0>, 'z3': <module at remote 0x7ffff6a5eb30>, 'Z3Formula': <ABCMeta(__module__='citoolkit.specifications.z3_formula', __doc__=' The Z3Formula class encodes a Z3 formula specification. The Z3Formula must support\n    the bitblasting tactic as all the language_size() and sample() computations are performed\n    by transforming the Z3Formula spec into an equivalent BoolFormula Spec. All main_vars must\n    also be either Bool or BitVec variables.\n\n    As an additional restriction, the specification reserves all variable names of the form\n    "CI_Internal_(*)_(...(truncated), 
    locals={'__name__': '__main__', '__doc__': ' Tests for the Z3Formula class', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py') at remote 0x7ffff6c8ac10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff6c73900>, '__file__': '/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py', '__cached__': None, 'pytest': <module at remote 0x7ffff6a585e0>, 'z3': <module at remote 0x7ffff6a5eb30>, 'Z3Formula': <ABCMeta(__module__='citoolkit.specifications.z3_formula', __doc__=' The Z3Formula class encodes a Z3 formula specification. The Z3Formula must support\n    the bitblasting tactic as all the language_size() and sample() computations are performed\n    by transforming the Z3Formula spec into an equivalent BoolFormula Spec. All main_vars must\n    also be either Bool or BitVec variables.\n\n    As an additional restriction, the specification reserves all variable names of the form\n    "CI_Internal_(*)_(...(truncated), flags=<optimized out>, arena=<optimized out>) at ../Python/pythonrun.c:1240
#51 0x000000000061f71d in pyrun_file (fp=0x96bab0, filename='/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py', start=<optimized out>, 
    globals={'__name__': '__main__', '__doc__': ' Tests for the Z3Formula class', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py') at remote 0x7ffff6c8ac10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff6c73900>, '__file__': '/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py', '__cached__': None, 'pytest': <module at remote 0x7ffff6a585e0>, 'z3': <module at remote 0x7ffff6a5eb30>, 'Z3Formula': <ABCMeta(__module__='citoolkit.specifications.z3_formula', __doc__=' The Z3Formula class encodes a Z3 formula specification. The Z3Formula must support\n    the bitblasting tactic as all the language_size() and sample() computations are performed\n    by transforming the Z3Formula spec into an equivalent BoolFormula Spec. All main_vars must\n    also be either Bool or BitVec variables.\n\n    As an additional restriction, the specification reserves all variable names of the form\n    "CI_Internal_(*)_(...(truncated), 
    locals={'__name__': '__main__', '__doc__': ' Tests for the Z3Formula class', '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py') at remote 0x7ffff6c8ac10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff6c73900>, '__file__': '/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py', '__cached__': None, 'pytest': <module at remote 0x7ffff6a585e0>, 'z3': <module at remote 0x7ffff6a5eb30>, 'Z3Formula': <ABCMeta(__module__='citoolkit.specifications.z3_formula', __doc__=' The Z3Formula class encodes a Z3 formula specification. The Z3Formula must support\n    the bitblasting tactic as all the language_size() and sample() computations are performed\n    by transforming the Z3Formula spec into an equivalent BoolFormula Spec. All main_vars must\n    also be either Bool or BitVec variables.\n\n    As an additional restriction, the specification reserves all variable names of the form\n    "CI_Internal_(*)_(...(truncated), closeit=1, flags=0x7fffffffdb98) at ../Python/pythonrun.c:1138
#52 0x000000000061f22a in pyrun_simple_file (flags=0x7fffffffdb98, closeit=1, filename='/home/ericvin/Projects/CIToolkit/tests/test_z3_formula.py', fp=0x96bab0) at ../Python/pythonrun.c:449
#53 PyRun_SimpleFileExFlags (fp=0x96bab0, filename=<optimized out>, closeit=1, flags=0x7fffffffdb98) at ../Python/pythonrun.c:482
#54 0x0000000000612a77 in pymain_run_file (cf=0x7fffffffdb98, config=0x96c4e0) at ../Modules/main.c:373
#55 pymain_run_python (exitcode=0x7fffffffdb90) at ../Modules/main.c:598
#56 Py_RunMain () at ../Modules/main.c:677
#57 0x00000000005eeb7d in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:731
#58 0x00007ffff7c0f565 in __libc_start_main (main=0x5eeb40 <main>, argc=2, argv=0x7fffffffdd88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffdd78) at ../csu/libc-start.c:332
#59 0x00000000005eea7e in _start ()

@Eric-Vin
Copy link
Contributor Author

Eric-Vin commented Feb 8, 2022

Hi Mate,
I just wanted to check in and see if you'd had a chance to take a look at this?
Thanks!
-Eric

@Eric-Vin
Copy link
Contributor Author

Eric-Vin commented Jun 22, 2022

Hi Mate,
I just wanted to follow up on this again. Happy to help however I can.

@msoos
Copy link
Collaborator

msoos commented Jul 7, 2022

Hi,

I know I am incredibly late, but I have been having some personal issues. Sorry. Anyway.

I spent today fixing up pycryptosat, it can now be built stand-alone. Not released yet, so now I wanna test with your tool :) I'm getting a build failure thanks to -Werror, it's the worst flag ever invented by mankind (because compilers improve, create new warnings, and hence builds break non-stop, literally nothing can be re-built 6 months later):

        creating /tmp/tmpwveb7npd/tmp/pip-build-env-nj03wuzh/overlay/lib/python3.10/site-packages/numpy/distutils/checks
        WARN: CCompilerOpt.dist_test[630] : CCompilerOpt._dist_test_spawn[764] : Command (gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -flto=auto -fPIC -c /tmp/pip-build-env-nj03wuzh/overlay/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knl.c -o /tmp/tmpwveb7npd/tmp/pip-build-env-nj03wuzh/overlay/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knl.o -MMD -MF /tmp/tmpwveb7npd/tmp/pip-build-env-nj03wuzh/overlay/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knl.o.d -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512er -mavx512pf -Werror) failed with exit status 1 output ->
        In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include/immintrin.h:51,
                         from /tmp/pip-build-env-nj03wuzh/overlay/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knl.c:14:
        In function ‘_mm512_exp2a23_round_pd’,
            inlined from ‘main’ at /tmp/pip-build-env-nj03wuzh/overlay/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knl.c:21:17:
        /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include/avx512erintrin.h:55:20: error: ‘__W’ is used uninitialized [-Werror=uninitialized]
           55 |   return (__m512d) __builtin_ia32_exp2pd_mask ((__v8df) __A,
              |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           56 |                                                (__v8df) __W,
              |                                                ~~~~~~~~~~~~~
           57 |                                                (__mmask8) -1, __R);
              |                                                ~~~~~~~~~~~~~~~~~~~
        /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include/avx512erintrin.h: In function ‘main’:
        /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include/avx512erintrin.h:54:11: note: ‘__W’ was declared here
           54 |   __m512d __W;
              |           ^~~
        In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include/immintrin.h:53:
        In function ‘_mm512_mask_prefetch_i64scatter_pd’,
            inlined from ‘main’ at /tmp/pip-build-env-nj03wuzh/overlay/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knl.c:23:5:
        /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include/avx512pfintrin.h:180:3: error: ‘base’ may be used uninitialized [-Werror=maybe-uninitialized]
          180 |   __builtin_ia32_scatterpfqpd (__mask, (__v8di) __index, __addr, __scale,
              |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          181 |                                __hint);
              |                                ~~~~~~~
        <built-in>: In function ‘main’:
        <built-in>: note: by argument 3 of type ‘const void *’ to ‘__builtin_ia32_scatterpfqpd’ declared here
        /tmp/pip-build-env-nj03wuzh/overlay/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knl.c:18:9: note: ‘base’ declared here
           18 |     int base[128];
              |         ^~~~
        cc1: all warnings being treated as errors
        
        WARN: CCompilerOpt.feature_test[1563] : testing failed
        WARN: CCompilerOpt.generate_dispatch_header[2362] : dispatch header dir build/src.linux-x86_64-3.10/numpy/distutils/include does not exist, creating it
        error: library mach has Fortran sources but no Fortran compiler found
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for scipy
  Failed to build scipy
  ERROR: Could not build wheels for scipy, which is required to install pyproject.toml-based projects
  WARNING: You are using pip version 22.0.2; however, version 22.1.2 is available.
  You should consider upgrading via the '/home/soos/.cache/pypoetry/virtualenvs/citoolkit-mHZilUvy-py3.10/bin/python -m pip install --upgrade pip' command.

And then it all fails:

  at /usr/lib/python3.10/site-packages/poetry/utils/env.py:1195 in _run
      1191│                 output = subprocess.check_output(
      1192│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1193│                 )
      1194│         except CalledProcessError as e:
    → 1195│             raise EnvCommandError(e, input=input_)
      1196│ 
      1197│         return decode(output)
      1198│ 
      1199│     def execute(self, bin, *args, **kwargs):

  • Installing urllib3 (1.26.10)
make: *** [Makefile:9: make_env] Error 1

I'll try to do something... though fixing numpy is really not my kind of thing :D Anyway, can you please check the code at: https://github.com/meelgroup/cryptominisat branch devel (the default, but please do check)? You should be able to just go to cryptominisat/python issue sudo python setup.py install and it should work. The difference is that you don't need anything built with cmake or anything, it really just works through setup.py.

I'm using the latest Arch release, so that's why it's breaking, likely -- I have a new compiler, which warns more. Classic.

@msoos
Copy link
Collaborator

msoos commented Jul 7, 2022

Yay! I just managed to do everything. After fixing the above, I had an issue where my /usr/lib/python3.10 got corrupted(!?). So I had to reinstall all of python. But anyway, in the end I made it work :) With the devel branch of cryptominisat from meelgroup, it works:

soos@tiresias:CIToolkit$ make
poetry install
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: citoolkit (0.1.0)
poetry shell
Spawning shell within /home/soos/.cache/pypoetry/virtualenvs/citoolkit-mHZilUvy-py3.10
soos@tiresias:CIToolkit$ . /home/soos/.cache/pypoetry/virtualenvs/citoolkit-mHZilUvy-py3.10/bin/activate
(citoolkit-mHZilUvy-py3.10) soos@tiresias:CIToolkit$ pytest -s -k test_z3_formula_basic
==================================================================================================================== test session starts ====================================================================================================================
platform linux -- Python 3.10.5, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
Using --randomly-seed=580533358
rootdir: /home/soos/development/sat_solvers/CIToolkit, configfile: pyproject.toml
plugins: randomly-3.12.0
collected 30 items / 30 deselected                                                                                                                                                                                                                          

================================================================================================================== 30 deselected in 0.08s ===================================================================================================================
(citoolkit-mHZilUvy-py3.10) soos@tiresias:CIToolkit$ 

@msoos
Copy link
Collaborator

msoos commented Jul 7, 2022

If you are happy with the above, I plan to release that CryptoMiniSat into the master branch (will rename it main). Let me know what you think!

Mate

@Eric-Vin
Copy link
Contributor Author

Eric-Vin commented Jul 8, 2022

Hi Mate,
No worries at all about the delay and thanks so much for taking a look! I actually limited the number of sampling calls in the test (from 1000 to 100) a couple months ago so it would pass as this bug hasn't come up anywhere else (sorry about the mixup!), but unfortunately it was only crashing on one of the seeds for the later calls. I just set it back and I'm still getting the crash on my machine with the new cryptominisat version. Would you mind pulling the newest push to the CIToolkit develop branch and seeing if you can reproduce it?

The best way to do that would be, inside the poetry virtual environment, navigate to the CiToolkit/tests directory and run python3 ./test_z3_formula.py, which should leave out pytest and also let you use pdb.

Thanks again!
Eric

@msoos
Copy link
Collaborator

msoos commented Jul 8, 2022

Hi,

Here's my output:

soos@tiresias:CIToolkit$ git checkout remotes/origin/develop
[ stuff cut out ]
soos@tiresias:tests$ git rev-parse HEAD
d09e25777fc35a70b52230b3384fda74b7dfddd7
soos@tiresias:tests$ git diff
[ empty response, i.e no diff ]
soos@tiresias:tests$ git diff --cached
[ empty response, i.e no diff ]
soos@tiresias:CIToolkit$ make update
poetry update
Updating dependencies
Resolving dependencies... (27.5s)

Writing lock file

Package operations: 24 installs, 0 updates, 0 removals

  • Installing pycparser (2.21)
  • Installing qdldl (0.1.5.post2)
  • Installing six (1.16.0)
  • Installing tomli (2.0.1)
  • Installing typing-extensions (4.3.0)
  • Installing cffi (1.15.1)
  • Installing cycler (0.11.0)
  • Installing dill (0.3.5.1)
  • Installing ecos (2.0.10)
  • Installing fonttools (4.34.4)
  • Installing kiwisolver (1.4.3)
  • Installing osqp (0.6.2.post5)
  • Installing pillow (9.2.0)
  • Installing python-dateutil (2.8.2)
  • Installing scs (3.2.0)
  • Installing setuptools-scm (7.0.4)
  • Installing boolexpr (2.4)
  • Installing cvxpy (1.2.1)
  • Installing matplotlib (3.5.2)
  • Installing multiprocess (0.70.13)
  • Installing psutil (5.9.1)
  • Installing pyeda (0.28.0)
  • Installing pytest-repeat (0.9.1)
  • Installing z3-solver (4.9.1.0)
soos@tiresias:CIToolkit$ make
poetry install -E dev
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: citoolkit (0.1.0)
poetry shell
Spawning shell within /home/soos/.cache/pypoetry/virtualenvs/citoolkit-mHZilUvy-py3.10
soos@tiresias:CIToolkit$ . /home/soos/.cache/pypoetry/virtualenvs/citoolkit-mHZilUvy-py3.10/bin/activate
(citoolkit-mHZilUvy-py3.10) soos@tiresias:CIToolkit$ cd tests/
(citoolkit-mHZilUvy-py3.10) soos@tiresias:tests$ python ./test_z3_formula.py 
[ there was some wait here, but eventually got my prompt back]
(citoolkit-mHZilUvy-py3.10) soos@tiresias:tests$ echo $?
0
(citoolkit-mHZilUvy-py3.10) soos@tiresias:tests$ pytest -s -k test_z3_formula_basic
========================================================================= test session starts =========================================================================
platform linux -- Python 3.10.5, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
Using --randomly-seed=1714858940
rootdir: /home/soos/development/sat_solvers/CIToolkit, configfile: pyproject.toml
plugins: repeat-0.9.1, randomly-3.12.0
collected 68 items / 67 deselected / 1 selected                                                                                                                       

test_z3_formula.py .

================================================================== 1 passed, 67 deselected in 2.94s ===================================================================
(citoolkit-mHZilUvy-py3.10) soos@tiresias:tests$ git rev-parse HEAD
d09e25777fc35a70b52230b3384fda74b7dfddd7

So it all seems good? Notice that I am using pycryptosat that was specifically built from cryptominisat version:

soos@tiresias:cryptominisat$ git rev-parse HEAD
9822d8902d0952c1083d7215306bc8bff913375a

In other words it's:

commit 9822d8902d0952c1083d7215306bc8bff913375a (HEAD -> devel, githubmeel/devel)
Author: Mate Soos <[email protected]>
Date:   Fri Jul 8 01:04:30 2022 +0200

    Fixing uninstall for python

Where githubmeel is:

soos@tiresias:cryptominisat$ git remote -v | grep githubmeel
githubmeel      [email protected]:meelgroup/cryptominisat.git (fetch)
githubmeel      [email protected]:meelgroup/cryptominisat.git (push)

My python version is:

soos@tiresias:cryptominisat$ python --version
Python 3.10.5
soos@tiresias:cryptominisat$ python-config --cflags
-I/usr/include/python3.10 -I/usr/include/python3.10  -Wno-unused-result -Wsign-compare -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -flto=auto -ffat-lto-objects -DNDEBUG -g -fwrapv -O3 -Wall
soos@tiresias:cryptominisat$ python-config --ldflags
 -L/usr/lib  -lcrypt -ldl  -lm -lm 
soos@tiresias:cryptominisat$ python-config --libs
 -lcrypt -ldl  -lm -lm 

I am running Arch Linux, gcc is:

soos@tiresias:cryptominisat$ gcc --version
gcc (GCC) 12.1.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Do you know why I may not be getting the same error?

Thanks for looking into this,

Mate

@Eric-Vin
Copy link
Contributor Author

Eric-Vin commented Jul 8, 2022

Hey Mate,

I did a full reinstall of everything and it runs and passes all tests now, so I must have left some remnant of old code or been using an incorrect version. Sorry about the mixup and thanks so much for your help with all of this!

Thanks again!
Eric

@Eric-Vin Eric-Vin closed this as completed Jul 8, 2022
@msoos
Copy link
Collaborator

msoos commented Jul 8, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants