How to read the tables?
Meaning of the model-checker's resultsModel-checker on the instrumented program | Observed state in Litmus | Picture |
VERIFICATION FAILED | YES | |
VERIFICATION SUCCESSFUL | NO |
Reading the results in the experiments tables
The table below is an excerpt of former results we obtained for x86/podwr experiments.
- In the first column, we can read the name of the original Litmus test (with a link to the file), and the expected violation cycle. The second column gathers the results of the model-checkers for SC. As we are interested in WMM bugs, the verification for SC should be .
- The three next columns concern TSO model. The last column tells which result we should get for this program under TSO, namely . We see in the third column that Satabs indeed finds the correct result when checking the instrumented program. ESBMC in this example is missing the counterexample. Threader is either returning an error, or killed by timeout (here, it is an error).
- Note that there are two columns for TSO results: in the first one, we instrument all the delays; in the second one, we instrument only one delay per cycle.
- We also experiment under PSO, RMO and Power, and with additional model-checkers: additional columns of results (and expected results) appear after TSO's columns.
Name | SC | TSO (all delay/cyc) | TSO (1 delay/cyc) | Expected (TSO) | ... |
x86/podwr000 Fre PodWR Fre PodWR |
SatAbs: 0.60sec ESBMC: 0.30sec Threader: 7.92sec ... |
Cycles SatAbs: 3.94sec (x6.56) ESBMC: 8.34sec Threader: ERROR/TIMEOUT[1] 491.99sec ... |
Cycles SatAbs: 2.43sec (x4.05) ESBMC: 7.58sec Threader: ERROR/TIMEOUT[1] 182.60sec ... |
... |
x86/podwr (tooling date Sun Jan 22 02:23:23 GMT 2012)
This family corresponds to litmus tests involving write-read pairs, to exercise typical x86 reordering of write-read pairs scenarios. Therefore they should answer Yes in litmus terms, or Verification Failed in verification terms, as soon as the model is x86/TSO or a weaker one.
Name | SC | CAV'11 | TSO (all delay/cyc) | TSO (1 delay/cyc) | TSO (opt delay/cyc) | Expected (TSO) | PSO (all delay/cyc) | PSO (1 delay/cyc) | PSO (opt delay/cyc) | Expected (PSO) | RMO (all delay/cyc) | RMO (1 delay/cyc) | RMO (opt delay/cyc) | Expected (RMO) | Power (all delay/cyc) | Power (1 delay/cyc) | Power (opt delay/cyc) | Expected (Power) |
x86/podwr000 |
SatAbs: 0.57sec Satabs-prefix-first: 0.36sec CBMC-WMM: 0.05sec CBMC-SC: 0.05sec CImpact: 0.11sec Checkfence: 0.01sec ESBMC: 0.07sec Threader-monolithic: 0.12sec Threader-owicki-gries: 0.10sec Threader-rely-guarantee: 0.09sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: sc): 0.55sec |
Cycles
SatAbs: 7.51sec (x13.17) Satabs-prefix-first: 15.27sec CBMC-WMM: 0.07sec CBMC-SC: 5.69sec CImpact: 0.36sec ESBMC: 0.10sec Threader-monolithic: ERROR/TIMEOUT[130] 602.21sec Threader-owicki-gries: ERROR/TIMEOUT[130] 635.04sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 620.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): 0.40sec |
Cycles
SatAbs: 7.97sec (x13.98) Satabs-prefix-first: 13.02sec CBMC-WMM: 0.07sec CBMC-SC: 5.50sec CImpact: 0.38sec Checkfence: 0.01sec ESBMC: 0.08sec Threader-monolithic: ERROR/TIMEOUT[130] 645.51sec Threader-owicki-gries: ERROR/TIMEOUT[130] 654.92sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 589.78sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): 0.40sec |
Cycles
SatAbs: 2.31sec (x4.05) Satabs-prefix-first: 6.90sec CBMC-WMM: 0.06sec CBMC-SC: 0.86sec CImpact: 0.25sec ESBMC: 0.07sec Threader-monolithic: ERROR/TIMEOUT[1] 220.89sec Threader-owicki-gries: ERROR/TIMEOUT[1] 232.29sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 229.49sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): 0.59sec |
Cycles
SatAbs: 2.45sec (x4.29) Satabs-prefix-first: 6.54sec CBMC-WMM: 0.07sec CBMC-SC: 0.88sec CImpact: 0.27sec ESBMC: 0.10sec Threader-monolithic: ERROR/TIMEOUT[1] 216.65sec Threader-owicki-gries: ERROR/TIMEOUT[1] 236.62sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 226.56sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): 0.37sec |
Cycles
SatAbs: 8.10sec (x14.21) Satabs-prefix-first: 12.91sec CBMC-WMM: 0.07sec CBMC-SC: 5.67sec CImpact: 0.35sec ESBMC: 0.08sec Threader-monolithic: ERROR/TIMEOUT[130] 600.25sec Threader-owicki-gries: ERROR/TIMEOUT[130] 596.85sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 601.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: |
Cycles
SatAbs: 2.59sec (x4.54) Satabs-prefix-first: 7.25sec CBMC-WMM: 0.04sec CBMC-SC: 0.88sec CImpact: 0.27sec ESBMC: 0.09sec Threader-monolithic: ERROR/TIMEOUT[1] 234.56sec Threader-owicki-gries: ERROR/TIMEOUT[1] 218.74sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 217.38sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: |
Cycles
SatAbs: 2.31sec (x4.05) Satabs-prefix-first: 6.58sec CBMC-WMM: 0.07sec CBMC-SC: 0.86sec CImpact: 0.26sec ESBMC: 0.07sec Threader-monolithic: ERROR/TIMEOUT[1] 215.07sec Threader-owicki-gries: ERROR/TIMEOUT[1] 224.18sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 234.22sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: |
Cycles
SatAbs: 7.80sec (x13.68) Satabs-prefix-first: 13.09sec CBMC-WMM: 0.07sec CBMC-SC: 5.32sec CImpact: 0.38sec ESBMC: 0.10sec Threader-monolithic: ERROR/TIMEOUT[130] 795.32sec Threader-owicki-gries: ERROR/TIMEOUT[130] 694.18sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 730.50sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 3.57sec (x6.26) Satabs-prefix-first: 9.17sec CBMC-WMM: 0.08sec CBMC-SC: 1.00sec CImpact: 0.27sec ESBMC: 0.10sec Threader-monolithic: ERROR/TIMEOUT[1] 218.40sec Threader-owicki-gries: ERROR/TIMEOUT[1] 228.64sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 221.15sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 2.52sec (x4.42) Satabs-prefix-first: 9.03sec CBMC-WMM: 0.09sec CBMC-SC: 1.13sec CImpact: 0.28sec ESBMC: 0.10sec Threader-monolithic: ERROR/TIMEOUT[1] 221.46sec Threader-owicki-gries: ERROR/TIMEOUT[1] 225.54sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 222.32sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 8.01sec (x14.05) Satabs-prefix-first: 13.02sec CBMC-WMM: 0.06sec CBMC-SC: 5.48sec CImpact: 0.39sec ESBMC: 0.08sec Threader-monolithic: ERROR/TIMEOUT[130] 676.45sec Threader-owicki-gries: ERROR/TIMEOUT[130] 776.29sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 770.68sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 2.37sec (x4.15) Satabs-prefix-first: 6.49sec CBMC-WMM: 0.08sec CBMC-SC: 0.83sec CImpact: 0.24sec ESBMC: 0.08sec Threader-monolithic: ERROR/TIMEOUT[1] 221.41sec Threader-owicki-gries: ERROR/TIMEOUT[1] 217.81sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 218.29sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 2.42sec (x4.24) Satabs-prefix-first: 7.11sec CBMC-WMM: 0.08sec CBMC-SC: 0.85sec CImpact: 0.29sec ESBMC: 0.09sec Threader-monolithic: ERROR/TIMEOUT[1] 222.60sec Threader-owicki-gries: ERROR/TIMEOUT[1] 216.85sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 238.74sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
||||
x86/podwr001 |
SatAbs: 2.23sec Satabs-prefix-first: 0.94sec CBMC-WMM: 0.07sec CBMC-SC: 0.06sec CImpact: 242.47sec Checkfence: 0.01sec ESBMC: 0.08sec Threader-monolithic: 0.13sec Threader-owicki-gries: 0.11sec Threader-rely-guarantee: 0.14sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: sc): 0.50sec |
Cycles
SatAbs: 24.52sec (x10.99) Satabs-prefix-first: 59.02sec CBMC-WMM: 0.12sec CBMC-SC: 13.96sec CImpact: ERROR/TIMEOUT[8] 451.20sec ESBMC: 0.11sec Threader-monolithic: ERROR/TIMEOUT[130] 897.17sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.10sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.36sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): 0.55sec |
Cycles
SatAbs: 29.20sec (x13.09) Satabs-prefix-first: 66.56sec CBMC-WMM: 0.10sec CBMC-SC: 10.15sec CImpact: ERROR/TIMEOUT[8] 423.36sec Checkfence: 0.01sec ESBMC: 0.10sec Threader-monolithic: ERROR/TIMEOUT[130] 896.77sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.25sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.24sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): 0.48sec |
Cycles
SatAbs: 7.06sec (x3.16) Satabs-prefix-first: 8.80sec CBMC-WMM: 0.11sec CBMC-SC: 1.08sec CImpact: ERROR/TIMEOUT[8] 331.69sec ESBMC: 0.09sec Threader-monolithic: ERROR/TIMEOUT[1] 197.76sec Threader-owicki-gries: ERROR/TIMEOUT[1] 255.52sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 256.25sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): 0.46sec |
Cycles
SatAbs: 6.22sec (x2.78) Satabs-prefix-first: 8.33sec CBMC-WMM: 0.12sec CBMC-SC: 1.04sec CImpact: ERROR/TIMEOUT[8] 325.55sec ESBMC: 0.09sec Threader-monolithic: ERROR/TIMEOUT[1] 256.57sec Threader-owicki-gries: ERROR/TIMEOUT[1] 256.22sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 227.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): 0.53sec |
Cycles
SatAbs: 29.06sec (x13.03) Satabs-prefix-first: 63.37sec CBMC-WMM: 0.12sec CBMC-SC: 15.96sec CImpact: ERROR/TIMEOUT[8] 464.42sec ESBMC: 0.11sec Threader-monolithic: ERROR/TIMEOUT[130] 897.28sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.96sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 896.80sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: |
Cycles
SatAbs: 7.98sec (x3.57) Satabs-prefix-first: 8.30sec CBMC-WMM: 0.11sec CBMC-SC: 1.12sec CImpact: ERROR/TIMEOUT[8] 306.55sec ESBMC: 0.08sec Threader-monolithic: ERROR/TIMEOUT[1] 245.43sec Threader-owicki-gries: ERROR/TIMEOUT[1] 258.20sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 256.38sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: |
Cycles
SatAbs: 6.57sec (x2.94) Satabs-prefix-first: 8.12sec CBMC-WMM: 0.11sec CBMC-SC: 1.00sec CImpact: ERROR/TIMEOUT[8] 333.16sec ESBMC: 0.09sec Threader-monolithic: ERROR/TIMEOUT[1] 257.93sec Threader-owicki-gries: ERROR/TIMEOUT[1] 239.83sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 234.72sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: |
Cycles
SatAbs: 29.26sec (x13.12) Satabs-prefix-first: 72.09sec CBMC-WMM: 0.14sec CBMC-SC: 14.05sec CImpact: ERROR/TIMEOUT[8] 329.89sec ESBMC: 0.11sec Threader-monolithic: ERROR/TIMEOUT[130] 861.08sec Threader-owicki-gries: ERROR/TIMEOUT[130] 874.04sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 896.40sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 8.01sec (x3.59) Satabs-prefix-first: 8.37sec CBMC-WMM: 0.13sec CBMC-SC: 1.22sec CImpact: ERROR/TIMEOUT[8] 317.91sec ESBMC: 0.06sec Threader-monolithic: ERROR/TIMEOUT[1] 234.36sec Threader-owicki-gries: ERROR/TIMEOUT[1] 192.15sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 253.84sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 6.59sec (x2.95) Satabs-prefix-first: 8.51sec CBMC-WMM: 0.11sec CBMC-SC: 1.11sec CImpact: ERROR/TIMEOUT[8] 235.61sec ESBMC: 0.09sec Threader-monolithic: ERROR/TIMEOUT[1] 234.48sec Threader-owicki-gries: ERROR/TIMEOUT[1] 216.57sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 257.84sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 27.32sec (x12.25) Satabs-prefix-first: 67.16sec CBMC-WMM: 0.12sec CBMC-SC: 13.26sec CImpact: ERROR/TIMEOUT[8] 376.41sec ESBMC: 0.10sec Threader-monolithic: ERROR/TIMEOUT[130] 895.72sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.75sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.50sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 6.42sec (x2.87) Satabs-prefix-first: 8.45sec CBMC-WMM: 0.15sec CBMC-SC: 1.09sec CImpact: ERROR/TIMEOUT[8] 255.58sec ESBMC: 0.09sec Threader-monolithic: ERROR/TIMEOUT[1] 257.70sec Threader-owicki-gries: ERROR/TIMEOUT[1] 237.76sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 254.91sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: 6.38sec (x2.86) Satabs-prefix-first: 8.12sec CBMC-WMM: 0.12sec CBMC-SC: 1.05sec CImpact: ERROR/TIMEOUT[8] 339.55sec ESBMC: 0.08sec Threader-monolithic: ERROR/TIMEOUT[1] 246.25sec Threader-owicki-gries: ERROR/TIMEOUT[1] 220.12sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 241.22sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |