diff --git a/.gitignore b/.gitignore
deleted file mode 100644
index 6ba9fa58a5f656f8f4a3093622d8ea4a9b739bcb..0000000000000000000000000000000000000000
--- a/.gitignore
+++ /dev/null
@@ -1,13 +0,0 @@
-auto/
-ltximg/
-
-valmari_fig4/benchmark.pdf
-valmari_fig4/benchmark.tex
-valmari_fig4/benchmark.tmp
-valmari_fig4/benchmarks.png
-prism/bench/
-dfa/bench/
-*.pdf
-*.tex
-/dfa/bench
-/wta/bench
diff --git a/README.md b/README.md
index d26e0bcb32833a1d3bec7799bf626525ada7687e..886bfa386a23ff6d5e3a9d221d528f1ca94cdacf 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,2 @@
# Benchmarks for CoPaR
-
-This repository contains tooling and raw results from benchmarking
-[CoPaR](https://git8.cs.fau.de/software/copar).
-
-This repo contains benchmarks for:
-
-- Deterministic Finite Automata in `dfa/`
-- Adaptions of the benchmark suite of the PRISM model checker in `prism/`
-- Weighted Tree Automata in `wta/`
+### This repo has moved to https://gitlab.cs.fau.de/hpd/copar-benchmarks
diff --git a/artefakt_rebuttal.org b/artefakt_rebuttal.org
deleted file mode 100644
index 4897d287081eb72ca1a5050a2fe9afd65c1cc864..0000000000000000000000000000000000000000
--- a/artefakt_rebuttal.org
+++ /dev/null
@@ -1,397 +0,0 @@
-#+TITLE: Messungen zum Artifact-Rebuttal
-#+DATE: <2018-12-07 Fr>
-#+AUTHOR: Hans-Peter Deifel
-
-Alle Messungen bis auf das Paper wurden mit dem Artefakt-Binary durchgeführt auf
-den im Artefakt mitgelieferten Dateien.
-
-* Ergebnisse im Paper
-** DFA 10^3
-
- #+NAME: paper_dfa_1000
- | N | t_p (s) | t_i (s) | t_r (s) | t_v (s) |
- |------+---------+---------+---------+---------|
- | 1000 | 2.4 | 0.76 | 0.36 | 1.03 |
- | 2000 | 4.96 | 1.58 | 0.74 | 2.52 |
- | 3000 | 7.39 | 2.11 | 1.4 | 4.33 |
- | 4000 | 10.2 | 3.2 | 1.67 | 5.74 |
- | 5000 | 13.06 | 4.05 | 2.1 | 7.45 |
-
-** DFA 10^4
-
- #+NAME: paper_dfa_10000
- | N | t_p (s) | t_i (s) | t_r (s) | t_v (s) |
- |------+---------+---------+---------+---------|
- | 600 | 44.75 | 1.82 | 2.88 | 7.84 |
- | 700 | 50.93 | 4.29 | 3.18 | 9.42 |
- | 800 | 60.78 | 2.54 | 4.16 | 11.04 |
- | 900 | 68.34 | 2.76 | 4.6 | 12.71 |
- | 1000 | 75.79 | 3.05 | 5.21 | 14.32 |
-
-** PRISM
-
- #+NAME: paper_prism
- | | t_p(s) | t_i(s) | t_r(s) | t_v(s) |
- |---+--------+--------+--------+--------|
- | 1 | 2.39 | 0.66 | 1.09 | 1.19 |
- | 1 | 9.66 | 2.83 | 5.43 | 5.58 |
- | 2 | 0.47 | 0.28 | 0.57 | 0.12 |
- | 3 | 4.89 | 3.01 | 5.45 | 0.91 |
- | 3 | 12.59 | 6.17 | 15.12 | 2.45 |
-
-* Ergebnisse am Desktop
-
- Desktop hat 16GB RAM. Ausgabe von =lscpu=:
-
- #+BEGIN_EXAMPLE
- Architecture: x86_64
- CPU op-mode(s): 32-bit, 64-bit
- Byte Order: Little Endian
- Address sizes: 39 bits physical, 48 bits virtual
- CPU(s): 4
- On-line CPU(s) list: 0-3
- Thread(s) per core: 1
- Core(s) per socket: 4
- Socket(s): 1
- NUMA node(s): 1
- Vendor ID: GenuineIntel
- CPU family: 6
- Model: 94
- Model name: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
- Stepping: 3
- CPU MHz: 3203.065
- CPU max MHz: 3600,0000
- CPU min MHz: 800,0000
- BogoMIPS: 6386.00
- Virtualization: VT-x
- L1d cache: 32K
- L1i cache: 32K
- L2 cache: 256K
- L3 cache: 6144K
- NUMA node0 CPU(s): 0-3
- Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d
- #+END_EXAMPLE
-
-** DFA 10^3
-
- #+NAME: desktop_dfa_1000
- | N | tp | ti | tr | tv |
- |------+-------+------+------+------|
- | 1000 | 2.56 | 0.82 | 0.37 | 1.06 |
- | 2000 | 5.29 | 1.67 | 0.80 | 2.65 |
- | 3000 | 7.83 | 2.32 | 1.52 | 4.24 |
- | 4000 | 10.68 | 3.40 | 1.74 | 6.54 |
- | 5000 | 13.66 | 4.28 | 2.22 | 7.73 |
-
-** DFA 10^4
-
- #+NAME: desktop_dfa_10000
- | N | tp | ti | tr | tv |
- |------+-------+------+------+-------|
- | 600 | 46.42 | 1.86 | 3.02 | 8.64 |
- | 700 | 52.50 | 4.55 | 3.32 | 10.25 |
- | 800 | 62.34 | 2.52 | 4.36 | 11.53 |
- | 900 | 70.08 | 2.78 | 4.92 | 13.85 |
- | 1000 | 78.11 | 3.14 | 5.49 | 14.64 |
-
-** PRISM
-
- #+NAME: desktop_prism
- | | tp | ti | tr | tv |
- |----+-------+------+-------+------|
- | 1. | 2.54 | 0.70 | 1.11 | 1.19 |
- | 1. | 10.26 | 3.00 | 5.61 | 6.09 |
- | 2. | 0.51 | 0.29 | 0.60 | 0.08 |
- | 3. | 5.33 | 3.09 | 5.78 | 0.78 |
- | 3. | 13.75 | 6.36 | 16.05 | 2.78 |
-
-* Ergebnisse am Laptop
-
- Der Laptop hat 8GB und folgendes =lscpu=:
-
- #+BEGIN_EXAMPLE
- Architecture: x86_64
- CPU op-mode(s): 32-bit, 64-bit
- Byte Order: Little Endian
- Address sizes: 39 bits physical, 48 bits virtual
- CPU(s): 4
- On-line CPU(s) list: 0-3
- Thread(s) per core: 1
- Core(s) per socket: 4
- Socket(s): 1
- NUMA node(s): 1
- Vendor ID: GenuineIntel
- CPU family: 6
- Model: 94
- Model name: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
- Stepping: 3
- CPU MHz: 2763.879
- CPU max MHz: 3600.0000
- CPU min MHz: 800.0000
- BogoMIPS: 6386.00
- Virtualization: VT-x
- L1d cache: 32K
- L1i cache: 32K
- L2 cache: 256K
- L3 cache: 6144K
- NUMA node0 CPU(s): 0-3
- Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d
- #+END_EXAMPLE
-
-** DFA 10^3
-
- #+NAME: laptop_dfa_1000
- | N | t_p (s) | t_i (s) | t_r (s) | t_v (s) |
- |------+---------+---------+---------+---------|
- | 1000 | 3.78 | 1.17 | 0.49 | 1.49 |
- | 2000 | 7.95 | 2.44 | 1.03 | 3.42 |
- | 3000 | 11.84 | 3.27 | 1.91 | 5.36 |
- | 4000 | 16.33 | 5.04 | 2.22 | 7.43 |
- | 5000 | 20.70 | 6.37 | 2.80 | 9.44 |
-
-** DFA 10^4
-
- #+NAME: laptop_dfa_10000
- | N | t_p (s) | t_i (s) | t_r (s) | t_v (s) |
- |-----+---------+---------+---------+---------|
- | 600 | 67.28 | 3.02 | 4.52 | 10.43 |
- | 700 | 80.67 | 7.68 | 5.12 | 12.65 |
-
- Rest ging nicht wegen RAM
-
-** PRISM
-
- #+NAME: laptop_prism
- | File | t_p (s) | t_i (s) | t_r (s) | t_v (s) |
- |------+---------+---------+---------+---------|
- | 1 | 4.20 | 1.11 | 1.80 | 2.18 |
- | 1 | 19.13 | 4.74 | 9.42 | 8.20 |
- | 2 | 1.09 | 0.46 | 1.17 | 0.12 |
- | 3 | 9.86 | 4.54 | 9.96 | 1.23 |
- | 3 | 28.01 | 9.91 | 27.27 | 3.42 |
-
-** Faktor zw Laptop und Desktop
-*** DFA 10^3
-
- #+CALL: factor(a=desktop_dfa_1000, b=laptop_dfa_1000)
-
- #+RESULTS:
- | N | t_p (s) | t_i (s) | t_r (s) | t_v (s) |
- |------+---------+---------+---------+---------|
- | 1000 | 1.48 | 1.43 | 1.32 | 1.41 |
- | 2000 | 1.5 | 1.46 | 1.29 | 1.29 |
- | 3000 | 1.51 | 1.41 | 1.26 | 1.26 |
- | 4000 | 1.53 | 1.48 | 1.28 | 1.14 |
- | 5000 | 1.52 | 1.49 | 1.26 | 1.22 |
-
-*** DFA 10^4
-
- #+CALL: factor(a=desktop_dfa_10000, b=laptop_dfa_10000)
-
- #+RESULTS:
- | N | t_p (s) | t_i (s) | t_r (s) | t_v (s) |
- |-----+---------+---------+---------+---------|
- | 600 | 1.45 | 1.62 | 1.5 | 1.21 |
- | 700 | 1.54 | 1.69 | 1.54 | 1.23 |
-
-*** PRISM
-
- #+CALL: factor(a=desktop_prism, b=laptop_prism)
-
- #+RESULTS:
- | File | t_p (s) | t_i (s) | t_r (s) | t_v (s) |
- |------+---------+---------+---------+---------|
- | 1 | 1.65 | 1.59 | 1.62 | 1.83 |
- | 1 | 1.86 | 1.58 | 1.68 | 1.35 |
- | 2 | 2.14 | 1.59 | 1.95 | 1.5 |
- | 3 | 1.85 | 1.47 | 1.72 | 1.58 |
- | 3 | 2.04 | 1.56 | 1.7 | 1.23 |
-
-* Ergebnisse in VM am Desktop
-
- VM hatte 8 von 16 GB RAM des Hostsystems
-
-** DFA 10^3
-
- #+NAME: vm_dfa_1000
- | N | tp | ti | tr | tv |
- |------+-------+------+------+------|
- | 1000 | 3.33 | 0.82 | 0.39 | 1.25 |
- | 2000 | 6.65 | 1.69 | 0.83 | 3.55 |
- | 3000 | 10.20 | 2.29 | 1.62 | 4.97 |
- | 4000 | 13.89 | 3.48 | 1.90 | 6.77 |
- | 5000 | 17.31 | 4.31 | 2.34 | 9.20 |
-
-** DFA 10^4
-
- #+NAME: vm_dfa_10000
- | N | tp | ti | tr | tv |
- |-----+-------+------+------+-------|
- | 600 | 82.08 | 1.94 | 3.24 | 9.65 |
- | 700 | 93.74 | 4.58 | 3.55 | 11.13 |
-
- Mehr ging nicht wegen RAM
-
-** PRISM
-
- #+NAME: vm_prism
- | | tp | ti | tr | tv |
- |----+-------+------+-------+------|
- | 1. | 2.63 | 0.74 | 1.21 | 1.41 |
- | 1. | 10.98 | 3.20 | 6.17 | 6.34 |
- | 2. | 0.55 | 0.31 | 0.65 | 0.11 |
- | 3. | 5.59 | 3.22 | 6.08 | 0.86 |
- | 3. | 14.46 | 6.55 | 17.19 | 2.95 |
-
-** Faktor zu Desktop
-*** DFA 10^3
-
- #+CALL: factor(a=desktop_dfa_1000, b=vm_dfa_1000)
-
- #+RESULTS:
- | N | tp | ti | tr | tv |
- |------+------+------+------+------|
- | 1000 | 1.3 | 1.0 | 1.05 | 1.18 |
- | 2000 | 1.26 | 1.01 | 1.04 | 1.34 |
- | 3000 | 1.3 | 0.99 | 1.07 | 1.17 |
- | 4000 | 1.3 | 1.02 | 1.09 | 1.04 |
- | 5000 | 1.27 | 1.01 | 1.05 | 1.19 |
-
-*** DFA 10^4
-
- #+CALL: factor(a=desktop_dfa_10000, b=vm_dfa_10000)
-
- #+RESULTS:
- | N | tp | ti | tr | tv |
- |-----+------+------+------+------|
- | 600 | 1.77 | 1.04 | 1.07 | 1.12 |
- | 700 | 1.79 | 1.01 | 1.07 | 1.09 |
-
-*** PRISM
-
- #+CALL: factor(a=desktop_prism, b=vm_prism)
-
- #+RESULTS:
- | | tp | ti | tr | tv |
- |---+------+------+------+------|
- | 1 | 1.04 | 1.06 | 1.09 | 1.18 |
- | 1 | 1.07 | 1.07 | 1.1 | 1.04 |
- | 2 | 1.08 | 1.07 | 1.08 | 1.38 |
- | 3 | 1.05 | 1.04 | 1.05 | 1.1 |
- | 3 | 1.05 | 1.03 | 1.07 | 1.06 |
-
-* Ergebnisse auf Thorstens Rechner
-
- Der Rechner hat theoretisch 16GB, allerdings kann mein Account davon nicht
- alle Nutzen (vll ca 10?). Ausgabe von =lscpu=:
-
- #+BEGIN_EXAMPLE
- processor : 0
- vendor_id : GenuineIntel
- cpu family : 6
- model : 60
- model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
- stepping : 3
- microcode : 0x1c
- cpu MHz : 3979.623
- cache size : 8192 KB
- physical id : 0
- siblings : 8
- core id : 0
- cpu cores : 4
- apicid : 0
- initial apicid : 0
- fpu : yes
- fpu_exception : yes
- cpuid level : 13
- wp : yes
- flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
- bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
- bogomips : 7186.95
- clflush size : 64
- cache_alignment : 64
- address sizes : 39 bits physical, 48 bits virtual
- power management:
- #+END_EXAMPLE
-
-** DFA 10^3
-
- | N | t_p(s) | t_i(s) | t_r(s) | t_v(s) |
- |------+--------+--------+--------+--------|
- | 1000 | 2.43 | 0.81 | 0.37 | 1.02 |
- | 2000 | 4.99 | 1.69 | 0.78 | 2.62 |
- | 3000 | 7.36 | 2.16 | 1.42 | 4.26 |
- | 4000 | 10.12 | 3.45 | 1.71 | 6.16 |
- | 5000 | 12.95 | 4.42 | 2.18 | 7.77 |
-
-** DFA 10^4
-
- | N | t_p(s) | t_i(s) | t_r(s) | t_v(s) |
- |-----+--------+--------+--------+--------|
- | 600 | 47.32 | 1.84 | 3.17 | 7.87 |
- | 700 | 53.92 | 4.39 | 3.26 | 9.26 |
-
-** PRISM
-
- | File | t_p(s) | t_i(s) | t_r(s) | t_v(s) |
- |------+--------+--------+--------+--------|
- | 1 | 2.39 | 0.70 | 1.13 | 1.17 |
- | 1 | 9.74 | 3.00 | 5.73 | 5.60 |
- | 2 | 0.49 | 0.31 | 0.62 | 0.09 |
- | 3 | 5.18 | 3.39 | 6.20 | 0.76 |
- | 3 | 13.27 | 7.01 | 17.29 | 2.21 |
-
-* Faktoren von VM zu Thorsten
-** DFA 10^3
-
- | N | tp | ti | tr | tv |
- |------+-----------+------------+-----------+-----------|
- | 1000 | 1.3703704 | 1.0123457 | 1.0540541 | 1.2254902 |
- | 2000 | 1.3326653 | 1. | 1.0641026 | 1.3549618 |
- | 3000 | 1.4166667 | 1.0601852 | 1.1408451 | 1.1666667 |
- | 4000 | 1.3725296 | 1.0086957 | 1.1111111 | 1.0990260 |
- | 5000 | 1.3366795 | 0.97511312 | 1.0733945 | 1.1840412 |
- #+TBLFM: @2$2=3.33/2.43::@2$3=0.82/0.81::@2$4=0.39/0.37::@2$5=1.25/1.02::@3$2=6.65/4.99::@3$3=1.69/1.69::@3$4=0.83/0.78::@3$5=3.55/2.62::@4$2=10.20/7.20::@4$3=2.29/2.16::@4$4=1.62/1.42::@4$5=4.97/4.26::@5$2=13.89/10.12::@5$3=3.48/3.45::@5$4=1.9/1.71::@5$5=6.77/6.16::@6$2=17.31/12.95::@6$3=4.31/4.42::@6$4=2.34/2.18::@6$5=9.20/7.77
-
- D.h faktor so um 1-1.1, außer beim parsen, da konsistent 1.3. (Valmari
- hat schwankend zw. 1 und 2)
-
-** DFA 10^4
-
- | N | tp | ti | tr | tv |
- |-----+-----------+-----------+-----------+-----------|
- | 600 | 1.7345731 | 1.0543478 | 1.0220820 | 1.2261753 |
- | 700 | 1.7385015 | 1.0432802 | 1.0889571 | 1.2019438 |
- #+TBLFM: @2$2=82.08/47.32::@2$3=1.94/1.84::@2$4=3.24/3.17::@2$5=9.65/7.87::@3$2=93.74/53.92::@3$3=4.58/4.39::@3$4=3.55/3.26::@3$5=11.13/9.26
-
- D.h bei ti und rt ein faktor von ca 1.05, bei tv ein faktor von 1.2
- und bei tp ein faktor von 1.7
-
-** PRISM
-
- | | tp | ti | tr | tv |
- |----+-----------+------------+------------+-----------|
- | | | x | x | x |
- | 1. | 1.1004184 | 1.0571429 | 1.0707965 | 1.2051282 |
- | 1. | 1.1273101 | 1.0666667 | 1.0767888 | 1.1321429 |
- | 2. | 1.1224490 | 1. | 1.0483871 | 1.2222222 |
- | 3. | 1.0791506 | 0.94985251 | 0.98064516 | 1.1315789 |
- | 3. | 1.0896760 | 0.93437946 | 0.99421631 | 1.3348416 |
- #+TBLFM: @2$2=2.63/2.39::@3$2=10.98/9.74::@3$3=0.74/0.7::@3$4=1.21/1.13::@3$5=1.41/1.17::@4$2=0.55/0.49::@4$3=3.20/3.0::@4$4=6.17/5.73::@4$5=6.34/5.6::@5$2=5.59/5.18::@5$3=0.31/0.31::@5$4=0.65/0.62::@5$5=0.11/0.09::@6$2=14.46/13.27::@6$3=3.22/3.39::@6$4=6.08/6.2::@6$5=0.86/0.76::@7$3=6.55/7.01::@7$4=17.19/17.29::@7$5=2.95/2.21
-
- D.h alle Faktoren eigentlich so um 1-1.1 manchmal sogar ein wenig
- schneller. Valmari eher bso bei 1.2.
-
-* Code
-** Faktor ausrechnen
-
- #+NAME: factor
- #+BEGIN_SRC python :var a=[] :var b=[] :colnames yes :rownames yes
- table = []
- for i in range(min(len(a), len(b))):
- row = []
- for j in range(min(len(a[i]), len(b[i]))):
- row.append(round(b[i][j] / a[i][j], 2))
- table.append(row)
- return table
- #+END_SRC
diff --git a/cabal.project.freeze b/cabal.project.freeze
deleted file mode 100644
index ef2054cdc8ebcd7ee7f06d56d00681485b101a4d..0000000000000000000000000000000000000000
--- a/cabal.project.freeze
+++ /dev/null
@@ -1,147 +0,0 @@
-constraints: any.Cabal ==2.2.0.1,
- any.Glob ==0.10.0,
- any.HUnit ==1.6.0.0,
- any.Only ==0.1,
- any.QuickCheck ==2.12.6.1,
- QuickCheck +templatehaskell,
- any.StateVar ==1.1.1.1,
- any.abstract-deque ==0.3,
- abstract-deque -usecas,
- any.abstract-par ==0.3.3,
- any.aeson ==1.4.2.0,
- aeson -bytestring-builder -cffi -developer -fast,
- any.ansi-terminal ==0.8.2,
- ansi-terminal -example,
- any.ansi-wl-pprint ==0.6.8.2,
- ansi-wl-pprint -example,
- any.array ==0.5.2.0,
- any.attoparsec ==0.13.2.2,
- attoparsec -developer,
- any.base ==4.11.1.0,
- any.base-compat ==0.10.5,
- any.base-compat-batteries ==0.10.5,
- any.base-orphans ==0.8,
- any.binary ==0.8.5.1,
- any.bytestring ==0.10.8.2,
- any.cabal-doctest ==1.0.6,
- any.call-stack ==0.1.0,
- any.case-insensitive ==1.2.0.11,
- any.cassava ==0.5.1.0,
- cassava -bytestring--lt-0_10_4,
- any.cereal ==0.5.7.0,
- cereal -bytestring-builder,
- any.clock ==0.7.2,
- clock -llvm,
- any.code-page ==0.2,
- any.colour ==2.3.4,
- any.containers ==0.5.11.0,
- any.contravariant ==1.5,
- contravariant -safe +semigroups +statevar +tagged,
- copar -release,
- any.criterion ==1.5.3.0,
- criterion -embed-data-files -fast,
- any.criterion-measurement ==0.1.1.0,
- criterion-measurement -fast,
- any.data-default-class ==0.1.2.0,
- any.deepseq ==1.4.3.0,
- any.dense-linear-algebra ==0.1.0.0,
- any.deriving-compat ==0.5.2,
- deriving-compat +base-4-9 +new-functor-classes +template-haskell-2-11,
- any.directory ==1.3.1.5,
- any.dlist ==0.8.0.5,
- any.doctest ==0.16.0.1,
- any.erf ==2.0.0.0,
- any.exceptions ==0.10.0,
- any.extra ==1.6.14,
- any.filepath ==1.4.2,
- any.ghc ==8.4.4,
- any.ghc-boot ==8.4.4,
- any.ghc-boot-th ==8.4.4,
- any.ghc-paths ==0.1.0.9,
- any.ghc-prim ==0.5.2.0,
- any.ghci ==8.4.4,
- any.happy ==1.19.9,
- happy +small_base,
- any.hashable ==1.2.7.0,
- hashable -examples +integer-gmp +sse2 -sse41,
- any.haskell-lexer ==1.0.2,
- any.hpc ==0.6.0.3,
- any.hsc2hs ==0.68.3,
- hsc2hs -in-ghc-tree,
- any.hspec ==2.6.0,
- any.hspec-core ==2.6.0,
- any.hspec-discover ==2.6.0,
- any.hspec-expectations ==0.8.2,
- any.hspec-megaparsec ==2.0.0,
- hspec-megaparsec -dev,
- any.ieee754 ==0.8.0,
- any.integer-gmp ==1.0.2.0,
- any.integer-logarithms ==1.0.2.2,
- integer-logarithms -check-bounds +integer-gmp,
- any.js-flot ==0.8.3,
- any.js-jquery ==3.3.1,
- any.json ==0.9.3,
- json -generic -mapdict -parsec -pretty +split-base,
- any.math-functions ==0.3.1.0,
- math-functions -system-expm1,
- any.megaparsec ==7.0.4,
- megaparsec -dev,
- any.microlens ==0.4.10,
- any.microlens-ghc ==0.4.10,
- any.microlens-mtl ==0.1.11.1,
- any.microlens-platform ==0.3.11,
- any.microlens-th ==0.4.2.3,
- microlens-th +inlining,
- any.microstache ==1.0.1.1,
- any.monad-par ==0.3.4.8,
- monad-par -chaselev -newgeneric,
- any.monad-par-extras ==0.3.3,
- any.mtl ==2.2.2,
- any.mwc-random ==0.14.0.0,
- any.optparse-applicative ==0.14.3.0,
- any.parallel ==3.2.2.0,
- any.parsec ==3.1.13.0,
- any.parser-combinators ==1.0.0,
- parser-combinators -dev,
- any.pretty ==1.1.3.6,
- any.pretty-show ==1.9.5,
- any.prettyprinter ==1.2.1,
- prettyprinter -buildreadme,
- any.prettyprinter-ansi-terminal ==1.1.1.2,
- any.prettyprinter-convert-ansi-wl-pprint ==1.1,
- any.primitive ==0.6.4.0,
- any.process ==1.6.3.0,
- any.quickcheck-io ==0.2.0,
- any.random ==1.1,
- any.rts ==1.0,
- any.scientific ==0.3.6.2,
- scientific -bytestring-builder -integer-simple,
- any.setenv ==0.1.1.3,
- any.statistics ==0.15.0.0,
- any.stm ==2.4.5.1,
- any.syb ==0.7,
- any.tagged ==0.8.6,
- tagged +deepseq +transformers,
- any.template-haskell ==2.13.0.0,
- any.terminfo ==0.4.1.1,
- any.text ==1.2.3.1,
- any.text-short ==0.1.2,
- text-short -asserts,
- any.tf-random ==0.5,
- any.th-abstraction ==0.2.10.0,
- any.time ==1.8.0.2,
- any.time-locale-compat ==0.1.1.5,
- time-locale-compat -old-locale,
- any.transformers ==0.5.5.0,
- any.transformers-compat ==0.6.2,
- transformers-compat -five +five-three -four +generic-deriving +mtl -three -two,
- any.unix ==2.7.2.2,
- any.unordered-containers ==0.2.9.0,
- unordered-containers -debug,
- any.uuid-types ==1.0.3,
- any.vector ==0.12.0.2,
- vector +boundschecks -internalchecks -unsafechecks -wall,
- any.vector-algorithms ==0.8.0.1,
- vector-algorithms +bench +boundschecks -internalchecks -llvm +properties -unsafechecks,
- any.vector-binary-instances ==0.2.5.1,
- any.vector-th-unbox ==0.2.1.6
diff --git a/dfa/bench.py b/dfa/bench.py
deleted file mode 100755
index daf665162dc96b0d0ec39134f82cf6f5346d9f03..0000000000000000000000000000000000000000
--- a/dfa/bench.py
+++ /dev/null
@@ -1,248 +0,0 @@
-#!/usr/bin/env python
-
-import sys
-import json
-import subprocess
-import os
-import re
-import numpy as np
-import scipy.stats as st
-import argparse
-
-samples = 10
-
-
-def usage(msg):
- print("Usage: %s %s" % (sys.argv[0], msg))
- sys.exit(1)
-
-
-def filename(ext, n, m, i, f):
- if f is None:
- return "bench/dfa_%d_%d_%d.%s" % (n, m, i, ext)
- else:
- return "bench/dfa_%d_%d_%d_f%d.%s" % (n, m, i, f, ext)
-
-
-def coalg_file(n, m, i, f):
- return filename("coalgebra", n, m, i, f)
-
-
-def valmari_file(n, m, i, f):
- return filename("valmari", n, m, i, f)
-
-
-def dot_file(n, m, i, f):
- return filename("dot", n, m, i, f)
-
-
-def generate(args):
- generator = args.generator
- n = args.n
- m = args.m
- dot = args.dot
- final = args.final
-
- os.makedirs("bench", exist_ok=True)
-
- for i in range(0, samples):
- if os.path.exists(coalg_file(n, m, i, final)):
- continue
- cmd = [generator, str(n), str(m),
- ("copar:%s" % coalg_file(n, m, i, final)),
- ("valmari:%s" % valmari_file(n, m, i, final))]
- if dot:
- cmd.append("dot:%s" % dot_file(n, m, i, final))
- if final is not None:
- cmd.append("--final")
- cmd.append(str(final))
- subprocess.run(cmd)
-
-
-def run_one(copar, valmari, n, m, i, final, noopt, nofunctrans):
- # Run copar on it
- copar_args = [copar, 'refine', '--stats-json', coalg_file(n, m, i, final)]
-
- if noopt:
- copar_args.append("--disable-optimizations")
-
- if nofunctrans:
- copar_args.append("--no-functor-transforms")
-
- out = subprocess.run(copar_args,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.PIPE)
- stats = json.loads(out.stderr.decode('utf-8'))
-
- # Run valmari on it
- valmari_open_file = open(valmari_file(n, m, i, final))
- out = subprocess.run(['perf', 'stat', valmari],
- stdin=valmari_open_file,
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE,
- env={'LC_ALL': 'C'})
- valmari_time = re.search(r'(\d+.\d+)\W+seconds time elapsed',
- out.stderr.decode('utf-8'))[1]
- valmari_states = re.search(r'\d+', out.stdout.decode('utf-8'))[0]
-
- stats['n'] = int(n)
- stats['m'] = int(m)
- stats['i'] = int(i)
- stats['valmari-time'] = valmari_time
- stats['valmari-states'] = valmari_states
-
- return stats
-
-
-def confidence(vals):
- """Compute the 95% confidence intervall (CI) for the mean with the student
-distribution.
-
-Returns a tuple of (mean, lower, upper), where lower and upper are the bounds
-of the CI"""
-
- # For a larger sample size (> 30), we could also use the normal
- # distribution.
- #
- # This code is taken from
- # https://stackoverflow.com/questions/15033511/compute-a-confidence-interval-from-sample-data/34474255#34474255
-
- mean = np.mean(vals)
- ci = st.t.interval(0.95, len(vals)-1, loc=np.mean(vals),
- scale=st.sem(vals))
-
- return (mean, ci[0], ci[1])
-
-
-def stddev(vals):
- """Compute the mean and standard deviation intervall on a sample.
-
-This uses the corrected sample standard deviation."""
-
- # see also:
- # https://en.wikipedia.org/wiki/Standard_deviation#Corrected_sample_standard_deviation
- mean = np.mean(vals)
- std = np.std(vals, ddof=1)
-
- return (mean, std)
-
-
-def print_row(d, stddev):
- values = [
- d['n'],
- d['m'],
- d['i'],
- d['states'],
- d['edges'],
- d['initial-partition-size'],
- d['final-partition-size'],
- d['explicit-final-partition-size'],
- d['size1-skipped']
- ]
-
- for k in ['overall-duration', 'parse-duration', 'algorithm-duration',
- 'initialize-duration', 'refine-duration', 'valmari-time']:
- values.append(d[k])
- if stddev:
- values.append(d[k + '-stddev'])
-
- values.append(d['valmari-states'])
-
- print('\t'.join(str(x) for x in values))
-
-
-def run(args):
- copar = args.copar
- valmari = args.valmari
- n = args.n
- m = args.m
- final = args.final
- noopt = args.noopt
- nofunctrans = args.nofunctrans
- sample_count = samples if args.samples is None else args.samples
-
- results = [run_one(copar, valmari, n, m, i, final, noopt, nofunctrans)
- for i in range(0, sample_count)]
-
- # Do some sanity checks on the samles
- if len(results) != sample_count:
- pass
-
- def allequal(lst, key):
- return all(x[key] == lst[0][key] for x in lst)
-
- equal_keys = ['n', 'm', 'states', 'edges', 'initial-partition-size']
-
- for k in equal_keys:
- if not allequal(results, k):
- raise(Exception("Samples don't match for key '%s'" % k))
-
- def confidencekey(vals, k):
- return confidence(list(float(x[k]) for x in vals))
-
- def stddevkey(vals, k):
- return stddev(list(float(x[k]) for x in vals))
-
- combined = results[0].copy()
- combined['i'] = sample_count
-
- for k in ['overall-duration', 'parse-duration', 'initialize-duration',
- 'refine-duration', 'algorithm-duration', 'valmari-time']:
- ci = stddevkey(results, k)
- combined[k] = str(ci[0])
- combined[k + '-stddev'] = str(ci[1])
-
- if args.indiv:
- for res in results:
- print_row(res, stddev=False)
- else:
- print_row(combined, stddev=args.stddev)
-
-
-if __name__ == "__main__":
- parser = argparse.ArgumentParser()
- subparsers = parser.add_subparsers(required=True)
-
- gen_parser = subparsers.add_parser('generate')
- gen_parser.add_argument('generator')
- gen_parser.add_argument('n', type=int)
- gen_parser.add_argument('m', type=int)
- gen_parser.add_argument('--dot', action='store_true')
- gen_parser.add_argument('--final', type=int,
- help="Percentage of final states",
- metavar="PERCENT")
- gen_parser.set_defaults(func=generate)
-
- run_parser = subparsers.add_parser('run')
- run_parser.add_argument('copar')
- run_parser.add_argument('valmari')
- run_parser.add_argument('n', type=int)
- run_parser.add_argument('m', type=int)
- run_parser.add_argument('--indiv', action='store_true',
- help="report individual samples")
- run_parser.add_argument('--stddev', action='store_true',
- help="report stddev for timings")
- run_parser.add_argument('--final', type=int,
- help="Percentage of final states",
- metavar="PERCENT")
- run_parser.add_argument('--noopt', action='store_true',
- help="run copar without --disable-optimizations")
- run_parser.add_argument('--nofunctrans', action='store_true',
- help="run copar without --no-functor-transforms")
- run_parser.add_argument('--samples', type=int,
- help="number of samples to run")
- run_parser.set_defaults(func=run)
-
- args = parser.parse_args()
- args.func(args)
-
- # if len(sys.argv) < 2:
- # usage("generate|run ARGUMENTS..")
- # sys.exit(1)
-
- # if sys.argv[1] == "generate":
- # generate(sys.argv[2:])
- # elif sys.argv[1] == "run":
- # run(sys.argv[2:])
- # else:
- # usage("generate|run ARGUMENTS..")
diff --git a/dfa/benchmark.org b/dfa/benchmark.org
deleted file mode 100644
index 10f9f65089f9d005835235bd4889658da1d9fd08..0000000000000000000000000000000000000000
--- a/dfa/benchmark.org
+++ /dev/null
@@ -1,749 +0,0 @@
-#+TITLE: Random DFAs
-#+LATEX_HEADER: \usepackage[a4paper,margin=1in,landscape]{geometry}
-
-* Generating the benchmarks
-
- Our generator accepts two parameters:
-
- - n :: Number of states
- - m :: Size of the input alphabet
-
- Generating the output file for n=10000 and m=10000 already almost eats up all
- 16GB of RAM, so we can't go any higher than that.
-
- #+BEGIN_SRC sh :results output silent
- for n in 10 20 30 40 50 60 70 80 90 100 200 300 400 500 600 700 800 900 1000 2000 3000 4000 5000 10000; do
- for m in 10 100 1000 10000; do
- python bench.py generate ../../copar/bin/random-dfa $n $m
- done
- done
- #+END_SRC
-
-* Executing the benchmarks
-
-The fields in the following table are:
-
-- N :: Number of states of the automaton
-- M :: Size of the input alphabet of the automaton
-- i :: Number of sample for individual benchmark or sample count for aggregated
- values.
-- States :: Number of states of the coalgebra
-- Edges :: Number of edges in the coalgebra
-- I :: Size of initial partition
-- Q :: Size of final partition
-- Qs1 :: Size of final partition restricted to sort 1
-- t :: Overall time for CoPaR in seconds
-- tp :: Time to parse the input in seconds
-- ta :: Overall time to execute the refinement algorithm
-- ti :: Time for the initialization procedure
-- tr :: Time for the actual partition refinement
-- tv :: Overall time for Valmari in seconds
-- Qv :: Size of final partition that valmari's tool computes
-
-For aggregated benchmarks, each timing result also has an additional column with
-the standard derivation of the samples
-
-Important correctness checks are:
-
-- The number of states should be the same in the coalgebra and automaton
-- The number of edges in the coalgebra should be $n*m$
-- The size of the initial partition must be 2
-- The size of the final partition should be the same as for valmari
-
-** Plotting Code
-
-The following python code can plot the tables below and produce an inline image
-into this document.
-
-
-#+NAME: plot
-#+BEGIN_SRC python :results file :exports results :var data="" :var filename=""
-import matplotlib.pyplot as plt
-
-'''Turn the table data into x and y data'''
-x = [a[0] for a in data]
-t = [a[9] for a in data]
-t_s = [a[10] for a in data]
-tp = [a[11] for a in data]
-tp_s = [a[12] for a in data]
-ta = [a[13] for a in data]
-ta_s = [a[14] for a in data]
-ti = [a[15] for a in data]
-ti_s = [a[16] for a in data]
-tr = [a[17] for a in data]
-tr_s = [a[18] for a in data]
-tv = [a[19] for a in data]
-tv_s = [a[20] for a in data]
-
-''' Plot the x and y data'''
-plt.errorbar(x, t, yerr=t_s, label="t")
-plt.errorbar(x, tp, yerr=tp_s, label="tp")
-plt.errorbar(x, ta, yerr=ta_s, label="ta")
-plt.errorbar(x, tr, yerr=tr_s, label="tr")
-plt.errorbar(x, ti, yerr=ti_s, label="ti")
-plt.errorbar(x, tv, yerr=tv_s, label="tv")
-
-''' Set the x and y labels on the graph '''
-plt.xlabel("Number of states")
-plt.ylabel("Seconds")
-
-''' Create the legend '''
-plt.legend(loc="upper left")
-
-''' Save the PNG file '''
-plt.savefig(filename)
-
-''' Return the PNG file path to OrgMode '''
-return(filename)
-#+END_SRC
-
-** TODO Test
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opt" "t (s)" "tp (s)" "ta (s)" "ti (s)" "tr (s)" "tv (s)" "Qv")
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 10 10 --indiv
-#+end_src
-
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opt | t (s) | tp (s) | ta (s) | ti (s) | tr (s) | tv (s) | Qv |
-|----+----+---+--------+-------+---+----+------+-----+-------------+-------------+-------------+------------+------------+-------------+-----|
-| 10 | 10 | 0 | 10 | 100 | 2 | 10 | 10 | 83 | 0.001046352 | 0.00092647 | 8.813e-05 | 2.5603e-05 | 4.8467e-05 | 0.0013978 | 10 |
-| 10 | 10 | 1 | 10 | 100 | 2 | 10 | 10 | 81 | 0.001332556 | 0.00117165 | 0.000121423 | 4.205e-05 | 6.1448e-05 | 0.001185295 | 10 |
-| 10 | 10 | 2 | 10 | 100 | 2 | 10 | 10 | 86 | 0.001131453 | 0.001002346 | 9.7326e-05 | 2.4585e-05 | 5.9035e-05 | 0.001117232 | 10 |
-| 10 | 10 | 3 | 10 | 100 | 2 | 10 | 10 | 76 | 0.001090468 | 0.000967138 | 9.1729e-05 | 2.4952e-05 | 5.2869e-05 | 0.001147177 | 10 |
-| 10 | 10 | 4 | 10 | 100 | 2 | 10 | 10 | 70 | 0.001085314 | 0.000961097 | 9.1407e-05 | 2.6115e-05 | 5.0625e-05 | 0.001228354 | 10 |
-| 10 | 10 | 5 | 10 | 100 | 2 | 10 | 10 | 77 | 0.001098224 | 0.000950174 | 9.715e-05 | 2.6022e-05 | 5.0376e-05 | 0.001149149 | 10 |
-| 10 | 10 | 6 | 10 | 100 | 2 | 10 | 10 | 75 | 0.001075821 | 0.000940202 | 0.000101156 | 2.7531e-05 | 5.9542e-05 | 0.001141678 | 10 |
-| 10 | 10 | 7 | 10 | 100 | 2 | 10 | 10 | 81 | 0.001051644 | 0.000930808 | 8.7346e-05 | 2.6147e-05 | 4.7219e-05 | 0.001227956 | 10 |
-| 10 | 10 | 8 | 10 | 100 | 2 | 10 | 10 | 78 | 0.001059422 | 0.000935757 | 9.0775e-05 | 2.6297e-05 | 4.999e-05 | 0.001297999 | 10 |
-| 10 | 10 | 9 | 10 | 100 | 2 | 10 | 10 | 79 | 0.001121328 | 0.000993801 | 9.2128e-05 | 2.6818e-05 | 5.0163e-05 | 0.001188937 | 10 |
-
-** For m = 1:
-*** Generating some benchmarks
-
-#+begin_src sh :results output silent
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000 200000 200000 210000 220000 230000 240000 250000 260000 270000 280000 290000 300000; do
- python bench.py generate ../../copar/bin/random-dfa $n 1
-done
-#+end_src
-
-*** And running them
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "sdt" "tp (s)" "sdtp" "ta (s)" "sdta" "ti (s)" "sdti" "tr (s)" "sdtr" "tv (s)" "sdtv" "Qv")
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000 200000 210000 220000 230000 240000 250000 260000 270000 280000 290000 300000; do
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 1 --stddev --nofunctrans
-done
-#+end_src
-
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | sdt | tp (s) | sdtp | ta (s) | sdta | ti (s) | sdti | tr (s) | sdtr | tv (s) | sdtv | Qv |
-|--------+---+----+--------+--------+---+--------+--------+-------+--------------------+---------------------+--------------------+----------------------+--------------------+---------------------+---------------------+-----------------------+--------------------+---------------------+---------------------+-----------------------+--------|
-| 100000 | 1 | 10 | 100000 | 100000 | 2 | 65489 | 65489 | 8071 | 3.9017160891999993 | 0.05725542243446023 | 1.7041072878999999 | 0.007666172923625777 | 1.9023876757 | 0.06490987848505304 | 0.2628267297 | 0.003272320490714361 | 1.5989905591 | 0.0658406983383148 | 0.13759999169999998 | 0.0065012827508490615 | 65489 |
-| 110000 | 1 | 10 | 110000 | 110000 | 2 | 72513 | 72513 | 9297 | 4.313956430299999 | 0.0763833477465056 | 1.949160167 | 0.009027583663742882 | 2.1389264483 | 0.08096080778904134 | 0.30003091430000006 | 0.00510612940724545 | 1.7878511580999998 | 0.08900092769490782 | 0.1531381338 | 0.006417484978214387 | 72513 |
-| 120000 | 1 | 10 | 120000 | 120000 | 2 | 78444 | 78444 | 9790 | 4.8029536083 | 0.11912637839624582 | 2.1499455626999997 | 0.015364837081170035 | 2.3896450291 | 0.12655058491231658 | 0.3351564949 | 0.005116840454431979 | 1.9936482227999999 | 0.12438977802181102 | 0.1976445005 | 0.06662102777240582 | 78444 |
-| 130000 | 1 | 10 | 130000 | 130000 | 2 | 85315 | 85315 | 10567 | 5.2303333725000005 | 0.14216674785304523 | 2.3350624718999997 | 0.015434810994377981 | 2.5886367938 | 0.1231368250495237 | 0.3720805978 | 0.004435277321512478 | 2.1608228643000005 | 0.12243514299040262 | 0.20366009489999998 | 0.009002729886375587 | 85315 |
-| 140000 | 1 | 10 | 140000 | 140000 | 2 | 91961 | 91961 | 11636 | 5.7241506135 | 0.18089497938779125 | 2.5227946168999997 | 0.04041226205771596 | 2.8734782348 | 0.19157855600495133 | 0.37903946050000004 | 0.0055335089050379035 | 2.4276934805000003 | 0.194103518528028 | 0.2238437406 | 0.009396875043746323 | 91961 |
-| 150000 | 1 | 10 | 150000 | 150000 | 2 | 98627 | 98627 | 11706 | 6.187801929800001 | 0.14064425090177476 | 2.6582514965 | 0.009003423452304189 | 3.1697628569000003 | 0.13611337872223803 | 0.2999485256 | 0.004355764391675269 | 2.6839304120999996 | 0.1374033748576153 | 0.2515853332 | 0.014174216184394798 | 98627 |
-| 160000 | 1 | 10 | 160000 | 160000 | 2 | 104809 | 104809 | 12764 | 6.7382136996 | 0.15245636348209132 | 2.8366039933 | 0.015161039687297094 | 3.3935111766000006 | 0.15093821038539926 | 0.46547556329999995 | 0.006831967314617837 | 2.8632204123999996 | 0.1490646566678644 | 0.2813251122 | 0.011347243669986634 | 104809 |
-| 170000 | 1 | 10 | 170000 | 170000 | 2 | 111232 | 111232 | 14138 | 7.2908625285 | 0.10604295432427215 | 3.09530805 | 0.03603531460563363 | 3.7597713653 | 0.12118258888916565 | 0.5004929244 | 0.006790266960715258 | 3.1806287245999996 | 0.10781123989114733 | 0.3073801929 | 0.013086131717466363 | 111232 |
-| 180000 | 1 | 10 | 180000 | 180000 | 2 | 118197 | 118197 | 14269 | 7.7940563974 | 0.2543156188997473 | 3.2715667038 | 0.009360892657020248 | 4.092677974399999 | 0.2619800187273766 | 0.525093342 | 0.007190986044884863 | 3.4845772176000005 | 0.2651429743565608 | 0.3501228732 | 0.016837178135142662 | 118197 |
-| 190000 | 1 | 10 | 190000 | 190000 | 2 | 123893 | 123893 | 14728 | 8.269414321400001 | 0.19220873052742554 | 3.4378605475 | 0.020742001772012533 | 4.3756709237 | 0.1867233205842528 | 0.5445772935000001 | 0.02997447405159672 | 3.7418600049 | 0.16409710010971268 | 0.3724915993 | 0.016211865333268422 | 123893 |
-| 200000 | 1 | 10 | 200000 | 200000 | 2 | 131252 | 131252 | 16562 | 8.7204120381 | 0.260072753804474 | 3.6344360476 | 0.016315930797572084 | 4.6021848058999995 | 0.24376077112686112 | 0.5754112171 | 0.00753933981008275 | 3.9305835423 | 0.2554540636025266 | 0.4100586315 | 0.018201735614278045 | 131252 |
-| 210000 | 1 | 10 | 210000 | 210000 | 2 | 137501 | 137501 | 17008 | 9.2838852824 | 0.2636020019857964 | 3.8083384965000002 | 0.00953027592476456 | 4.9315308102 | 0.24319350760405056 | 0.5869515222999999 | 0.005476529581826432 | 4.2464882237 | 0.24324445568951442 | 0.441084476 | 0.023085429113395166 | 137501 |
-| 220000 | 1 | 10 | 220000 | 220000 | 2 | 144061 | 144061 | 17847 | 9.770356903799998 | 0.3423008046584935 | 4.0041314226 | 0.01303346631043801 | 5.2126720673 | 0.3285456949555346 | 0.6358050346999999 | 0.003022973889400297 | 4.469972052800001 | 0.3372079781421167 | 0.466137511 | 0.02001933439908453 | 144061 |
-| 230000 | 1 | 10 | 230000 | 230000 | 2 | 150310 | 150310 | 17731 | 10.437322718299999 | 0.22941617352716984 | 4.2013492154 | 0.05899131321541619 | 5.627988751199999 | 0.2635210887153482 | 0.6455574729 | 0.007678134687488006 | 4.868182401100001 | 0.25893084670217076 | 0.5156026173 | 0.022520583055545204 | 150310 |
-| 240000 | 1 | 10 | 240000 | 240000 | 2 | 157139 | 157139 | 18612 | 11.053093357799998 | 0.33411355973433937 | 4.3785478234 | 0.025313675755905325 | 6.043547060300001 | 0.3334691533111559 | 0.6885990058 | 0.004772154958912903 | 5.235913457100001 | 0.3284099648007952 | 0.5510596194 | 0.024897984793143372 | 157139 |
-| 250000 | 1 | 10 | 250000 | 250000 | 2 | 164155 | 164155 | 20499 | 11.5662374633 | 0.2625704687079291 | 4.5883679078 | 0.014908850614635781 | 6.3090031014 | 0.25471836122993463 | 0.7311154823999999 | 0.00533183466125695 | 5.459369843300001 | 0.2558493631410093 | 0.6040049302 | 0.009586109716559724 | 164155 |
-| 260000 | 1 | 10 | 260000 | 260000 | 2 | 170681 | 170681 | 20415 | 11.6456015015 | 0.511989377271459 | 4.7059698737 | 0.12513626926548418 | 6.2599540109 | 0.4130910662631789 | 0.6555105109 | 0.10337670349981325 | 5.4029216802 | 0.3887658337407677 | 0.6083744114 | 0.034852307242699845 | 170681 |
-| 270000 | 1 | 10 | 270000 | 270000 | 2 | 177419 | 177419 | 22189 | 12.120666825499999 | 0.3737239336482957 | 4.8034746198 | 0.01835509221979868 | 6.633201435299999 | 0.35387444814978264 | 0.7363434314 | 0.0031265120905228346 | 5.767584987699999 | 0.34512662981485487 | 0.6364424622 | 0.027411122932964365 | 177419 |
-| 280000 | 1 | 10 | 280000 | 280000 | 2 | 183598 | 183598 | 23254 | 12.940827799400001 | 0.3948481113814338 | 4.965828097099999 | 0.02554346533283756 | 7.025901951100001 | 0.37191392383248734 | 0.8191934889999999 | 0.007455230101518104 | 6.0883975169 | 0.3627755680944626 | 0.6621498792 | 0.033948444495518634 | 183598 |
-| 290000 | 1 | 10 | 290000 | 290000 | 2 | 190044 | 190044 | 22976 | 13.3304674734 | 0.36445051870292144 | 5.1312368327 | 0.013064971257863454 | 7.2310452817 | 0.36289207727580786 | 0.8502582112999999 | 0.0037351564859896816 | 6.2581376613 | 0.3563549113115732 | 0.7004045213 | 0.024056932759703036 | 190044 |
-| 300000 | 1 | 10 | 300000 | 300000 | 2 | 197133 | 197133 | 23844 | 14.146797660499999 | 0.31595827791135156 | 5.2838587488 | 0.018411110755737675 | 7.897829865599999 | 0.3353595236408498 | 0.8807967731999999 | 0.002439527016899235 | 6.8851801584 | 0.34025879458180663 | 0.7427352719 | 0.030209104626308197 | 197133 |
-
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "sdt" "tp (s)" "sdtp" "ta (s)" "sdta" "ti (s)" "sdti" "tr (s)" "sdtr" "tv (s)" "sdtv" "Qv")
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000 200000 210000 220000 230000 240000 250000 260000 270000 280000 290000 300000; do
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 1 --stddev
-done
-#+end_src
-
-#+NAME: results1
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | sdt | tp (s) | sdtp | ta (s) | sdta | ti (s) | sdti | tr (s) | sdtr | tv (s) | sdtv | Qv |
-|--------+---+----+--------+--------+---+--------+--------+-------+--------------------+---------------------+--------------------+-----------------------+--------------------+----------------------+---------------------+-----------------------+--------------------+---------------------+---------------------+-----------------------+--------|
-| 100000 | 1 | 10 | 100000 | 100000 | 2 | 65489 | 65489 | 8071 | 3.1723742583 | 0.06415135633353668 | 1.0352353053 | 0.008941562892079086 | 1.9350310748 | 0.059410960798478435 | 0.32343377709999993 | 0.006503732380012458 | 1.4825559133 | 0.06182975711003514 | 0.13457228840000002 | 0.0061834340026262645 | 65489 |
-| 110000 | 1 | 10 | 110000 | 110000 | 2 | 72513 | 72513 | 9297 | 3.6060606505000004 | 0.07120835312896302 | 1.1337366444 | 0.0066497616379870466 | 2.1326008449 | 0.06899518842479228 | 0.37291921390000005 | 0.009062006583398122 | 1.7116194235999997 | 0.07486105411993155 | 0.1531423643 | 0.009425071171266949 | 72513 |
-| 120000 | 1 | 10 | 120000 | 120000 | 2 | 78444 | 78444 | 9790 | 3.9819581869000005 | 0.08295924676715061 | 1.2491448927 | 0.008382980979176068 | 2.3497672478 | 0.08960022035531837 | 0.4194873646 | 0.008112127594262736 | 1.8800125135999999 | 0.09098119072406534 | 0.1770516467 | 0.007224653623244186 | 78444 |
-| 130000 | 1 | 10 | 130000 | 130000 | 2 | 85315 | 85315 | 10567 | 4.4078494884 | 0.11759337156384515 | 1.3737832236999998 | 0.009853487052357706 | 2.5904541815999997 | 0.12985226650810058 | 0.4653538676 | 0.009431514691067346 | 2.0676565862000005 | 0.12518907212331976 | 0.1988111277 | 0.004361050542692717 | 85315 |
-| 140000 | 1 | 10 | 140000 | 140000 | 2 | 91961 | 91961 | 11636 | 4.8345211587 | 0.18598447828239828 | 1.4702093769999998 | 0.00526703848044785 | 2.8969718773000004 | 0.18768478695801893 | 0.522174264 | 0.046844514809917 | 2.3171949742000004 | 0.231287985327397 | 0.22347404060000003 | 0.01314437739599008 | 91961 |
-| 150000 | 1 | 10 | 150000 | 150000 | 2 | 98627 | 98627 | 11706 | 5.1577071244 | 0.10743990008037858 | 1.6936769373 | 0.03179318810342249 | 3.1119092698 | 0.11610573189464778 | 0.5696047548000001 | 0.010090814422831586 | 2.4752281476 | 0.11124271004914135 | 0.24985176729999997 | 0.012456361594495972 | 98627 |
-| 160000 | 1 | 10 | 160000 | 160000 | 2 | 104809 | 104809 | 12764 | 5.4344598328 | 0.14671501495323758 | 1.7836474614 | 0.009964746328457117 | 3.2475654382999997 | 0.13978295339543426 | 0.5652916232 | 0.002392113968621664 | 2.609053951 | 0.1398729333907138 | 0.2762560884 | 0.006841057884760942 | 104809 |
-| 170000 | 1 | 10 | 170000 | 170000 | 2 | 111232 | 111232 | 14138 | 6.245343322000001 | 0.12009578467332085 | 1.8606334966999998 | 0.02741443041452057 | 3.7771725017 | 0.11137133528898206 | 0.6416360163999999 | 0.01595195544619899 | 3.0529216406 | 0.11120009895874813 | 0.3069283973 | 0.010424249298240946 | 111232 |
-| 180000 | 1 | 10 | 180000 | 180000 | 2 | 118197 | 118197 | 14269 | 6.6310759754 | 0.2253434734039179 | 1.9310191868 | 0.01289471358647633 | 4.0831294374 | 0.23827453971489956 | 0.6716207292 | 0.016394009117171423 | 3.3307862904000003 | 0.24442357796736197 | 0.3474492079 | 0.013999878438761125 | 118197 |
-| 190000 | 1 | 10 | 190000 | 190000 | 2 | 123893 | 123893 | 14728 | 7.0231597877 | 0.09399855550806477 | 2.2149782845 | 0.013126799196048424 | 4.356570219499999 | 0.09521435695870631 | 0.7373797438 | 0.0575174924076721 | 3.5380053205000004 | 0.10378347745460388 | 0.37080809929999997 | 0.01290979756291998 | 123893 |
-| 200000 | 1 | 10 | 200000 | 200000 | 2 | 131252 | 131252 | 16562 | 7.522432313 | 0.2633420321550606 | 2.3628339816 | 0.039989943606690156 | 4.6675009988 | 0.2132461130121612 | 0.7622307122 | 0.009604337545853697 | 3.8178427712 | 0.21696400809832647 | 0.4085155762 | 0.017721784682142964 | 131252 |
-| 210000 | 1 | 10 | 210000 | 210000 | 2 | 137501 | 137501 | 17008 | 7.9078188707 | 0.22101361497446462 | 2.4432533651999995 | 0.015650174443881663 | 4.9470993974 | 0.22727214564348047 | 0.9518358890000002 | 0.015518576165034272 | 3.9050474077000006 | 0.22542975036092622 | 0.44033106270000005 | 0.01727193852812063 | 137501 |
-| 220000 | 1 | 10 | 220000 | 220000 | 2 | 144061 | 144061 | 17847 | 8.4310485219 | 0.2898905581083682 | 2.5987084524 | 0.009502650210868985 | 5.2729674842000005 | 0.27919121144959574 | 0.8455970423 | 0.01824497086380857 | 4.327157211800001 | 0.28910724973041124 | 0.46449658029999996 | 0.01736415985291904 | 144061 |
-| 230000 | 1 | 10 | 230000 | 230000 | 2 | 150310 | 150310 | 17731 | 8.6320747989 | 0.33668340920034806 | 2.6197801442000004 | 0.05256162205418141 | 5.4315238363 | 0.2747458205147567 | 0.9960376444 | 0.028735197336432615 | 4.336221116799999 | 0.25835249305193325 | 0.5015617587 | 0.021268906232720087 | 150310 |
-| 240000 | 1 | 10 | 240000 | 240000 | 2 | 157139 | 157139 | 18612 | 9.009050490400002 | 0.3128766363819569 | 2.7155297847 | 0.023178125088956805 | 5.700408186600001 | 0.2879543985300683 | 1.0367297358000003 | 0.013734694710037488 | 4.5622825022 | 0.2876871470532377 | 0.5279500610000001 | 0.02206413574130229 | 157139 |
-| 250000 | 1 | 10 | 250000 | 250000 | 2 | 164155 | 164155 | 20499 | 9.4270514669 | 0.23225929149256214 | 2.8463226528 | 0.01506100528301562 | 5.951184781199999 | 0.2039882530457369 | 0.9299208275999999 | 0.057306714844376085 | 4.9170371191 | 0.18694913347225156 | 0.5725323586 | 0.012914493862261942 | 164155 |
-| 260000 | 1 | 10 | 260000 | 260000 | 2 | 170681 | 170681 | 20415 | 9.6284610064 | 0.27863865455263176 | 2.9354543054 | 0.011552204060273432 | 6.042820605099999 | 0.26088610696446574 | 1.0819876318000001 | 0.013253391610787106 | 4.8494240756 | 0.2592568608766787 | 0.5946690359 | 0.03364498517483961 | 170681 |
-| 270000 | 1 | 10 | 270000 | 270000 | 2 | 177419 | 177419 | 22189 | 10.3441837675 | 0.34038473596017615 | 3.0790200003 | 0.04722014457879415 | 6.5974961401000005 | 0.30192774188595645 | 1.1574601788999999 | 0.014557294361249015 | 5.3245052121 | 0.30141106044101496 | 0.6368050635000001 | 0.028727929151588963 | 177419 |
-| 280000 | 1 | 10 | 280000 | 280000 | 2 | 183598 | 183598 | 23254 | 10.591835607099998 | 0.35271658898500896 | 3.1467475046 | 0.02026936342148749 | 6.7210618943 | 0.3420285354390056 | 1.0249986697 | 0.005550453719610728 | 5.566262730699999 | 0.3325643977911153 | 0.6547289464999999 | 0.034210139168846056 | 183598 |
-| 290000 | 1 | 10 | 290000 | 290000 | 2 | 190044 | 190044 | 22976 | 10.8875411057 | 0.3231777853754279 | 3.2379748348 | 0.015718810790019754 | 6.9012326078 | 0.3570489487250581 | 1.0157828841 | 0.0067663978067151525 | 5.745269477999999 | 0.34550046535345835 | 0.6994596174 | 0.020991553070787917 | 190044 |
-| 300000 | 1 | 10 | 300000 | 300000 | 2 | 197133 | 197133 | 23844 | 11.6383832293 | 0.32088026618713805 | 3.3229065135 | 0.028968215816625995 | 7.5626338512 | 0.2829614452296093 | 0.9824864115 | 0.13146291683857875 | 6.3761597855 | 0.2835986824686698 | 0.74688904 | 0.030483611878687856 | 197133 |
-
-
-**** How many of those models are minimal
-
-Thorsten asks how many of those models are already minimal, because the thesis
-currently reads "very few".
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "tp (s)" "ta (s)" "ti (s)" "tr (s)" "tv (s)" "Qv")
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000 200000 210000 220000 230000 240000 250000 260000 270000 280000 290000 300000; do
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 1 --indiv
-done
-#+end_src
-
-#+NAME: results1indiv
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | tp (s) | ta (s) | ti (s) | tr (s) | tv (s) | Qv |
-|--------+---+---+--------+--------+---+--------+--------+-------+--------------+-------------+-------------+-------------+-------------+-------------+--------|
-| 100000 | 1 | 0 | 100000 | 100000 | 2 | 65489 | 65489 | 8071 | 3.090418942 | 1.029257521 | 1.861948383 | 0.328673706 | 1.392791488 | 0.134144684 | 65489 |
-| 100000 | 1 | 1 | 100000 | 100000 | 2 | 65912 | 65912 | 8413 | 3.108938053 | 1.010879236 | 1.898772498 | 0.324852464 | 1.448839488 | 0.128740365 | 65912 |
-| 100000 | 1 | 2 | 100000 | 100000 | 2 | 65551 | 65551 | 8087 | 3.226284016 | 1.002811598 | 2.024931788 | 0.332001372 | 1.567582393 | 0.131146099 | 65551 |
-| 100000 | 1 | 3 | 100000 | 100000 | 2 | 65473 | 65473 | 7919 | 3.051887477 | 1.009408854 | 1.845523985 | 0.323038515 | 1.381551998 | 0.135409477 | 65473 |
-| 100000 | 1 | 4 | 100000 | 100000 | 2 | 65668 | 65668 | 8130 | 3.131196133 | 1.004897268 | 1.92738016 | 0.321685576 | 1.482568015 | 0.136392373 | 65668 |
-| 100000 | 1 | 5 | 100000 | 100000 | 2 | 65857 | 65857 | 8264 | 3.138151577 | 1.019564955 | 1.917969114 | 0.330225925 | 1.463897169 | 0.133613369 | 65857 |
-| 100000 | 1 | 6 | 100000 | 100000 | 2 | 65709 | 65709 | 8190 | 3.125807861 | 1.012800151 | 1.914870332 | 0.32505943 | 1.464654751 | 0.134245446 | 65709 |
-| 100000 | 1 | 7 | 100000 | 100000 | 2 | 65777 | 65777 | 8199 | 3.103733509 | 1.006876374 | 1.898138763 | 0.322296575 | 1.452699306 | 0.136855992 | 65777 |
-| 100000 | 1 | 8 | 100000 | 100000 | 2 | 65443 | 65443 | 7867 | 3.089412709 | 1.003632113 | 1.887783014 | 0.331641437 | 1.430431613 | 0.134312071 | 65443 |
-| 100000 | 1 | 9 | 100000 | 100000 | 2 | 65551 | 65551 | 8353 | 3.175007591 | 1.007407203 | 1.966208791 | 0.323649855 | 1.519172145 | 0.133161502 | 65551 |
-| 110000 | 1 | 0 | 110000 | 110000 | 2 | 72513 | 72513 | 9297 | 3.583623638 | 1.115609103 | 2.135438479 | 0.382718716 | 1.709379538 | 0.156874112 | 72513 |
-| 110000 | 1 | 1 | 110000 | 110000 | 2 | 72154 | 72154 | 9232 | 3.627570506 | 1.103570537 | 2.189885346 | 0.36544056 | 1.781507161 | 0.144085569 | 72154 |
-| 110000 | 1 | 2 | 110000 | 110000 | 2 | 71771 | 71771 | 9054 | 3.562237099 | 1.105080032 | 2.123648638 | 0.364379437 | 1.716464571 | 0.154521312 | 71771 |
-| 110000 | 1 | 3 | 110000 | 110000 | 2 | 72036 | 72036 | 9000 | 3.465402602 | 1.104175201 | 2.028329005 | 0.382252627 | 1.582947338 | 0.148894505 | 72036 |
-| 110000 | 1 | 4 | 110000 | 110000 | 2 | 71937 | 71937 | 8951 | 3.558752008 | 1.105804525 | 2.120581865 | 0.37868275 | 1.698635252 | 0.144977146 | 71937 |
-| 110000 | 1 | 5 | 110000 | 110000 | 2 | 71792 | 71792 | 9055 | 3.603305467 | 1.103168135 | 2.161728608 | 0.366633907 | 1.751642064 | 0.153111461 | 71792 |
-| 110000 | 1 | 6 | 110000 | 110000 | 2 | 71770 | 71770 | 8890 | 3.600811378 | 1.109044686 | 2.157865917 | 0.382314179 | 1.732870663 | 0.14781034 | 71770 |
-| 110000 | 1 | 7 | 110000 | 110000 | 2 | 71872 | 71872 | 9105 | 3.526468424 | 1.115787775 | 2.078221198 | 0.38139662 | 1.654133087 | 0.143199962 | 71872 |
-| 110000 | 1 | 8 | 110000 | 110000 | 2 | 71821 | 71821 | 9209 | 3.554558287 | 1.106931016 | 2.101587211 | 0.380784721 | 1.677697086 | 0.156406139 | 71821 |
-| 110000 | 1 | 9 | 110000 | 110000 | 2 | 71886 | 71886 | 8496 | 3.4223644 | 1.113231918 | 1.978985167 | 0.374148003 | 1.540877846 | 0.150616037 | 71886 |
-| 120000 | 1 | 0 | 120000 | 120000 | 2 | 78444 | 78444 | 9790 | 4.044083052 | 1.22089555 | 2.44686854 | 0.424892606 | 1.975084357 | 0.173854233 | 78444 |
-| 120000 | 1 | 1 | 120000 | 120000 | 2 | 79064 | 79064 | 9263 | 3.710149224 | 1.214804905 | 2.097213639 | 0.40627247 | 1.643010328 | 0.16670665 | 79064 |
-| 120000 | 1 | 2 | 120000 | 120000 | 2 | 78524 | 78524 | 9654 | 3.938946788 | 1.217641276 | 2.346316501 | 0.425755958 | 1.87338334 | 0.17600581 | 78524 |
-| 120000 | 1 | 3 | 120000 | 120000 | 2 | 78399 | 78399 | 9588 | 3.971771578 | 1.228776231 | 2.369599418 | 0.407793587 | 1.91415154 | 0.173603882 | 78399 |
-| 120000 | 1 | 4 | 120000 | 120000 | 2 | 78746 | 78746 | 9978 | 3.988636977 | 1.229441895 | 2.375189285 | 0.428107565 | 1.899082552 | 0.169995152 | 78746 |
-| 120000 | 1 | 5 | 120000 | 120000 | 2 | 78615 | 78615 | 9663 | 3.938008292 | 1.232103189 | 2.328706998 | 0.428644184 | 1.852547968 | 0.169057935 | 78615 |
-| 120000 | 1 | 6 | 120000 | 120000 | 2 | 78527 | 78527 | 9645 | 3.970203162 | 1.22990592 | 2.361789463 | 0.428056652 | 1.886277794 | 0.175214768 | 78527 |
-| 120000 | 1 | 7 | 120000 | 120000 | 2 | 78774 | 78774 | 9675 | 3.953384523 | 1.229521193 | 2.346187673 | 0.427421114 | 1.87109108 | 0.174559227 | 78774 |
-| 120000 | 1 | 8 | 120000 | 120000 | 2 | 78247 | 78247 | 9857 | 3.981651613 | 1.233029007 | 2.372190025 | 0.426377975 | 1.898751486 | 0.181812051 | 78247 |
-| 120000 | 1 | 9 | 120000 | 120000 | 2 | 78928 | 78928 | 9693 | 3.829601322 | 1.218223075 | 2.234109856 | 0.424610468 | 1.739492118 | 0.163196799 | 78928 |
-| 130000 | 1 | 0 | 130000 | 130000 | 2 | 85315 | 85315 | 10567 | 4.496630254 | 1.350185864 | 2.725690284 | 0.463947662 | 2.209744665 | 0.197830476 | 85315 |
-| 130000 | 1 | 1 | 130000 | 130000 | 2 | 85142 | 85142 | 10403 | 4.342937024 | 1.337201086 | 2.560209662 | 0.46648121 | 2.041729717 | 0.188478467 | 85142 |
-| 130000 | 1 | 2 | 130000 | 130000 | 2 | 84925 | 84925 | 10388 | 4.340346355 | 1.343248689 | 2.549159688 | 0.470840552 | 2.026643502 | 0.196708577 | 84925 |
-| 130000 | 1 | 3 | 130000 | 130000 | 2 | 85122 | 85122 | 10224 | 4.325072141 | 1.346636564 | 2.532176279 | 0.465016216 | 2.015463288 | 0.192874629 | 85122 |
-| 130000 | 1 | 4 | 130000 | 130000 | 2 | 84972 | 84972 | 10050 | 4.157620772 | 1.345743388 | 2.364910478 | 0.468773206 | 1.844103122 | 0.200949646 | 84972 |
-| 130000 | 1 | 5 | 130000 | 130000 | 2 | 84995 | 84995 | 10729 | 4.445922674 | 1.341322827 | 2.681902811 | 0.468660831 | 2.140002118 | 0.226240593 | 84995 |
-| 130000 | 1 | 6 | 130000 | 130000 | 2 | 85356 | 85356 | 9815 | 4.14208991 | 1.34305509 | 2.354709522 | 0.470155475 | 1.832324281 | 0.196692155 | 85356 |
-| 130000 | 1 | 7 | 130000 | 130000 | 2 | 84961 | 84961 | 10513 | 4.406610962 | 1.328715031 | 2.649823902 | 0.465283683 | 2.108863585 | 0.195631672 | 84961 |
-| 130000 | 1 | 8 | 130000 | 130000 | 2 | 85019 | 85019 | 10795 | 4.34992193 | 1.351010514 | 2.553603859 | 0.468264295 | 2.033701538 | 0.190065701 | 85019 |
-| 130000 | 1 | 9 | 130000 | 130000 | 2 | 85572 | 85572 | 10741 | 4.341899661 | 1.34878337 | 2.545454159 | 0.467400417 | 2.026376139 | 0.197790981 | 85572 |
-| 140000 | 1 | 0 | 140000 | 140000 | 2 | 91961 | 91961 | 11636 | 4.785633961 | 1.429755155 | 2.908650967 | 0.508993845 | 2.344014886 | 0.224317586 | 91961 |
-| 140000 | 1 | 1 | 140000 | 140000 | 2 | 92261 | 92261 | 11446 | 4.869175696 | 1.440850625 | 2.969629031 | 0.490015676 | 2.423266125 | 0.214010397 | 92261 |
-| 140000 | 1 | 2 | 140000 | 140000 | 2 | 91700 | 91700 | 10883 | 4.525186491 | 1.429946253 | 2.623965743 | 0.589262975 | 1.979096417 | 0.21107713 | 91700 |
-| 140000 | 1 | 3 | 140000 | 140000 | 2 | 91529 | 91529 | 10873 | 4.763031452 | 1.43469886 | 2.867981765 | 0.488977323 | 2.322682854 | 0.21543578 | 91529 |
-| 140000 | 1 | 4 | 140000 | 140000 | 2 | 91629 | 91629 | 11305 | 4.786165978 | 1.427493488 | 2.900078154 | 0.57321664 | 2.245201199 | 0.23427863 | 91629 |
-| 140000 | 1 | 5 | 140000 | 140000 | 2 | 91880 | 91880 | 11470 | 4.83428426 | 1.429748273 | 2.944194491 | 0.49067493 | 2.397151544 | 0.220462138 | 91880 |
-| 140000 | 1 | 6 | 140000 | 140000 | 2 | 91604 | 91604 | 11166 | 4.638033401 | 1.43262339 | 2.717577346 | 0.565435414 | 2.095427687 | 0.219351165 | 91604 |
-| 140000 | 1 | 7 | 140000 | 140000 | 2 | 91511 | 91511 | 10132 | 4.494097405 | 1.436518961 | 2.586711282 | 0.612464606 | 1.916895669 | 0.228088975 | 91511 |
-| 140000 | 1 | 8 | 140000 | 140000 | 2 | 92066 | 92066 | 11696 | 5.005829102 | 1.515282813 | 3.040166054 | 0.519872847 | 2.463503272 | 0.221593645 | 92066 |
-| 140000 | 1 | 9 | 140000 | 140000 | 2 | 91879 | 91879 | 11445 | 4.873020022 | 1.432733643 | 2.977947848 | 0.488562877 | 2.432931357 | 0.225284717 | 91879 |
-| 150000 | 1 | 0 | 150000 | 150000 | 2 | 98627 | 98627 | 11706 | 4.87874908 | 1.6648079 | 2.871211159 | 0.557953727 | 2.246941746 | 0.232801971 | 98627 |
-| 150000 | 1 | 1 | 150000 | 150000 | 2 | 98261 | 98261 | 11998 | 5.053498174 | 1.638635224 | 3.073569116 | 0.566411168 | 2.441170162 | 0.260842737 | 98261 |
-| 150000 | 1 | 2 | 150000 | 150000 | 2 | 98235 | 98235 | 12071 | 5.020194853 | 1.652143113 | 3.024943915 | 0.564898091 | 2.394127067 | 0.249948478 | 98235 |
-| 150000 | 1 | 3 | 150000 | 150000 | 2 | 98312 | 98312 | 11705 | 4.993322009 | 1.639803476 | 3.005850198 | 0.559600904 | 2.381452894 | 0.250668565 | 98312 |
-| 150000 | 1 | 4 | 150000 | 150000 | 2 | 98743 | 98743 | 12062 | 5.027550332 | 1.634849775 | 3.048277529 | 0.560757771 | 2.42163756 | 0.244345537 | 98743 |
-| 150000 | 1 | 5 | 150000 | 150000 | 2 | 98223 | 98223 | 12114 | 5.122875915 | 1.643787445 | 3.137857417 | 0.563333104 | 2.508676615 | 0.235187336 | 98223 |
-| 150000 | 1 | 6 | 150000 | 150000 | 2 | 98058 | 98058 | 12112 | 5.029908619 | 1.641798555 | 3.048464151 | 0.566356261 | 2.415904032 | 0.251638182 | 98058 |
-| 150000 | 1 | 7 | 150000 | 150000 | 2 | 98210 | 98210 | 11960 | 5.082807396 | 1.623055534 | 3.118177094 | 0.563246208 | 2.488820486 | 0.244642605 | 98210 |
-| 150000 | 1 | 8 | 150000 | 150000 | 2 | 97960 | 97960 | 11597 | 4.832574492 | 1.634116768 | 2.859206384 | 0.563952597 | 2.228513891 | 0.245505624 | 97960 |
-| 150000 | 1 | 9 | 150000 | 150000 | 2 | 98422 | 98422 | 12311 | 5.254205812 | 1.641570462 | 3.241477264 | 0.561048378 | 2.614893926 | 0.25921745 | 98422 |
-| 160000 | 1 | 0 | 160000 | 160000 | 2 | 104809 | 104809 | 12764 | 5.440070645 | 1.738349582 | 3.29496342 | 0.573287414 | 2.650413678 | 0.272580126 | 104809 |
-| 160000 | 1 | 1 | 160000 | 160000 | 2 | 104736 | 104736 | 12041 | 5.164445291 | 1.740211137 | 3.046503543 | 0.554132548 | 2.420454444 | 0.270962312 | 104736 |
-| 160000 | 1 | 2 | 160000 | 160000 | 2 | 105357 | 105357 | 13124 | 5.419971635 | 1.745238982 | 3.263228991 | 0.556133519 | 2.634783843 | 0.274127138 | 105357 |
-| 160000 | 1 | 3 | 160000 | 160000 | 2 | 105082 | 105082 | 12438 | 5.229333287 | 1.747621002 | 3.107000357 | 0.558363145 | 2.47616425 | 0.263994342 | 105082 |
-| 160000 | 1 | 4 | 160000 | 160000 | 2 | 105268 | 105268 | 12063 | 5.082048681 | 1.74809226 | 2.960825826 | 0.555949584 | 2.331742656 | 0.263435602 | 105268 |
-| 160000 | 1 | 5 | 160000 | 160000 | 2 | 104256 | 104256 | 12487 | 5.422671634 | 1.746827086 | 3.271733078 | 0.557495636 | 2.643176684 | 0.267293552 | 104256 |
-| 160000 | 1 | 6 | 160000 | 160000 | 2 | 104686 | 104686 | 12465 | 5.23858361 | 1.735106293 | 3.08249264 | 0.558101199 | 2.453024637 | 0.271405331 | 104686 |
-| 160000 | 1 | 7 | 160000 | 160000 | 2 | 105070 | 105070 | 12893 | 5.425021324 | 1.74997715 | 3.271989538 | 0.55255752 | 2.647834339 | 0.286615979 | 105070 |
-| 160000 | 1 | 8 | 160000 | 160000 | 2 | 104067 | 104067 | 12612 | 5.466025941 | 1.72077701 | 3.344051429 | 0.552210851 | 2.719644446 | 0.278253136 | 104067 |
-| 160000 | 1 | 9 | 160000 | 160000 | 2 | 104683 | 104683 | 12955 | 5.445543346 | 1.736855611 | 3.302319902 | 0.552927518 | 2.678168574 | 0.27573911 | 104683 |
-| 170000 | 1 | 0 | 170000 | 170000 | 2 | 111232 | 111232 | 14138 | 6.168860346 | 1.794201456 | 3.79426348 | 0.646182461 | 3.050866849 | 0.316425533 | 111232 |
-| 170000 | 1 | 1 | 170000 | 170000 | 2 | 111515 | 111515 | 14069 | 6.291022894 | 1.806047037 | 3.901511142 | 0.649299151 | 3.184940069 | 0.312489109 | 111515 |
-| 170000 | 1 | 2 | 170000 | 170000 | 2 | 111410 | 111410 | 13322 | 6.065010338 | 1.820236643 | 3.625553865 | 0.646420322 | 2.91112181 | 0.286363926 | 111410 |
-| 170000 | 1 | 3 | 170000 | 170000 | 2 | 111530 | 111530 | 13559 | 6.083466882 | 1.79696291 | 3.702714542 | 0.650556016 | 2.953389729 | 0.315408512 | 111530 |
-| 170000 | 1 | 4 | 170000 | 170000 | 2 | 111153 | 111153 | 13532 | 6.034167436 | 1.799491065 | 3.622846207 | 0.629077068 | 2.926017228 | 0.313858897 | 111153 |
-| 170000 | 1 | 5 | 170000 | 170000 | 2 | 111203 | 111203 | 13432 | 6.016587597 | 1.799698199 | 3.602595388 | 0.649027157 | 2.885823847 | 0.29558384 | 111203 |
-| 170000 | 1 | 6 | 170000 | 170000 | 2 | 111830 | 111830 | 13548 | 6.081547831 | 1.803101912 | 3.692775572 | 0.647747248 | 2.947152311 | 0.306056469 | 111830 |
-| 170000 | 1 | 7 | 170000 | 170000 | 2 | 111722 | 111722 | 14225 | 6.118880544 | 1.801466529 | 3.731702997 | 0.650617257 | 2.981227424 | 0.302745422 | 111722 |
-| 170000 | 1 | 8 | 170000 | 170000 | 2 | 111333 | 111333 | 13460 | 5.951473301 | 1.798359163 | 3.536319187 | 0.626822294 | 2.841167646 | 0.293359484 | 111333 |
-| 170000 | 1 | 9 | 170000 | 170000 | 2 | 111104 | 111104 | 14024 | 6.322780834 | 1.803281674 | 3.937282214 | 0.648205194 | 3.221344806 | 0.308468407 | 111104 |
-| 180000 | 1 | 0 | 180000 | 180000 | 2 | 118197 | 118197 | 14269 | 6.399882746 | 1.887676568 | 3.909145115 | 0.688854581 | 3.115447653 | 0.337145292 | 118197 |
-| 180000 | 1 | 1 | 180000 | 180000 | 2 | 117899 | 117899 | 15008 | 6.59193564 | 1.904407536 | 4.08653053 | 0.665274738 | 3.346812896 | 0.371029427 | 117899 |
-| 180000 | 1 | 2 | 180000 | 180000 | 2 | 117666 | 117666 | 13561 | 6.180263714 | 1.885404815 | 3.646186864 | 0.654902445 | 2.917396039 | 0.322655201 | 117666 |
-| 180000 | 1 | 3 | 180000 | 180000 | 2 | 118429 | 118429 | 14730 | 6.683809119 | 1.887848416 | 4.181829556 | 0.65788037 | 3.450755587 | 0.33928549 | 118429 |
-| 180000 | 1 | 4 | 180000 | 180000 | 2 | 117919 | 117919 | 14765 | 6.645367305 | 1.883886051 | 4.160680592 | 0.691621936 | 3.396849828 | 0.330699572 | 117919 |
-| 180000 | 1 | 5 | 180000 | 180000 | 2 | 117831 | 117831 | 14368 | 6.570270358 | 1.899077345 | 4.07657936 | 0.688312904 | 3.316043289 | 0.362467943 | 117831 |
-| 180000 | 1 | 6 | 180000 | 180000 | 2 | 118164 | 118164 | 14146 | 6.253575237 | 1.880088227 | 3.760370092 | 0.662409327 | 2.991572834 | 0.334099838 | 118164 |
-| 180000 | 1 | 7 | 180000 | 180000 | 2 | 118052 | 118052 | 14371 | 6.445557412 | 1.878072219 | 3.955840363 | 0.687813284 | 3.196005552 | 0.347213376 | 118052 |
-| 180000 | 1 | 8 | 180000 | 180000 | 2 | 118236 | 118236 | 14815 | 6.889289486 | 1.907757451 | 4.385984832 | 0.679527108 | 3.633801247 | 0.349703641 | 118236 |
-| 180000 | 1 | 9 | 180000 | 180000 | 2 | 118382 | 118382 | 14925 | 6.736632409 | 1.891914072 | 4.247302756 | 0.68559706 | 3.489265896 | 0.334848585 | 118382 |
-| 190000 | 1 | 0 | 190000 | 190000 | 2 | 123893 | 123893 | 14728 | 6.900986764 | 2.161582434 | 4.286226596 | 0.734762604 | 3.472067038 | 0.353948463 | 123893 |
-| 190000 | 1 | 1 | 190000 | 190000 | 2 | 124576 | 124576 | 15159 | 6.898488004 | 2.174624047 | 4.27912098 | 0.700296369 | 3.498568863 | 0.35830138 | 124576 |
-| 190000 | 1 | 2 | 190000 | 190000 | 2 | 125003 | 125003 | 15028 | 6.710198557 | 2.160130688 | 4.099515534 | 0.730724067 | 3.290501527 | 0.34505851 | 125003 |
-| 190000 | 1 | 3 | 190000 | 190000 | 2 | 124525 | 124525 | 15349 | 7.034451859 | 2.17570743 | 4.4134223 | 0.714822027 | 3.618546002 | 0.351501335 | 124525 |
-| 190000 | 1 | 4 | 190000 | 190000 | 2 | 124747 | 124747 | 15192 | 7.002408049 | 2.172273501 | 4.385477998 | 0.716428478 | 3.590030986 | 0.349493967 | 124747 |
-| 190000 | 1 | 5 | 190000 | 190000 | 2 | 124560 | 124560 | 15457 | 7.05067543 | 2.157656357 | 4.436947595 | 0.847961622 | 3.506773273 | 0.371769263 | 124560 |
-| 190000 | 1 | 6 | 190000 | 190000 | 2 | 124538 | 124538 | 15810 | 6.994375326 | 2.178560202 | 4.37305179 | 0.833612472 | 3.459327275 | 0.374734634 | 124538 |
-| 190000 | 1 | 7 | 190000 | 190000 | 2 | 124769 | 124769 | 15347 | 6.859897755 | 2.172018642 | 4.242157485 | 0.715965149 | 3.445706969 | 0.385739932 | 124769 |
-| 190000 | 1 | 8 | 190000 | 190000 | 2 | 124494 | 124494 | 15160 | 6.925273713 | 2.15971988 | 4.322267457 | 0.716420296 | 3.526599735 | 0.384269638 | 124494 |
-| 190000 | 1 | 9 | 190000 | 190000 | 2 | 124557 | 124557 | 15285 | 6.897091465 | 2.18135559 | 4.271128314 | 0.718135681 | 3.472719135 | 0.35732001 | 124557 |
-| 200000 | 1 | 0 | 200000 | 200000 | 2 | 131252 | 131252 | 16562 | 7.501766674 | 2.290347737 | 4.733828831 | 0.777096111 | 3.87381184 | 0.424507374 | 131252 |
-| 200000 | 1 | 1 | 200000 | 200000 | 2 | 130414 | 130414 | 15929 | 7.314470042 | 2.300023248 | 4.546142615 | 0.74427866 | 3.719243714 | 0.428758608 | 130414 |
-| 200000 | 1 | 2 | 200000 | 200000 | 2 | 130472 | 130472 | 15645 | 7.532480312 | 2.412615713 | 4.6471952 | 0.793080252 | 3.769570721 | 0.411414107 | 130472 |
-| 200000 | 1 | 3 | 200000 | 200000 | 2 | 131201 | 131201 | 16704 | 7.921693365 | 2.28837716 | 5.120909431 | 0.763193167 | 4.274570419 | 0.379625197 | 131201 |
-| 200000 | 1 | 4 | 200000 | 200000 | 2 | 130338 | 130338 | 16019 | 7.333021768 | 2.302058129 | 4.547763537 | 0.777164588 | 3.689658429 | 0.401041409 | 130338 |
-| 200000 | 1 | 5 | 200000 | 200000 | 2 | 131308 | 131308 | 16144 | 7.461368238 | 2.329180847 | 4.645205799 | 0.777043676 | 3.785984536 | 0.434666287 | 131308 |
-| 200000 | 1 | 6 | 200000 | 200000 | 2 | 130142 | 130142 | 15625 | 7.263421771 | 2.297130478 | 4.488145182 | 0.775779486 | 3.630773898 | 0.385860234 | 130142 |
-| 200000 | 1 | 7 | 200000 | 200000 | 2 | 130897 | 130897 | 15694 | 7.089368287 | 2.294700687 | 4.323368884 | 0.761099811 | 3.442399339 | 0.402031887 | 130897 |
-| 200000 | 1 | 8 | 200000 | 200000 | 2 | 131111 | 131111 | 15630 | 7.153170481 | 2.296633907 | 4.375205575 | 0.758395281 | 3.536094842 | 0.406151482 | 131111 |
-| 200000 | 1 | 9 | 200000 | 200000 | 2 | 131658 | 131658 | 15813 | 7.345773525 | 2.290669569 | 4.585029803 | 0.761367922 | 3.740556123 | 0.384697184 | 131658 |
-| 210000 | 1 | 0 | 210000 | 210000 | 2 | 137501 | 137501 | 17008 | 8.041410976 | 2.403339336 | 5.123146016 | 0.942056976 | 4.091394929 | 0.409698156 | 137501 |
-| 210000 | 1 | 1 | 210000 | 210000 | 2 | 137135 | 137135 | 15957 | 7.741675231 | 2.396214712 | 4.837474255 | 0.943700366 | 3.803742428 | 0.421952086 | 137135 |
-| 210000 | 1 | 2 | 210000 | 210000 | 2 | 137287 | 137287 | 15418 | 7.212113427 | 2.377236587 | 4.291066239 | 0.940134423 | 3.261723983 | 0.424028043 | 137287 |
-| 210000 | 1 | 3 | 210000 | 210000 | 2 | 137678 | 137678 | 16996 | 7.852423325 | 2.400356141 | 4.948782914 | 0.944696767 | 3.914108096 | 0.435767402 | 137678 |
-| 210000 | 1 | 4 | 210000 | 210000 | 2 | 137689 | 137689 | 16467 | 7.706523853 | 2.408438126 | 4.788697876 | 0.948561505 | 3.749344946 | 0.414052643 | 137689 |
-| 210000 | 1 | 5 | 210000 | 210000 | 2 | 137048 | 137048 | 16582 | 7.944378905 | 2.402098191 | 5.03439182 | 0.951142524 | 3.995750467 | 0.426074004 | 137048 |
-| 210000 | 1 | 6 | 210000 | 210000 | 2 | 137675 | 137675 | 17013 | 7.872924876 | 2.393076388 | 4.97605133 | 0.947243336 | 3.939171511 | 0.456742565 | 137675 |
-| 210000 | 1 | 7 | 210000 | 210000 | 2 | 137782 | 137782 | 17222 | 7.851843478 | 2.386854521 | 4.953363162 | 0.942257674 | 3.922594883 | 0.472598536 | 137782 |
-| 210000 | 1 | 8 | 210000 | 210000 | 2 | 137457 | 137457 | 17269 | 7.941092926 | 2.400592639 | 5.019741544 | 0.96243323 | 3.969171412 | 0.456810118 | 137457 |
-| 210000 | 1 | 9 | 210000 | 210000 | 2 | 137666 | 137666 | 16679 | 7.65857879 | 2.381422353 | 4.777175065 | 0.940556494 | 3.748671504 | 0.421552087 | 137666 |
-| 220000 | 1 | 0 | 220000 | 220000 | 2 | 144061 | 144061 | 17847 | 8.669095907 | 2.545385024 | 5.537661442 | 0.861795041 | 4.581711535 | 0.434233353 | 144061 |
-| 220000 | 1 | 1 | 220000 | 220000 | 2 | 143962 | 143962 | 17924 | 8.318288059 | 2.512176282 | 5.279486794 | 0.841207079 | 4.343627504 | 0.478695353 | 143962 |
-| 220000 | 1 | 2 | 220000 | 220000 | 2 | 144022 | 144022 | 17751 | 8.578085453 | 2.518105343 | 5.476536271 | 0.858895534 | 4.525636131 | 0.453601706 | 144022 |
-| 220000 | 1 | 3 | 220000 | 220000 | 2 | 143831 | 143831 | 16007 | 7.698838517 | 2.524429064 | 4.629894943 | 0.858350426 | 3.63611909 | 0.457531583 | 143831 |
-| 220000 | 1 | 4 | 220000 | 220000 | 2 | 144524 | 144524 | 17382 | 8.273610384 | 2.527772666 | 5.206568723 | 0.858430052 | 4.254333768 | 0.446028672 | 144524 |
-| 220000 | 1 | 5 | 220000 | 220000 | 2 | 144219 | 144219 | 16792 | 8.064943676 | 2.535638022 | 4.988759882 | 0.864193856 | 4.030824969 | 0.465423993 | 144219 |
-| 220000 | 1 | 6 | 220000 | 220000 | 2 | 144053 | 144053 | 17449 | 8.342951384 | 2.528450286 | 5.273498833 | 0.863906134 | 4.315790117 | 0.458714232 | 144053 |
-| 220000 | 1 | 7 | 220000 | 220000 | 2 | 143664 | 143664 | 17853 | 8.604525287 | 2.54432029 | 5.478018839 | 0.863750754 | 4.519996727 | 0.489311152 | 143664 |
-| 220000 | 1 | 8 | 220000 | 220000 | 2 | 144603 | 144603 | 17247 | 8.038770401 | 2.531384236 | 4.969336395 | 0.859721024 | 4.016490586 | 0.438631087 | 144603 |
-| 220000 | 1 | 9 | 220000 | 220000 | 2 | 144014 | 144014 | 17534 | 8.253180489 | 2.533094252 | 5.189130014 | 0.845358612 | 4.248644903 | 0.450726977 | 144014 |
-| 230000 | 1 | 0 | 230000 | 230000 | 2 | 150310 | 150310 | 17731 | 8.310385661 | 2.627288746 | 5.126576882 | 1.012838289 | 4.012415445 | 0.472023176 | 150310 |
-| 230000 | 1 | 1 | 230000 | 230000 | 2 | 150614 | 150614 | 18286 | 8.850914286 | 2.653181854 | 5.555712184 | 1.032394359 | 4.424215084 | 0.534918245 | 150614 |
-| 230000 | 1 | 2 | 230000 | 230000 | 2 | 150045 | 150045 | 18470 | 8.901776764 | 2.651087917 | 5.642981332 | 1.018398605 | 4.523348677 | 0.480107241 | 150045 |
-| 230000 | 1 | 3 | 230000 | 230000 | 2 | 151151 | 151151 | 19032 | 9.161537892 | 2.624767484 | 5.925766944 | 1.026628763 | 4.79818448 | 0.492935426 | 151151 |
-| 230000 | 1 | 4 | 230000 | 230000 | 2 | 150433 | 150433 | 19473 | 8.893856489 | 2.639158704 | 5.644482115 | 1.021033059 | 4.522405807 | 0.519523569 | 150433 |
-| 230000 | 1 | 5 | 230000 | 230000 | 2 | 151042 | 151042 | 19176 | 8.874304794 | 2.615782409 | 5.642420608 | 1.027906535 | 4.515114968 | 0.499868164 | 151042 |
-| 230000 | 1 | 6 | 230000 | 230000 | 2 | 150671 | 150671 | 17738 | 8.430467737 | 2.616624791 | 5.258739268 | 1.017196006 | 4.139772202 | 0.513665986 | 150671 |
-| 230000 | 1 | 7 | 230000 | 230000 | 2 | 150828 | 150828 | 18530 | 8.680881471 | 2.624730177 | 5.495091539 | 1.055239464 | 4.339090618 | 0.525135532 | 150828 |
-| 230000 | 1 | 8 | 230000 | 230000 | 2 | 150549 | 150549 | 18563 | 8.976734652 | 2.6574307 | 5.692591009 | 1.028878404 | 4.563868735 | 0.498219728 | 150549 |
-| 230000 | 1 | 9 | 230000 | 230000 | 2 | 151016 | 151016 | 18035 | 8.559480359 | 2.616094984 | 5.383410728 | 1.019994484 | 4.261937361 | 0.5079374 | 151016 |
-| 240000 | 1 | 0 | 240000 | 240000 | 2 | 157139 | 157139 | 18612 | 8.815972912 | 2.74773998 | 5.48311889 | 1.087140152 | 4.292111659 | 0.57189365 | 157139 |
-| 240000 | 1 | 1 | 240000 | 240000 | 2 | 157608 | 157608 | 19252 | 9.157400044 | 2.771516517 | 5.792792991 | 1.104238642 | 4.585645223 | 0.540382115 | 157608 |
-| 240000 | 1 | 2 | 240000 | 240000 | 2 | 157646 | 157646 | 20024 | 9.781766985 | 2.770280889 | 6.365296777 | 1.103147254 | 5.156231439 | 0.531953967 | 157646 |
-| 240000 | 1 | 3 | 240000 | 240000 | 2 | 156622 | 156622 | 19369 | 9.367733297 | 2.758797231 | 6.017452665 | 1.094654537 | 4.817963485 | 0.549178664 | 156622 |
-| 240000 | 1 | 4 | 240000 | 240000 | 2 | 157345 | 157345 | 19699 | 9.758490449 | 2.770374206 | 6.338868771 | 1.120063219 | 5.115686793 | 0.522267916 | 157345 |
-| 240000 | 1 | 5 | 240000 | 240000 | 2 | 157221 | 157221 | 19449 | 9.15696264 | 2.770948598 | 5.783570148 | 1.110696326 | 4.569368463 | 0.553962177 | 157221 |
-| 240000 | 1 | 6 | 240000 | 240000 | 2 | 156929 | 156929 | 19481 | 9.33711475 | 2.758312879 | 5.992392294 | 1.085176846 | 4.802884048 | 0.561701544 | 156929 |
-| 240000 | 1 | 7 | 240000 | 240000 | 2 | 157431 | 157431 | 19737 | 9.480237235 | 2.740959893 | 6.092215797 | 1.084837778 | 4.902966712 | 0.502511967 | 157431 |
-| 240000 | 1 | 8 | 240000 | 240000 | 2 | 157398 | 157398 | 19422 | 9.343251503 | 2.725190387 | 6.000660595 | 1.073390483 | 4.825764661 | 0.532866407 | 157398 |
-| 240000 | 1 | 9 | 240000 | 240000 | 2 | 157057 | 157057 | 18490 | 8.582717593 | 2.703775099 | 5.302892817 | 1.056925038 | 4.144808802 | 0.536938491 | 157057 |
-| 250000 | 1 | 0 | 250000 | 250000 | 2 | 164155 | 164155 | 20499 | 9.60557815 | 2.822196904 | 6.136753035 | 0.934586401 | 5.096747938 | 0.570337366 | 164155 |
-| 250000 | 1 | 1 | 250000 | 250000 | 2 | 163799 | 163799 | 20090 | 9.765207872 | 2.822791811 | 6.28078652 | 0.9555997 | 5.221866039 | 0.5819578 | 163799 |
-| 250000 | 1 | 2 | 250000 | 250000 | 2 | 163795 | 163795 | 19435 | 9.122273726 | 2.817267273 | 5.690582319 | 0.955516537 | 4.630858254 | 0.574494614 | 163795 |
-| 250000 | 1 | 3 | 250000 | 250000 | 2 | 163882 | 163882 | 20565 | 9.733671852 | 2.83933094 | 6.230632543 | 1.104118097 | 5.022094244 | 0.546038473 | 163882 |
-| 250000 | 1 | 4 | 250000 | 250000 | 2 | 163488 | 163488 | 19736 | 9.42577444 | 2.827970226 | 6.003941266 | 0.950475659 | 4.946863585 | 0.579136603 | 163488 |
-| 250000 | 1 | 5 | 250000 | 250000 | 2 | 163040 | 163040 | 20113 | 9.585754001 | 2.825224281 | 6.101184877 | 0.949185342 | 5.048083201 | 0.573078765 | 163040 |
-| 250000 | 1 | 6 | 250000 | 250000 | 2 | 164270 | 164270 | 19386 | 9.19282763 | 2.840616946 | 5.751106153 | 0.942298045 | 4.702955477 | 0.572438797 | 164270 |
-| 250000 | 1 | 7 | 250000 | 250000 | 2 | 163551 | 163551 | 19810 | 9.267103056 | 2.833960096 | 5.830650066 | 0.948962675 | 4.778918365 | 0.549960572 | 163551 |
-| 250000 | 1 | 8 | 250000 | 250000 | 2 | 163657 | 163657 | 20139 | 9.437199933 | 2.807412327 | 5.96740992 | 0.929643275 | 4.933262311 | 0.58148418 | 163657 |
-| 250000 | 1 | 9 | 250000 | 250000 | 2 | 164412 | 164412 | 20005 | 9.573767442 | 2.835445074 | 6.092619835 | 0.929378761 | 5.057647231 | 0.571809639 | 164412 |
-| 260000 | 1 | 0 | 260000 | 260000 | 2 | 170681 | 170681 | 20415 | 9.4548144 | 2.930642649 | 5.889017916 | 1.105724873 | 4.671796122 | 0.550283763 | 170681 |
-| 260000 | 1 | 1 | 260000 | 260000 | 2 | 169673 | 169673 | 20584 | 9.590741172 | 2.915821966 | 6.051150871 | 1.078916389 | 4.858871158 | 0.606429393 | 169673 |
-| 260000 | 1 | 2 | 260000 | 260000 | 2 | 169714 | 169714 | 20638 | 9.866056656 | 2.910540007 | 6.279885217 | 1.092830522 | 5.074247433 | 0.617775182 | 169714 |
-| 260000 | 1 | 3 | 260000 | 260000 | 2 | 170275 | 170275 | 20738 | 9.97812243 | 2.915090305 | 6.389943714 | 1.094404504 | 5.182674507 | 0.55613254 | 170275 |
-| 260000 | 1 | 4 | 260000 | 260000 | 2 | 170569 | 170569 | 20990 | 9.789765173 | 2.911049224 | 6.195948999 | 1.086773541 | 4.996391502 | 0.641643796 | 170569 |
-| 260000 | 1 | 5 | 260000 | 260000 | 2 | 170612 | 170612 | 19707 | 9.111092709 | 2.911439929 | 5.577325876 | 1.076905109 | 4.388617693 | 0.562255785 | 170612 |
-| 260000 | 1 | 6 | 260000 | 260000 | 2 | 170532 | 170532 | 21402 | 9.908742138 | 2.90222627 | 6.324748882 | 1.102120351 | 5.112509739 | 0.597304417 | 170532 |
-| 260000 | 1 | 7 | 260000 | 260000 | 2 | 170002 | 170002 | 19791 | 9.437899396 | 2.910546682 | 5.894809996 | 1.101619829 | 4.681988044 | 0.626214093 | 170002 |
-| 260000 | 1 | 8 | 260000 | 260000 | 2 | 170169 | 170169 | 20431 | 9.857592193 | 2.906946936 | 6.27132285 | 1.09868213 | 5.061145086 | 0.586025964 | 170169 |
-| 260000 | 1 | 9 | 260000 | 260000 | 2 | 170493 | 170493 | 20194 | 9.385833214 | 2.908785965 | 5.846454599 | 1.10610514 | 4.629300513 | 0.581113361 | 170493 |
-| 270000 | 1 | 0 | 270000 | 270000 | 2 | 177419 | 177419 | 22189 | 10.689381872 | 3.080717335 | 6.89093731 | 1.194186556 | 5.580800268 | 0.59785875 | 177419 |
-| 270000 | 1 | 1 | 270000 | 270000 | 2 | 176846 | 176846 | 21447 | 10.299967542 | 3.019579518 | 6.634314668 | 1.167149517 | 5.352308427 | 0.644087616 | 176846 |
-| 270000 | 1 | 2 | 270000 | 270000 | 2 | 177200 | 177200 | 21096 | 9.852972777 | 3.037923152 | 6.166210003 | 1.173188819 | 4.876581258 | 0.592539765 | 177200 |
-| 270000 | 1 | 3 | 270000 | 270000 | 2 | 176612 | 176612 | 21777 | 10.282605222 | 3.041046279 | 6.591089515 | 1.14796408 | 5.327337908 | 0.612531435 | 176612 |
-| 270000 | 1 | 4 | 270000 | 270000 | 2 | 177296 | 177296 | 20837 | 9.776640558 | 3.038335321 | 6.087881823 | 1.17535362 | 4.795954983 | 0.638223713 | 177296 |
-| 270000 | 1 | 5 | 270000 | 270000 | 2 | 176611 | 176611 | 21444 | 10.500645576 | 3.053852266 | 6.740246334 | 1.146873701 | 5.477494592 | 0.63675463 | 176611 |
-| 270000 | 1 | 6 | 270000 | 270000 | 2 | 176888 | 176888 | 22016 | 10.538372465 | 3.070766221 | 6.815931971 | 1.210888556 | 5.481491521 | 0.640227132 | 176888 |
-| 270000 | 1 | 7 | 270000 | 270000 | 2 | 176571 | 176571 | 21891 | 10.765028727 | 3.033038125 | 7.032778841 | 1.189691099 | 5.726028081 | 0.628098181 | 176571 |
-| 270000 | 1 | 8 | 270000 | 270000 | 2 | 176523 | 176523 | 22648 | 10.795500125 | 3.035613661 | 7.061060684 | 1.165833564 | 5.780602599 | 0.671489039 | 176523 |
-| 270000 | 1 | 9 | 270000 | 270000 | 2 | 176549 | 176549 | 21041 | 10.198250889 | 3.030090479 | 6.522306993 | 1.16941795 | 5.238104309 | 0.651520711 | 176549 |
-| 280000 | 1 | 0 | 280000 | 280000 | 2 | 183598 | 183598 | 23254 | 10.707345637 | 3.12594362 | 6.846144329 | 1.042242054 | 5.676629961 | 0.678939611 | 183598 |
-| 280000 | 1 | 1 | 280000 | 280000 | 2 | 183411 | 183411 | 22220 | 10.565084738 | 3.109234517 | 6.717684789 | 1.029231563 | 5.564469962 | 0.693535243 | 183411 |
-| 280000 | 1 | 2 | 280000 | 280000 | 2 | 184056 | 184056 | 23451 | 11.091024335 | 3.096977688 | 7.312582168 | 1.037470722 | 6.102368581 | 0.613608554 | 184056 |
-| 280000 | 1 | 3 | 280000 | 280000 | 2 | 183148 | 183148 | 22543 | 10.541255626 | 3.101419148 | 6.695550284 | 1.020266867 | 5.549581829 | 0.688000785 | 183148 |
-| 280000 | 1 | 4 | 280000 | 280000 | 2 | 183020 | 183020 | 22790 | 10.563007126 | 3.103572056 | 6.725438094 | 1.02641916 | 5.572920965 | 0.659520079 | 183020 |
-| 280000 | 1 | 5 | 280000 | 280000 | 2 | 183175 | 183175 | 22252 | 10.74211594 | 3.127961086 | 6.869822513 | 1.032502365 | 5.711579782 | 0.61829214 | 183175 |
-| 280000 | 1 | 6 | 280000 | 280000 | 2 | 183674 | 183674 | 22445 | 10.846812933 | 3.130509433 | 6.966774772 | 1.03071281 | 5.813079371 | 0.643883661 | 183674 |
-| 280000 | 1 | 7 | 280000 | 280000 | 2 | 183458 | 183458 | 22356 | 10.558548553 | 3.099913925 | 6.726646702 | 1.035148717 | 5.565872566 | 0.614459156 | 183458 |
-| 280000 | 1 | 8 | 280000 | 280000 | 2 | 183022 | 183022 | 20824 | 9.800448926 | 3.122052285 | 5.98520867 | 1.040402131 | 4.820086203 | 0.626547431 | 183022 |
-| 280000 | 1 | 9 | 280000 | 280000 | 2 | 183171 | 183171 | 22565 | 10.545346725 | 3.131769483 | 6.66417774 | 1.041565284 | 5.497697244 | 0.68598763 | 183171 |
-| 290000 | 1 | 0 | 290000 | 290000 | 2 | 190044 | 190044 | 22976 | 10.92077822 | 3.214080215 | 6.934844505 | 1.014736808 | 5.790384 | 0.691925568 | 190044 |
-| 290000 | 1 | 1 | 290000 | 290000 | 2 | 189806 | 189806 | 23292 | 10.843386214 | 3.215425947 | 6.852131514 | 1.021619776 | 5.697462828 | 0.703619281 | 189806 |
-| 290000 | 1 | 2 | 290000 | 290000 | 2 | 190197 | 190197 | 22354 | 10.397205167 | 3.226330292 | 6.393833166 | 1.012785386 | 5.251509419 | 0.733568292 | 190197 |
-| 290000 | 1 | 3 | 290000 | 290000 | 2 | 190324 | 190324 | 23564 | 11.441253374 | 3.220414777 | 7.508093618 | 1.027947969 | 6.347256823 | 0.664929487 | 190324 |
-| 290000 | 1 | 4 | 290000 | 290000 | 2 | 190272 | 190272 | 22463 | 10.536742866 | 3.218774317 | 6.551271321 | 1.028759609 | 5.390552493 | 0.700022212 | 190272 |
-| 290000 | 1 | 5 | 290000 | 290000 | 2 | 190136 | 190136 | 23381 | 11.236277163 | 3.206357594 | 7.31621001 | 1.024936209 | 6.111820107 | 0.691577028 | 190136 |
-| 290000 | 1 | 6 | 290000 | 290000 | 2 | 190678 | 190678 | 23072 | 10.9148769 | 3.223648571 | 6.924738285 | 1.030530888 | 5.761589518 | 0.711657323 | 190678 |
-| 290000 | 1 | 7 | 290000 | 290000 | 2 | 190946 | 190946 | 22889 | 10.56511376 | 3.227557879 | 6.560143815 | 1.021750427 | 5.407384 | 0.690981759 | 190946 |
-| 290000 | 1 | 8 | 290000 | 290000 | 2 | 190053 | 190053 | 23433 | 10.908398565 | 3.209551958 | 6.929929038 | 1.01080225 | 5.787722801 | 0.707466444 | 190053 |
-| 290000 | 1 | 9 | 290000 | 290000 | 2 | 189700 | 189700 | 23564 | 11.167754785 | 3.198865208 | 7.264370827 | 1.024548376 | 6.059374361 | 0.658562025 | 189700 |
-| 300000 | 1 | 0 | 300000 | 300000 | 2 | 197133 | 197133 | 23844 | 11.290930841 | 3.292479571 | 7.254842007 | 0.739282109 | 6.051628224 | 0.684718157 | 197133 |
-| 300000 | 1 | 1 | 300000 | 300000 | 2 | 196650 | 196650 | 24747 | 11.747530758 | 3.266119091 | 7.742190827 | 1.050865719 | 6.551144967 | 0.728950319 | 196650 |
-| 300000 | 1 | 2 | 300000 | 300000 | 2 | 196403 | 196403 | 24759 | 12.062785334 | 3.295742322 | 7.946142871 | 0.738346077 | 6.751236901 | 0.74227076 | 196403 |
-| 300000 | 1 | 3 | 300000 | 300000 | 2 | 196270 | 196270 | 24044 | 11.276732836 | 3.303501308 | 7.234669539 | 1.049454402 | 6.043991894 | 0.74733691 | 196270 |
-| 300000 | 1 | 4 | 300000 | 300000 | 2 | 196354 | 196354 | 24166 | 11.519037176 | 3.292506598 | 7.472094511 | 1.049057767 | 6.282284293 | 0.781757696 | 196354 |
-| 300000 | 1 | 5 | 300000 | 300000 | 2 | 196996 | 196996 | 24758 | 11.517524274 | 3.285781745 | 7.486309733 | 1.05492982 | 6.289337414 | 0.76825964 | 196996 |
-| 300000 | 1 | 6 | 300000 | 300000 | 2 | 197385 | 197385 | 24911 | 12.271921845 | 3.408291823 | 8.101506133 | 1.096258241 | 6.864540233 | 0.727420201 | 197385 |
-| 300000 | 1 | 7 | 300000 | 300000 | 2 | 196983 | 196983 | 24370 | 11.700670367 | 3.284961744 | 7.659590979 | 1.046525693 | 6.472569943 | 0.710060576 | 196983 |
-| 300000 | 1 | 8 | 300000 | 300000 | 2 | 196764 | 196764 | 23885 | 11.355408019 | 3.28628696 | 7.327721993 | 1.052316196 | 6.135234582 | 0.773097422 | 196764 |
-| 300000 | 1 | 9 | 300000 | 300000 | 2 | 196322 | 196322 | 24157 | 11.656118582 | 3.275123115 | 7.642015947 | 1.04603912 | 6.454546005 | 0.724228232 | 196322 |
-
-#+begin_src python :var data=results1indiv
-return (all(x[3] != x[6] for x in data), data[0][3], data[0][6])
-#+end_src
-
-#+RESULTS:
-| True | 100000 | 65489 |
-
-=> /no/ model was minimal
-
-*** And plotting
-
-#+CALL: plot(results1, "results1.png")
-
-#+RESULTS:
-[[file:results1.png]]
-
-** For m = 2
-*** Generating some benchmarks
-
-#+begin_src sh :results output silent
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000; do
- python bench.py generate ../../copar/bin/random-dfa $n 2
-done
-#+end_src
-
-*** And running them
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "σ" "tp (s)" "σ" "ta (s)" "σ" "ti (s)" "σ" "tr (s)" "σ" "tv (s)" "σ" "Qv")
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000; do
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 2 --stddev
-done
-#+end_src
-
-#+NAME: results2
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | σ | tp (s) | σ | ta (s) | σ | ti (s) | σ | tr (s) | σ | tv (s) | σ | Qv |
-|--------+---+----+--------+--------+---+--------+--------+--------+--------------------+----------------------+--------------------+----------------------+--------------------+----------------------+---------------------+----------------------+--------------------+---------------------+---------------------+----------------------+--------|
-| 100000 | 2 | 10 | 100000 | 200000 | 2 | 100000 | 100000 | 176835 | 3.4030988053 | 0.047125905819625266 | 1.4113720969999999 | 0.020157652579552755 | 1.7480437949999998 | 0.030617105080139137 | 0.4651039364 | 0.004331701728800437 | 1.2271866079 | 0.02960202148844354 | 0.2616005265 | 0.004401878111343841 | 100000 |
-| 110000 | 2 | 10 | 110000 | 220000 | 2 | 110000 | 110000 | 198390 | 3.9692688889000003 | 0.0370434112058736 | 1.5157029322 | 0.02116665725333869 | 2.0486332889999996 | 0.040191720781762134 | 0.4946425016 | 0.013523587264140786 | 1.4991156887000003 | 0.03442819266484039 | 0.30168761980000003 | 0.007153435060528811 | 110000 |
-| 120000 | 2 | 10 | 120000 | 240000 | 2 | 119999 | 119999 | 211802 | 4.3143812377 | 0.028623553516576524 | 1.7607175563 | 0.01395808199964705 | 2.2709801406 | 0.019540291251429913 | 0.6559281311999998 | 0.016807148943218927 | 1.5497294746999999 | 0.0211686298572404 | 0.3521036861 | 0.010123212916930498 | 119999 |
-| 130000 | 2 | 10 | 130000 | 260000 | 2 | 130000 | 130000 | 233351 | 4.711098789299999 | 0.07164931362575287 | 1.8374207242 | 0.015974690038318053 | 2.5580884159 | 0.07464571058990088 | 0.45663371779999995 | 0.06532202237008912 | 1.8270874564 | 0.05325031657468783 | 0.3873578492 | 0.00808873090733974 | 130000 |
-| 140000 | 2 | 10 | 140000 | 280000 | 2 | 140000 | 140000 | 255450 | 5.126386760700001 | 0.07043815136254557 | 1.9956747868 | 0.018427064561353488 | 2.7867161994 | 0.073077457235916 | 0.49504377580000003 | 0.0638811650849501 | 1.9893636328 | 0.06274443335469422 | 0.4376762827999999 | 0.0092923256658709 | 140000 |
-| 150000 | 2 | 10 | 150000 | 300000 | 2 | 149999 | 149999 | 268773 | 5.552826151500001 | 0.06095217378481222 | 2.1609233121 | 0.026029046999351532 | 3.0173265741 | 0.05415723778561011 | 0.5094614781 | 0.007915398330318239 | 2.1829203786999996 | 0.0632789354781126 | 0.48105161770000004 | 0.004698776514825223 | 149999 |
-| 160000 | 2 | 10 | 160000 | 320000 | 2 | 160000 | 160000 | 287497 | 6.0156567696000005 | 0.1014958200025798 | 2.3278508021 | 0.030676236665497392 | 3.2862890981 | 0.08842134639178673 | 0.5667389131 | 0.061818697661962456 | 2.3874675003 | 0.06963184602365943 | 0.530804924 | 0.008374295955608495 | 160000 |
-| 170000 | 2 | 10 | 170000 | 340000 | 2 | 170000 | 170000 | 305026 | 6.4638692547000005 | 0.06069976545804155 | 2.4802088854 | 0.01090849282529911 | 3.5534037927 | 0.05792099698647397 | 0.5959597545000002 | 0.008029045672763618 | 2.5980215775 | 0.05470590971316728 | 0.5807772137 | 0.008923135110179324 | 170000 |
-| 180000 | 2 | 10 | 180000 | 360000 | 2 | 180000 | 180000 | 318552 | 7.0769255199 | 0.06648812882039208 | 2.6026580344 | 0.021088489126152892 | 3.8877335147000003 | 0.13692635138399228 | 0.8762553760999999 | 0.019006836766069967 | 2.8003821182 | 0.06175237500886775 | 0.6318085971 | 0.008747586811742026 | 180000 |
-| 190000 | 2 | 10 | 190000 | 380000 | 2 | 190000 | 190000 | 336100 | 7.471944982100001 | 0.1193043264263505 | 2.7358027763 | 0.030596172663756225 | 4.172442589299999 | 0.14609929644360664 | 0.9283796991 | 0.021647195894253953 | 2.9708602129999995 | 0.09071021481473823 | 0.6796285844 | 0.014169458167688257 | 190000 |
-
-*** Plot
-
-#+CALL: plot(results2, "results2.png")
-
-#+RESULTS:
-[[file:results2.png]]
-
-** For m = 2 and 10% finality
-*** Generating some benchmarks
-
-#+begin_src sh :results output silent
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000; do
- python bench.py generate --final 10 ../../copar/bin/random-dfa $n 2
-done
-#+end_src
-
-*** And running them
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "sdt" "tp (s)" "sdtp" "ta (s)" "sdta" "ti (s)" "sdti" "tr (s)" "sdtr" "tv (s)" "sdtv" "Qv")
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000; do
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 2 --stddev --final 10
-done
-#+end_src
-
-#+NAME: results2f
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | sdt | tp (s) | sdtp | ta (s) | sdta | ti (s) | sdti | tr (s) | sdtr | tv (s) | sdtv | Qv |
-|--------+---+----+--------+--------+---+--------+--------+--------+--------------------+----------------------+--------------------+----------------------+--------------------+----------------------+---------------------+----------------------+--------------------+----------------------+---------------------+-----------------------+--------|
-| 100000 | 2 | 10 | 100000 | 200000 | 2 | 100000 | 100000 | 160352 | 3.0936953513 | 0.015467818313428423 | 1.4124696133 | 0.005943672222068957 | 1.4574320088000001 | 0.017000822749439697 | 0.42203191440000004 | 0.004520660260577265 | 0.9797939001000001 | 0.018105061634433976 | 0.2695312297 | 0.0069361300204224706 | 100000 |
-| 110000 | 2 | 10 | 110000 | 220000 | 2 | 109999 | 109999 | 176657 | 3.5784416617 | 0.01508095774483378 | 1.5098241521 | 0.004227983506536663 | 1.6316288086 | 0.012239722621501683 | 0.4385282887 | 0.006495824492836184 | 1.1376198136 | 0.014767401631046848 | 0.31170653470000004 | 0.006848544046025738 | 109999 |
-| 120000 | 2 | 10 | 120000 | 240000 | 2 | 119999 | 119999 | 191218 | 3.9010189149000007 | 0.021209169527059025 | 1.7667944406 | 0.008668441762979982 | 1.8386231196 | 0.02984760040605402 | 0.5990321109 | 0.014804404733360246 | 1.1508279068 | 0.021326970463733233 | 0.3524312471 | 0.004316307776005543 | 119999 |
-| 130000 | 2 | 10 | 130000 | 260000 | 2 | 130000 | 130000 | 208170 | 4.2108411467 | 0.030429870784823763 | 1.8758927709999997 | 0.014720813295526257 | 2.0216873166 | 0.02944166619615172 | 0.5635494682000001 | 0.006692369051929955 | 1.3816380033999998 | 0.029729760194715515 | 0.3983277031 | 0.0024183986266932186 | 130000 |
-| 140000 | 2 | 10 | 140000 | 280000 | 2 | 139999 | 139999 | 225471 | 4.6525796292 | 0.08226665976530252 | 2.0223978788 | 0.02782859024824926 | 2.2457517173 | 0.06419042143691525 | 0.43292923410000006 | 0.07437008468651318 | 1.5179340291999999 | 0.028571780925593183 | 0.4464015892 | 0.008337702174814625 | 139999 |
-| 150000 | 2 | 10 | 150000 | 300000 | 2 | 150000 | 150000 | 240331 | 4.9966277785 | 0.05561432863860171 | 2.1685035476999994 | 0.011273739740092388 | 2.4161792503 | 0.05218254165871221 | 0.4491742889 | 0.04435758243758258 | 1.6487869333999998 | 0.03158639366001329 | 0.4903136348 | 0.0035377975274226065 | 150000 |
-| 160000 | 2 | 10 | 160000 | 320000 | 2 | 159999 | 159999 | 256952 | 5.355174741499999 | 0.01181544267095224 | 2.3217772239000003 | 0.009423441961012942 | 2.5912845906 | 0.009615174702699738 | 0.4742331759000001 | 0.00567499301007789 | 1.774249567 | 0.00497004110981295 | 0.5373223992 | 0.006688940145895668 | 159999 |
-| 170000 | 2 | 10 | 170000 | 340000 | 2 | 169999 | 169999 | 273299 | 5.757032531199999 | 0.02661150144562761 | 2.4794814604999997 | 0.012372017151358293 | 2.8041343125 | 0.015352615509927985 | 0.5093437867 | 0.005037497584885556 | 1.9293367177999996 | 0.009725668899383278 | 0.58678976 | 0.006087850747350132 | 169999 |
-| 180000 | 2 | 10 | 180000 | 360000 | 2 | 180000 | 180000 | 289908 | 6.1609400376 | 0.03188833978682093 | 2.6309119938999994 | 0.00791867193670523 | 3.0293672266000002 | 0.035760581136465915 | 0.5431468218 | 0.006141095107119764 | 2.0932277568999997 | 0.0332985663387355 | 0.6429327362999999 | 0.007674756785721323 | 180000 |
-| 190000 | 2 | 10 | 190000 | 380000 | 2 | 190000 | 190000 | 307527 | 6.778789601 | 0.06991943979616976 | 2.7624881334999998 | 0.010831227316887074 | 3.3906752104999995 | 0.1397668131349721 | 0.8261375824 | 0.00893049817644368 | 2.2897163365 | 0.043731496282112814 | 0.6881156208999999 | 0.012255461536873485 | 190000 |
-
-*** And plotting them
-
-#+CALL: plot(results2f, "results2f.png")
-
-#+RESULTS:
-[[file:results2f.png]]
-
-** For m = 2 and 0% finality
-*** Generating some benchmarks
-
-#+begin_src sh :results output silent
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000; do
- python bench.py generate --final 0 ../../copar/bin/random-dfa $n 2
-done
-#+end_src
-
-*** And running them
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "sdt" "tp (s)" "sdtp" "ta (s)" "sdta" "ti (s)" "sdti" "tr (s)" "sdtr" "tv (s)" "sdtv" "Qv")
-for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000; do
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 2 --stddev --final 0
-done
-#+end_src
-
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | sdt | tp (s) | sdtp | ta (s) | sdta | ti (s) | sdti | tr (s) | sdtr | tv (s) | sdtv | Qv |
-|--------+---+----+--------+--------+---+---+-----+------+--------------------+----------------------+--------------------+----------------------+--------------------+-----------------------+---------------------+-----------------------+------------------------+------------------------+---------------------+-----------------------+----|
-| 100000 | 2 | 10 | 100000 | 200000 | 1 | 1 | 1 | 0 | 1.9286943634 | 0.00878107688302739 | 1.3837160297 | 0.008180649292894082 | 0.4171904952 | 0.003120493919147368 | 0.3641316149 | 0.0032160904550749985 | 1.0225800000000001e-05 | 2.1150455734517354e-07 | 0.1024724376 | 0.006073081115665208 | 1 |
-| 110000 | 2 | 10 | 110000 | 220000 | 1 | 1 | 1 | 0 | 2.2182804934999996 | 0.01412802800557772 | 1.4834582776 | 0.00738948570673236 | 0.4289651487 | 0.0055944857852055704 | 0.3764284657 | 0.005279970052798764 | 1.19171e-05 | 3.2553220851058316e-06 | 0.11168206160000001 | 0.0021092801129365952 | 1 |
-| 120000 | 2 | 10 | 120000 | 240000 | 1 | 1 | 1 | 0 | 2.4764642443 | 0.012935219317291618 | 1.7297758319 | 0.006418203929536284 | 0.5863125886 | 0.009762956606817539 | 0.5235213498 | 0.01025042704825142 | 1.0723899999999999e-05 | 2.6905367081267974e-07 | 0.1266497688 | 0.00756630532110257 | 1 |
-| 130000 | 2 | 10 | 130000 | 260000 | 1 | 1 | 1 | 0 | 2.5684661536 | 0.006060926053458022 | 1.8325813717 | 0.004199590188170929 | 0.5551686706 | 0.0026852027358281088 | 0.4815377142 | 0.0030480000798772133 | 1.0317600000000001e-05 | 3.9572190010438157e-07 | 0.1354930515 | 0.003756031590903835 | 1 |
-| 140000 | 2 | 10 | 140000 | 280000 | 1 | 1 | 1 | 0 | 2.8226806149000003 | 0.04107118493117886 | 1.9613554895 | 0.00623230387960241 | 0.6217279724999999 | 0.03774254529931291 | 0.33219782079999993 | 0.05402562365110335 | 1.21055e-05 | 3.7677216590401157e-06 | 0.1476372001 | 0.0057871447980288974 | 1 |
-| 150000 | 2 | 10 | 150000 | 300000 | 1 | 1 | 1 | 0 | 3.0114131203 | 0.02544552047477694 | 2.1203642880999998 | 0.008153746807082824 | 0.6600289866 | 0.006795559380887498 | 0.347185776 | 0.006751335332853107 | 1.1084300000000002e-05 | 2.0330387245587713e-06 | 0.1582478537 | 0.004157373435323506 | 1 |
-| 160000 | 2 | 10 | 160000 | 320000 | 1 | 1 | 1 | 0 | 3.2158140727999998 | 0.015839540316263775 | 2.2723906333999997 | 0.006996337136486422 | 0.7061300778999999 | 0.003739872502854762 | 0.373723791 | 0.004235039390227649 | 1.15646e-05 | 3.6649735848676696e-06 | 0.16917011399999998 | 0.0026638888864231733 | 1 |
-| 170000 | 2 | 10 | 170000 | 340000 | 1 | 1 | 1 | 0 | 3.4702468513999998 | 0.0509720428289049 | 2.4451624861 | 0.029551826817010306 | 0.7609187292999999 | 0.013753158673689668 | 0.404949446 | 0.004671610869945024 | 1.05429e-05 | 2.703024848819066e-07 | 0.1830823594 | 0.006077950328655399 | 1 |
-| 180000 | 2 | 10 | 180000 | 360000 | 1 | 1 | 1 | 0 | 3.650301588 | 0.014122237424641738 | 2.5787219889 | 0.011997897515023564 | 0.8084984614999999 | 0.00382175626616949 | 0.4325368038 | 0.001311000193778083 | 1.02732e-05 | 4.443144031976556e-07 | 0.19291228479999997 | 0.00398230081639267 | 1 |
-| 190000 | 2 | 10 | 190000 | 380000 | 1 | 1 | 1 | 0 | 4.0598550992 | 0.03655234344413544 | 2.7205397726999996 | 0.015378199595674087 | 0.9662325937 | 0.10889493389502679 | 0.7044185453 | 0.008811645674713221 | 1.0971300000000001e-05 | 3.923952230992513e-07 | 0.20248723759999998 | 0.0030261990594331093 | 1 |
-
-** For m = 1000:
-
-#+BEGIN_SRC sh :results output silent
-for n in 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000; do
- python bench.py generate ../../copar/bin/random-dfa $n 1000
-done
-#+END_SRC
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "sdt" "tp (s)" "sdtp" "ta (s)" "sdta" "ti (s)" "sdti" "tr (s)" "sdtr" "tv (s)" "sdtv" "Qv")
-for n in 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000; do
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 1000 --stddev --nofunctrans
-done
-#+end_src
-
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | sdt | tp (s) | sdtp | ta (s) | sdta | ti (s) | sdti | tr (s) | sdtr | tv (s) | sdtv | Qv |
-|------+------+----+--------+---------+---+------+------+---------+-------------------+----------------------+--------------------+----------------------+--------------------+----------------------+--------------+-----------------------+---------------------+-----------------------+--------------------+----------------------+------|
-| 1000 | 1000 | 10 | 1000 | 1000000 | 2 | 1000 | 1000 | 997998 | 3.9533156947 | 0.011002913439092422 | 3.1950151519 | 0.012356134887847729 | 0.7512120445 | 0.011239427555034565 | 0.3620109859 | 0.006701712453569692 | 0.3155020266 | 0.006691475865527507 | 1.0712054533999997 | 0.01653342562226392 | 1000 |
-| 1100 | 1000 | 10 | 1100 | 1100000 | 2 | 1100 | 1100 | 1097961 | 4.3701553551 | 0.015494278323821673 | 3.5279982033 | 0.009613643482103808 | 0.8343428652 | 0.011940829640151796 | 0.3988682328 | 0.006982419022379785 | 0.3525587549 | 0.006717777060771679 | 1.2101320207000001 | 0.02285978749012452 | 1100 |
-| 1200 | 1000 | 10 | 1200 | 1200000 | 2 | 1200 | 1200 | 1198002 | 4.7980000412 | 0.025899568868215973 | 3.8535088401000004 | 0.023463041540423543 | 0.9359110436 | 0.009696634861305436 | 0.4516177406 | 0.00894257321455952 | 0.3941434675 | 0.0032382300431677185 | 1.3848342052 | 0.03178452275534523 | 1200 |
-| 1300 | 1000 | 10 | 1300 | 1300000 | 2 | 1300 | 1300 | 1297980 | 5.174526376099999 | 0.015473966022638494 | 4.1554920699 | 0.010599095389132018 | 1.0095937982999998 | 0.011678465689753345 | 0.4864740198 | 0.006749029917787903 | 0.42563662580000006 | 0.007148089005487441 | 1.5224950211 | 0.019849208114581882 | 1300 |
-| 1400 | 1000 | 10 | 1400 | 1400000 | 2 | 1400 | 1400 | 1397912 | 5.5841241614 | 0.017194308572012134 | 4.48448128 | 0.012257064605230078 | 1.089672036 | 0.012331696297551112 | 0.5215628853 | 0.004909497714163821 | 0.4615644913 | 0.010132308836309516 | 1.6689043449 | 0.016312588975287756 | 1400 |
-| 1500 | 1000 | 10 | 1500 | 1500000 | 2 | 1500 | 1500 | 1497965 | 5.9623769231 | 0.046782942319537155 | 4.7854645485 | 0.028274372855761787 | 1.1664068236 | 0.020009936157051142 | 0.544806253 | 0.01972441506278639 | 0.5086029899000001 | 0.004057096938679593 | 1.8024757562000002 | 0.025080828008935546 | 1500 |
-| 1600 | 1000 | 10 | 1600 | 1600000 | 2 | 1600 | 1600 | 1597977 | 6.3001005989 | 0.021079788043247123 | 5.0446699955 | 0.011706946966419577 | 1.2439826050999998 | 0.012902023532078491 | 0.5636539833 | 0.010026688550342238 | 0.4158968506 | 0.009316903495588802 | 1.9512469113000002 | 0.015498110040682259 | 1600 |
-| 1700 | 1000 | 10 | 1700 | 1700000 | 2 | 1700 | 1700 | 1697999 | 6.8181373291 | 0.04768740025244082 | 5.3489446795 | 0.050821897845646204 | 1.4569464741999998 | 0.011898390779945658 | 0.6821315177 | 0.0066429662291499015 | 0.4497148234 | 0.01179709122261191 | 2.1008681020999997 | 0.017493430063319576 | 1700 |
-| 1800 | 1000 | 10 | 1800 | 1800000 | 2 | 1800 | 1800 | 1797969 | 7.2023435628 | 0.06276683614308662 | 5.769271290400001 | 0.02040555043753859 | 1.4207522051 | 0.04471515856236278 | 0.7175096355 | 0.020056793607776843 | 0.5715630545 | 0.021527475799908602 | 2.2076871809 | 0.03296824675743875 | 1800 |
-| 1900 | 1000 | 10 | 1900 | 1900000 | 2 | 1900 | 1900 | 1898012 | 7.5665405717 | 0.06220257139169582 | 6.060004332899999 | 0.022453768686637596 | 1.4931699382999999 | 0.041532090749656074 | 0.7137491964 | 0.021448931504701424 | 0.6389280077 | 0.019773789498324538 | 2.4230123612 | 0.06387080178483137 | 1900 |
-| 2000 | 1000 | 10 | 2000 | 2000000 | 2 | 2000 | 2000 | 1997904 | 8.0639794634 | 0.08248153745537908 | 6.4399256459 | 0.07693042641425599 | 1.609800233 | 0.012785432616659714 | 0.7680619713 | 0.013815734340735344 | 0.6888247968999999 | 0.00989100877264004 | 2.5814963057999996 | 0.02307845851893694 | 2000 |
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "sdt" "tp (s)" "sdtp" "ta (s)" "sdta" "ti (s)" "sdti" "tr (s)" "sdtr" "tv (s)" "sdtv" "Qv")
-for n in 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000; do
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 1000 --stddev
-done
-#+end_src
-
-#+NAME: results1000
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | sdt | tp (s) | sdtp | ta (s) | sdta | ti (s) | sdti | tr (s) | sdtr | tv (s) | sdtv | Qv |
-|------+------+----+--------+---------+---+------+------+---------+--------------------+-----------------------+--------------------+----------------------+--------------------+----------------------+---------------------+-----------------------+---------------------+-----------------------+--------------------+----------------------+------|
-| 1000 | 1000 | 10 | 1000 | 1000000 | 2 | 1000 | 1000 | 997998 | 3.4170901261000006 | 0.0067332911363214626 | 2.4400518157 | 0.012751085512302314 | 0.9700435225999999 | 0.00970206397709066 | 0.41540584669999997 | 0.00506517945359832 | 0.4819094799 | 0.0066852566220028985 | 1.0732738749 | 0.032397918884447534 | 1000 |
-| 1100 | 1000 | 10 | 1100 | 1100000 | 2 | 1100 | 1100 | 1097961 | 3.7839400801999994 | 0.02054243005075528 | 2.6923810361 | 0.01430132872516138 | 1.0838880011 | 0.012209366463486233 | 0.45925104040000003 | 0.006389879793225777 | 0.5437107408000001 | 0.007094074865848301 | 1.2343653796 | 0.046358644222388744 | 1100 |
-| 1200 | 1000 | 10 | 1200 | 1200000 | 2 | 1200 | 1200 | 1198002 | 3.9745562989 | 0.02849241548298448 | 2.719424793 | 0.01675680154232307 | 1.246594623 | 0.015534617230713086 | 0.49883183249999996 | 0.00924928186737593 | 0.4107449004 | 0.009692795378012436 | 1.3924885328 | 0.045409804321302843 | 1200 |
-| 1300 | 1000 | 10 | 1300 | 1300000 | 2 | 1300 | 1300 | 1297980 | 4.4492492853000005 | 0.019011021239560805 | 2.935811995 | 0.01336577945727504 | 1.4779213829 | 0.08254178277905756 | 0.5365245862000001 | 0.007709042658200236 | 0.6079340167 | 0.008997140674001144 | 1.4928702701999998 | 0.022580296655694658 | 1300 |
-| 1400 | 1000 | 10 | 1400 | 1400000 | 2 | 1400 | 1400 | 1397912 | 4.753583547100002 | 0.01964103887049429 | 3.4506703865999997 | 0.017531483434986516 | 1.2932429607999998 | 0.015339050721789663 | 0.5846364146 | 0.010854382347369648 | 0.6054716699 | 0.01145623639581948 | 1.6770205635 | 0.0292648210010643 | 1400 |
-| 1500 | 1000 | 10 | 1500 | 1500000 | 2 | 1500 | 1500 | 1497965 | 5.2460979974 | 0.06296848781374169 | 3.6766706373 | 0.06038250665653121 | 1.5588874043999998 | 0.01104492509639117 | 1.0406573998000002 | 0.010493977730771424 | 0.40563505910000003 | 0.006857974236933447 | 1.8193636309000003 | 0.04728729571439115 | 1500 |
-| 1600 | 1000 | 10 | 1600 | 1600000 | 2 | 1600 | 1600 | 1597977 | 5.5934895831 | 0.020471690649198758 | 3.8378448388 | 0.01405062980592564 | 1.7442780326999998 | 0.016801329216519365 | 1.1766017755000002 | 0.010817165560994954 | 0.44630750559999993 | 0.010926126526067979 | 1.9773537042999998 | 0.021751180636291764 | 1600 |
-| 1700 | 1000 | 10 | 1700 | 1700000 | 2 | 1700 | 1700 | 1697999 | 5.8844434041 | 0.025355074221741828 | 4.0209844924 | 0.01936016953688846 | 1.8514450776000004 | 0.01098843935670959 | 1.1342198513999997 | 0.0066834373252539206 | 0.5865765241 | 0.010631404738858001 | 2.1336306657 | 0.0248035664484013 | 1700 |
-| 1800 | 1000 | 10 | 1800 | 1800000 | 2 | 1800 | 1800 | 1797969 | 6.0798966787 | 0.03714316285262671 | 4.1842880729 | 0.016958763574648844 | 1.8828436436 | 0.03186802850868568 | 0.7512876949 | 0.012112711545569563 | 0.6483303171999999 | 0.020828759639626687 | 2.2825130664999995 | 0.03591125038883075 | 1800 |
-| 1900 | 1000 | 10 | 1900 | 1900000 | 2 | 1900 | 1900 | 1898012 | 6.310596887999999 | 0.09207001119016445 | 4.323571235299999 | 0.04513265558681511 | 1.9738281792999999 | 0.051358992309827774 | 0.7882479636 | 0.017551971166439598 | 0.6708881243 | 0.021509786422262756 | 2.4061289706 | 0.049279654362565456 | 1900 |
-| 2000 | 1000 | 10 | 2000 | 2000000 | 2 | 2000 | 2000 | 1997904 | 6.793026988199999 | 0.0405905959258501 | 4.606611127900001 | 0.018356435490485965 | 2.1718201212 | 0.02686927001218402 | 0.8559822423 | 0.012914019713495746 | 0.7510526754999999 | 0.01756542404255711 | 2.6220723597999998 | 0.05447934428571054 | 2000 |
-
-
-#+CALL: plot(results1000, "results1000.png")
-
-#+RESULTS:
-[[file:results1000.png]]
-
-*** Quantifying the effect of the one-block optimization
-
-Let's test the timings with and without optimizations on one single DFA.
-
-**** With all optimizations
-:PROPERTIES:
-:ID: c21020da-1d9c-4553-bffe-478b4e35aa90
-:END:
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "tp (s)" "ta (s)" "ti (s)" "tr (s)" "tv (s)" "Qv")
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1
-#+end_src
-
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | tp (s) | ta (s) | ti (s) | tr (s) | tv (s) | Qv |
-|------+------+---+--------+---------+---+------+------+--------+-------------+-------------+-------------+-------------+-------------+-------------+------|
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 997998 | 3.380971816 | 2.406927892 | 0.967057876 | 0.421992475 | 0.47148897 | 1.011007518 | 1000 |
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 997998 | 3.371089411 | 2.388749696 | 0.975496846 | 0.415187236 | 0.489191121 | 1.040577816 | 1000 |
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 997998 | 3.389716249 | 2.409769645 | 0.973066032 | 0.408821936 | 0.493088452 | 1.0411288 | 1000 |
-
-**** Without any optimization
-:PROPERTIES:
-:ID: 849f9d1f-1106-4bee-8270-7b46a09f709a
-:END:
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "tp (s)" "ta (s)" "ti (s)" "tr (s)" "tv (s)" "Qv")
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --noopt --nofunctrans
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --noopt --nofunctrans
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --noopt --nofunctrans
-#+end_src
-
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | tp (s) | ta (s) | ti (s) | tr (s) | tv (s) | Qv |
-|------+------+---+--------+---------+---+------+------+------+--------------+-------------+--------------+-------------+--------------+-------------+------|
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 0 | 22.870625628 | 3.26410958 | 19.599263489 | 0.384769247 | 19.137524603 | 1.038918053 | 1000 |
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 0 | 22.738491632 | 3.160893761 | 19.570250212 | 0.364611475 | 19.130150657 | 1.033282541 | 1000 |
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 0 | 22.679763636 | 3.160448145 | 19.511865875 | 0.354412205 | 19.082196797 | 1.036891031 | 1000 |
-
-**** With just the functor rewrite
-:PROPERTIES:
-:ID: d2559cc7-c179-48cf-aaab-c7f17a29ea1a
-:END:
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "tp (s)" "ta (s)" "ti (s)" "tr (s)" "tv (s)" "Qv")
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --noopt
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --noopt
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --noopt
-#+end_src
-
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | tp (s) | ta (s) | ti (s) | tr (s) | tv (s) | Qv |
-|------+------+---+--------+---------+---+------+------+------+--------------+-------------+--------------+-------------+--------------+-------------+------|
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 0 | 46.8651896 | 2.412941424 | 44.44513298 | 0.424391142 | 43.94959211 | 1.026554102 | 1000 |
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 0 | 46.922508866 | 2.421724742 | 44.493428228 | 0.42204586 | 43.997902631 | 1.045297534 | 1000 |
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 0 | 46.938332303 | 2.40965665 | 44.521220406 | 0.423164636 | 44.024494492 | 1.057795996 | 1000 |
-
-**** With just the 1-elem optimization
-:PROPERTIES:
-:ID: d17f77cb-9a07-4c9f-8366-533e07075e6b
-:END:
-
-#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Qs1" "Opts" "t (s)" "tp (s)" "ta (s)" "ti (s)" "tr (s)" "tv (s)" "Qv")
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --nofunctrans
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --nofunctrans
-python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000 --indiv --samples 1 --nofunctrans
-#+end_src
-
-#+RESULTS:
-| N | M | i | States | Edges | I | Q | Qs1 | Opts | t (s) | tp (s) | ta (s) | ti (s) | tr (s) | tv (s) | Qv |
-|------+------+---+--------+---------+---+------+------+--------+-------------+-------------+-------------+-------------+-------------+-------------+------|
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 997998 | 3.890574258 | 3.151214053 | 0.732321905 | 0.35274363 | 0.307295302 | 1.012714586 | 1000 |
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 997998 | 3.886333612 | 3.152135172 | 0.727229289 | 0.350018448 | 0.305030783 | 1.045386717 | 1000 |
-| 1000 | 1000 | 0 | 1000 | 1000000 | 2 | 1000 | 1000 | 997998 | 3.886510967 | 3.146301307 | 0.733262327 | 0.354048267 | 0.306785068 | 1.033495542 | 1000 |
-
-
-** For m = 10000
-
- #+BEGIN_SRC sh :hlines yes :colnames '("N" "M" "States" "Edges" "I" "Q" "Qs1" "t (s)" "tp (s)" "ta (s)" "ti (s)" "tr (s)" "tv (s)" "Qv")
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 500 10000
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 600 10000
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 700 10000
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 800 10000
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 900 10000
- python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 10000
- #+END_SRC
-
-
- #+NAME: results10000
- #+RESULTS:
- | N | M | States | Edges | I | Q | Qs1 | t (s) | tp (s) | ta (s) | ti (s) | tr (s) | tv (s) | Qv |
- |------+-------+--------+----------+---+------+------+--------------+--------------+-------------+-------------+-------------+--------------+------|
- | 500 | 10000 | 500 | 5000000 | 2 | 500 | 500 | 42.104668963 | 37.582293662 | 4.493314329 | 1.518858584 | 2.386916363 | 6.366641481 | 500 |
- | 600 | 10000 | 600 | 6000000 | 2 | 600 | 600 | 50.191903227 | 44.747513385 | 5.40920263 | 1.820207355 | 2.88154887 | 7.835340678 | 600 |
- | 700 | 10000 | 700 | 7000000 | 2 | 700 | 700 | 59.318446986 | 50.925788701 | 8.350924719 | 4.28553676 | 3.176143156 | 9.419061177 | 700 |
- | 800 | 10000 | 800 | 8000000 | 2 | 800 | 800 | 68.472073706 | 60.783380535 | 7.642012395 | 2.535035704 | 4.159210276 | 11.039647443 | 800 |
- | 900 | 10000 | 900 | 9000000 | 2 | 900 | 900 | 76.874599881 | 68.344624307 | 8.477049112 | 2.759185939 | 4.596666911 | 12.712082944 | 900 |
- | 1000 | 10000 | 1000 | 10000000 | 2 | 1000 | 1000 | 85.269853921 | 75.788964993 | 9.422514059 | 3.052157078 | 5.207201202 | 14.320693046 | 1000 |
-
- #+BEGIN_SRC python :results file :exports results :var data=results10000
-
- import matplotlib.pyplot as plt
-
- '''Turn the table data into x and y data'''
- x = [a[0] for a in data]
- t = [a[7] for a in data]
- tp = [a[8] for a in data]
- tr = [a[9] for a in data]
- tv = [a[10] for a in data]
-
- ''' Plot the x and y data'''
- a, = plt.plot(x, t, label="t", marker='v')
- b, = plt.plot(x, tp, label="tp", marker='o')
- c, = plt.plot(x, tr, label="tr", marker='x')
- d, = plt.plot(x, tv, label="tv", marker='s')
-
- ''' Set the x and y labels on the graph '''
- plt.xlabel("Number of states")
- plt.ylabel("Seconds")
-
- ''' Create the legend '''
- plt.legend(handles=[a,b,c,d],loc="upper left")
-
- ''' Save the PNG file '''
- filename = "results10000.png"
- plt.savefig(filename)
-
- ''' Return the PNG file path to OrgMode '''
- return(filename)
- #+END_SRC
-
- #+RESULTS:
- [[file:results10000.png]]
-
-
-* Colophone
-
-#+begin_src elisp
-(require 'ob-shell)
-(require 'ob-python)
-#+end_src
-
-#+RESULTS:
-: ob-shell
-
diff --git a/dfa/results1.png b/dfa/results1.png
deleted file mode 100644
index 6bbcbe38a2bd9e0c5b0b2a3400ef4e9eb5a788e1..0000000000000000000000000000000000000000
Binary files a/dfa/results1.png and /dev/null differ
diff --git a/dfa/results1000.png b/dfa/results1000.png
deleted file mode 100644
index fe3f84a7e9c378cab81335ea7c583dc86bd459bb..0000000000000000000000000000000000000000
Binary files a/dfa/results1000.png and /dev/null differ
diff --git a/dfa/results10000.png b/dfa/results10000.png
deleted file mode 100644
index e8c339f63393314724beba86501cd8e66de7a7f2..0000000000000000000000000000000000000000
Binary files a/dfa/results10000.png and /dev/null differ
diff --git a/dfa/results2.png b/dfa/results2.png
deleted file mode 100644
index caf12c1a037683507788f6921fce50841d998965..0000000000000000000000000000000000000000
Binary files a/dfa/results2.png and /dev/null differ
diff --git a/dfa/results2f.png b/dfa/results2f.png
deleted file mode 100644
index dcf910186ff16bf53dc4e344e1d2188d74261d74..0000000000000000000000000000000000000000
Binary files a/dfa/results2f.png and /dev/null differ
diff --git a/optimizations/optimizations.org b/optimizations/optimizations.org
deleted file mode 100644
index 73c77e9e64476ab91a2b98c3c97508cd23ba4ccc..0000000000000000000000000000000000000000
--- a/optimizations/optimizations.org
+++ /dev/null
@@ -1,123 +0,0 @@
-#+TITLE: Meassuring the Impact of Optimizations
-
-* The Data
-
-#+NAME: the-data
-| Benchmark | Time1 | Time2 | Time3 | Mean | Error |
-|---------------+-----------+-----------+-----------+-----------+--------------|
-| wlan both opt | 39.363733 | 39.464646 | 39.064738 | 39.297706 | 0.12007124 |
-| wlan no opt | 62.803240 | 62.809634 | 62.884818 | 62.832564 | 0.026192118 |
-| wlan func | 40.482061 | 40.554590 | 40.829683 | 40.622111 | 0.10587668 |
-| wlan 1elem | 62.322413 | 62.416543 | 62.852129 | 62.530362 | 0.16316227 |
-|---------------+-----------+-----------+-----------+-----------+--------------|
-| fms both opt | 21.111599 | 20.959398 | 20.955570 | 21.008856 | 0.051383551 |
-| fms no opt | 35.451870 | 35.385039 | 35.830079 | 35.555663 | 0.13855786 |
-| fms func | 35.379264 | 35.544466 | 35.531165 | 35.484965 | 0.052989795 |
-| fms 1elem | 21.331782 | 21.307057 | 21.295322 | 21.311387 | 0.010745458 |
-|---------------+-----------+-----------+-----------+-----------+--------------|
-| dfa both opt | 3.3809203 | 3.3856634 | 3.3604341 | 3.3756726 | 7.7412997e-3 |
-| dfa no opt | 22.870626 | 22.738492 | 22.679764 | 22.762961 | 0.056439090 |
-| dfa func | 46.865190 | 46.922509 | 46.938332 | 46.908677 | 0.022218095 |
-| dfa 1elem | 3.8905743 | 3.8863336 | 3.8865110 | 3.8878063 | 1.3849471e-3 |
-#+TBLFM: $5=vmean($2..$4)::$6=vsdev($2..$4)/sqrt(3)::@2$2..@2$4=remote(d39b3290-297b-4063-8464-a011a048eff4, @$#$11)::@4$2..@4$4=remote(eb6cd726-026c-49ec-974a-1b6285d0fabb, @$#$11)::@6$2..@6$4=remote(cdef09e0-b92a-46db-9ed5-be0cde788441, @$#$11)::@8$2..@8$4=remote(966b0dc7-cf00-4594-88ae-80332696166e, @$#$11)::@10$2..@10$4=remote(c21020da-1d9c-4553-bffe-478b4e35aa90, @$#$10)::@12$2..@12$4=remote(d2559cc7-c179-48cf-aaab-c7f17a29ea1a, @$#$10)::@3$2..@3$4=remote(408056f6-53de-4920-ba53-9308bb2fbf15, @$#$11)::@5$2..@5$4=remote(74566779-0749-4bbd-9556-80f857144793, @$#$11)::@7$2..@7$4=remote(8850d72d-8f23-495e-b3af-a06deb25be53, @$#$11)::@9$2..@9$4=remote(7e19eb5e-4009-4bc1-89a3-4e4e5c7f643e, @$#$11)::@11$2..@11$4=remote(849f9d1f-1106-4bee-8270-7b46a09f709a, @$#$10)::@13$2..@13$4=remote(d17f77cb-9a07-4c9f-8366-533e07075e6b, @$#$10)
-
-* Individual Tables
-
-These are the input for pgfplots.
-
-All confidence intervals are the standard error times 10.
-
-** No Optimizations
-
-| Benchmark | Time | CI |
-|-----------+-----------+------------|
-| wlan | 62.832564 | 0.26192118 |
-| fms | 35.555663 | 1.3855786 |
-| dfa | 22.762961 | 0.5643909 |
-#+TBLFM: @2$2=remote(the-data, @3$5)::@2$3=10*remote(the-data, @3$6)::@3$2=remote(the-data, @7$5)::@3$3=10*remote(the-data, @7$6)::@4$2=remote(the-data, @11$5)::@4$3=10*remote(the-data, @11$6)
-
-** Just Functor Rewrite
-
-| Benchmark | Time | CI |
-|-----------+-----------+------------|
-| wlan | 40.622111 | 1.0587668 |
-| fms | 35.484965 | 0.52989795 |
-| dfa | 46.908677 | 0.22218095 |
-#+TBLFM: @2$2=remote(the-data, @4$5)::@2$3=10*remote(the-data, @4$6)::@3$2=remote(the-data, @8$5)::@3$3=10*remote(the-data, @8$6)::@4$2=remote(the-data, @12$5)::@4$3=10*remote(the-data, @12$6)
-
-** Just one-element blocks
-
-| Benchmark | Time | CI |
-|-----------+-----------+-------------|
-| wlan | 62.530362 | 1.6316227 |
-| fms | 21.311387 | 0.10745458 |
-| dfa | 3.8878063 | 0.013849471 |
-#+TBLFM: @2$2=remote(the-data, @5$5)::@2$3=10*remote(the-data, @5$6)::@3$2=remote(the-data, @9$5)::@3$3=10*remote(the-data, @9$6)::@4$2=remote(the-data, @13$5)::@4$3=10*remote(the-data, @13$6)
-
-** Both optimizations
-
-| Benchmark | Time | CI |
-|-----------+-----------+-------------|
-| wlan | 39.297706 | 1.2007124 |
-| fms | 21.008856 | 0.51383551 |
-| dfa | 3.3756726 | 0.077412997 |
-#+TBLFM: @2$2=remote(the-data, @2$5)::@2$3=10*remote(the-data, @2$6)::@3$2=remote(the-data, @6$5)::@3$3=10*remote(the-data, @6$6)::@4$2=remote(the-data, @10$5)::@4$3=10*remote(the-data, @10$6)
-
-* Old stuff :ARCHIVE:
-** The "skip one-element blocks" Optimization
-*** The Data
-
-#+NAME: one-elem-data
-| Benchmark | Time1 | Time2 | Time3 | Mean | Error |
-|------------+-----------+-----------+-----------+-----------+--------------|
-| wlan opt | 39.976493 | 39.832630 | 39.784828 | 39.864650 | 0.057598739 |
-| wlan noopt | 40.482061 | 40.554590 | 40.829683 | 40.622111 | 0.10587668 |
-|------------+-----------+-----------+-----------+-----------+--------------|
-| fms opt | 21.126185 | 20.870302 | 20.996653 | 20.997713 | 0.073868962 |
-| fms noopt | 35.581014 | 36.222743 | 36.020706 | 35.941488 | 0.18943837 |
-|------------+-----------+-----------+-----------+-----------+--------------|
-| dfa opt | 3.8669068 | 3.8483511 | 3.8465348 | 3.8539309 | 6.5091019e-3 |
-| dfa noopt | 47.182955 | 47.271392 | 47.166293 | 47.20688 | 0.032612647 |
-#+TBLFM: $6=vsdev($2..$4)/sqrt(3)::@2$2..@2$4=remote(d39b3290-297b-4063-8464-a011a048eff4, @$#$11)::@3$2..@3$4=remote(eb6cd726-026c-49ec-974a-1b6285d0fabb, @$#$11)::@4$2..@4$4=remote(cdef09e0-b92a-46db-9ed5-be0cde788441, @$#$11)::@5$2..@5$4=remote(966b0dc7-cf00-4594-88ae-80332696166e, @$#$11)::@6$2..@6$4=remote(c21020da-1d9c-4553-bffe-478b4e35aa90, @$#$10)::@7$2..@7$4=remote(d2559cc7-c179-48cf-aaab-c7f17a29ea1a, @$#$10)
-
-*** Making the data easier for pgfplot
-**** Opt
-
-| Benchmark | Time | CI |
-|-----------+-----------+-------------|
-| wlan | 39.86465 | 0.57598739 |
-| fms | 20.997713 | 0.73868962 |
-| dfa | 3.8539309 | 0.065091019 |
-#+TBLFM: @2$2=remote(one-elem-data, @2$5)::@2$3=10*remote(one-elem-data, @2$6)::@3$2=remote(one-elem-data, @4$5)::@3$3=10*remote(one-elem-data, @4$6)::@4$2=remote(one-elem-data, @6$5)::@4$3=10*remote(one-elem-data, @6$6)
-
-**** Noopt
-
-| Benchmark | Time | CI |
-|-----------+-----------+------------|
-| wlan | 40.622111 | 1.0587668 |
-| fms | 35.941488 | 1.8943837 |
-| dfa | 47.20688 | 0.32612647 |
-#+TBLFM: @2$2=remote(one-elem-data, @3$5)::@2$3=10*remote(one-elem-data, @3$6)::@3$2=remote(one-elem-data, @5$5)::@3$3=10*remote(one-elem-data, @5$6)::@4$2=remote(one-elem-data, @7$5)::@4$3=10*remote(one-elem-data, @7$6)
-
-** Functor expression rewriting
-*** The Data
-
-#+NAME: func-rewrite-data
-| Benchmark | Time1 | Time2 | Time3 | Mean | Error |
-|------------+-----------+-----------+-----------+-----------+--------------|
-| wlan noopt | 62.322413 | 62.416543 | 62.852129 | 62.530362 | 0.16316227 |
-|------------+-----------+-----------+-----------+-----------+--------------|
-| fms noopt | 21.331782 | 21.307057 | 21.295322 | 21.311387 | 0.010745458 |
-|------------+-----------+-----------+-----------+-----------+--------------|
-| dfa noopt | 3.6995118 | 3.6943322 | 3.6743715 | 3.6894052 | 7.6641028e-3 |
-#+TBLFM: $5=vmean($2..$4)::$6=vsdev($2..$4)/sqrt(3)::@2$2..@2$4=remote(74566779-0749-4bbd-9556-80f857144793, @$#$11)::@3$2..@3$4=remote(7e19eb5e-4009-4bc1-89a3-4e4e5c7f643e, @$#$11)::@4$2..@4$4=remote(d17f77cb-9a07-4c9f-8366-533e07075e6b, @$#$10)
-
-*** Making the data easier for pgfplot
-*** Noopt
-
-| Benchmark | Time | CI |
-|-----------+-----------+-------------|
-| wlan | 62.530362 | 1.6316227 |
-| fms | 21.311387 | 0.10745458 |
-| dfa | 3.6894052 | 0.076641028 |
-#+TBLFM: $2=remote(func-rewrite-data, @@#$5)::$3=10*remote(func-rewrite-data, @@#$6)
diff --git a/prism/bench.py b/prism/bench.py
deleted file mode 100755
index e903a491a3bfe1904627a5ee19a10dfda4ece595..0000000000000000000000000000000000000000
--- a/prism/bench.py
+++ /dev/null
@@ -1,54 +0,0 @@
-#!/usr/bin/python
-
-import subprocess
-import os
-import sys
-import re
-
-
-def extract_time(perf_output):
- "returns the measured time from the perf output in seconds"
- m = re.search(r'(\d+.\d+)\W+seconds time elapsed', str(perf_output))
- return m[1]
-
-
-def run_copar(copar, input_file):
- process = subprocess.run(
- ['perf', 'stat', copar],
- stdin=input_file,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.PIPE,
- env={'LC_ALL': 'C'})
-
- try:
- process.check_returncode()
- except subprocess.CalledProcessError as e:
- err = os.fdopen(sys.stderr.fileno(), 'wb')
- err.write(process.stderr)
- err.flush()
- raise e
-
- return extract_time(process.stderr)
-
-
-def run_bench(copar, name, filename):
- with open(filename) as model_file:
- time = run_copar(copar, model_file)
-
- print("%s" % time)
-
-
-def main(args):
- if len(args) != 4:
- print("Usage: %s EXECUTABLE MODEL_NAME MODEL_FILE" % args[0])
- sys.exit(1)
-
- copar_binary = args[1]
- model_name = args[2]
- model_file = args[3]
-
- run_bench(copar_binary, model_name, model_file)
-
-
-if __name__ == "__main__":
- main(sys.argv)
diff --git a/prism/benchmark.org b/prism/benchmark.org
deleted file mode 100644
index b6254761f79719fed055ce99e1a8b22ed2d72eab..0000000000000000000000000000000000000000
--- a/prism/benchmark.org
+++ /dev/null
@@ -1,1554 +0,0 @@
-#+TITLE: PRISM Benchmarks
-#+LATEX_HEADER: \usepackage[a4paper,margin=1in,landscape]{geometry}
-
-* Introduction
-
- This uses the extensive bechmark-suite of [[https://www.prismmodelchecker.org/][PRISM]] to test correctness and
- performance of our implementation. In the following, we exclusively use PRISM
- version ~4.4~, available [[https://github.com/prismmodelchecker/prism/archive/v4.4.tar.gz][from github]].
-
-** PRISM
-
- PRISM supports the following model types:
-
- - mdp :: Markov Decision Process
- - dtmc :: Discrete-time Markov chain
- - ctmc :: Continuous-time Markov chain
- - pta :: Extension to MDPs with real-time behaviour
-
- We can translate those into category theoretic functors:
-
- | Model type | Functor |
- |------------+---------|
- | mdp | $P(DX)$ |
- | ctmc | $DX$ |
- | dtmc | $R^X$ |
- | pta | TODO |
-
- An initial partition $I$ can be represented by pairing the functor with $I$:
- $I\times FX$.
-
- PRISM has a directory called ~prism-examples~, from which we can extract
- quite a few models for our benchmarks.
-
-** Models
-
- From the examples shipped with PRISM, we can use the following models.
- Columns in this tables are defined as follows:
-
- - Name :: Descriptive name
- - Model file :: PRISM model file, relative to the ~prism-examples~ directory.
- - Model type :: Model type, as defined above
- - Constants :: Constant that need to be defined in the model. These can be
- used to parameterise benchmarks.
-
- #+NAME: models
- | Name | Model file | Model type | Constants |
- |------------------------+--------------------------------------------+------------+-----------------------------|
- | [[Bounded Retransmission Protocol][brp]] | brp/brp.pm | dtmc | N, MAX |
- | [[Cell in Wireless Communication Network][cell]] | cell/cell.sm | ctmc | N |
- | [[Dependable Cluster][cluster]] | cluster/cluster.sm | ctmc | N, T |
- | [[Shared coin protocol][coin]] | consensus/coinN.nm | mdp | N, K, k |
- | [[IEEE 802.3 CSMA/CD][csma]] | csma/csmaN_K.nm | mdp | N, K, k |
- | [[Dice][dice]] | dice/dice.pm | dtmc | |
- | [[Dice][two_dice_knuth]] | dice/two_dice_knuth.pm | dtmc | |
- | [[Dice][two_dice]] | dice/two_dice.nm | mdp | |
- | [[Dining cryptographers][dining_crypt]] | dining_crypt/dining_cryptN.nm | mdp | N, k |
- | [[Embedded Control System][embedded]] | embedded/embedded.sm | ctmc | MAX_COUNT, T |
- | [[Firewire][firewire_abst_firewire]] | firewire/abst/firewire.nm | mdp | delay, fast |
- | [[Firewire][firewire_abst_deadline]] | firewire/abst/deadline.nm | mdp | deadline, delay, fast |
- | [[Firewire][firewire_impl_firewire]] | firewire/impl/firewire.nm | mdp | delay, fast |
- | [[Firewire][firewire_impl_deadline]] | firewire/impl/deadline.nm | mdp | deadline, delay, fast |
- | [[Flexible Manufacturing System][fms]] | fms/fms.sm | ctmc | n |
- | [[Kanban][kanban]] | kanban/kanban.sm | ctmc | t |
- | [[id:fcc30abf-3473-429b-ade2-0900e6825042][leader_sync]] | leader/synchronous/leaderN_K.pm | dtmc | L |
- | [[id:e6b7cd19-fdf8-4c34-8a16-5429102fe2b6][leader_async]] | leader/asynchronous/leaderN.nm | mdp | N, K |
- | [[id:2fc0e60e-721b-47b7-9606-dfb32118fe47][nacl]] | molecules/nacl.sm | ctmc | N1, N2, T, i |
- | [[id:2fc0e60e-721b-47b7-9606-dfb32118fe47][knacl]] | molecules/knacl.sm | ctmc | N1, N2, N3, T, i |
- | [[id:2fc0e60e-721b-47b7-9606-dfb32118fe47][mc]] | molecules/mc.sm | ctmc | N1, N2, T, i |
- | [[id:e8ac9d4d-0ed9-4cc2-85ea-359a5261442c][mutual]] | mutual/mutualN.nm | mdp | N |
- | [[id:8f53a247-e80e-4b29-b0e9-34bd29aa91ff][peer2peer]] | peer2peer/peer2peerN_K.sm | ctmc | N, K, T |
- | [[id:eb035abb-aa5a-4edd-a193-605fbfb386d5][phil_original]] | phil/original/philN.nm | mdp | N |
- | [[id:2d736b3d-a5c8-4485-9065-0724cd79efcd][phil_nofair]] | phil/nofair/phil-nofairN.nm | mdp | K |
- | [[id:7e4f230d-5ee9-4d00-84bd-1b2a936e2fee][poll]] | polling/pollN.sm | ctmc | T |
- | [[id:754e893c-474b-4a3b-a870-7bb69d4c4a0b][rabin]] | rabin/rabinN.nm | mdp | N |
- | [[id:94785b23-df89-4daa-a615-ba97a16661a0][beauquier]] | self-stabilisation/beauquier/beauquierN.nm | mdp | N, K, k |
- | [[id:ab8fc607-28eb-4618-9369-20353be3b7ec][herman]] | self-stabilisation/herman/hermanN.pm | dtmc | N, K, k |
- | [[id:c6c91d7d-0c6e-41ff-8610-7f1cd020393d][israeli-jalfon]] | self-stabilisation/israeli-jalfon/ijN.nm | mdp | N, K, k |
- | [[id:f4d75ba1-5bd9-427f-846a-c233037887d8][tandem]] | tandem/tandem.sm | ctmc | T, c |
- | [[id:c808971a-8c52-41c3-8de0-42aebe332be5][wlan]] | wlan/wlanK.nm | mdp | TRANS_TIME_MAX, k, K |
- | [[id:c808971a-8c52-41c3-8de0-42aebe332be5][wlan_collide]] | wlan/wlanK_collide.nm | mdp | COL, TRANS_TIME_MAX, k, K |
- | [[id:c808971a-8c52-41c3-8de0-42aebe332be5][wlan_time_bounded]] | wlan/wlanK_time_bounded.nm | mdp | TRANS_TIME_MAX, DEADLINE, K |
- | [[id:e8a80143-4f49-4109-8f58-950514ada685][zeroconf]] | zeroconf/zeroconf.nm | mdp | N, K, err, reset |
- | [[id:e8a80143-4f49-4109-8f58-950514ada685][zeroconf_time_bounded]] | zeroconf/zeroconf_time_bounded.nm | mdp | N, K, T, bound, reset |
-
- We have to import this table into python for later reference:
-
- #+BEGIN_SRC python :session :var models_table=models :results value silent
- import re
- models = []
- for t in models_table:
- models.append({'name': re.sub(r"\[\[.*\]\[(.*)\]\]", r"\1", t[0]), 'path': t[1], 'type': t[2], 'constants': t[3]})
- #+END_SRC
-
-** Tooling
-
- We will use python to translate models and generate benchmark results.
-
- #+BEGIN_SRC python :session :results output silent :eval never-export
- import os
- import subprocess
- import json
- import re
- #+END_SRC
-
- The following environment definitions probably need to be adapted for
- different systems:
-
- #+BEGIN_SRC python :session :results value silent :eval never-export
- config = {'prism_cmd': '../../prism-4.4/prism/bin/prism',
- 'prism_examples': '../../prism-4.4/prism-examples',
- 'copar_cmd': '../../copar/bin/copar',
- 'prism_converter_cmd': '../../copar/bin/prism-converter',
- }
- #+END_SRC
-
- We need to define a few python functions and variables to use later:
-
- #+BEGIN_SRC python :session :results output silent :eval never-export
- def const_assignment_to_dict(const_assignment):
- clist = const_assignment.strip().split(',')
- if '' in clist:
- clist.remove('')
- d = {}
- for assignment in clist:
- x = assignment.split('=', 1)
- d[x[0]] = x[1]
- return d
-
- def replace_consts_in_path(path, const_assignment):
- "returns the unassigned constants"
- const_dict = const_assignment_to_dict(const_assignment)
- new_const_assignment = ""
- for key, value in const_dict.items():
- upkey = key.upper()
- if upkey in path:
- path = path.replace(upkey, value, 1)
- else:
- new_const_assignment += "%s=%s," % (key,value)
- return (path, new_const_assignment.rstrip(','))
-
- def const_base_path(path, const_assignments):
- return "bench/%s_%s" % (path, const_assignments) \
- if const_assignments else ("bench/%s" % path)
-
- def export_model_impl(path, const_assignments):
- path_with_consts = const_base_path(path, const_assignments)
- tra_file = "%s.tra" % path_with_consts
- sta_file = "%s.sta" % path_with_consts
- if os.path.isfile(tra_file):
- return
- os.makedirs(os.path.dirname(tra_file), exist_ok=True)
- model_path = "%s/%s" % (config['prism_examples'], path)
- subprocess.run([config['prism_cmd'],
- '-exporttrans', tra_file,
- '-exportstates', sta_file,
- '-const', const_assignments,
- model_path])
-
- def find_model(name):
- for m in models:
- if m['name'] == name:
- return m
- raise Exception("Model %s not found" % name)
-
- def export_model(name, const_assignments):
- m = find_model(name)
- path, const_assignments = replace_consts_in_path(m['path'], const_assignments)
- return export_model_impl(path, const_assignments)
-
- def convert_model(name, const_assignments, initial_partition):
- model = find_model(name)
- path, const_assignments = replace_consts_in_path(model['path'], const_assignments)
- path_with_consts = const_base_path(path, const_assignments)
- tra_file = "%s.tra" % path_with_consts
- sta_file = "%s.sta" % path_with_consts
- coalgebra_file = path_with_consts + ".coalgebra"
- if os.path.isfile(coalgebra_file):
- return
- if initial_partition:
- subprocess.run([config['prism_converter_cmd'], '--model-type',
- model['type'], '--states-file', sta_file,
- "--partition-on-variables", initial_partition,
- tra_file, coalgebra_file])
- else:
- subprocess.run([config['prism_converter_cmd'], '--model-type',
- model['type'], '--states-file', sta_file,
- tra_file, coalgebra_file])
-
- def partition_model(name, const_assignments, disableopts, nofunctrans):
- model = find_model(name)
- path, const_assignments = replace_consts_in_path(model['path'], const_assignments)
- path_with_consts = const_base_path(path, const_assignments)
- coalgebra_file = path_with_consts + ".coalgebra"
- cmd = [config['copar_cmd'], 'refine', '--stats-json', coalgebra_file]
- if disableopts:
- cmd.append('--disable-optimizations')
- if nofunctrans:
- cmd.append('--no-functor-transforms')
- out = subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
- print(out.stderr.decode('utf-8'))
- return json.loads(out.stderr.decode('utf-8'))
-
- def run_all(model, const_assignments, initial_partition, disableopts, nofunctrans):
- export_model(model, const_assignments)
- convert_model(model, const_assignments, initial_partition)
- return partition_model(model, const_assignments, disableopts, nofunctrans)
- #+END_SRC
-
-** Valmari's Implementation
-
- We also compare the running time of our implementation against a direct
- implementation of markov chain lumping that also supports MDPs. This is
- expected to be a lot faster since it's written in hand optimized C++ and
- specialized for its use case instead of being generic like ours.
-
- First, we need to define where the executable for this implementation is:
-
- #+BEGIN_SRC python :session :eval never-export :results output silent
- config['valmari_cmd'] = '../../valmari_cpp/mdpmin'
- #+END_SRC
-
- And then some python to generate input files and run it.
-
- #+BEGIN_SRC python :session :eval never-export :results output silent
- def convert_model_valmari(name, const_assignments, initial_partition):
- model = find_model(name)
- path, const_assignments = replace_consts_in_path(model['path'], const_assignments)
- path_with_consts = const_base_path(path, const_assignments)
- tra_file = "%s.tra" % path_with_consts
- sta_file = "%s.sta" % path_with_consts
- output_file = path_with_consts + ".valmari"
- if os.path.isfile(output_file):
- return
- if initial_partition:
- subprocess.run([config['prism_converter_cmd'],
- '--model-type', model['type'],
- '--output-format', 'valmari',
- '--states-file', sta_file,
- "--partition-on-variables", initial_partition,
- tra_file, output_file])
- else:
- subprocess.run([config['prism_converter_cmd'],
- '--model-type', model['type'],
- '--output-format', 'valmari',
- '--states-file', sta_file,
- tra_file, output_file])
-
- def run_valmari(name, const_assignments):
- model = find_model(name)
- path, const_assignments = replace_consts_in_path(model['path'], const_assignments)
- path_with_consts = const_base_path(path, const_assignments)
- valmari_file = open(path_with_consts + ".valmari")
- perf = subprocess.Popen(['perf', 'stat', config['valmari_cmd']],
- stdin=valmari_file, stdout=subprocess.PIPE,
- stderr=subprocess.PIPE, env={'LC_ALL': 'C'})
- sort1 = subprocess.run(['runhaskell', 'valmari-extract-sort1.hs', model['type']],
- stdin=perf.stdout, stdout=subprocess.PIPE)
- (_, out) = perf.communicate()
- m = re.search(r'(\d+.\d+)\W+seconds time elapsed', out.decode('utf-8'))
- return (m[1], sort1.stdout.decode('utf-8').strip())
-
- def run_all_valmari(name, const_assignments, initial_partition):
- convert_model_valmari(name, const_assignments, initial_partition)
- return run_valmari(name, const_assignments)
- #+END_SRC
-
-** Running Benchmarks
-
-#+BEGIN_SRC python :session :eval never-export :results output silent :exports none
-def run_benchmark(model, const_assignments, initial_partition, disableopts, nofunctrans):
- stats = run_all(model, const_assignments, initial_partition, disableopts, nofunctrans)
- valmari = run_all_valmari(model, const_assignments, initial_partition)
- return [model, const_assignments, initial_partition,
- stats['states'],
- stats['edges'],
- stats['explicit-states'],
- stats['initial-partition-size'],
- stats['final-partition-size'],
- stats['explicit-final-partition-size'],
- stats['size1-skipped'],
- stats['overall-duration'],
- stats['parse-duration'],
- stats['initialize-duration'],
- stats['refine-duration'],
- stats['algorithm-duration'],
- valmari[0],
- valmari[1]]
-#+END_SRC
-
-#+NAME: bench
-#+BEGIN_SRC python :session :eval no-export :var table="" :colnames no :hlines yes
- [['Model', 'Consts', 'Partition', "States", "Edges", "Sort 0", "I", "Q", 'Q_0', 'Opts', 't(s)', 't_p(s)', 't_i(s)', 't_r(s)', 't_a(s)', 't_valmari(s)', 'Q_0v'], None] + \
- list(map(lambda t: run_benchmark(t[0], t[1], t[2], False, False), table[2:]))
-#+END_SRC
-
-#+NAME: bench_noopt
-#+BEGIN_SRC python :session :eval no-export :var table="" :colnames no :hlines yes
- [['Model', 'Consts', 'Partition', "States", "Edges", "Sort 0", "I", "Q", 'Q_0', 'Opts', 't(s)', 't_p(s)', 't_i(s)', 't_r(s)', 't_a(s)', 't_valmari(s)', 'Q_0v'], None] + \
- list(map(lambda t: run_benchmark(t[0], t[1], t[2], True, False), table[2:]))
-#+END_SRC
-
-#+NAME: bench_nofunctrans
-#+BEGIN_SRC python :session :eval no-export :var table="" :colnames no :hlines yes
- [['Model', 'Consts', 'Partition', "States", "Edges", "Sort 0", "I", "Q", 'Q_0', 'Opts', 't(s)', 't_p(s)', 't_i(s)', 't_r(s)', 't_a(s)', 't_valmari(s)', 'Q_0v'], None] + \
- list(map(lambda t: run_benchmark(t[0], t[1], t[2], False, True), table[2:]))
-#+END_SRC
-
-#+NAME: bench_sloooow
-#+BEGIN_SRC python :session :eval no-export :var table="" :colnames no :hlines yes
- [['Model', 'Consts', 'Partition', "States", "Edges", "Sort 0", "I", "Q", 'Q_0', 'Opts', 't(s)', 't_p(s)', 't_i(s)', 't_r(s)', 't_a(s)', 't_valmari(s)', 'Q_0v'], None] + \
- list(map(lambda t: run_benchmark(t[0], t[1], t[2], True, True), table[2:]))
-#+END_SRC
-
-* Benchmarks
-** Bounded Retransmission Protocol
-
- #+BEGIN_QUOTE
- This case study is based on the bounded retransmission protocol (BRP) [HSV94], a variant of the alternating bit protocol.
-
- Its parameters are:
-
- N = number of chunks in a file
- MAX = maximum number of retransmissions
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/brp.php
- #+END_QUOTE
-
- Benchmarks:
-
- #+NAME: brp_benchmarks
- | Model | Constants | Partition |
- |-------+------------+-----------|
- | brp | N=16,MAX=2 | s |
- | brp | N=16,MAX=3 | s |
- | brp | N=16,MAX=4 | s |
- | brp | N=16,MAX=5 | s |
- | brp | N=32,MAX=2 | s |
- | brp | N=32,MAX=3 | s |
- | brp | N=32,MAX=4 | s |
- | brp | N=32,MAX=5 | s |
- | brp | N=64,MAX=2 | s |
- | brp | N=64,MAX=3 | s |
- | brp | N=64,MAX=4 | s |
- | brp | N=64,MAX=5 | s |
-
- Results:
-
- #+CALL: bench(brp_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------+------------+-----------+--------+-------+--------+---+------+------+------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | brp | N=16,MAX=2 | s | 677 | 867 | 677 | 7 | 329 | 329 | 52 | 0.0096711520 | 0.0055319890 | 0.0013735710 | 0.0018519030 | 0.0033773790 | 0.002082710 | NA |
- | brp | N=16,MAX=3 | s | 886 | 1155 | 886 | 7 | 442 | 442 | 88 | 0.0116424020 | 0.0063141920 | 0.0018566350 | 0.0022196060 | 0.0044309990 | 0.002534837 | NA |
- | brp | N=16,MAX=4 | s | 1095 | 1443 | 1095 | 7 | 555 | 555 | 120 | 0.0156490230 | 0.0087450750 | 0.0019053060 | 0.0034745740 | 0.0057855190 | 0.002608192 | NA |
- | brp | N=16,MAX=5 | s | 1304 | 1731 | 1304 | 7 | 668 | 668 | 152 | 0.0187181940 | 0.0099388020 | 0.0030763620 | 0.0035504570 | 0.0071181880 | 0.002846267 | NA |
- | brp | N=32,MAX=2 | s | 1349 | 1731 | 1349 | 7 | 649 | 649 | 100 | 0.0187313390 | 0.0102275120 | 0.0032966890 | 0.0031392650 | 0.0066961280 | 0.002826721 | NA |
- | brp | N=32,MAX=3 | s | 1766 | 2307 | 1766 | 7 | 874 | 874 | 167 | 0.0253537910 | 0.0125933750 | 0.0034613680 | 0.0057390080 | 0.0095412560 | 0.003196347 | NA |
- | brp | N=32,MAX=4 | s | 2183 | 2883 | 2183 | 7 | 1099 | 1099 | 233 | 0.0329036780 | 0.0169290770 | 0.0044749150 | 0.0082289000 | 0.0133349350 | 0.003782915 | NA |
- | brp | N=32,MAX=5 | s | 2600 | 3459 | 2600 | 7 | 1324 | 1324 | 295 | 0.0408756860 | 0.0207791250 | 0.0071231710 | 0.0096075050 | 0.0175274810 | 0.004091965 | NA |
- | brp | N=64,MAX=2 | s | 2693 | 3459 | 2693 | 7 | 1289 | 1289 | 196 | 0.0379403510 | 0.0196948050 | 0.0067873590 | 0.0079025110 | 0.0157109670 | 0.004156036 | NA |
- | brp | N=64,MAX=3 | s | 3526 | 4611 | 3526 | 7 | 1738 | 1738 | 328 | 0.0551088630 | 0.0282033910 | 0.0077104360 | 0.0135217460 | 0.0224298520 | 0.004877520 | NA |
- | brp | N=64,MAX=4 | s | 4359 | 5763 | 4359 | 7 | 2187 | 2187 | 456 | 0.0679899860 | 0.0338266070 | 0.0121471660 | 0.0163239430 | 0.0298915000 | 0.005788835 | NA |
- | brp | N=64,MAX=5 | s | 5192 | 6915 | 5192 | 7 | 2636 | 2636 | 583 | 0.0840906920 | 0.0399630440 | 0.0158103130 | 0.0209396970 | 0.0387673500 | 0.116052312 | NA |
-
-** Cell in Wireless Communication Network
-
- #+BEGIN_QUOTE
- This case study is based on a single cell in a wireless communication network. It is taken from [HMPT00].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/cell.php
- #+END_QUOTE
-
- Benchmarks:
-
- There is only a single state variable ~s~. If we base our initial partition
- on that, we distinguish all states immediately.
-
- #+NAME: cell_benchmarks
- | Model | Constants | Partition |
- |-------+-------------+-----------|
- | cell | N=50,T=0.5 | |
- | cell | N=100,T=0.5 | |
- | cell | N=200,T=0.5 | |
- | cell | N=300,T=0.5 | |
- | cell | N=400,T=0.5 | |
- | cell | N=500,T=0.5 | |
-
- Results:
-
- #+CALL: bench(cell_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------+-------------+-----------+--------+-------+--------+-----+-----+-----+------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | cell | N=50,T=0.5 | | 51 | 100 | 51 | 50 | 51 | 51 | 98 | 0.0011218680 | 0.0009294180 | 0.0000494130 | 0.0000435180 | 0.0001277970 | 0.001563127 | NA |
- | cell | N=100,T=0.5 | | 101 | 200 | 101 | 80 | 101 | 101 | 178 | 0.0015188440 | 0.0010590700 | 0.0003003410 | 0.0000425270 | 0.0003747630 | 0.001673470 | NA |
- | cell | N=200,T=0.5 | | 201 | 400 | 201 | 160 | 201 | 201 | 358 | 0.0021876160 | 0.0016676340 | 0.0002400200 | 0.0000813670 | 0.0003676030 | 0.001733180 | NA |
- | cell | N=300,T=0.5 | | 301 | 600 | 301 | 251 | 301 | 301 | 549 | 0.0029263770 | 0.0021101180 | 0.0003710050 | 0.0000992930 | 0.0005347260 | 0.002071309 | NA |
- | cell | N=400,T=0.5 | | 401 | 800 | 401 | 351 | 401 | 401 | 749 | 0.0042151170 | 0.0030344660 | 0.0005944010 | 0.0001019410 | 0.0008068710 | 0.002196061 | NA |
- | cell | N=500,T=0.5 | | 501 | 1000 | 501 | 451 | 501 | 501 | 949 | 0.0043326070 | 0.0029545140 | 0.0007268790 | 0.0001279380 | 0.0009650510 | 0.002397520 | NA |
-
-** Dependable Cluster
-
- #+BEGIN_QUOTE
- This case study is based on a dependable cluster of workstations, taken from [HHK00].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/cluster.php
- #+END_QUOTE
-
- Benchmarks:
-
- #+NAME: cluster_benchmarks
- | Model | Constants | Partition |
- |---------+-----------+-----------|
- | cluster | N=4,T=10 | |
- | cluster | N=8,T=10 | |
- | cluster | N=16,T=10 | |
- | cluster | N=32,T=10 | |
-
- Results:
-
- #+CALL: bench(cluster_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |---------+-----------+-----------+--------+--------+--------+------+-------+-------+-------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | cluster | N=4,T=10 | | 820 | 3616 | 820 | 227 | 565 | 565 | 1058 | 0.0133275730 | 0.0077306660 | 0.0021054550 | 0.0023376670 | 0.0046840240 | 0.003796554 | NA |
- | cluster | N=8,T=10 | | 2772 | 12832 | 2772 | 453 | 1917 | 1917 | 3867 | 0.0482514870 | 0.0252573650 | 0.0067393290 | 0.0122252760 | 0.0203239000 | 0.008798572 | NA |
- | cluster | N=16,T=10 | | 10132 | 48160 | 10132 | 953 | 7021 | 7021 | 15796 | 0.2476780700 | 0.1366680710 | 0.0286580230 | 0.0623478880 | 0.0968610530 | 0.163654355 | NA |
- | cluster | N=32,T=10 | | 38676 | 186400 | 38676 | 2062 | 26829 | 26829 | 66228 | 0.9490274650 | 0.4146188670 | 0.1122056580 | 0.3070002710 | 0.4420124300 | 0.157405829 | NA |
-
-** Shared coin protocol
-
- #+BEGIN_QUOTE
- This case study is based on the shared coin protocol from the randomised consensus algorithm of Aspnes and Herlihy [AH90].
- For more information, see: http://www.prismmodelchecker.org/casestudies/consensus_prism.php
- #+END_QUOTE
-
- We could only test up to coin4, as coin6 already uses all available RAM to
- convert the model.
-
- Benchmarks:
-
- #+NAME: consensus_benchmarks
- | Model | Constants | Partition |
- |-------+--------------+-----------------|
- | coin | N=2,K=2,k=10 | pc1,pc2 |
- | coin | N=2,K=4,k=10 | pc1,pc2 |
- | coin | N=4,K=2,k=10 | pc1,pc2,pc3,pc4 |
- | coin | N=4,K=4,k=10 | pc1,pc2,pc3,pc4 |
-
- Results:
-
- #+CALL: bench(consensus_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------+--------------+-----------------+--------+--------+--------+-----+-------+-------+------+--------------+--------------+--------------+--------------+--------------+--------------+-------|
- | coin | N=2,K=2,k=10 | pc1,pc2 | 672 | 892 | 272 | 18 | 231 | 95 | 4 | 0.0087244350 | 0.0045174220 | 0.0013927310 | 0.0020353580 | 0.0035664660 | 0.001915857 | 95 |
- | coin | N=2,K=4,k=10 | pc1,pc2 | 1312 | 1756 | 528 | 18 | 439 | 183 | 4 | 0.0168299880 | 0.0088780660 | 0.0027004320 | 0.0039363490 | 0.0071328000 | 0.002738838 | 183 |
- | coin | N=4,K=2,k=10 | pc1,pc2,pc3,pc4 | 83200 | 135776 | 22656 | 260 | 18522 | 5392 | 48 | 1.9382094360 | 0.6608932970 | 0.5348946720 | 0.5962997930 | 1.1876570350 | 0.113754275 | 5392 |
- | coin | N=4,K=4,k=10 | pc1,pc2,pc3,pc4 | 158976 | 260192 | 43136 | 260 | 35482 | 10352 | 48 | 3.8120987810 | 1.2882729670 | 0.9272679960 | 1.3343738850 | 2.3771375450 | 0.200260367 | 10352 |
-
-** IEEE 802.3 CSMA/CD
-
- #+BEGIN_QUOTE
- This case study concerns the IEEE 802.3 CSMA/CD (Carrier Sense, Multiple Access with Collision Detection) protocol
-
- model files csmaN_K.nm
- property file: csma.pctl
-
- where N is the number of stations and K is the maximum backoff
-
- For more information on the probabilistic timed automata see: http://www.prismmodelchecker.org/casestudies/csma.php
-
- The PRISM model uses the integer semantics given in [KNPS06].
- #+END_QUOTE
-
- Benchmarks:
-
- We partition on the state of the bus ~b~.
-
- #+NAME: csma_benchmarks
- | Model | Constants | Partition |
- |-------+-------------+-----------|
- | csma | N=2,K=2,k=1 | b |
- | csma | N=2,K=4,k=1 | b |
- | csma | N=2,K=6,k=1 | b |
- | csma | N=3,K=2,k=1 | b |
- | csma | N=4,K=2,k=1 | b |
-
- Results:
-
- #+CALL: bench(csma_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------+-------------+-----------+---------+---------+--------+---+-------+------+------+---------------+---------------+--------------+---------------+---------------+--------------+------|
- | csma | N=2,K=2,k=1 | b | 2092 | 2336 | 1038 | 5 | 470 | 233 | 19 | 0.0271229570 | 0.0134353980 | 0.0055199520 | 0.0059881320 | 0.0120362210 | 0.003081025 | 233 |
- | csma | N=2,K=4,k=1 | b | 15946 | 18582 | 7958 | 5 | 1992 | 997 | 109 | 0.2394990420 | 0.1079331770 | 0.0432150540 | 0.0709119940 | 0.1202429350 | 0.013782941 | 997 |
- | csma | N=2,K=6,k=1 | b | 133506 | 159860 | 66718 | 5 | 14829 | 7436 | 487 | 2.9227050260 | 0.9680199220 | 0.5949931450 | 0.9674891040 | 1.6568496520 | 0.128931912 | 7436 |
- | csma | N=3,K=2,k=1 | b | 75306 | 94318 | 36850 | 6 | 4686 | 2281 | 20 | 1.3873632270 | 0.5463126360 | 0.2537259070 | 0.4778206100 | 0.7738012150 | 0.090490568 | 2281 |
- | csma | N=4,K=2,k=1 | b | 1587466 | 2152572 | 761962 | 7 | 21024 | 9830 | 27 | 50.1628253620 | 16.9687094990 | 7.5975967340 | 21.3369999780 | 31.0321613660 | 2.081963159 | 9830 |
-
-** Dice
-
- This models a 6-sided dice using fair coins. Partitioning on ~s~ (internal
- model state) collapses final dice results and distinguishes all other states.
- Partitioning on ~d~ (the dice result) distinguishes all states, which just
- shows that no internal state is redundant.
-
- Benchmarks:
-
- #+NAME: dice_benchmarks
- | Model | Constants | Partition |
- |----------------+-----------+-----------|
- | dice | | s |
- | two_dice_knuth | | s |
- | two_dice | | s1,s2 |
-
- Results:
-
- #+CALL: bench(dice_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |----------------+--------+-----------+--------+-------+--------+----+-----+-----+------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | dice | | s | 13 | 20 | 13 | 8 | 8 | 8 | 8 | 0.0006871810 | 0.0005694540 | 0.0000342090 | 0.0000119660 | 0.0000621000 | 0.001404938 | NA |
- | two_dice_knuth | | s | 45 | 79 | 45 | 35 | 35 | 35 | 35 | 0.0011354970 | 0.0009655350 | 0.0000685430 | 0.0000131480 | 0.0000985090 | 0.001483642 | NA |
- | two_dice | | s1,s2 | 423 | 690 | 169 | 66 | 160 | 64 | 203 | 0.0050796840 | 0.0033219290 | 0.0006923820 | 0.0005149300 | 0.0014069710 | 0.001946452 | 64 |
-
-** Dining cryptographers
-
- #+BEGIN_QUOTE
- This case study is based on the well known dining cryptographers problem [Cha88].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/dining_crypt.php
- #+END_QUOTE
-
- Benchmarks:
-
- #+NAME: dining_crypt_benchmarks
- | Model | Constants | Partition |
- |--------------+-----------+-----------|
- | dining_crypt | N=3 | pay |
- | dining_crypt | N=4 | pay |
- | dining_crypt | N=5 | pay |
- | dining_crypt | N=6 | pay |
- | dining_crypt | N=7 | pay |
-
- Results:
-
- #+CALL: bench(dining_crypt_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |--------------+--------+-----------+---------+---------+--------+----+------+------+------+---------------+---------------+---------------+---------------+---------------+--------------+------|
- | dining_crypt | N=3 | pay | 1000 | 1396 | 380 | 7 | 96 | 36 | 16 | 0.0115788870 | 0.0065676890 | 0.0026811490 | 0.0015388800 | 0.0044264420 | 0.002342207 | 36 |
- | dining_crypt | N=4 | pay | 6705 | 10260 | 2165 | 9 | 335 | 115 | 55 | 0.0942949780 | 0.0439388580 | 0.0230355970 | 0.0213027110 | 0.0471209270 | 0.006517673 | 115 |
- | dining_crypt | N=5 | pay | 42552 | 69474 | 11850 | 11 | 1014 | 318 | 84 | 0.8393567260 | 0.3363783460 | 0.2121007970 | 0.2415892790 | 0.4793911600 | 0.040772024 | 318 |
- | dining_crypt | N=6 | pay | 258349 | 442106 | 63063 | 13 | 2842 | 805 | 119 | 5.7126363640 | 2.0802571770 | 1.1695632410 | 1.7726151170 | 3.4571072280 | 0.260343728 | 805 |
- | dining_crypt | N=7 | pay | 1514800 | 2685512 | 328760 | 15 | 7536 | 1928 | 160 | 42.2796849430 | 14.7078751920 | 11.5979644660 | 12.3266122930 | 26.0055200610 | 1.656931802 | 1928 |
-
-** Embedded Control System
-
- #+BEGIN_QUOTE
- This case study models a simple embedded system, as described in [KNP04c].
- It is closely based on the example of [MCT94].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/embedded.php
- #+END_QUOTE
-
- This uses the model [[/embedded.sm/]]
-
- Benchmarks:
-
- #+NAME: embedded_benchmarks
- | Model | Constants | Partition |
- |----------+--------------------+-----------|
- | embedded | MAX_COUNT=2,T=12 | i,a,m |
- | embedded | MAX_COUNT=2,T=14 | i,a,m |
- | embedded | MAX_COUNT=2,T=0 | i,a,m |
- | embedded | MAX_COUNT=60,T=0 | i,a,m |
- | embedded | MAX_COUNT=70,T=0 | i,a,m |
- | embedded | MAX_COUNT=100,T=0 | i,a,m |
- | embedded | MAX_COUNT=500,T=0 | i,a,m |
- | embedded | MAX_COUNT=1000,T=0 | i,a,m |
-
- Results:
-
- #+CALL: bench(embedded_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |----------+--------------------+-----------+--------+---------+--------+-----+-------+-------+--------+---------------+---------------+--------------+---------------+---------------+--------------+------|
- | embedded | MAX_COUNT=2,T=12 | i,a,m | 3478 | 14639 | 3478 | 157 | 568 | 568 | 725 | 0.1000065050 | 0.0548361140 | 0.0136441340 | 0.0261467390 | 0.0421925780 | 0.013208914 | NA |
- | embedded | MAX_COUNT=2,T=14 | i,a,m | 3478 | 14639 | 3478 | 157 | 568 | 568 | 725 | 0.0972915520 | 0.0517521910 | 0.0137723260 | 0.0263733620 | 0.0425782980 | 0.013174102 | NA |
- | embedded | MAX_COUNT=2,T=0 | i,a,m | 3478 | 14639 | 3478 | 157 | 568 | 568 | 725 | 0.0966324330 | 0.0512371760 | 0.0136189920 | 0.0262566010 | 0.0423206290 | 0.013252364 | NA |
- | embedded | MAX_COUNT=60,T=0 | i,a,m | 52488 | 221525 | 52488 | 157 | 5556 | 5556 | 9948 | 1.8095733020 | 0.8240218270 | 0.3609584160 | 0.4951166810 | 0.9139563520 | 0.197722292 | NA |
- | embedded | MAX_COUNT=70,T=0 | i,a,m | 60938 | 257195 | 60938 | 157 | 6416 | 6416 | 11548 | 2.1071708480 | 0.9535734410 | 0.4265075780 | 0.5761428870 | 1.0710210680 | 0.220429190 | NA |
- | embedded | MAX_COUNT=100,T=0 | i,a,m | 86288 | 364205 | 86288 | 157 | 8996 | 8996 | 16348 | 3.0536448800 | 1.3787377890 | 0.4321661420 | 0.8209543500 | 1.5314433370 | 0.311546032 | NA |
- | embedded | MAX_COUNT=500,T=0 | i,a,m | 424288 | 1791005 | 424288 | 157 | 43396 | 43396 | 80348 | 18.8280785360 | 7.4157094000 | 3.7931441200 | 5.0072190250 | 9.4383797080 | 1.605090640 | NA |
- | embedded | MAX_COUNT=1000,T=0 | i,a,m | 846788 | 3574505 | 846788 | 157 | 86396 | 86396 | 160348 | 39.3540056840 | 17.4606916590 | 7.9101838110 | 10.6530826300 | 20.0306799400 | 3.271844717 | NA |
-
-
- The same results without 1-elem-optimization:
-
- #+CALL: bench_noopt(embedded_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |----------+--------------------+-----------+--------+---------+--------+-----+-------+-------+------+---------------+---------------+--------------+---------------+---------------+--------------+------|
- | embedded | MAX_COUNT=2,T=12 | i,a,m | 3478 | 14639 | 3478 | 157 | 568 | 568 | 0 | 0.1001943700 | 0.0524606130 | 0.0135043360 | 0.0274145950 | 0.0432767850 | 0.013087436 | NA |
- | embedded | MAX_COUNT=2,T=14 | i,a,m | 3478 | 14639 | 3478 | 157 | 568 | 568 | 0 | 0.0996656840 | 0.0515371400 | 0.0137546470 | 0.0275381250 | 0.0436520370 | 0.013195463 | NA |
- | embedded | MAX_COUNT=2,T=0 | i,a,m | 3478 | 14639 | 3478 | 157 | 568 | 568 | 0 | 0.0994936990 | 0.0515434150 | 0.0137538940 | 0.0273789150 | 0.0434910870 | 0.013170084 | NA |
- | embedded | MAX_COUNT=60,T=0 | i,a,m | 52488 | 221525 | 52488 | 157 | 5556 | 5556 | 0 | 1.8221398340 | 0.8219622070 | 0.3597875990 | 0.5115648390 | 0.9292901920 | 0.190747923 | NA |
- | embedded | MAX_COUNT=70,T=0 | i,a,m | 60938 | 257195 | 60938 | 157 | 6416 | 6416 | 0 | 2.1222394910 | 0.9547011360 | 0.4222102910 | 0.5932027720 | 1.0840538490 | 0.227079494 | NA |
- | embedded | MAX_COUNT=100,T=0 | i,a,m | 86288 | 364205 | 86288 | 157 | 8996 | 8996 | 0 | 3.0582449950 | 1.3704334160 | 0.4266735130 | 0.8404935300 | 1.5447937590 | 0.310357914 | NA |
- | embedded | MAX_COUNT=500,T=0 | i,a,m | 424288 | 1791005 | 424288 | 157 | 43396 | 43396 | 0 | 18.8984776070 | 7.3305746790 | 3.7833043010 | 5.1593155550 | 9.5775605760 | 1.609142877 | NA |
- | embedded | MAX_COUNT=1000,T=0 | i,a,m | 846788 | 3574505 | 846788 | 157 | 86396 | 86396 | 0 | 39.2988154120 | 17.3241690020 | 7.8508650210 | 10.8467155700 | 20.1387386290 | 3.390845831 | NA |
-
-** Firewire
-
- #+BEGIN_QUOTE
- This case study concerns the Tree Identify Protocol of the IEEE 1394 High Performance Serial Bus (called ``FireWire'').
-
- These are MDP models, manually constructed from probabilistic timed automaton (PTA) models,
- using the "digital clocks" semantics [KNPS06]. You can also find the PTA models, in the directory ../pta/firewire.
-
- We consider the following probabilistic timed automata models of the root contention part of the
- tree identify protocol, which are based on probabilistic I/O automata models presented in [SV99].
-
- impl: which consists of the parallel composition of two nodes (Node1 and Node2),
- and two communication channels (Wire12 for messages from Node1 to Node2,
- and Wire21 for messages from Node2 to Node1) and corresponds to the
- system Impl of [SV99].
-
- abst: which is represented by a single probabilistic timed automaton and is an abstraction of Impl
- based on the the probabilistic I/O automaton I1 of [SV99].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/firewire.php
- #+END_QUOTE
-
- Benchmarks:
-
- For the "abst" benchmarks, we partition on the local state state of the
- protocol ~s~. For the "impl" benchmarks we partition on the local state of
- wire 12 ~w12~.
-
- #+NAME: firewire_benchmarks
- | Model | Constants | Partition |
- |------------------------+--------------------------------+-----------|
- | firewire_abst_firewire | delay=36,fast=0.5 | s |
- | firewire_abst_firewire | delay=3,fast=0.2:0.1:0.8 | s |
- | firewire_abst_firewire | delay=36,fast=0.2:0.1:0.8 | s |
- | firewire_abst_deadline | deadline=200,delay=36,fast=0.5 | s |
- | firewire_abst_deadline | deadline=300,delay=36,fast=0.5 | s |
- | firewire_abst_deadline | deadline=400,delay=36,fast=0.5 | s |
- | firewire_abst_deadline | deadline=500,delay=36,fast=0.5 | s |
- | firewire_abst_deadline | deadline=600,delay=36,fast=0.5 | s |
- | firewire_impl_firewire | delay=36,fast=0.5 | w12 |
- | firewire_impl_firewire | delay=3,fast=0.2:0.1:0.8 | w12 |
- | firewire_impl_firewire | delay=36,fast=0.2:0.1:0.8 | w12 |
- | firewire_impl_deadline | deadline=200,delay=3,fast=0.5 | w12 |
- | firewire_impl_deadline | deadline=300,delay=3,fast=0.5 | w12 |
- | firewire_impl_deadline | deadline=400,delay=3,fast=0.5 | w12 |
-
- Results:
-
- #+CALL: bench(firewire_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |------------------------+--------------------------------+-----------+---------+---------+--------+----+--------+--------+--------+---------------+--------------+--------------+---------------+---------------+--------------+--------|
- | firewire_abst_firewire | delay=36,fast=0.5 | s | 1965 | 2600 | 776 | 13 | 1623 | 776 | 183 | 0.0252118430 | 0.0121053230 | 0.0057877890 | 0.0053013720 | 0.0119495800 | 0.003340243 | 776 |
- | firewire_abst_firewire | delay=3,fast=0.2:0.1:0.8 | s | 1305 | 1412 | 611 | 13 | 1227 | 611 | 50 | 0.0151854790 | 0.0077784750 | 0.0025718150 | 0.0036091760 | 0.0065828230 | 0.002662910 | 611 |
- | firewire_abst_firewire | delay=36,fast=0.2:0.1:0.8 | s | 1965 | 2600 | 776 | 13 | 1623 | 776 | 183 | 0.0241420600 | 0.0113885000 | 0.0054158060 | 0.0052211330 | 0.0116302240 | 0.003275894 | 776 |
- | firewire_abst_deadline | deadline=200,delay=36,fast=0.5 | s | 163757 | 209250 | 67901 | 15 | 35260 | 15848 | 5929 | 3.8682574500 | 1.3507531240 | 1.0458554720 | 1.1846375430 | 2.3532160650 | 0.176546731 | 15848 |
- | firewire_abst_deadline | deadline=300,delay=36,fast=0.5 | s | 351965 | 458048 | 142810 | 15 | 136910 | 63708 | 25835 | 9.1705285110 | 2.9618801350 | 2.0662920860 | 3.3795052140 | 5.7368827450 | 0.492995413 | 63708 |
- | firewire_abst_deadline | deadline=400,delay=36,fast=0.5 | s | 548465 | 718048 | 220410 | 15 | 231876 | 107370 | 63243 | 14.5446419200 | 4.6610660740 | 3.1752422440 | 5.4435062210 | 9.0879450400 | 0.816190931 | 107370 |
- | firewire_abst_deadline | deadline=500,delay=36,fast=0.5 | s | 744965 | 978048 | 298010 | 15 | 327376 | 151270 | 94043 | 19.9813400310 | 6.4153091910 | 4.2919166960 | 7.5645346800 | 12.4521966490 | 1.263053838 | 151270 |
- | firewire_abst_deadline | deadline=600,delay=36,fast=0.5 | s | 941465 | 1238048 | 375610 | 15 | 422876 | 195170 | 124843 | 27.0856920960 | 8.6422967040 | 5.7980549930 | 10.2229868990 | 16.8033036870 | 1.517745482 | 195170 |
- | firewire_impl_firewire | delay=36,fast=0.5 | w12 | 691024 | 960548 | 212268 | 8 | 131382 | 62836 | 29632 | 20.3819706330 | 5.7341866800 | 3.7580567140 | 9.4194338870 | 13.8074737560 | 0.968866380 | 62836 |
- | firewire_impl_firewire | delay=3,fast=0.2:0.1:0.8 | w12 | 9612 | 11104 | 4093 | 8 | 7362 | 3631 | 553 | 0.1694758160 | 0.0870325480 | 0.0294502430 | 0.0417651470 | 0.0745785150 | 0.123132247 | 3631 |
- | firewire_impl_firewire | delay=36,fast=0.2:0.1:0.8 | w12 | 691024 | 960548 | 212268 | 8 | 131382 | 62836 | 29633 | 20.3538908920 | 5.7030014760 | 3.7557613730 | 9.4185034920 | 13.8091226960 | 0.965448462 | 62836 |
- | firewire_impl_deadline | deadline=200,delay=3,fast=0.5 | w12 | 192016 | 224278 | 80980 | 8 | 4021 | 1967 | 264 | 3.4042976910 | 1.5151843100 | 0.9815302800 | 0.6092578520 | 1.7277678430 | 0.151578484 | 1967 |
- | firewire_impl_deadline | deadline=300,delay=3,fast=0.5 | w12 | 497567 | 572016 | 213805 | 8 | 10432 | 5133 | 100 | 9.8533852310 | 4.2797769220 | 2.3338217920 | 2.4103991290 | 5.0971449630 | 0.407009032 | 5133 |
- | firewire_impl_deadline | deadline=400,delay=3,fast=0.5 | w12 | 1020411 | 1180934 | 434364 | 8 | 240169 | 118151 | 16363 | 22.1560649610 | 8.7138424910 | 4.8081857930 | 6.3958608920 | 12.0307396340 | 1.105685453 | 118151 |
-
-** Flexible Manufacturing System
-
- #+BEGIN_QUOTE
- This case study is based on the flexible manufacturing system of [CT93].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/fms.php
- #+END_QUOTE
-
- Bechmarks:
-
- We don't need an explicit initial partition, because the instrinsic partition
- of the coalgebra already leads to a maximally fine final partition.
-
- #+NAME: fms_benchmarks
- | Model | Constants | Partition |
- |-------+-----------+-----------|
- | fms | n=1 | |
- | fms | n=2 | |
- | fms | n=3 | |
- | fms | n=4 | |
- | fms | n=5 | |
- | fms | n=6 | |
-
- Results:
-
- #+CALL: bench(fms_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------+--------+-----------+--------+---------+--------+------+--------+--------+---------+---------------+--------------+--------------+--------------+--------------+--------------+------|
- | fms | n=1 | | 54 | 155 | 54 | 27 | 54 | 54 | 144 | 0.0011190190 | 0.0009196850 | 0.0000526740 | 0.0000540440 | 0.0001321580 | 0.001270014 | NA |
- | fms | n=2 | | 810 | 3699 | 810 | 129 | 810 | 810 | 3607 | 0.0114879660 | 0.0074573830 | 0.0018583150 | 0.0011231400 | 0.0032124940 | 0.004057395 | NA |
- | fms | n=3 | | 6520 | 37394 | 6520 | 376 | 6520 | 6520 | 38537 | 0.1230228560 | 0.0727707760 | 0.0205872780 | 0.0166235890 | 0.0409569130 | 0.163422677 | NA |
- | fms | n=4 | | 35910 | 237120 | 35910 | 815 | 35910 | 35910 | 248896 | 0.8787897430 | 0.4779690720 | 0.1196028640 | 0.1624789370 | 0.3375174170 | 0.213602261 | NA |
- | fms | n=5 | | 152712 | 1111482 | 152712 | 1299 | 152712 | 152712 | 1163734 | 4.9372687240 | 2.4584752490 | 0.6822492420 | 1.1002566890 | 2.0805720360 | 1.214758409 | NA |
- | fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 4369694 | 21.3837698990 | 9.9432312680 | 2.9128952850 | 5.5583560150 | 9.6761859970 | 5.843124489 | NA |
-
-
-*** Comparing with and without optimization
-
-We run a single benchmark multiple times. Once with and once without optimizations.
-
-#+NAME: fms_benchmark_single
-| Model | Constants | Partition |
-|-------+-----------+-----------|
-| fms | n=6 | |
-| fms | n=6 | |
-| fms | n=6 | |
-
-**** With both optimizations
-:PROPERTIES:
-:ID: cdef09e0-b92a-46db-9ed5-be0cde788441
-:END:
-
-#+CALL: bench(fms_benchmark_single)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------+--------+-----------+--------+---------+--------+------+--------+--------+---------+---------------+--------------+--------------+--------------+--------------+--------------+------|
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 4369694 | 21.1115990510 | 9.8550286160 | 2.8687953810 | 5.4316401000 | 9.5064537000 | 5.708263962 | NA |
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 4369694 | 20.9593982310 | 9.7634182720 | 2.8666464320 | 5.4096516480 | 9.4901079220 | 5.613742001 | NA |
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 4369694 | 20.9555704890 | 9.7799698680 | 2.8857570390 | 5.4510723400 | 9.5017096000 | 5.685050507 | NA |
-
-**** Without any optimization
-:PROPERTIES:
-:ID: 8850d72d-8f23-495e-b3af-a06deb25be53
-:END:
-
-#+CALL: bench_sloooow(fms_benchmark_single)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------+--------+-----------+--------+---------+--------+------+--------+--------+------+---------------+--------------+--------------+---------------+---------------+--------------+------|
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 0 | 35.4518703620 | 9.9894868350 | 2.9299455570 | 19.7603050700 | 23.8772346850 | 5.701320777 | NA |
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 0 | 35.3850394920 | 9.8971921010 | 2.9127417480 | 19.7676467020 | 23.8871916030 | 5.798974060 | NA |
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 0 | 35.8300788970 | 9.9196566430 | 2.9461043890 | 20.1929301170 | 24.3299005100 | 5.656880994 | NA |
-
-**** With just functor rewrites
-:PROPERTIES:
-:ID: 966b0dc7-cf00-4594-88ae-80332696166e
-:END:
-
-#+CALL: bench_noopt(fms_benchmark_single)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------+--------+-----------+--------+---------+--------+------+--------+--------+------+---------------+--------------+--------------+---------------+---------------+--------------+------|
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 0 | 35.3792636600 | 9.8840455080 | 2.9389656800 | 19.7589163770 | 23.9099881210 | 5.741194174 | NA |
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 0 | 35.5444664040 | 9.9008855000 | 2.9221486960 | 19.9320237960 | 24.0572613560 | 5.697179509 | NA |
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 0 | 35.5311650510 | 9.9824189200 | 2.9167611140 | 19.8211857990 | 23.9367806740 | 5.675679034 | NA |
-
-**** With just 1-elem optimization
-:PROPERTIES:
-:ID: 7e19eb5e-4009-4bc1-89a3-4e4e5c7f643e
-:END:
-
-#+CALL: bench_nofunctrans(fms_benchmark_single)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------+--------+-----------+--------+---------+--------+------+--------+--------+---------+---------------+--------------+--------------+--------------+--------------+--------------+------|
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 4369694 | 21.3317819830 | 9.9422740590 | 2.9165776070 | 5.5338563370 | 9.6616926620 | 5.676567183 | NA |
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 4369694 | 21.3070570780 | 9.9521240360 | 2.9318476720 | 5.5074560910 | 9.6391451150 | 5.714505330 | NA |
-| fms | n=6 | | 537768 | 4205670 | 537768 | 1766 | 537768 | 537768 | 4369694 | 21.2953219580 | 9.9278855310 | 2.8900129830 | 5.5345639560 | 9.6430584370 | 5.873457181 | NA |
-
-** Kanban
-
- #+BEGIN_QUOTE
- This case study is based on the Kanban flexible manufacturing system of [CT96].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/kanban.php
- #+END_QUOTE
-
- Benchmarks:
-
- No need for an explicit partition, the coalgebra itself partitions pretty well.
-
- #+NAME: kanban_benchmarks
- | Model | Constants | Partition |
- |--------+-----------+-----------|
- | kanban | t=1 | |
- | kanban | t=2 | |
- | kanban | t=3 | |
- | kanban | t=4 | |
-
- Results:
-
- #+CALL: bench(kanban_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |--------+--------+-----------+--------+---------+--------+-----+--------+--------+---------+---------------+--------------+--------------+--------------+---------------+--------------+------|
- | kanban | t=1 | | 160 | 616 | 160 | 63 | 160 | 160 | 584 | 0.0035293670 | 0.0027577500 | 0.0002127660 | 0.0002967780 | 0.0005807870 | 0.001957585 | NA |
- | kanban | t=2 | | 4600 | 28120 | 4600 | 138 | 4600 | 4600 | 31996 | 0.0845046560 | 0.0525366560 | 0.0133163970 | 0.0102044590 | 0.0263515370 | 0.151409559 | NA |
- | kanban | t=3 | | 58400 | 446400 | 58400 | 188 | 58400 | 58400 | 484533 | 1.8091822700 | 0.8656578580 | 0.2273689800 | 0.4913825500 | 0.8316374780 | 0.379954446 | NA |
- | kanban | t=4 | | 454475 | 3979850 | 454475 | 205 | 454475 | 454475 | 4145639 | 20.6806881570 | 9.0696400970 | 2.5312314400 | 7.1832748850 | 10.2826909110 | 5.079570712 | NA |
-
-** Leader Election Protocol
-*** Synchronous
-:PROPERTIES:
-:ID: fcc30abf-3473-429b-ade2-0900e6825042
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on the synchronous leader election protocol of Itai & Rodeh [IR90].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/synchronous_leader.php
- #+END_QUOTE
-
- Benchmarks:
-
- We initially partition on the respective local states of all processes.
-
- #+NAME: leader_sync_benchmarks
- | Model | Constants | Partition |
- |-------------+-------------+----------------|
- | leader_sync | N=3,K=2,L=1 | s1,s2,s3 |
- | leader_sync | N=3,K=3,L=1 | s1,s2,s3 |
- | leader_sync | N=3,K=4,L=1 | s1,s2,s3 |
- | leader_sync | N=4,K=2,L=1 | s1,s2,s3,s4 |
- | leader_sync | N=4,K=3,L=1 | s1,s2,s3,s4 |
- | leader_sync | N=4,K=4,L=1 | s1,s2,s3,s4 |
- | leader_sync | N=5,K=2,L=1 | s1,s2,s3,s4,s5 |
- | leader_sync | N=5,K=3,L=1 | s1,s2,s3,s4,s5 |
- | leader_sync | N=5,K=4,L=1 | s1,s2,s3,s4,s5 |
-
- Results:
-
- #+CALL: bench(leader_sync_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------------+-------------+----------------+--------+-------+--------+---+----+-----+------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | leader_sync | N=3,K=2,L=1 | s1,s2,s3 | 26 | 33 | 26 | 4 | 8 | 8 | 3 | 0.0011261370 | 0.0008873720 | 0.0000668850 | 0.0000975260 | 0.0001830760 | 0.001425685 | NA |
- | leader_sync | N=3,K=3,L=1 | s1,s2,s3 | 69 | 95 | 69 | 4 | 8 | 8 | 4 | 0.0015097720 | 0.0010825010 | 0.0000699380 | 0.0000615120 | 0.0001584650 | 0.001376350 | NA |
- | leader_sync | N=3,K=4,L=1 | s1,s2,s3 | 147 | 210 | 147 | 4 | 8 | 8 | 5 | 0.0022074270 | 0.0018903660 | 0.0001109040 | 0.0000890860 | 0.0002318850 | 0.001483183 | NA |
- | leader_sync | N=4,K=2,L=1 | s1,s2,s3,s4 | 61 | 76 | 61 | 4 | 10 | 10 | 9 | 0.0012056580 | 0.0010085850 | 0.0000633110 | 0.0000579530 | 0.0001401530 | 0.001605176 | NA |
- | leader_sync | N=4,K=3,L=1 | s1,s2,s3,s4 | 274 | 354 | 274 | 4 | 10 | 10 | 22 | 0.0031891820 | 0.0024679180 | 0.0003274110 | 0.0001492080 | 0.0005299830 | 0.001830196 | NA |
- | leader_sync | N=4,K=4,L=1 | s1,s2,s3,s4 | 812 | 1067 | 812 | 4 | 10 | 10 | 41 | 0.0086410250 | 0.0058045520 | 0.0011613200 | 0.0005618120 | 0.0021124430 | 0.002080188 | NA |
- | leader_sync | N=5,K=2,L=1 | s1,s2,s3,s4,s5 | 141 | 172 | 141 | 4 | 12 | 12 | 11 | 0.0020531090 | 0.0016603120 | 0.0001839930 | 0.0000896080 | 0.0003088410 | 0.001703827 | NA |
- | leader_sync | N=5,K=3,L=1 | s1,s2,s3,s4,s5 | 1050 | 1292 | 1050 | 4 | 12 | 12 | 64 | 0.0118074260 | 0.0079930730 | 0.0012989110 | 0.0007590220 | 0.0027172380 | 0.002489031 | NA |
- | leader_sync | N=5,K=4,L=1 | s1,s2,s3,s4,s5 | 4244 | 5267 | 4244 | 4 | 12 | 12 | 125 | 0.0492064960 | 0.0303649000 | 0.0077673220 | 0.0064060210 | 0.0154437500 | 0.004702621 | NA |
-
-*** Asynchronous
-:PROPERTIES:
-:ID: e6b7cd19-fdf8-4c34-8a16-5429102fe2b6
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on the asynchronous leader election protocol of Itai & Rodeh [IR90].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/asynchronous_leader.php
- #+END_QUOTE
-
- Benchmarks:
-
- We initially partition on the respective local states of all processes.
-
- #+NAME: leader_async_benchmarks
- | Model | Constants | Partition |
- |--------------+-----------+----------------|
- | leader_async | N=3,K=1 | s1,s2,s3 |
- | leader_async | N=4,K=1 | s1,s2,s3,s4 |
- | leader_async | N=5,K=1 | s1,s2,s3,s4,s5 |
-
- Results:
-
- #+CALL: bench(leader_async_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |--------------+---------+----------------+--------+--------+--------+-----+-------+------+------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | leader_async | N=3,K=1 | s1,s2,s3 | 937 | 1227 | 364 | 57 | 385 | 171 | 100 | 0.0114401660 | 0.0059803830 | 0.0026568900 | 0.0018486450 | 0.0047440420 | 0.002268829 | 171 |
- | leader_async | N=4,K=1 | s1,s2,s3,s4 | 9424 | 13396 | 3172 | 221 | 2974 | 1169 | 724 | 0.1549121930 | 0.0635229990 | 0.0411555370 | 0.0399712740 | 0.0853628400 | 0.009218520 | 1169 |
- | leader_async | N=5,K=1 | s1,s2,s3,s4,s5 | 92284 | 139350 | 27299 | 868 | 22626 | 7948 | 5064 | 1.9666992650 | 0.7098582570 | 0.5072619060 | 0.5892218320 | 1.1746346370 | 0.121196583 | 7948 |
-
-** Molecules
-:PROPERTIES:
-:ID: 2fc0e60e-721b-47b7-9606-dfb32118fe47
-:END:
-
- #+BEGIN_QUOTE
- This case study models several simple molecular reactions:
-
- - Na + Cl <-> Na+ + Cl-
- - K + Na + 2Cl <-> K+ + Na- + 2Cl-
- - Mg + 2Cl <-> Mg+2 + 2Cl-
-
- These are taken from Ehud Shapiro's lecture notes on Biomolecular Processes as Concurrent Computation [Sha].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/molecules.php
- #+END_QUOTE
-
- Benchmarks:
-
- We partition on the current number of ~Na~ or ~Mg~ atoms at the current
- state.
-
- #+NAME: molecules_benchmarks
- | Model | Constants | Partition |
- |-------+---------------------------------------+-----------|
- | nacl | N1=10,N2=10,T=0:0.001:0.006,i=4 | na |
- | nacl | N1=10,N2=10,T=0:0.001:0.006,i=0 | na |
- | nacl | N1=10,N2=10,T=0,i=0 | na |
- | knacl | N1=10,N2=10,N3=10,T=0:0.001:0.006,i=4 | na |
- | knacl | N1=10,N2=10,N3=10,T=0:0.001:0.006,i=0 | na |
- | knacl | N1=10,N2=10,N3=10,T=0,i=0 | na |
- | mc | N1=10,N2=10,T=0:0.001:0.006,i=4 | mg |
- | mc | N1=10,N2=10,T=0:0.001:0.006,i=0 | mg |
- | mc | N1=10,N2=10,T=0i=0 | mg |
-
- Results:
-
- #+CALL: bench(molecules_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------+---------------------------------------+-----------+--------+-------+--------+----+----+-----+------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | nacl | N1=10,N2=10,T=0:0.001:0.006,i=4 | na | 11 | 20 | 11 | 11 | 11 | 11 | 19 | 0.0006936910 | 0.0005806090 | 0.0000383150 | 0.0000131620 | 0.0000692840 | 0.001467669 | NA |
- | nacl | N1=10,N2=10,T=0:0.001:0.006,i=0 | na | 11 | 20 | 11 | 11 | 11 | 11 | 19 | 0.0007232610 | 0.0006209010 | 0.0000312240 | 0.0000126100 | 0.0000607890 | 0.001385457 | NA |
- | nacl | N1=10,N2=10,T=0,i=0 | na | 11 | 20 | 11 | 11 | 11 | 11 | 19 | 0.0008047070 | 0.0006996310 | 0.0000318730 | 0.0000132890 | 0.0000628290 | 0.001539412 | NA |
- | knacl | N1=10,N2=10,N3=10,T=0:0.001:0.006,i=4 | na | 66 | 220 | 66 | 66 | 66 | 66 | 218 | 0.0018716320 | 0.0016524530 | 0.0000806560 | 0.0000363960 | 0.0001412880 | 0.001596522 | NA |
- | knacl | N1=10,N2=10,N3=10,T=0:0.001:0.006,i=0 | na | 66 | 220 | 66 | 66 | 66 | 66 | 218 | 0.0024822640 | 0.0022006060 | 0.0001195270 | 0.0000290760 | 0.0001879910 | 0.001583101 | NA |
- | knacl | N1=10,N2=10,N3=10,T=0,i=0 | na | 66 | 220 | 66 | 66 | 66 | 66 | 218 | 0.0018328980 | 0.0016366420 | 0.0000831910 | 0.0000205740 | 0.0001278490 | 0.001707427 | NA |
- | mc | N1=10,N2=10,T=0:0.001:0.006,i=4 | mg | 36 | 110 | 36 | 36 | 36 | 36 | 109 | 0.0011844780 | 0.0010416150 | 0.0000503610 | 0.0000144850 | 0.0000809270 | 0.001584403 | NA |
- | mc | N1=10,N2=10,T=0:0.001:0.006,i=0 | mg | 36 | 110 | 36 | 36 | 36 | 36 | 109 | 0.0011972170 | 0.0010458730 | 0.0000496930 | 0.0000144850 | 0.0000803150 | 0.001653878 | NA |
- | mc | N1=10,N2=10,T=0i=0 | mg | 36 | 110 | 36 | 36 | 36 | 36 | 109 | 0.0011372140 | 0.0009994990 | 0.0000512750 | 0.0000140010 | 0.0000815090 | 0.001643996 | NA |
-
-** Mutual Exclusion
-:PROPERTIES:
-:ID: e8ac9d4d-0ed9-4cc2-85ea-359a5261442c
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on Pnueli and Zuck's [PZ86] probabilistic symmetric
- solution to the n-process mutual exclusion problem.
-
- For reference, these are the local states of each process:
- | No: | No: | Description: | State: |
- |-----+-----+--------------------------+--------------|
- | 0 | 0 | idle | uninterested |
- | 1 | 1 | trying | uninterested |
- | 2 | 2 | trying | enter |
- | 3 | 3 | trying | enter |
- | 4 | 7 | trying | high |
- | 5 | 8 | trying | high |
- | 6 | 9 | trying | tie |
- | 7 | 10 | trying | low |
- | 8 | 11 | trying | low |
- | 9 | 12 | trying | tie |
- | 10 | 13 | in critical section | high |
- | 11 | 14 | in critical section | high |
- | 12 | 15 | in critical section | high |
- | 13 | 17 | in critical section | high |
- | 14 | 18 | leaving critical section | admit |
- | 15 | 19 | leaving critical section | admit |
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/mutual.php
- #+END_QUOTE
-
- I don't include ~mutual8~ and ~mutual10~ here, since ~mutual8~ already
- produced an 11 GB transaction file, which my computer doesn't have the memory
- to even convert to CoPaR's input format.
-
- We partition on the state of the first process.
-
- Benchmarks:
-
- #+NAME: mutual_benchmarks
- | Model | Constants | Partition |
- |--------+-----------+-----------|
- | mutual | N=3 | p1 |
- | mutual | N=4 | p1 |
- | mutual | N=5 | p1 |
-
- Results:
-
- #+CALL: bench(mutual_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |--------+--------+-----------+---------+---------+--------+----+---------+--------+---------+---------------+---------------+---------------+---------------+---------------+--------------+--------|
- | mutual | N=3 | p1 | 10636 | 16992 | 2368 | 22 | 7978 | 2220 | 5921 | 0.2019455650 | 0.0703412350 | 0.0424574750 | 0.0769162890 | 0.1238906080 | 0.113024788 | 2220 |
- | mutual | N=4 | p1 | 157184 | 266576 | 27600 | 24 | 118224 | 26682 | 103462 | 4.3537811680 | 1.1411224410 | 0.7291466170 | 2.0216322970 | 3.0423430190 | 0.278805850 | 26682 |
- | mutual | N=5 | p1 | 2130240 | 3751600 | 308800 | 26 | 1604356 | 303608 | 1483271 | 85.9962595640 | 19.3640643730 | 16.7449184410 | 42.2897042540 | 62.1227017060 | 4.923615233 | 303608 |
-
-** Peer-to-Peer Protocol
-:PROPERTIES:
-:ID: 8f53a247-e80e-4b29-b0e9-34bd29aa91ff
-:END:
-
- #+BEGIN_QUOTE
- This case study models a very simple peer-to-peer protocol, loosely based on BitTorrent.
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/peer2peer.php
- #+END_QUOTE
-
- Benchmarks:
-
- The initial partition is derived from the current download process of client1.
-
- #+NAME: peer2peer_benchmarks
- | Model | Constants | Partition |
- |-----------+---------------+-----------------|
- | peer2peer | N=4,K=4,T=1.1 | b11,b12,b13,b14 |
- | peer2peer | N=5,K=4,T=1.1 | b11,b12,b13,b14 |
-
- Results:
-
- #+CALL: bench(peer2peer_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-----------+---------------+-----------------+---------+----------+---------+-----+------+------+------+---------------+---------------+---------------+---------------+---------------+--------------+------|
- | peer2peer | N=4,K=4,T=1.1 | b11,b12,b13,b14 | 65536 | 524289 | 65536 | 143 | 2595 | 2595 | 1423 | 3.7316735590 | 1.6372802810 | 0.6629852480 | 1.1867859750 | 1.9638201450 | 0.261887081 | NA |
- | peer2peer | N=5,K=4,T=1.1 | b11,b12,b13,b14 | 1048576 | 10485761 | 1048576 | 416 | 4575 | 4575 | 1738 | 89.1275273410 | 37.1055019560 | 16.3894437490 | 30.2417840130 | 49.5512363070 | 5.953450908 | NA |
-
-** PEPA
-
- #+BEGIN_QUOTE
- These examples are adapted from those on the PEPA webpage:
-
- http://www.dcs.ed.ac.uk/pepa/examples.html
- #+END_QUOTE
-
- We can't use this currently, as the model isn't in the normal PRISM language
- and requires the special flag ~-importpepa~ to use it. Our python currenlty
- doesn't support that.
-
-** Dining Philosophers
-*** Original
-:PROPERTIES:
-:ID: eb035abb-aa5a-4edd-a193-605fbfb386d5
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on Lehmann and Rabin's solution to the well known dining philosophers problem [LR81].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/phil.php
- #+END_QUOTE
-
- Benchmarks:
-
- For the initial partition, we differentiate on the current state of the first
- philosopher.
-
- Although the ~auto~ file goes to ~N=9~, the memory usage of CoPaR on ~N=6~ is
- already over the limit of my system (16GB).
-
- #+NAME: phil_original_benchmarks
- | Model | Constants | Partition |
- |---------------+-----------+-----------|
- | phil_original | N=3 | p1 |
- | phil_original | N=4 | p1 |
- | phil_original | N=5 | p1 |
-
- Results:
-
- #+CALL: bench(phil_original_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |---------------+--------+-----------+--------+---------+--------+----+--------+-------+--------+---------------+--------------+--------------+---------------+---------------+--------------+-------|
- | phil_original | N=3 | p1 | 4298 | 7038 | 956 | 18 | 3476 | 956 | 2918 | 0.0850968110 | 0.0385582620 | 0.0139462220 | 0.0284026650 | 0.0442237230 | 0.006178703 | 956 |
- | phil_original | N=4 | p1 | 53440 | 92656 | 9440 | 20 | 42227 | 9440 | 40917 | 1.3897395050 | 0.3895630360 | 0.3158503200 | 0.5947657080 | 0.9424930210 | 0.141154620 | 9440 |
- | phil_original | N=5 | p1 | 635298 | 1141830 | 93068 | 22 | 494654 | 93068 | 528298 | 23.2253278090 | 5.6642915180 | 4.6927226990 | 11.3302278460 | 16.7053319470 | 1.346441694 | 93068 |
-
-*** No Fair
-:PROPERTIES:
-:ID: 2d736b3d-a5c8-4485-9065-0724cd79efcd
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on Lehmann and Rabin's solution to the well known
- dining philosophers problem [LR81]. It considers the version presented in
- [DFP04] which removes the need to consider 'fairness' assumptions on the
- scheduling mechanism.
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/phil.php
- #+END_QUOTE
-
- As above, the initial partition is determined by the state of the first
- philosopher.
-
- Benchmarks:
-
- #+NAME: phil_nofair_benchmarks
- | Model | Constants | Partition |
- |-------------+-----------+-----------|
- | phil_nofair | N=3,K=1 | p1 |
- | phil_nofair | N=4,K=1 | p1 |
-
- Results:
-
- #+CALL: bench(phil_nofair_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------------+---------+-----------+--------+-------+--------+----+-------+------+-------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | phil_nofair | N=3,K=1 | p1 | 3650 | 5742 | 956 | 16 | 3023 | 956 | 2683 | 0.0592746560 | 0.0248430880 | 0.0122072230 | 0.0180669160 | 0.0312848240 | 0.005367682 | 956 |
- | phil_nofair | N=4,K=1 | p1 | 44904 | 75584 | 9440 | 18 | 36509 | 9440 | 38332 | 1.1399154410 | 0.3453988980 | 0.2401662920 | 0.4899106610 | 0.7572538220 | 0.113009046 | 9440 |
-
-** Cyclic Server Polling
-:PROPERTIES:
-:ID: 7e4f230d-5ee9-4d00-84bd-1b2a936e2fee
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on a cyclic server polling system, taken from [IT90].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/polling.php
- #+END_QUOTE
-
- Benchmarks:
-
- The initial partition is determined by the current station.
-
- #+NAME: poll_benchmarks
- | Model | Constants | Partition |
- |-------+-----------+-----------|
- | poll | N=2,T=50 | s |
- | poll | N=3,T=50 | s |
- | poll | N=4,T=50 | s |
- | poll | N=5,T=50 | s |
- | poll | N=6,T=50 | s |
- | poll | N=7,T=50 | s |
- | poll | N=8,T=50 | s |
- | poll | N=9,T=50 | s |
-
- Results:
-
- #+CALL: bench(poll_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------+----------+-----------+--------+-------+--------+-----+------+------+-------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | poll | N=2,T=50 | s | 12 | 22 | 12 | 10 | 12 | 12 | 18 | 0.0008483950 | 0.0006841860 | 0.0000355220 | 0.0000492120 | 0.0001176020 | 0.001614123 | NA |
- | poll | N=3,T=50 | s | 36 | 84 | 36 | 21 | 36 | 36 | 59 | 0.0011711260 | 0.0009708210 | 0.0000515500 | 0.0000792950 | 0.0001463870 | 0.001475102 | NA |
- | poll | N=4,T=50 | s | 96 | 272 | 96 | 36 | 96 | 96 | 189 | 0.0022220210 | 0.0017541980 | 0.0002040780 | 0.0001504100 | 0.0003847900 | 0.001608076 | NA |
- | poll | N=5,T=50 | s | 240 | 800 | 240 | 69 | 240 | 240 | 636 | 0.0050187920 | 0.0034426060 | 0.0005370600 | 0.0006597340 | 0.0012748220 | 0.002082991 | NA |
- | poll | N=6,T=50 | s | 576 | 2208 | 576 | 78 | 576 | 576 | 1427 | 0.0134682950 | 0.0085022040 | 0.0014425170 | 0.0023570030 | 0.0044400710 | 0.003906478 | NA |
- | poll | N=7,T=50 | s | 1344 | 5824 | 1344 | 117 | 1344 | 1344 | 4380 | 0.0438199740 | 0.0261821490 | 0.0060602420 | 0.0088203580 | 0.0156700730 | 0.146802595 | NA |
- | poll | N=8,T=50 | s | 3072 | 14848 | 3072 | 136 | 3072 | 3072 | 10300 | 0.0971086310 | 0.0464659790 | 0.0170125600 | 0.0280937530 | 0.0469848680 | 0.149056317 | NA |
- | poll | N=9,T=50 | s | 6912 | 36864 | 6912 | 171 | 6912 | 6912 | 26212 | 0.2728167060 | 0.1187546460 | 0.0438468550 | 0.0901830570 | 0.1413547160 | 0.174681553 | NA |
-
-** Mutual Exclusion à la Rabin
-:PROPERTIES:
-:ID: 754e893c-474b-4a3b-a870-7bb69d4c4a0b
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on Rabin's solution to the well known mutual exclusion problem [Rab82].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/rabin.php
- #+END_QUOTE
-
- Benchmarks:
-
- We partition on the state of the first process initially.
-
- Unfortunately, we can only test N upto 4, since the size of the transaction
- file for N=5 is already 5 GB.
-
- #+NAME: rabin_benchmarks
- | Model | Constants | Partition |
- |-------+-----------+-----------|
- | rabin | N=3 | p1 |
- | rabin | N=4 | p1 |
-
- #+CALL: bench(rabin_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------+--------+-----------+---------+---------+--------+---+--------+--------+------+---------------+---------------+---------------+---------------+---------------+--------------+--------|
- | rabin | N=3 | p1 | 73402 | 183438 | 27766 | 6 | 11348 | 6389 | 0 | 2.2583024290 | 0.7124565070 | 0.3522928110 | 1.0694487250 | 1.4813627410 | 0.130309093 | 6389 |
- | rabin | N=4 | p1 | 1839572 | 4808224 | 668836 | 7 | 240744 | 126402 | 0 | 90.3695585910 | 25.3093680120 | 11.9820889700 | 46.3475014590 | 61.5265400030 | 4.535178263 | 126402 |
-
-** Self Stabilisation
-*** Beauquier
-:PROPERTIES:
-:ID: 94785b23-df89-4daa-a615-ba97a16661a0
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on Beauquier, Gradinariu and Johnen's self-stabilising algorithm [BGJ99].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/self-stabilisation.php
- #+END_QUOTE
-
- Benchmarks:
-
- We partition on the state of the first process.
-
- #+NAME: beauquier_benchmarks
- | Model | Constants | Partition |
- |-----------+-------------------+-----------|
- | beauquier | N=3,K=0,k=0 | d1,p1 |
- | beauquier | N=3,K=0,k=1:2:3 | d1,p1 |
- | beauquier | N=3,K=1:1:100,k=3 | d1,p1 |
-
- Results:
-
- #+CALL: bench(beauquier_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-----------+-------------------+-----------+--------+-------+--------+---+-----+-----+------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | beauquier | N=3,K=0,k=0 | d1,p1 | 160 | 240 | 64 | 7 | 156 | 64 | 70 | 0.0054042780 | 0.0043894410 | 0.0003494090 | 0.0004709010 | 0.0008911640 | 0.001732362 | 64 |
- | beauquier | N=3,K=0,k=1:2:3 | d1,p1 | 160 | 240 | 64 | 7 | 156 | 64 | 70 | 0.0026586000 | 0.0018153220 | 0.0002680470 | 0.0004496140 | 0.0007550450 | 0.001701858 | 64 |
- | beauquier | N=3,K=1:1:100,k=3 | d1,p1 | 160 | 240 | 64 | 7 | 156 | 64 | 70 | 0.0026884210 | 0.0018925630 | 0.0002718390 | 0.0004044380 | 0.0007133480 | 0.001542218 | 64 |
-
-*** Herman
-:PROPERTIES:
-:ID: ab8fc607-28eb-4618-9369-20353be3b7ec
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on Herman's self-stabilising algorithm [Her90].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/self-stabilisation.php
- #+END_QUOTE
-
- Benchmarks:
-
- We partition on the state of the first process.
-
- #+NAME: herman_benchmarks
- | Model | Constants | Partition |
- |--------+--------------------+-----------|
- | herman | N=11,K=0,k=0 | x1 |
- | herman | N=13,K=0,k=0 | x1 |
- | herman | N=15,K=0,k=0 | x1 |
-
- Results:
-
- #+CALL: bench(herman_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |--------+--------------+-----------+--------+----------+--------+---+-------+-------+----------+---------------+---------------+--------------+---------------+---------------+--------------+------|
- | herman | N=11,K=0,k=0 | x1 | 2048 | 177148 | 2048 | 4 | 2048 | 2048 | 271341 | 0.5366140250 | 0.3206644650 | 0.0822131960 | 0.1083287110 | 0.2124922160 | 0.159943487 | NA |
- | herman | N=13,K=0,k=0 | x1 | 8192 | 1594324 | 8192 | 4 | 8192 | 8192 | 2146545 | 4.8963375250 | 2.7911046460 | 0.3606212620 | 1.1401700670 | 2.0782951680 | 1.317832894 | NA |
- | herman | N=15,K=0,k=0 | x1 | 32768 | 14348908 | 32768 | 4 | 32768 | 32768 | 24579335 | 49.8407456710 | 32.0083210790 | 3.4641624420 | 12.3775878660 | 17.6873562750 | 15.226739963 | NA |
-
-*** Israeli and Jalfon
-:PROPERTIES:
-:ID: c6c91d7d-0c6e-41ff-8610-7f1cd020393d
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on Israeli and Jalfon's self-stabilising algorithm [IJ90].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/self-stabilisation.php
- #+END_QUOTE
-
- Benchmarks:
-
- We partition on the state of the first process:
-
- #+NAME: ij_benchmarks
- | Model | Constants | Partition |
- |----------------+-------------------+-----------|
- | israeli-jalfon | N=3,K=0,k=0 | q1 |
- | israeli-jalfon | N=3,K=0,k=1:1:3 | q1 |
- | israeli-jalfon | N=3,K=1:1:100,k=0 | q1 |
-
- Results:
-
- #+CALL: bench(ij_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |----------------+-------------------+-----------+--------+-------+--------+---+----+-----+------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | israeli-jalfon | N=3,K=0,k=0 | q1 | 19 | 33 | 7 | 5 | 14 | 5 | 9 | 0.0012083740 | 0.0010020590 | 0.0000514580 | 0.0000869100 | 0.0001614120 | 0.001514328 | 5 |
- | israeli-jalfon | N=3,K=0,k=1:1:3 | q1 | 19 | 33 | 7 | 5 | 14 | 5 | 9 | 0.0009209260 | 0.0007316150 | 0.0000577930 | 0.0000717430 | 0.0001501250 | 0.001682336 | 5 |
- | israeli-jalfon | N=3,K=1:1:100,k=0 | q1 | 19 | 33 | 7 | 5 | 14 | 5 | 9 | 0.0009256730 | 0.0007310420 | 0.0000577220 | 0.0000682320 | 0.0001451640 | 0.001568259 | 5 |
-
-** Tandem Queueing Network
-:PROPERTIES:
-:ID: f4d75ba1-5bd9-427f-846a-c233037887d8
-:END:
-
- #+BEGIN_QUOTE
- This case study is based on a simple tandem queueing network, taken from [HMKS99].
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/tandem.php
- #+END_QUOTE
-
- Benchmarks:
-
- We partition on the current utilization of the first queue.
-
- #+NAME: tandem_benchmarks
- | Model | Constants | Partition |
- |--------+-----------+-----------|
- | tandem | T=1,c=3 | sc |
- | tandem | T=1,c=7 | sc |
- | tandem | T=1,c=15 | sc |
- | tandem | T=1,c=31 | sc |
- | tandem | T=1,c=63 | sc |
- | tandem | T=1,c=127 | sc |
- | tandem | T=1,c=255 | sc |
-
- Results:
-
- #+CALL: bench(tandem_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |--------+-----------+-----------+--------+--------+--------+------+--------+--------+--------+--------------+--------------+--------------+--------------+--------------+--------------+------|
- | tandem | T=1,c=3 | sc | 28 | 71 | 28 | 14 | 28 | 28 | 67 | 0.0012204660 | 0.0010410840 | 0.0000436570 | 0.0000566860 | 0.0001147310 | 0.001533843 | NA |
- | tandem | T=1,c=7 | sc | 120 | 363 | 120 | 30 | 120 | 120 | 306 | 0.0028063490 | 0.0021379790 | 0.0002532330 | 0.0002302300 | 0.0005184930 | 0.002015983 | NA |
- | tandem | T=1,c=15 | sc | 496 | 1619 | 496 | 62 | 496 | 496 | 1331 | 0.0098958290 | 0.0055752370 | 0.0018006410 | 0.0017059840 | 0.0038092730 | 0.003074152 | NA |
- | tandem | T=1,c=31 | sc | 2016 | 6819 | 2016 | 126 | 2016 | 2016 | 5251 | 0.0469872290 | 0.0229718530 | 0.0080544630 | 0.0129506950 | 0.0220459930 | 0.153096167 | NA |
- | tandem | T=1,c=63 | sc | 8128 | 27971 | 8128 | 254 | 8128 | 8128 | 20771 | 0.2275481120 | 0.1098518430 | 0.0420433780 | 0.0568686420 | 0.1060195480 | 0.157777373 | NA |
- | tandem | T=1,c=127 | sc | 32640 | 113283 | 32640 | 510 | 32640 | 32640 | 82531 | 0.9978470040 | 0.4330151870 | 0.1334547820 | 0.2972327800 | 0.5115283170 | 0.161372538 | NA |
- | tandem | T=1,c=255 | sc | 130816 | 455939 | 130816 | 1022 | 130816 | 130816 | 328931 | 4.8591040570 | 2.0670770960 | 0.9269812650 | 1.3728879970 | 2.4489077610 | 0.444020561 | NA |
-
-** WLAN
-:PROPERTIES:
-:ID: c808971a-8c52-41c3-8de0-42aebe332be5
-:END:
-
- #+BEGIN_QUOTE
- This case study concerns the IEEE 802.11 Wireless LAN
-
- We consider the scenario where two stations trying to send packets simultaneously.
-
- The PRISM model is taken from [KNS02a] using the integer semantics given in [KNS06]. In the model one time unit
- corresponds to 50\mu s and to ensure integer bounds, where neccessary, scaling lower bound constraints down and
- upper bound constraints up to sensure a sound abstraction.
-
- For more information on the probabilistic timed automata see: http://www.prismmodelchecker.org/casestudies/wlan.php
- #+END_QUOTE
-
- Benchmarks:
-
- We initially partition on the state of the medium: Are messages being sent
- and are they transmitted correctly or garbled.
-
- #+NAME: wlan_benchmarks
- | Model | Constants | Partition |
- |-------------------+------------------------------------+-----------|
- | wlan | TRANS_TIME_MAX=10,k=2,K=0 | c1,c2 |
- | wlan | TRANS_TIME_MAX=10,k=2,K=1 | c1,c2 |
- | wlan | TRANS_TIME_MAX=10,k=2,K=2 | c1,c2 |
- | wlan_collide | COL=2,TRANS_TIME_MAX=10,k=2,K=0 | c1,c2 |
- | wlan_collide | COL=2,TRANS_TIME_MAX=10,k=2,K=2 | c1,c2 |
- | wlan_collide | COL=2,TRANS_TIME_MAX=10,k=2,K=3 | c1,c2 |
- | wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 |
- | wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 |
-
- Results:
-
- #+CALL: bench(wlan_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-------------------+------------------------------------+-----------+---------+---------+--------+---+--------+--------+--------+---------------+---------------+--------------+---------------+---------------+--------------+--------|
- | wlan | TRANS_TIME_MAX=10,k=2,K=0 | c1,c2 | 65462 | 94146 | 28480 | 9 | 49602 | 21869 | 24891 | 1.4907680020 | 0.5026076160 | 0.2945986770 | 0.5831547880 | 0.9162903700 | 0.163609992 | 21869 |
- | wlan | TRANS_TIME_MAX=10,k=2,K=1 | c1,c2 | 65462 | 94146 | 28480 | 9 | 49602 | 21869 | 24891 | 1.4916662260 | 0.5080859340 | 0.2890311160 | 0.5838870710 | 0.9118207740 | 0.126401342 | 21869 |
- | wlan | TRANS_TIME_MAX=10,k=2,K=2 | c1,c2 | 65462 | 94146 | 28480 | 9 | 49602 | 21869 | 24891 | 1.4926573050 | 0.5059405690 | 0.2887227640 | 0.5868386100 | 0.9145294070 | 0.142771426 | 21869 |
- | wlan_collide | COL=2,TRANS_TIME_MAX=10,k=2,K=0 | c1,c2 | 65718 | 94452 | 28598 | 9 | 49602 | 21869 | 24886 | 1.5141671190 | 0.5187767530 | 0.2933001550 | 0.5922487230 | 0.9221007800 | 0.152518883 | 21869 |
- | wlan_collide | COL=2,TRANS_TIME_MAX=10,k=2,K=2 | c1,c2 | 65718 | 94452 | 28598 | 9 | 49602 | 21869 | 24886 | 1.4856360770 | 0.5008109760 | 0.2896682780 | 0.5805601160 | 0.9068050890 | 0.141182039 | 21869 |
- | wlan_collide | COL=2,TRANS_TIME_MAX=10,k=2,K=3 | c1,c2 | 65718 | 94452 | 28598 | 9 | 49602 | 21869 | 24886 | 1.4843975830 | 0.4955192090 | 0.2967677220 | 0.5828330410 | 0.9159187970 | 0.154033264 | 21869 |
- | wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 | 582327 | 771088 | 248503 | 9 | 236170 | 107865 | 103088 | 15.3710302540 | 5.1540280990 | 3.1144350580 | 5.6189957580 | 9.2669057860 | 0.937634369 | 107865 |
- | wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 1408676 | 1963522 | 607727 | 9 | 543056 | 243325 | 271999 | 39.7515383150 | 13.3537814010 | 6.1222019780 | 16.2128002410 | 23.8991977920 | 2.543385720 | 243325 |
-
-*** Comparing the 1-elem optimization
-
-We run a single benchmark multiple times. Once with and once without optimizations.
-
-#+NAME: wlan_benchmark_single
-| Model | Constants | Partition |
-|-------------------+------------------------------------+-----------|
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 |
-
-
-**** With optimizations
-:PROPERTIES:
-:ID: a61de2c3-45be-4184-b947-7ddcc41d451f
-:END:
-
-#+CALL: bench(wlan_benchmark_single)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------------------+------------------------------------+-----------+--------+--------+--------+---+--------+--------+--------+---------------+--------------+--------------+--------------+--------------+--------------+--------|
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 | 582327 | 771088 | 248503 | 9 | 236170 | 107865 | 103088 | 15.0906499200 | 5.1550493690 | 2.9967487970 | 5.4691189410 | 8.9839650110 | 0.906231199 | 107865 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 | 582327 | 771088 | 248503 | 9 | 236170 | 107865 | 103088 | 15.0476778900 | 5.1313616310 | 3.0210706480 | 5.4530511170 | 8.9922491330 | 0.921986645 | 107865 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 | 582327 | 771088 | 248503 | 9 | 236170 | 107865 | 103088 | 15.1975878580 | 5.1788464490 | 3.0373609690 | 5.5238989600 | 9.0837889770 | 0.913164837 | 107865 |
-
-**** Without optimizations
-:PROPERTIES:
-:ID: 82a6bf0c-547c-4e84-baa1-98c53379d7f3
-:END:
-
-#+CALL: bench_noopt(wlan_benchmark_single)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------------------+------------------------------------+-----------+--------+--------+--------+---+--------+--------+------+---------------+--------------+--------------+--------------+--------------+--------------+--------|
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 | 582327 | 771088 | 248503 | 9 | 236170 | 107865 | 0 | 15.3893624460 | 5.1767105960 | 3.0413877800 | 5.7070144040 | 9.2869304960 | 0.919612983 | 107865 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 | 582327 | 771088 | 248503 | 9 | 236170 | 107865 | 0 | 15.3638788430 | 5.1630418130 | 3.0081672330 | 5.7031240120 | 9.2438828940 | 0.895646868 | 107865 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=0 | c1,c2 | 582327 | 771088 | 248503 | 9 | 236170 | 107865 | 0 | 15.2745021350 | 5.1306704560 | 3.0418000640 | 5.6621348040 | 9.2231714690 | 0.988643949 | 107865 |
-
-*** Comparing the optimizations
-
-This time with a different benchmark
-
-#+NAME: wlan_benchmark_single2
-| Model | Constants | Partition |
-|-------------------+------------------------------------+-----------|
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 |
-
-
-**** With both optimizations
-:PROPERTIES:
-:ID: d39b3290-297b-4063-8464-a011a048eff4
-:END:
-
-#+CALL: bench(wlan_benchmark_single2)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------------------+------------------------------------+-----------+---------+---------+--------+---+--------+--------+--------+---------------+---------------+--------------+---------------+---------------+--------------+--------|
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 1408676 | 1963522 | 607727 | 9 | 543056 | 243325 | 271999 | 39.3637329540 | 13.2453896050 | 6.0643303750 | 16.0121625480 | 23.5999972580 | 2.532994467 | 243325 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 1408676 | 1963522 | 607727 | 9 | 543056 | 243325 | 271999 | 39.4646461980 | 13.0936489980 | 6.0869881880 | 16.2522690080 | 23.8950358450 | 2.475905320 | 243325 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 1408676 | 1963522 | 607727 | 9 | 543056 | 243325 | 271999 | 39.0647377280 | 13.0970151140 | 6.0643634280 | 15.9493142430 | 23.5259992150 | 2.651193004 | 243325 |
-
-**** Without any optimization
-:PROPERTIES:
-:ID: 408056f6-53de-4920-ba53-9308bb2fbf15
-:END:
-
-#+CALL: bench_sloooow(wlan_benchmark_single2)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------------------+------------------------------------+-----------+---------+---------+--------+----+---------+--------+------+---------------+---------------+--------------+---------------+---------------+--------------+--------|
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 2817352 | 3372198 | 607727 | 11 | 1038591 | 243325 | 0 | 62.8032399420 | 24.0298961220 | 8.0328573550 | 23.0904043580 | 33.8712151930 | 2.533598832 | 243325 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 2817352 | 3372198 | 607727 | 11 | 1038591 | 243325 | 0 | 62.8096335930 | 23.9130181410 | 8.0323240210 | 23.1153711080 | 33.9856103020 | 2.672804745 | 243325 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 2817352 | 3372198 | 607727 | 11 | 1038591 | 243325 | 0 | 62.8848181530 | 24.0303139130 | 8.0121181690 | 23.1079646660 | 33.9319790150 | 2.528242842 | 243325 |
-
-**** With only functor rewrite
-:PROPERTIES:
-:ID: eb6cd726-026c-49ec-974a-1b6285d0fabb
-:END:
-
-#+CALL: bench_noopt(wlan_benchmark_single2)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------------------+------------------------------------+-----------+---------+---------+--------+---+--------+--------+------+---------------+---------------+--------------+---------------+---------------+--------------+--------|
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 1408676 | 1963522 | 607727 | 9 | 543056 | 243325 | 0 | 40.4820614570 | 13.3353128550 | 6.1625531160 | 16.9365448810 | 24.6478678180 | 2.697483848 | 243325 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 1408676 | 1963522 | 607727 | 9 | 543056 | 243325 | 0 | 40.5545896510 | 13.4405492660 | 6.1304969200 | 16.8959012700 | 24.5962440200 | 2.516830279 | 243325 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 1408676 | 1963522 | 607727 | 9 | 543056 | 243325 | 0 | 40.8296828650 | 13.4560628920 | 6.1831625030 | 17.1048950840 | 24.8651696430 | 2.539904578 | 243325 |
-
-**** With only 1-elem optimization
-:PROPERTIES:
-:ID: 74566779-0749-4bbd-9556-80f857144793
-:END:
-
-#+CALL: bench_nofunctrans(wlan_benchmark_single2)
-
-#+RESULTS:
-| Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
-|-------------------+------------------------------------+-----------+---------+---------+--------+----+---------+--------+--------+---------------+---------------+--------------+---------------+---------------+--------------+--------|
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 2817352 | 3372198 | 607727 | 11 | 1038591 | 243325 | 272066 | 62.3224133500 | 24.0014545030 | 8.0163584000 | 22.5763681410 | 33.3185527870 | 2.538228830 | 243325 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 2817352 | 3372198 | 607727 | 11 | 1038591 | 243325 | 272066 | 62.4165430840 | 23.9578211250 | 8.0180667980 | 22.6588742730 | 33.4171848350 | 2.526273353 | 243325 |
-| wlan_time_bounded | TRANS_TIME_MAX=10,DEADLINE=100,K=1 | c1,c2 | 2817352 | 3372198 | 607727 | 11 | 1038591 | 243325 | 272066 | 62.8521293030 | 23.9960135000 | 8.1913510000 | 22.7039631120 | 33.7983183080 | 2.539817679 | 243325 |
-
-** Zeroconf
-:PROPERTIES:
-:ID: e8a80143-4f49-4109-8f58-950514ada685
-:END:
-
- #+BEGIN_QUOTE
- This case study concerns the IPv4 Zeroconf Protocol [CAG02]
-
- We consider the probabilistic timed automata models presented in [KNPS06] using
- the integer semantics also presented in the paper.
-
- For more information, see: http://www.prismmodelchecker.org/casestudies/zeroconf.php
-
- =====================================================================================
-
- PARAMETERS
-
- - reset :: reset is true/false dependent on whether the reset/norest model is to be analysed
- - loss :: probability of message (0.1, 0.01, 0.001 and 0)
- - K :: number of probes (4 in standard) 1:1:8
- - N :: number of concrete hosts, e.g. 20 or 1000 for small/large network
- - err :: error cost from 1e+6 to 1e+12
- - bound :: time bound from 0:50 (then set T to be 1+maximum value of bound in experiment)
- #+END_QUOTE
-
- Benchmarks:
-
- We partition on part of the current state of the environment.
-
- #+NAME: zeroconf_benchmarks
- | Model | Constants | Partition |
- |-----------------------+--------------------------------------+-----------|
- | zeroconf | N=1000,K=4,err=0,reset=true | n,n0,n1,b |
- | zeroconf | N=1000,K=4,err=0,reset=false | n,n0,n1,b |
- | zeroconf_time_bounded | N=1000,K=1,T=11,bound=10,reset=true | n,n0,n1,b |
- | zeroconf_time_bounded | N=1000,K=1,T=11,bound=10,reset=false | n,n0,n1,b |
-
- Results:
-
- #+CALL: bench(zeroconf_benchmarks)
-
- #+RESULTS:
- | Model | Consts | Partition | States | Edges | Sort 0 | I | Q | Q_0 | Opts | t(s) | t_p(s) | t_i(s) | t_r(s) | t_a(s) | t_valmari(s) | Q_0v |
- |-----------------------+--------------------------------------+-----------+--------+---------+--------+------+--------+--------+--------+---------------+--------------+--------------+--------------+---------------+--------------+--------|
- | zeroconf | N=1000,K=4,err=0,reset=true | n,n0,n1,b | 2443 | 2968 | 1088 | 16 | 1206 | 580 | 179 | 0.0358108630 | 0.0167680330 | 0.0057826350 | 0.0103080610 | 0.0170235500 | 0.003827381 | 578 |
- | zeroconf | N=1000,K=4,err=0,reset=false | n,n0,n1,b | 876995 | 1281359 | 307768 | 1722 | 435509 | 194292 | 195033 | 23.5491846280 | 7.4090193180 | 4.6009392190 | 8.3819097560 | 14.8418591750 | 1.670716063 | 194290 |
- | zeroconf_time_bounded | N=1000,K=1,T=11,bound=10,reset=true | n,n0,n1,b | 9523 | 11992 | 4214 | 17 | 4375 | 2102 | 1140 | 0.1552350690 | 0.0627752380 | 0.0336032210 | 0.0461910700 | 0.0833916190 | 0.125739644 | 2102 |
- | zeroconf_time_bounded | N=1000,K=1,T=11,bound=10,reset=false | n,n0,n1,b | 37160 | 51852 | 14834 | 59 | 16412 | 7525 | 6791 | 0.8208913870 | 0.2498574720 | 0.1718957150 | 0.3022022560 | 0.4914674350 | 0.122138130 | 7525 |
diff --git a/prism/valmari-extract-sort1.hs b/prism/valmari-extract-sort1.hs
deleted file mode 100644
index c12caeb9c9d488662f8a9d6e8991cc13bee468e0..0000000000000000000000000000000000000000
--- a/prism/valmari-extract-sort1.hs
+++ /dev/null
@@ -1,55 +0,0 @@
--- Usage:
---
--- mdpmin < input.valmari | runhaskell valmari-extract-sort1.hs mdp
---
--- This is a small script that extracts the size of sort 1 from the output of
--- the valmari tool for MDPs.
-
-
--- This exploits the fact that the valmari implementation prints a list of final
--- blocks that are descendants of a corresponding initial block. Since all
--- states that are _not_ in sort 1 share one block initially, we just have to
--- count the final blocks that origin form all other initial blocks.
---
--- This is particularly easy, since valmari omits the first block from the in-
--- and output. Since the first block is the block of non-sort-1 states in our
--- input, we just have to count all the states in all blocks printed in the
--- outpu.
-
-
-module Main where
-
-import System.Environment
-
--- | Expects the output of valmaris tool, already tokenized on whitespace and
--- returns the number of final blocks that are in sort 1.
---
---
--- The input format is essentially:
---
--- MC ::= <sizes> <l_transitions> <w_transitions> <blocks>
--- <sizes> ::= num_states num_choices num_l_trans num_w_trans num_blocks
--- <l_transitions> ::= <l_transition>*
--- <l_transition> ::= source choice target
--- <w_transitions> ::= <w_transition>*
--- <w_transition> ::= source weight target
--- <blocks> ::= <block>*
--- <block> ::= state* 0
-
-extractSort1 :: [String] -> Int
-extractSort1 (_ : _ : numLTrans' : numWTrans' : _ : rest) =
- let numTrans = read numLTrans' + read numWTrans'
- blocks = drop (numTrans * 3) rest -- Drop all the 'from' 'label' 'to' triples
- in length (filter (/= "0") blocks)
-
-
-main :: IO ()
-main = do
- args <- getArgs
- case args of
- ["mdp"] -> do
- sort1 <- (extractSort1 . words) <$> getContents
- print sort1
- _ -> do
- _ <- getContents -- Ignore input if not an mdp
- putStrLn "NA"
diff --git a/valmari_fig4/bench.py b/valmari_fig4/bench.py
deleted file mode 100755
index 75d84020e5ec2fd61fd670aa7b6b904430fd90cc..0000000000000000000000000000000000000000
--- a/valmari_fig4/bench.py
+++ /dev/null
@@ -1,60 +0,0 @@
-#!/usr/bin/python
-
-import sys
-import tempfile
-import subprocess
-import re
-import os
-
-
-def generate_model(output_file, h):
- subprocess.run(["./generate.py", str(h)], stdout=output_file)
-
-
-def extract_time(perf_output):
- "returns the measured time from the perf output in seconds"
- m = re.search(r'(\d+.\d+)\W+seconds time elapsed', str(perf_output))
- return m[1]
-
-
-def run_copar(copar, input_file):
- process = subprocess.run(
- ['perf', 'stat', copar],
- stdin=input_file,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.PIPE,
- env={'LC_ALL': 'C'})
-
- try:
- process.check_returncode()
- except subprocess.CalledProcessError as e:
- err = os.fdopen(sys.stderr.fileno(), 'wb')
- err.write(process.stderr)
- err.flush()
- raise e
-
- return extract_time(process.stderr)
-
-
-def run_bench(copar, h):
- model_file = tempfile.TemporaryFile()
- generate_model(model_file, h)
- model_file.seek(0)
- time = run_copar(copar, model_file)
-
- print("%d %s" % (h, time))
-
-
-def main(args):
- if len(args) != 3:
- print("Usage: %s EXECUTABLE h" % args[0])
- sys.exit(1)
-
- copar_binary = args[1]
- h = int(args[2])
-
- run_bench(copar_binary, h)
-
-
-if __name__ == "__main__":
- main(sys.argv)
diff --git a/valmari_fig4/benchmark.org b/valmari_fig4/benchmark.org
deleted file mode 100644
index b644a3d732bbfc82e6740fc6cdb5d576cab42613..0000000000000000000000000000000000000000
--- a/valmari_fig4/benchmark.org
+++ /dev/null
@@ -1,130 +0,0 @@
-#+TITLE: Figure 4 from Valmari & Franceschinis
-
-* Introduction
-
- This benchmark implements figure 4 from [fn:1]. It is intended to show that
- the runtime complexity is indeed $O(m \log n)$ and not $O(m \log^2 n)$.
-
-* Benchmarking
-
- First, we generate the model from figure 4 for each $h$ and measure the time
- to partition it.
-
- Replace ~../../copar/bin/copar~ by the path to the tested program and Beware:
- evaluating the following snippet can take a while.
-
- #+BEGIN_SRC sh :exports both :eval no-export
- echo h "time(s)"
- for h in {4..20}; do
- ./bench.py ../../copar/bin/copar $h
- done
- #+END_SRC
-
- #+NAME: benchresults
- #+RESULTS:
- | h | time(s) |
- | 4 | 0.002862292 |
- | 5 | 0.00316286 |
- | 6 | 0.004661828 |
- | 7 | 0.007075351 |
- | 8 | 0.013313165 |
- | 9 | 0.025658629 |
- | 10 | 0.055067783 |
- | 11 | 0.117350064 |
- | 12 | 0.24884803 |
- | 13 | 0.541005227 |
- | 14 | 1.120708321 |
- | 15 | 2.469870281 |
- | 16 | 5.183162128 |
- | 17 | 10.774933357 |
- | 18 | 22.683064288 |
- | 19 | 49.498858156 |
- | 20 | 109.218546203 |
-
-* Interpretation
-
- We now try to fit the data onto possible curves using the Python library
- /SciPy/.
-
- We first need to import a few python modules.
-
- #+BEGIN_SRC python :session :results none
- import numpy as np
- import matplotlib.pyplot as plt
- from scipy.optimize import curve_fit
- #+END_SRC
-
-** Possible functions to fit
-
- Here are the functions that our data could fit. As mentioned above, the
- complexity should be $n \log n$ and not $n \log^2 n$.
-
- - The good function, implementing $n \log n$:
-
- #+BEGIN_SRC python :session :results none
- def func_good(x, a, b):
- return a * x * np.log(x) + b
- #+END_SRC
-
- - The bad function, implementing $n \log^2 n$:
-
- #+BEGIN_SRC python :session :results none
- def func_bad(x, a, b):
- return a * x * (np.log(x)**2) + b
- #+END_SRC
-
-** Trying to fit the data
-
- Let's first import the data we generated earlier into python. A model for $h$
- has $n = 2^h$ states one transition per state.
-
- #+BEGIN_SRC python :session :var data=benchresults :results none
- xdata = np.array([2**a[0] for a in data[1:]])
- ydata = np.array([a[1] for a in data[1:]])
- #+END_SRC
-
- And let SciPy do its magic to calculate the fitting coefficients with the
- ~curve_fit~ function for both ~func_good~ and ~func_bad~.
-
- #+BEGIN_SRC python :session :exports both
- popt_good, pcov_good = curve_fit(func_good, xdata, ydata)
- popt_bad, pcov_bad = curve_fit(func_bad, xdata, ydata)
- [("", "a", "b"),
- ("good", popt_good[0], popt_good[1]),
- ("bad", popt_bad[0], popt_bad[1])
- ]
- #+END_SRC
-
- #+RESULTS:
- | | a | b |
- | good | 7.4470666850016445e-06 | -0.21839458264866904 |
- | bad | 5.417012783955984e-07 | 0.2258533257706899 |
-
-** Plot
-
- Finally, plot the real data and the fitted curves:
-
- #+BEGIN_SRC python :session :results file :exports both
- plt.clf()
- plt.xlabel("n")
- plt.ylabel("time(s)")
-
- plt.plot(xdata, func_good(xdata, *popt_good), label='good')
- plt.plot(xdata, func_bad(xdata, *popt_bad), label='bad')
- plt.plot(xdata, ydata, 'o', label='measurement', fillstyle='none')
-
- plt.legend()
-
- #plt.show()
- filename = "benchmarks.png"
- plt.savefig(filename, dpi=150)
- filename
- #+END_SRC
-
- #+RESULTS:
- [[file:benchmarks.png]]
-
-
-* Footnotes
-
-[fn:1] Valmari and Franceschinis: Simple O(m log n) Time Markov Chain Lumping
diff --git a/valmari_fig4/generate.py b/valmari_fig4/generate.py
deleted file mode 100755
index ea1f2bdf05bdbb4c0f9ad4701af96ebbcc20049a..0000000000000000000000000000000000000000
--- a/valmari_fig4/generate.py
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/usr/bin/env python
-
-import sys
-
-
-def gen_functor():
- print("3x(PX)\n")
-
-
-def gen_state(block, column, row):
- print("s_%d_%d: (%s, {s_%d_0})" % (column, row, block, column+1))
-
-
-def gen_final_state(h):
- print("s_%d_0: (2, {s_%d_0})" % (h+1, h+1))
-
-
-def gen_n_states(block, column, n):
- for row in range(0, n):
- gen_state(block, column, row)
-
-
-def gen_column(h, column):
- if column == 0:
- gen_n_states("0", column, 2**h)
- else:
- gen_n_states("1", column, 2**(column-1))
-
-
-def gen_all(h):
- gen_functor()
- gen_final_state(h)
- for column in reversed(range(0, h+1)):
- gen_column(h, column)
-
-
-def main(args):
- if len(args) != 2:
- print("Usage: %s h" % args[0])
- sys.exit(1)
-
- h = int(args[1])
- gen_all(h)
-
-
-if __name__ == "__main__":
- main(sys.argv)
diff --git a/wta/README.md b/wta/README.md
deleted file mode 100644
index 9f79db9fef8709fb4e1d53552b373ff7a4f0a118..0000000000000000000000000000000000000000
--- a/wta/README.md
+++ /dev/null
@@ -1,11 +0,0 @@
-# Benchmarks for Weighted Tree Automata
-
-The benchmarks in this directory are contained in an interactive Emacs org-mode
-file `benchmarks.org` that contains the commands used to generated the results
-as well as the results themselves.
-
-Unfortunately, Gitlab's rendering of org-mode markup is poor and hides the
-actual results, so this file is best viewed in a text editor (e.g. Emacs) or
-[pre-rendered as PDF][1].
-
-[1]: https://git8.cs.fau.de/software/copar-benchmarks/wikis/uploads/053845133bb13efa9bef4e6cc96743ba/benchmarks.pdf
diff --git a/wta/bench.py b/wta/bench.py
deleted file mode 100755
index 206032255b43d5a1be75782802e816abd2e5d4c0..0000000000000000000000000000000000000000
--- a/wta/bench.py
+++ /dev/null
@@ -1,392 +0,0 @@
-#!/usr/bin/env python3
-
-import argparse
-import os
-import sys
-import subprocess
-import json
-import numpy as np
-import scipy.stats as st
-
-samples = 5
-
-
-def from_maybe(default, x):
- return default if x is None else x
-
-
-# Edge Spec should be ("transitions", str) or ("zero-frequency", str)
-def coalg_file(states, monoid, symbols, edge_spec, different_values, i):
- if edge_spec[0] == "transitions":
- return "bench/wta_%s_%s_t%s_%s_%s_%d" % (monoid, symbols, edge_spec[1],
- from_maybe("", different_values), states, i)
- else:
- return "bench/wta_%s_%s_%s_%s_%s_%d" % (monoid, symbols, edge_spec[1],
- from_maybe("", different_values), states, i)
-
-
-def check_zerotrans(args):
- if args.zero_frequency:
- edge_spec = ("zero-frequency", args.zero_frequency)
- elif args.transitions:
- edge_spec = ("transitions", args.transitions)
- elif args.out_degree:
- edge_spec = ("transitions", args.out_degree) # HACK
- else:
- print("One of --zero-frequency and --transitions must be provided")
- sys.exit(1)
-
- args.edge_spec = edge_spec
-
-
-def generate(args):
- check_zerotrans(args)
-
- if args.out_degree:
- args.edge_spec = ("transitions", str(int(args.out_degree)*int(args.states)))
-
- generator = args.generator
- states = args.states
- monoid = args.monoid
- symbols = args.symbols
- different_values = args.different_values
- edge_spec = args.edge_spec
-
- os.makedirs("bench", exist_ok=True)
-
- for i in range(0, samples):
- f = coalg_file(states, monoid, symbols, edge_spec, different_values,
- i) + ".coalgebra"
-
- if os.path.exists(f):
- continue
-
- cmd = [generator, "--states", states, "--monoid", monoid, "--symbols", symbols]
- cmd += ["--" + edge_spec[0], edge_spec[1]]
- if not (different_values is None):
- cmd += ["--different-values", different_values]
-
- subprocess.run(cmd, stdout=open(f, "w+"))
-
-
-def run_one(args, i):
- copar = args.copar
- states = args.states
- monoid = args.monoid
- symbols = args.symbols
- different_values = args.different_values
- edge_spec = args.edge_spec
-
- f = coalg_file(states, monoid, symbols, edge_spec, different_values, i) + ".coalgebra"
-
- copar_args = [copar, 'refine', '--stats-json', f]
-
- try:
- out = subprocess.run(
- copar_args,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.PIPE,
- check=True)
- except subprocess.CalledProcessError as e:
- print("error: %s" % e.stderr.decode('utf-8'))
- raise
-
- stats = json.loads(out.stderr.decode('utf-8'))
-
- stats['monoid'] = monoid
- stats['symbols'] = symbols
- stats['edge-spec'] = str(edge_spec)
- stats['mon-values'] = from_maybe("", different_values)
- stats['i'] = i
-
- return stats
-
-
-def run_one_simple(args, i):
- copar = args.copar
- states = args.states
- monoid = args.monoid
- symbols = args.symbols
- edge_spec = args.edge_spec
- different_values= args.different_values
-
- f = coalg_file(states, monoid, symbols, edge_spec, different_values, i) + ".coalgebra"
-
- copar_args = [copar, 'refine', f]
-
- subprocess.run(
- copar_args,
- stdout=subprocess.DEVNULL,
- check=True)
-
-
-def confidence(vals):
- """Compute the 95% confidence intervall (CI) for the mean with the student
-distribution.
-
-Returns a tuple of (mean, lower, upper), where lower and upper are the bounds
-of the CI"""
-
- # For a larger sample size (> 30), we could also use the normal
- # distribution.
- #
- # This code is taken from
- # https://stackoverflow.com/questions/15033511/compute-a-confidence-interval-from-sample-data/34474255#34474255
-
- mean = np.mean(vals)
- ci = st.t.interval(
- 0.95, len(vals) - 1, loc=np.mean(vals), scale=st.sem(vals))
-
- return (mean, ci[0], ci[1])
-
-
-def stddev(vals):
- """Compute the mean and standard deviation intervall on a sample.
-
-This uses the corrected sample standard deviation."""
-
- # see also:
- # https://en.wikipedia.org/wiki/Standard_deviation#Corrected_sample_standard_deviation
- mean = np.mean(vals)
- std = np.std(vals, ddof=1)
-
- return (mean, std)
-
-
-def print_row(d, header, stddev):
- keys = [
- 'i', 'states', 'edges', 'initial-partition-size',
- 'final-partition-size', 'explicit-final-partition-size',
- 'size1-skipped'
- ]
-
- for k in [
- 'overall-duration', 'parse-duration', 'algorithm-duration',
- 'initialize-duration', 'refine-duration'
- ]:
- keys.append(k)
- if stddev:
- keys.append(k + '-stddev')
-
- values = [d[k] for k in keys]
-
- if header:
- print('\t'.join(keys))
- else:
- print('\t'.join(str(x) for x in values))
-
-
-def run(args):
- check_zerotrans(args)
- if args.out_degree:
- args.edge_spec = ("transitions", str(int(args.out_degree)*int(args.states)))
-
- results = [run_one(args, i) for i in range(0, samples)]
-
- def confidencekey(vals, k):
- return confidence(list(float(x[k]) for x in vals))
-
- def stddevkey(vals, k):
- return stddev(list(float(x[k]) for x in vals))
-
- combined = results[0].copy()
- combined['i'] = samples
-
- for k in [
- 'overall-duration', 'parse-duration', 'initialize-duration',
- 'refine-duration', 'algorithm-duration'
- ]:
- ci = stddevkey(results, k)
- combined[k] = str(ci[0])
- combined[k + '-stddev'] = str(ci[1])
-
- if args.indiv:
- if args.header:
- print_row(combined, True, stddev=False)
- for res in results:
- print_row(res, False, stddev=False)
- else:
- if args.header:
- print_row(combined, True, stddev=args.stddev)
- print_row(combined, False, stddev=args.stddev)
-
-
-def test(args, states):
- print("Trying %d..." % states)
-
- args.states = str(states)
- if args.out_degree:
- args.transitions = str(states * int(args.out_degree))
-
- generate(args)
-
- for i in range(0, samples):
- try:
- run_one_simple(args, i)
- except subprocess.CalledProcessError:
- return False
-
- return True
-
-
-def find_bad(args, good):
- states = good*2
-
- if test(args, states):
- return find_bad(args, states)
- else:
- return (good, states)
-
-
-def bisect_states(args):
- check_zerotrans(args)
-
- global samples
- samples = 1
-
- states = args.start_states
-
- good = args.good or 0
- bad = args.bad
-
- if bad is None:
- if good and states < good:
- states = good+1
-
- if test(args, states):
- (good, bad) = find_bad(args, states)
- else:
- bad = states
-
- while good+1 < bad:
- states = good + (bad-good)//2
- if test(args, states):
- good = states
- else:
- bad = states
-
- print("First bad state count: %d" % bad)
-
-
-def test_trans(args, transitions):
- print("Trying %d..." % transitions)
-
- args.transitions = str(transitions)
-
- generate(args)
-
- for i in range(0, samples):
- try:
- run_one_simple(args, i)
- except subprocess.CalledProcessError:
- return False
-
- return True
-
-
-def find_bad_trans(args, good):
- transitions = good*2
-
- if test_trans(args, transitions):
- return find_bad(args, transitions)
- else:
- return (good, transitions)
-
-
-def bisect_trans(args):
- # Hacks and hacks and hacks
- args.zero_frequency = None
- args.out_degree = None
-
- transitions = args.start_transitions
-
- global samples
- samples = 1
-
- good = args.good or 0
- bad = args.bad
-
- if bad is None:
- if good and transitions < good:
- transitions = good+1
-
- if test_trans(args, transitions):
- (good, bad) = find_bad_trans(args, transitions)
- else:
- bad = transitions
-
- while good+1 < bad:
- transitions = good + (bad-good)//2
- if test_trans(args, transitions):
- good = transitions
- else:
- bad = transitions
-
- print("First bad transition count: %d" % bad)
-
-
-def main():
- parser = argparse.ArgumentParser()
- subparsers = parser.add_subparsers(required=True)
-
- gen_parser = subparsers.add_parser('generate')
- gen_parser.add_argument('generator')
- gen_parser.add_argument('--states', required=True)
- gen_parser.add_argument('--monoid', required=True)
- gen_parser.add_argument('--symbols', required=True)
- gen_parser.add_argument('--zero-frequency')
- gen_parser.add_argument('--transitions')
- gen_parser.add_argument('--out-degree')
- gen_parser.add_argument('--different-values')
- gen_parser.set_defaults(func=generate)
-
- run_parser = subparsers.add_parser('run')
- run_parser.add_argument('copar')
- run_parser.add_argument('--states', required=True)
- run_parser.add_argument('--monoid', required=True)
- run_parser.add_argument('--symbols', required=True)
- run_parser.add_argument('--zero-frequency')
- run_parser.add_argument('--transitions')
- run_parser.add_argument('--out-degree')
- run_parser.add_argument('--different-values')
- run_parser.add_argument(
- '--stddev', action='store_true', help="report stddev for timings")
- run_parser.add_argument(
- '--indiv', action='store_true', help="report individual samples")
- run_parser.add_argument(
- '--header', action='store_true', help="Print header row for table")
- run_parser.set_defaults(func=run)
-
- bisect_parser = subparsers.add_parser('bisect')
- bisect_parser.add_argument('generator')
- bisect_parser.add_argument('copar')
- bisect_parser.add_argument('--monoid', required=True)
- bisect_parser.add_argument('--symbols', required=True)
- bisect_parser.add_argument('--zero-frequency')
- bisect_parser.add_argument('--transitions')
- bisect_parser.add_argument('--out-degree')
- bisect_parser.add_argument('--different-values')
- bisect_parser.add_argument('--start-states', type=int, default=50)
- bisect_parser.add_argument('--good', type=int)
- bisect_parser.add_argument('--bad', type=int)
-
- bisect_parser.set_defaults(func=bisect_states)
-
- bisect_trans_parser = subparsers.add_parser('bisect-trans')
- bisect_trans_parser.add_argument('generator')
- bisect_trans_parser.add_argument('copar')
- bisect_trans_parser.add_argument('--monoid', required=True)
- bisect_trans_parser.add_argument('--symbols', required=True)
- bisect_trans_parser.add_argument('--different-values')
- bisect_trans_parser.add_argument('--states', required=True)
- bisect_trans_parser.add_argument('--start-transitions', type=int, default=100)
- bisect_trans_parser.add_argument('--good', type=int)
- bisect_trans_parser.add_argument('--bad', type=int)
- bisect_trans_parser.set_defaults(func=bisect_trans)
-
- args = parser.parse_args()
- args.func(args)
-
-
-if __name__ == "__main__":
- main()
diff --git a/wta/benchmarks.org b/wta/benchmarks.org
deleted file mode 100644
index a72b8a117524c55b9f8efd15acba83b5c1f48577..0000000000000000000000000000000000000000
--- a/wta/benchmarks.org
+++ /dev/null
@@ -1,1561 +0,0 @@
-#+TITLE: Benchmarks for Weighted Tree Automata
-#+PROPERTY: header-args :exports both :eval never-export
-#+LATEX_HEADER: \usepackage[paperheight=30cm,paperwidth=40cm,margin=0.5cm]{geometry}
-
-* What fits in 16GB RAM
-
-Finding the number of states that copar can solve in 16GB RAM for a few
-different types of WTAs.
-
-** Powerset
-*** GHC 8.6.4 with symbols 0,8 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid powerset --symbols 0,8 --zero-frequency 0.7 --good 1400 --bad 1600 --start-states 1500
-#+END_SRC
-
-#+RESULTS:
-: Trying 1500...
-: Trying 1450...
-: Trying 1475...
-: Trying 1487...
-: Trying 1481...
-: Trying 1478...
-: Trying 1479...
-: First bad state count: 1479
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_powerset_0,8_0.7_1478*
-#+END_SRC
-
-#+RESULTS:
-: 82M bench/wta_powerset_0,8_0.7_1478_0.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_1.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_2.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_3.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_4.coalgebra
-
-*** GHC 8.4.4 with symbols 0,8 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid powerset --symbols 0,8 --zero-frequency 0.7 --start-states 1000
-#+END_SRC
-
-#+RESULTS:
-: Trying 1000...
-: Trying 2000...
-: Trying 1500...
-: Trying 1250...
-: Trying 1375...
-: Trying 1437...
-: Trying 1468...
-: Trying 1484...
-: Trying 1476...
-: Trying 1480...
-: Trying 1478...
-: Trying 1479...
-: First bad state count: 1479
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_powerset_0,8_0.7_1478*
-#+END_SRC
-
-#+RESULTS:
-: 82M bench/wta_powerset_0,8_0.7_1478_0.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_1.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_2.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_3.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_4.coalgebra
-
-**** Automaton size
-
-#+begin_src sh :hlines yes
-python bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,8 --zero-frequency 0.7 --states 1478
-python bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,8 --zero-frequency 0.7 --states 1478 --indiv --header
-#+end_src
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 5243856 | 10484756 | 10 | 18 | 2 | 0 | 115.339077581 | 34.439478263 | 79.458232767 | 27.712258503 | 37.668249537 |
-| 1 | 5245405 | 10487854 | 10 | 18 | 2 | 0 | 114.001430765 | 34.477874858 | 78.07613373 | 27.764347556 | 37.074617863 |
-| 2 | 5243549 | 10484142 | 10 | 18 | 2 | 0 | 114.862812706 | 34.668409012 | 78.756016051 | 27.741448159 | 37.09480403 |
-| 3 | 5241660 | 10480364 | 10 | 18 | 2 | 0 | 114.228588077 | 34.362576104 | 78.424601434 | 27.648526393 | 37.043127108 |
-| 4 | 5243032 | 10483108 | 10 | 18 | 2 | 0 | 114.692551901 | 34.827430372 | 78.414037199 | 27.723198479 | 38.276163878 |
-
-*** GHC 8.4.4 with symbols 1,0,4 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid powerset --symbols 1,0,4 --zero-frequency 0.7
-#+END_SRC
-
-#+RESULTS:
-: Trying 50...
-: Trying 100...
-: Trying 200...
-: Trying 150...
-: Trying 175...
-: Trying 162...
-: Trying 156...
-: Trying 153...
-: Trying 151...
-: Trying 152...
-: First bad state count: 152
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_powerset_1,0,4_0.7_152*
-#+END_SRC
-
-#+RESULTS:
-: 83M bench/wta_powerset_1,0,4_0.7_152_0.coalgebra
-: 83M bench/wta_powerset_1,0,4_0.7_152_1.coalgebra
-: 83M bench/wta_powerset_1,0,4_0.7_152_2.coalgebra
-: 83M bench/wta_powerset_1,0,4_0.7_152_3.coalgebra
-: 83M bench/wta_powerset_1,0,4_0.7_152_4.coalgebra
-
-**** Automaton Size
-
-#+begin_src sh :hlines yes
-python bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 1,0,4 --zero-frequency 0.7 --states 151
-python bench.py run ../../copar/bin/copar --monoid powerset --symbols 1,0,4 --zero-frequency 0.7 --states 151 --indiv --header
-#+end_src
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 4131380 | 12393591 | 7 | 69 | 4 | 0 | 92.530296805 | 32.925871573 | 58.292789973 | 13.413784138 | 32.25285307 |
-| 1 | 4131667 | 12394466 | 7 | 69 | 4 | 0 | 104.072447103 | 32.6810851 | 70.054731043 | 22.601698388 | 34.657530397 |
-| 2 | 4130798 | 12391837 | 7 | 69 | 4 | 0 | 104.65358267 | 32.641475967 | 70.686110886 | 22.881971907 | 34.580390995 |
-| 3 | 4131273 | 12393268 | 7 | 69 | 4 | 0 | 104.436764929 | 33.010643372 | 70.097279338 | 22.722527765 | 34.948100296 |
-| 4 | 4131755 | 12394730 | 7 | 69 | 4 | 0 | 103.940596569 | 32.874641172 | 69.748179673 | 22.561661198 | 34.530464508 |
-
-*** GHC 8.4.4 with symbols 4,3,2 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid powerset --symbols 4,3,2 --zero-frequency 0.7
-#+END_SRC
-
-#+RESULTS:
-: Trying 50...
-: Trying 100...
-: Trying 200...
-: Trying 150...
-: Trying 175...
-: Trying 187...
-: Trying 193...
-: Trying 190...
-: Trying 191...
-: First bad state count: 191
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_powerset_4,3,2_0.7_190*
-#+END_SRC
-
-#+RESULTS:
-: 83M bench/wta_powerset_4,3,2_0.7_190_0.coalgebra
-: 83M bench/wta_powerset_4,3,2_0.7_190_1.coalgebra
-: 83M bench/wta_powerset_4,3,2_0.7_190_2.coalgebra
-: 83M bench/wta_powerset_4,3,2_0.7_190_3.coalgebra
-: 83M bench/wta_powerset_4,3,2_0.7_190_4.coalgebra
-
-**** Automaton Size
-
-#+begin_src sh :hlines yes
-python bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 4,3,2 --zero-frequency 0.7 --states 190
-python bench.py run ../../copar/bin/copar --monoid powerset --symbols 4,3,2 --zero-frequency 0.7 --states 190 --indiv --header
-#+end_src
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 4150153 | 12416791 | 11 | 72964 | 190 | 7474719 | 141.894074912 | 32.94816078 | 104.894412823 | 19.27142335 | 73.292672481 |
-| 1 | 4149881 | 12415853 | 11 | 72964 | 190 | 7374059 | 143.936708293 | 32.959668876 | 106.847647981 | 19.335579933 | 74.389142529 |
-| 2 | 4147912 | 12410153 | 11 | 72964 | 190 | 7530637 | 143.039477927 | 32.922891848 | 106.049882108 | 19.238034371 | 73.786648681 |
-| 3 | 4149832 | 12415896 | 11 | 72964 | 190 | 7400820 | 141.899762867 | 32.763790399 | 105.08945468 | 19.081253852 | 73.103108932 |
-| 4 | 4147931 | 12410198 | 11 | 72964 | 190 | 7356721 | 142.277553489 | 33.004372363 | 105.135364747 | 19.217062907 | 72.82642293 |
-
-*** GHC 8.4.4 with symbols 0,0,0,0,0,3 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid powerset --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --start-states 2
-#+END_SRC
-
-#+RESULTS:
-: Trying 2...
-: Trying 4...
-: Trying 8...
-: Trying 16...
-: Trying 12...
-: Trying 10...
-: Trying 11...
-: First bad state count: 12
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_powerset_0,0,0,0,0,3_0.7_11*
-#+END_SRC
-
-#+RESULTS:
-: 47M bench/wta_powerset_0,0,0,0,0,3_0.7_11_0.coalgebra
-: 47M bench/wta_powerset_0,0,0,0,0,3_0.7_11_1.coalgebra
-: 47M bench/wta_powerset_0,0,0,0,0,3_0.7_11_2.coalgebra
-: 47M bench/wta_powerset_0,0,0,0,0,3_0.7_11_3.coalgebra
-: 47M bench/wta_powerset_0,0,0,0,0,3_0.7_11_4.coalgebra
-
-**** Automaton Size
-
-#+begin_src sh :hlines yes
-python bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --states 11
-python bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --states 11 --indiv --header
-#+end_src
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 1593503 | 9560952 | 5 | 98 | 2 | 0 | 44.433319213 | 19.470309466 | 24.352231629 | 11.34834025 | 10.606600771 |
-| 1 | 1594688 | 9568062 | 5 | 98 | 2 | 0 | 44.35660646 | 19.503070403 | 24.284648797 | 11.546398372 | 10.351405421 |
-| 2 | 1595746 | 9574410 | 5 | 98 | 2 | 0 | 44.979436845 | 19.622696684 | 24.756055326 | 11.448256352 | 10.958179969 |
-| 3 | 1594811 | 9568800 | 5 | 98 | 2 | 0 | 44.518512407 | 19.486959131 | 24.465812029 | 11.501689163 | 10.63837111 |
-| 4 | 1595187 | 9571056 | 5 | 98 | 2 | 0 | 44.449440224 | 19.537019031 | 24.300846734 | 11.389487771 | 10.5363747 |
-
-** Z,max
-*** GHC 8.4.4 with symbols 0,8 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid 'Z,max' --symbols 0,8 --zero-frequency 0.7
-#+END_SRC
-
-#+RESULTS:
-: Trying 50...
-: Trying 100...
-: Trying 200...
-: Trying 400...
-: Trying 800...
-: Trying 1600...
-: Trying 1200...
-: Trying 1400...
-: Trying 1500...
-: Trying 1450...
-: Trying 1475...
-: Trying 1462...
-: Trying 1456...
-: Trying 1453...
-: Trying 1451...
-: First bad state count: 1451
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_Z,max_0,8_0.7_1450*
-#+END_SRC
-
-#+RESULTS:
-: 182M bench/wta_Z,max_0,8_0.7_1450_0.coalgebra
-: 182M bench/wta_Z,max_0,8_0.7_1450_1.coalgebra
-: 182M bench/wta_Z,max_0,8_0.7_1450_2.coalgebra
-: 182M bench/wta_Z,max_0,8_0.7_1450_3.coalgebra
-: 182M bench/wta_Z,max_0,8_0.7_1450_4.coalgebra
-
-**** Automaton Size
-
-#+BEGIN_SRC sh
-python bench.py generate ../../copar/bin/random-wta --monoid 'Z,max' --symbols 0,8 --zero-frequency 0.7 --states 1450
-python bench.py run ../../copar/bin/copar --monoid 'Z,max' --symbols 0,8 --zero-frequency 0.7 --states 1450 --indiv --header
-#+END_SRC
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 5047826 | 10092752 | 1458 | 13050 | 1450 | 9457197 | 127.724360284 | 43.258927035 | 81.171137974 | 26.874901612 | 40.422676395 |
-| 1 | 5047514 | 10092128 | 1458 | 13050 | 1450 | 9456756 | 125.832804664 | 42.853972016 | 79.686342664 | 26.577174808 | 39.966384527 |
-| 2 | 5049772 | 10096644 | 1458 | 13050 | 1450 | 9460968 | 126.336593045 | 42.68312591 | 80.349875424 | 26.745235991 | 40.031230363 |
-| 3 | 5046916 | 10090932 | 1458 | 13050 | 1450 | 9455805 | 125.95962352 | 42.526542018 | 80.176310943 | 26.550162046 | 40.227319833 |
-| 4 | 5049229 | 10095558 | 1458 | 13050 | 1450 | 9460347 | 126.718920083 | 42.559013501 | 80.863052096 | 26.756405734 | 40.145229524 |
-
-*** GHC 8.4.4 with symbols 1,0,4 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid 'Z,max' --symbols 1,0,4 --zero-frequency 0.7
-#+END_SRC
-
-#+RESULTS:
-: Trying 50...
-: Trying 100...
-: Trying 200...
-: Trying 150...
-: Trying 175...
-: Trying 162...
-: Trying 156...
-: Trying 153...
-: Trying 151...
-: First bad state count: 151
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_Z,max_1,0,4_0.7_150*
-#+END_SRC
-
-**** File Size
-
-#+RESULTS:
-: 162M bench/wta_Z,max_1,0,4_0.7_150_0.coalgebra
-: 162M bench/wta_Z,max_1,0,4_0.7_150_1.coalgebra
-: 162M bench/wta_Z,max_1,0,4_0.7_150_2.coalgebra
-: 162M bench/wta_Z,max_1,0,4_0.7_150_3.coalgebra
-: 162M bench/wta_Z,max_1,0,4_0.7_150_4.coalgebra
-
-**** Automaton Size
-
-#+BEGIN_SRC sh
-python bench.py generate ../../copar/bin/random-wta --monoid 'Z,max' --symbols 1,0,4 --zero-frequency 0.7 --states 150
-python bench.py run ../../copar/bin/copar --monoid 'Z,max' --symbols 1,0,4 --zero-frequency 0.7 --states 150 --indiv --header
-#+END_SRC
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 4051480 | 12153896 | 155 | 90151 | 150 | 7087519 | 120.395749162 | 40.871788157 | 74.90886142 | 22.238988672 | 39.72932804 |
-| 1 | 4050707 | 12151583 | 155 | 90151 | 150 | 7087488 | 120.463605814 | 40.81720097 | 75.096220332 | 22.78192433 | 39.512466086 |
-| 2 | 4048842 | 12145992 | 155 | 90151 | 150 | 7083654 | 120.367574362 | 40.847872293 | 74.879748499 | 22.174812232 | 39.882652278 |
-| 3 | 4052208 | 12156090 | 155 | 90151 | 150 | 7090069 | 120.237306821 | 40.747046417 | 74.945644255 | 22.185753095 | 39.520022235 |
-| 4 | 4048823 | 12145929 | 155 | 90151 | 150 | 7084229 | 121.046334089 | 41.206127141 | 75.236966807 | 22.108849249 | 39.746848148 |
-
-*** GHC 8.4.4 with symbols 4,3,2 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid 'Z,max' --symbols 4,3,2 --zero-frequency 0.7 --start-states 100
-#+END_SRC
-
-#+RESULTS:
-: Trying 100...
-: Trying 200...
-: Trying 150...
-: Trying 175...
-: Trying 187...
-: Trying 193...
-: Trying 190...
-: Trying 188...
-: Trying 189...
-: First bad state count: 189
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_Z,max_4,3,2_0.7_188*
-#+END_SRC
-
-#+RESULTS:
-: 162M bench/wta_Z,max_4,3,2_0.7_188_0.coalgebra
-: 162M bench/wta_Z,max_4,3,2_0.7_188_1.coalgebra
-: 162M bench/wta_Z,max_4,3,2_0.7_188_2.coalgebra
-: 162M bench/wta_Z,max_4,3,2_0.7_188_3.coalgebra
-: 162M bench/wta_Z,max_4,3,2_0.7_188_4.coalgebra
-
-**** Automaton Size
-
-#+BEGIN_SRC sh
-python bench.py generate ../../copar/bin/random-wta --monoid 'Z,max' --symbols 4,3,2 --zero-frequency 0.7 --states 188
-python bench.py run ../../copar/bin/copar --monoid 'Z,max' --symbols 4,3,2 --zero-frequency 0.7 --states 188 --indiv --header
-#+END_SRC
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 4020872 | 12029832 | 197 | 71444 | 188 | 6045852 | 115.942447016 | 40.75075879 | 71.271681594 | 18.625916768 | 39.91853539 |
-| 1 | 4017121 | 12018620 | 197 | 71444 | 188 | 6040501 | 113.960683621 | 40.210261362 | 69.908854506 | 18.715510494 | 39.429755908 |
-| 2 | 4017092 | 12018743 | 197 | 71444 | 188 | 6040038 | 114.417975118 | 40.056454421 | 70.449543532 | 18.443439893 | 39.714702929 |
-| 3 | 4013171 | 12006729 | 197 | 71444 | 188 | 6034912 | 104.421419394 | 40.426583126 | 60.116296609 | 10.279226244 | 37.258447268 |
-| 4 | 4018149 | 12021895 | 197 | 71444 | 188 | 6042204 | 104.595285757 | 40.484957759 | 60.29935373 | 10.313693438 | 37.315322541 |
-
-*** GHC 8.4.4 with symbols 0,0,0,0,0,3 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid 'Z,max' --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --start-states 6
-#+END_SRC
-
-#+RESULTS:
-: Trying 6...
-: Trying 12...
-: Trying 9...
-: Trying 10...
-: Trying 11...
-: First bad state count: 12
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_Z,max_0,0,0,0,0,3_0.7_11*
-#+END_SRC
-
-#+RESULTS:
-: 79M bench/wta_Z,max_0,0,0,0,0,3_0.7_11_0.coalgebra
-: 79M bench/wta_Z,max_0,0,0,0,0,3_0.7_11_1.coalgebra
-: 79M bench/wta_Z,max_0,0,0,0,0,3_0.7_11_2.coalgebra
-: 79M bench/wta_Z,max_0,0,0,0,0,3_0.7_11_3.coalgebra
-: 79M bench/wta_Z,max_0,0,0,0,0,3_0.7_11_4.coalgebra
-
-**** Automaton Size
-
-#+BEGIN_SRC sh
-python bench.py generate ../../copar/bin/random-wta --monoid 'Z,max' --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --states 11
-python bench.py run ../../copar/bin/copar --monoid 'Z,max' --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --states 11 --indiv --header
-#+END_SRC
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 1595450 | 9572634 | 14 | 473731 | 11 | 2679440 | 59.567674089 | 22.815217966 | 34.656233896 | 10.712862563 | 21.527223791 |
-| 1 | 1596274 | 9577578 | 14 | 473599 | 11 | 2680789 | 59.667949978 | 22.964709813 | 34.595654672 | 10.671440318 | 21.504327033 |
-| 2 | 1593533 | 9561132 | 14 | 473725 | 11 | 2676188 | 59.60896951 | 22.965296594 | 34.468009878 | 10.694780509 | 21.357284246 |
-| 3 | 1593597 | 9561516 | 14 | 473516 | 11 | 2675256 | 59.618517937 | 22.900218752 | 34.517770264 | 10.699678956 | 21.404437124 |
-| 4 | 1595682 | 9574026 | 14 | 473759 | 11 | 2679419 | 59.829033166 | 22.84743033 | 34.719394797 | 10.690592405 | 21.632768706 |
-
-** Word,or
-*** GHC 8.4.4 with symbols 0,8 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid 'Word,or' --symbols 0,8 --zero-frequency 0.7 --start-states 1000
-#+END_SRC
-
-#+RESULTS:
-: Trying 1000...
-: Trying 2000...
-: Trying 1500...
-: Trying 1250...
-: Trying 1375...
-: Trying 1437...
-: Trying 1406...
-: Trying 1421...
-: Trying 1413...
-: Trying 1409...
-: Trying 1407...
-: Trying 1408...
-: First bad state count: 1409
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_Word,or_0,8_0.7_1408*
-#+END_SRC
-
-#+RESULTS:
-: 164M bench/wta_Word,or_0,8_0.7_1408_0.coalgebra
-: 164M bench/wta_Word,or_0,8_0.7_1408_1.coalgebra
-: 165M bench/wta_Word,or_0,8_0.7_1408_2.coalgebra
-: 165M bench/wta_Word,or_0,8_0.7_1408_3.coalgebra
-: 165M bench/wta_Word,or_0,8_0.7_1408_4.coalgebra
-
-**** Automaton Size
-
-#+BEGIN_SRC sh
-python bench.py generate ../../copar/bin/random-wta --monoid 'Word,or' --symbols 0,8 --zero-frequency 0.7 --states 1408
-python bench.py run ../../copar/bin/copar --monoid 'Word,or' --symbols 0,8 --zero-frequency 0.7 --states 1408 --indiv --header
-#+END_SRC
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 4757493 | 9512170 | 1416 | 12672 | 1408 | 8913294 | 122.529064018 | 49.673619826 | 69.872621378 | 24.487186873 | 37.636595831 |
-| 1 | 4757305 | 9511794 | 1416 | 12672 | 1408 | 8912679 | 121.871882143 | 49.468009438 | 69.396700752 | 24.548033181 | 37.526833514 |
-| 2 | 4760364 | 9517912 | 1416 | 12672 | 1408 | 8918643 | 122.195401637 | 49.367139075 | 69.859922188 | 24.551202679 | 37.745627532 |
-| 3 | 4760232 | 9517648 | 1416 | 12672 | 1408 | 8918564 | 121.744240716 | 49.628682967 | 69.111452032 | 24.573090085 | 37.819264634 |
-| 4 | 4759753 | 9516690 | 1416 | 12672 | 1408 | 8917624 | 118.671354206 | 49.213411686 | 66.440058464 | 25.139216435 | 33.058420248 |
-
-*** GHC 8.4.4 with symbols 1,0,4 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid 'Word,or' --symbols 1,0,4 --zero-frequency 0.7 --start-states 120
-#+END_SRC
-
-#+RESULTS:
-: Trying 120...
-: Trying 240...
-: Trying 180...
-: Trying 150...
-: Trying 135...
-: Trying 142...
-: Trying 146...
-: Trying 148...
-: Trying 149...
-: First bad state count: 149
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_Word,or_1,0,4_0.7_148*
-#+END_SRC
-
-#+RESULTS:
-: 151M bench/wta_Word,or_1,0,4_0.7_148_0.coalgebra
-: 151M bench/wta_Word,or_1,0,4_0.7_148_1.coalgebra
-: 151M bench/wta_Word,or_1,0,4_0.7_148_2.coalgebra
-: 151M bench/wta_Word,or_1,0,4_0.7_148_3.coalgebra
-: 151M bench/wta_Word,or_1,0,4_0.7_148_4.coalgebra
-
-**** Automaton Size
-
-#+BEGIN_SRC sh
-python bench.py generate ../../copar/bin/random-wta --monoid 'Word,or' --symbols 1,0,4 --zero-frequency 0.7 --states 148
-python bench.py run ../../copar/bin/copar --monoid 'Word,or' --symbols 1,0,4 --zero-frequency 0.7 --states 148 --indiv --header
-#+END_SRC
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 3889583 | 11668189 | 153 | 87765 | 148 | 6805177 | 117.573809962 | 44.159477335 | 69.176479615 | 20.97881901 | 36.97313268 |
-| 1 | 3891836 | 11675004 | 153 | 87765 | 148 | 6809318 | 118.060187599 | 44.059742822 | 69.709799716 | 20.986764025 | 37.089169403 |
-| 2 | 3889498 | 11667962 | 153 | 87765 | 148 | 6805428 | 117.680086041 | 43.801892931 | 69.639925802 | 20.888711457 | 36.894495934 |
-| 3 | 3890675 | 11671481 | 153 | 87765 | 148 | 6807190 | 118.213230214 | 44.026034118 | 69.934017821 | 20.807216331 | 36.920223682 |
-| 4 | 3889335 | 11667465 | 153 | 87765 | 148 | 6804884 | 118.020940476 | 44.097067581 | 69.813347698 | 20.923033139 | 36.907214948 |
-
-*** GHC 8.4.4 with symbols 4,3,2 and zero-freq 0.7
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid 'Word,or' --symbols 4,3,2 --zero-frequency 0.7
-#+END_SRC
-
-#+RESULTS:
-: Trying 50...
-: Trying 100...
-: Trying 200...
-: Trying 150...
-: Trying 175...
-: Trying 187...
-: Trying 181...
-: Trying 184...
-: Trying 185...
-: Trying 186...
-: First bad state count: 187
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_Word,or_4,3,2_0.7_186*
-#+END_SRC
-
-#+RESULTS:
-: 152M bench/wta_Word,or_4,3,2_0.7_186_0.coalgebra
-: 152M bench/wta_Word,or_4,3,2_0.7_186_1.coalgebra
-: 152M bench/wta_Word,or_4,3,2_0.7_186_2.coalgebra
-: 152M bench/wta_Word,or_4,3,2_0.7_186_3.coalgebra
-: 152M bench/wta_Word,or_4,3,2_0.7_186_4.coalgebra
-
-**** Automaton Size
-
-#+BEGIN_SRC sh
-python bench.py generate ../../copar/bin/random-wta --monoid 'Word,or' --symbols 4,3,2 --zero-frequency 0.7 --states 186
-python bench.py run ../../copar/bin/copar --monoid 'Word,or' --symbols 4,3,2 --zero-frequency 0.7 --states 186 --indiv --header
-#+END_SRC
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 3890269 | 11638838 | 195 | 69940 | 186 | 5848699 | 114.733470175 | 43.499787726 | 67.5948969 | 17.660572753 | 37.835620352 |
-| 1 | 3893818 | 11649123 | 195 | 69940 | 186 | 5854005 | 114.756947419 | 43.702682275 | 67.505722972 | 17.675712966 | 37.818639736 |
-| 2 | 3893265 | 11647592 | 195 | 69940 | 186 | 5854134 | 115.418354168 | 43.606980048 | 68.133828807 | 17.837028275 | 37.992011256 |
-| 3 | 3892206 | 11644421 | 195 | 69940 | 186 | 5852423 | 113.998060941 | 43.769845642 | 66.618120946 | 17.720427448 | 37.714434677 |
-| 4 | 3892036 | 11643814 | 195 | 69940 | 186 | 5851296 | 115.965290499 | 44.138476758 | 68.234040288 | 17.763408679 | 38.356640369 |
-
-*** GHC 8.4.4 with symbols 0,0,0,0,0,3 and zero-freq 0.7
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid 'Word,or' --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --start-states 6
-#+END_SRC
-
-#+RESULTS:
-: Trying 6...
-: Trying 12...
-: Trying 9...
-: Trying 10...
-: Trying 11...
-: First bad state count: 12
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_Word,or_0,0,0,0,0,3_0.7_11*
-#+END_SRC
-
-#+RESULTS:
-: 77M bench/wta_Word,or_0,0,0,0,0,3_0.7_11_0.coalgebra
-: 77M bench/wta_Word,or_0,0,0,0,0,3_0.7_11_1.coalgebra
-: 77M bench/wta_Word,or_0,0,0,0,0,3_0.7_11_2.coalgebra
-: 77M bench/wta_Word,or_0,0,0,0,0,3_0.7_11_3.coalgebra
-: 77M bench/wta_Word,or_0,0,0,0,0,3_0.7_11_4.coalgebra
-
-**** Automaton Size
-
-#+BEGIN_SRC sh
-python bench.py generate ../../copar/bin/random-wta --monoid 'Word,or' --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --states 11
-python bench.py run ../../copar/bin/copar --monoid 'Word,or' --symbols 0,0,0,0,0,3 --zero-frequency 0.7 --states 11 --indiv --header
-#+END_SRC
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 1594413 | 9566412 | 14 | 473587 | 11 | 2676959 | 61.53939505 | 24.79942316 | 34.477272309 | 10.822832613 | 21.313535924 |
-| 1 | 1596360 | 9578094 | 14 | 473404 | 11 | 2681034 | 61.147599248 | 24.824030133 | 34.101896988 | 10.793184 | 20.989457877 |
-| 2 | 1595591 | 9573480 | 14 | 473700 | 11 | 2679546 | 61.459443653 | 24.801616514 | 34.389540062 | 10.893824596 | 21.133849133 |
-| 3 | 1593825 | 9562884 | 14 | 473580 | 11 | 2676789 | 61.147557505 | 24.98879381 | 33.942496577 | 10.818043846 | 20.839884809 |
-| 4 | 1596528 | 9579102 | 14 | 473600 | 11 | 2681048 | 61.864094058 | 24.884111862 | 34.762990208 | 10.893522867 | 21.485077775 |
-* Same Question, but with limited transition count
-** Powerset
-*** GHC 8.4.4 with symbols 0,8 and transitions 500
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/{random-wta,copar} --monoid powerset --symbols 0,8 --transitions 5000
-#+END_SRC
-
-#+RESULTS:
-: Trying 1000...
-: Trying 2000...
-: Trying 1500...
-: Trying 1250...
-: Trying 1375...
-: Trying 1437...
-: Trying 1468...
-: Trying 1484...
-: Trying 1476...
-: Trying 1480...
-: Trying 1478...
-: Trying 1479...
-: First bad state count: 1479
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_powerset_0,8_0.7_1478*
-#+END_SRC
-
-#+RESULTS:
-: 82M bench/wta_powerset_0,8_0.7_1478_0.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_1.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_2.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_3.coalgebra
-: 82M bench/wta_powerset_0,8_0.7_1478_4.coalgebra
-
-**** Automaton size
-
-#+begin_src sh :hlines yes
-python bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,8 --zero-frequency 0.7 --states 1478
-python bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,8 --zero-frequency 0.7 --states 1478 --indiv --header
-#+end_src
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 5243856 | 10484756 | 10 | 18 | 2 | 0 | 115.339077581 | 34.439478263 | 79.458232767 | 27.712258503 | 37.668249537 |
-| 1 | 5245405 | 10487854 | 10 | 18 | 2 | 0 | 114.001430765 | 34.477874858 | 78.07613373 | 27.764347556 | 37.074617863 |
-| 2 | 5243549 | 10484142 | 10 | 18 | 2 | 0 | 114.862812706 | 34.668409012 | 78.756016051 | 27.741448159 | 37.09480403 |
-| 3 | 5241660 | 10480364 | 10 | 18 | 2 | 0 | 114.228588077 | 34.362576104 | 78.424601434 | 27.648526393 | 37.043127108 |
-| 4 | 5243032 | 10483108 | 10 | 18 | 2 | 0 | 114.692551901 | 34.827430372 | 78.414037199 | 27.723198479 | 38.276163878 |
-
-*** GHC 8.4.4 with symbols 0,8 and out-degree 1000
-
-#+BEGIN_SRC sh
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid powerset --symbols 0,8 --out-degree 1000 --good 1000
-#+END_SRC
-
-#+RESULTS:
-: Trying 1001...
-: Trying 2002...
-: Trying 4004...
-: Trying 8008...
-: Trying 6006...
-: Trying 7007...
-: Trying 7507...
-: Trying 7257...
-: Trying 7382...
-: Trying 7444...
-: Trying 7413...
-: Trying 7428...
-: Trying 7420...
-: Trying 7416...
-: Trying 7414...
-: Trying 7415...
-: First bad state count: 7416
-
-**** File Size
-
-#+BEGIN_SRC sh :results output
-ls -sh bench/wta_powerset_0,8_t7415000__7415*
-#+END_SRC
-
-#+RESULTS:
-: 120M bench/wta_powerset_0,8_t7415000__7415_0.coalgebra
-: 120M bench/wta_powerset_0,8_t7415000__7415_1.coalgebra
-: 120M bench/wta_powerset_0,8_t7415000__7415_2.coalgebra
-: 120M bench/wta_powerset_0,8_t7415000__7415_3.coalgebra
-: 120M bench/wta_powerset_0,8_t7415000__7415_4.coalgebra
-
-**** Automaton Size
-
-#+begin_src sh :hlines yes
-python bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,8 --transitions 7415000 --states 7415
-python bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,8 --transitions 7415000 --states 7415 --indiv --header
-#+end_src
-
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 7422415 | 14830000 | 10 | 18 | 2 | 0 | 157.872966092 | 51.221314999 | 104.615248956 | 33.7012898 | 58.664739269 |
-| 1 | 7422415 | 14830000 | 10 | 18 | 2 | 0 | 157.428380106 | 51.433510926 | 103.926494829 | 33.216537153 | 58.775334589 |
-| 2 | 7422415 | 14830000 | 10 | 18 | 2 | 0 | 156.808413994 | 51.343226544 | 103.40051371 | 32.969125294 | 58.865798525 |
-| 3 | 7422415 | 14830000 | 10 | 18 | 2 | 0 | 157.791363724 | 51.873644617 | 103.855838314 | 33.05713406 | 58.931293578 |
-| 4 | 7422415 | 14830000 | 10 | 18 | 2 | 0 | 157.126256596 | 51.400301031 | 103.666329167 | 32.976280595 | 58.830566981 |
-
-* Next Try
-
-We now use the following parameters:
-
-- Signature :: 4×X^k k ∈ { 0, ..., 5 }
-- GHC :: 8.4.4
-
-** Powerset
-*** 4xX mit out-degree 500
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid powerset --symbols 0,4 --out-degree 500 --good 8000 --start-states 12000 --bad 16000
-#+END_SRC
-
-#+RESULTS:
-: Trying 12000...
-: Trying 14000...
-: Trying 15000...
-: Trying 14500...
-: Trying 14250...
-: Trying 14375...
-: Trying 14312...
-: Trying 14281...
-: Trying 14296...
-: Trying 14288...
-: Trying 14284...
-: Trying 14282...
-: First bad state count: 14282
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_powerset_0,4_*__14281_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: power_1_500_file
-#+RESULTS:
-| 118M | bench/wta_powerset_0.4_t7140500__14281_0.coalgebra |
-| 118M | bench/wta_powerset_0.4_t7140500__14281_1.coalgebra |
-| 118M | bench/wta_powerset_0.4_t7140500__14281_2.coalgebra |
-| 118M | bench/wta_powerset_0.4_t7140500__14281_3.coalgebra |
-| 118M | bench/wta_powerset_0.4_t7140500__14281_4.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,4 --out-degree 500 --states 14281
-./bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,4 --out-degree 500 --states 14281 --indiv --header
-#+END_SRC
-
-#+NAME: power_1_500
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 7154781 | 14281000 | 6 | 10 | 2 | 0 | 155.635043092 | 50.511676534 | 103.239235361 | 30.135940008 | 61.445316625 |
-| 1 | 7154781 | 14281000 | 6 | 10 | 2 | 0 | 155.345664537 | 50.391815858 | 103.046448356 | 29.52881681 | 61.823502784 |
-| 2 | 7154781 | 14281000 | 6 | 10 | 2 | 0 | 154.746389702 | 50.356318464 | 102.487287332 | 29.561887582 | 61.245616512 |
-| 3 | 7154781 | 14281000 | 6 | 10 | 2 | 0 | 156.681042711 | 50.477922803 | 104.306181251 | 30.132009036 | 62.231759233 |
-| 4 | 7154781 | 14281000 | 6 | 10 | 2 | 0 | 156.206931801 | 50.552217812 | 103.775304876 | 30.106012782 | 61.793692847 |
-
-*** 4xX mit out-degree 20
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid powerset --symbols 0,4 --out-degree 20 --good 8000 --start-states 14281
-#+END_SRC
-
-#+RESULTS:
-: Trying 14281...
-: Trying 28562...
-: Trying 57124...
-: Trying 114248...
-: Trying 228496...
-: Trying 456992...
-: Trying 342744...
-: Trying 399868...
-: Trying 371306...
-: Trying 357025...
-: Trying 349884...
-: Trying 353454...
-: Trying 355239...
-: Trying 356132...
-: Trying 355685...
-: Trying 355462...
-: Trying 355573...
-: Trying 355629...
-: Trying 355657...
-: Trying 355671...
-: Trying 355678...
-: Trying 355681...
-: Trying 355683...
-: Trying 355684...
-: First bad state count: 355685
-
-**** File Size
-
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,4 --out-degree 20 --states 355684
-./bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,4 --out-degree 20 --states 355684 --indiv --header
-#+END_SRC
-
-#+NAME: power_1_20
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 7469364 | 14227360 | 6 | 1768728 | 355684 | 10883373 | 344.399732107 | 61.942569869 | 262.572157652 | 32.047148894 | 209.333188906 |
-| 1 | 7469364 | 14227360 | 6 | 1768935 | 355684 | 10883077 | 340.732038227 | 60.912573518 | 260.176273796 | 31.928906963 | 208.837646082 |
-| 2 | 7469364 | 14227360 | 6 | 1768908 | 355684 | 10885631 | 340.838739608 | 61.120402905 | 260.008098908 | 31.951679391 | 208.775189688 |
-| 3 | 7469364 | 14227360 | 6 | 1768880 | 355684 | 10880372 | 340.237713603 | 61.448840572 | 259.027481579 | 31.936433445 | 207.318948419 |
-| 4 | 7469364 | 14227360 | 6 | 1768875 | 355684 | 10886155 | 342.792217839 | 61.487509806 | 261.434824046 | 32.045849362 | 209.995181997 |
-
-*** 4xX mit out-degree 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid powerset --symbols 0,4 --out-degree 50 --good 130000 --bad 140000
-#+END_SRC
-
-#+RESULTS:
-: Trying 135000...
-: Trying 132500...
-: Trying 131250...
-: Trying 131875...
-: Trying 132187...
-: Trying 132031...
-: Trying 132109...
-: Trying 132148...
-: Trying 132167...
-: Trying 132177...
-: Trying 132182...
-: Trying 132179...
-: Trying 132178...
-: First bad state count: 132179
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_powerset_0,4_*_132178_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: power_1_file
-#+RESULTS:
-| 117M | bench/wta_powerset_0.4_t6608900__132178_0.coalgebra |
-| 117M | bench/wta_powerset_0.4_t6608900__132178_1.coalgebra |
-| 117M | bench/wta_powerset_0.4_t6608900__132178_2.coalgebra |
-| 117M | bench/wta_powerset_0.4_t6608900__132178_3.coalgebra |
-| 117M | bench/wta_powerset_0.4_t6608900__132178_4.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,4 --out-degree 50 --states 132177
-./bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,4 --out-degree 50 --states 132177 --indiv --header
-#+END_SRC
-
-#+NAME: power_1
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 6741027 | 13217700 | 6 | 660882 | 132177 | 9323005 | 265.849947508 | 53.024255888 | 200.012469965 | 32.711091055 | 156.049078211 |
-| 1 | 6741027 | 13217700 | 6 | 660884 | 132177 | 9336757 | 266.027492928 | 53.656205959 | 199.630533624 | 33.784156055 | 154.30670604 |
-| 2 | 6741027 | 13217700 | 6 | 660885 | 132177 | 9295860 | 266.131064164 | 53.031692734 | 200.333990554 | 33.839128343 | 154.752397766 |
-| 3 | 6741027 | 13217700 | 6 | 660882 | 132177 | 9287608 | 266.57863175 | 53.338030998 | 200.492577153 | 33.341400299 | 155.694877544 |
-| 4 | 6741027 | 13217700 | 6 | 660884 | 132177 | 9347998 | 265.39360727 | 52.816564437 | 199.812521066 | 33.25935421 | 155.166041015 |
-
-*** 4xX^2 mit out-degree 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid powerset --symbols 0,0,4 --out-degree 50 --start-states 80000
-#+END_SRC
-
-#+RESULTS:
-: Trying 80000...
-: Trying 160000...
-: Trying 120000...
-: Trying 100000...
-: Trying 90000...
-: Trying 95000...
-: Trying 97500...
-: Trying 98750...
-: Trying 98125...
-: Trying 98437...
-: Trying 98593...
-: Trying 98671...
-: Trying 98632...
-: Trying 98651...
-: Trying 98661...
-: Trying 98666...
-: Trying 98668...
-: Trying 98669...
-: Trying 98670...
-: First bad state count: 98671
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_powerset_0,0,4_*_98670_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: power_2_file
-#+RESULTS:
-| 123M | bench/wta_powerset_0.0.4_t4933500__98670_0.coalgebra |
-| 123M | bench/wta_powerset_0.0.4_t4933500__98670_1.coalgebra |
-| 123M | bench/wta_powerset_0.0.4_t4933500__98670_2.coalgebra |
-| 123M | bench/wta_powerset_0.0.4_t4933500__98670_3.coalgebra |
-| 123M | bench/wta_powerset_0.0.4_t4933500__98670_4.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,0,4 --out-degree 50 --states 98670
-./bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,0,4 --out-degree 50 --states 98670 --indiv --header
-#+END_SRC
-
-#+NAME: power_2
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 5032170 | 14800500 | 6 | 5031861 | 98670 | 11563672 | 318.18771425 | 46.231443471 | 259.512847417 | 31.400595851 | 212.323222545 |
-| 1 | 5032170 | 14800500 | 6 | 5031879 | 98670 | 11565958 | 318.887658897 | 46.483533968 | 259.856085361 | 31.407316023 | 212.38857748 |
-| 2 | 5032170 | 14800500 | 6 | 5031848 | 98670 | 11565284 | 318.529242826 | 46.499900763 | 259.501330137 | 31.347128979 | 210.955648567 |
-| 3 | 5032170 | 14800500 | 6 | 5031845 | 98670 | 11558234 | 317.809993575 | 46.522121526 | 258.840719384 | 31.371523591 | 211.925301877 |
-| 4 | 5032170 | 14800500 | 6 | 5031882 | 98670 | 11551538 | 316.309936896 | 46.212332602 | 257.508264904 | 31.362333309 | 210.039264838 |
-
-*** 4xX^3 mit out-degree 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid powerset --symbols 0,0,0,4 --out-degree 50 --start-states 60000
-#+END_SRC
-
-#+RESULTS:
-: Trying 60000...
-: Trying 120000...
-: Trying 90000...
-: Trying 75000...
-: Trying 82500...
-: Trying 86250...
-: Trying 84375...
-: Trying 85312...
-: Trying 84843...
-: Trying 85077...
-: Trying 84960...
-: Trying 85018...
-: Trying 84989...
-: Trying 85003...
-: Trying 85010...
-: Trying 85014...
-: Trying 85016...
-: Trying 85017...
-: First bad state count: 85018
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_powerset_0,0,0,4_*_85017_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: power_3_file
-#+RESULTS:
-| 138M | bench/wta_powerset_0.0.0.4_t4250850__85017_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,0,0,4 --out-degree 50 --states 85016
-./bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,0,0,4 --out-degree 50 --states 85016 --indiv --header
-#+END_SRC
-
-#+NAME: power_3
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 4335816 | 17003200 | 6 | 4335816 | 85016 | 11932210 | 261.096600392 | 47.614392976 | 203.643561011 | 20.446586332 | 167.587154416 |
-
-*** 4xX^4 mit out-degree 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid powerset --symbols 0,0,0,0,4 --out-degree 50 --start-states 48000
-#+END_SRC
-
-#+RESULTS:
-: Trying 48000...
-: Trying 96000...
-: Trying 72000...
-: Trying 60000...
-: Trying 54000...
-: Trying 57000...
-: Trying 58500...
-: Trying 59250...
-: Trying 59625...
-: Trying 59437...
-: Trying 59531...
-: Trying 59578...
-: Trying 59601...
-: Trying 59589...
-: Trying 59595...
-: Trying 59598...
-: Trying 59596...
-: First bad state count: 59596
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_powerset_0,0,0,0,4_*_59596_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: power_4_file
-#+RESULTS:
-| 119M | bench/wta_powerset_0.0.0.0.4_t2979800__59596_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,0,0,0,4 --out-degree 50 --states 59596
-./bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,0,0,0,4 --out-degree 50 --states 59596 --indiv --header
-#+END_SRC
-
-#+NAME: power_4
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 3039396 | 14899000 | 6 | 3039396 | 59596 | 13024716 | 200.034099951 | 41.065064963 | 152.485306449 | 25.383501709 | 121.316305478 |
-
-*** 4xX^5 mit out-degree 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid powerset --symbols 0,0,0,0,0,4 --out-degree 50 --start-states 40000
-#+END_SRC
-
-#+RESULTS:
-: Trying 40000...
-: Trying 80000...
-: Trying 60000...
-: Trying 50000...
-: Trying 45000...
-: Trying 47500...
-: Trying 48750...
-: Trying 49375...
-: Trying 49687...
-: Trying 49531...
-: Trying 49453...
-: Trying 49414...
-: Trying 49394...
-: Trying 49384...
-: Trying 49379...
-: Trying 49377...
-: Trying 49376...
-: First bad state count: 49376
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_powerset_0,0,0,0,0,4_*_49375_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: power_5_file
-#+RESULTS:
-| 116M | bench/wta_powerset_0.0.0.0.0.4_t2468750__49375_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid powerset --symbols 0,0,0,0,0,4 --out-degree 50 --states 49375
-./bench.py run ../../copar/bin/copar --monoid powerset --symbols 0,0,0,0,0,4 --out-degree 50 --states 49375 --indiv --header
-#+END_SRC
-
-#+NAME: power_5
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 2518125 | 14812500 | 6 | 2518125 | 49375 | 13940781 | 164.986249384 | 38.769441334 | 119.843574846 | 24.415685672 | 90.703821725 |
-
-** Z,max
-*** 4xX mit out-degree 50 und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Z,max --symbols 0,4 --out-degree 50 --good 100000 --bad 130000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 115000...
-: Trying 107500...
-: Trying 111250...
-: Trying 113125...
-: Trying 114062...
-: Trying 114531...
-: Trying 114765...
-: Trying 114882...
-: Trying 114941...
-: Trying 114911...
-: Trying 114896...
-: Trying 114889...
-: Trying 114885...
-: Trying 114887...
-: Trying 114888...
-: First bad state count: 114889
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Z,max_0,4_*_114888_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: zmax_1_file
-#+RESULTS:
-| 122M | bench/wta_Z.max_0.4_t5744400_50_114888_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Z,max --symbols 0,4 --out-degree 50 --different-values 50 --states 114888
-./bench.py run ../../copar/bin/copar --monoid Z,max --symbols 0,4 --out-degree 50 --different-values 50 --states 114888 --indiv --header
-#+END_SRC
-
-#+NAME: zmax_1
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 5859288 | 11488800 | 416 | 574439 | 114888 | 6911761 | 181.011843864 | 58.957254337 | 111.080014285 | 34.573326846 | 66.278452727 |
-
-*** 4xX^2 mit out-degree 50 und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Z,max --symbols 0,0,4 --out-degree 50 --good 50000 --bad 100000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 75000...
-: Trying 87500...
-: Trying 93750...
-: Trying 96875...
-: Trying 95312...
-: Trying 94531...
-: Trying 94921...
-: Trying 95116...
-: Trying 95214...
-: Trying 95263...
-: Trying 95287...
-: Trying 95299...
-: Trying 95293...
-: Trying 95290...
-: Trying 95288...
-: First bad state count: 95288
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Z,max_0,0,4_*_95287_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: zmax_2_file
-#+RESULTS:
-| 136M | bench/wta_Z.max_0.0.4_t4764350_50_95287_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Z,max --symbols 0,0,4 --out-degree 50 --different-values 50 --states 95287
-./bench.py run ../../copar/bin/copar --monoid Z,max --symbols 0,0,4 --out-degree 50 --different-values 50 --states 95287 --indiv --header
-#+END_SRC
-
-#+NAME: zmax_2
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 4859637 | 14293050 | 404 | 4859343 | 95287 | 7100778 | 222.092892824 | 54.223932252 | 154.676164911 | 30.811423501 | 108.364036654 |
-
-*** 4xX^3 mit out-degree und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Z,max --symbols 0,0,0,4 --out-degree 50 --good 58471 --bad 73000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 65735...
-: Trying 69367...
-: Trying 71183...
-: Trying 70275...
-: Trying 70729...
-: Trying 70502...
-: Trying 70615...
-: Trying 70672...
-: Trying 70643...
-: Trying 70657...
-: Trying 70664...
-: Trying 70660...
-: Trying 70662...
-: Trying 70661...
-: First bad state count: 70661
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Z,max_0,0,0,4_*_70660_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: zmax_3_file
-#+RESULTS:
-| 127M | bench/wta_Z.max_0.0.0.4_t3533000_50_70660_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Z,max --symbols 0,0,0,4 --out-degree 50 --different-values 50 --states 70660
-./bench.py run ../../copar/bin/copar --monoid Z,max --symbols 0,0,0,4 --out-degree 50 --different-values 50 --states 70660 --indiv --header
-#+END_SRC
-
-#+NAME: zmax_3
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 3603660 | 14132000 | 397 | 3603660 | 70660 | 9226293 | 172.420794076 | 49.287429 | 115.080725376 | 25.88631871 | 82.245759651 |
-
-*** 4xX^4 mit out-degree und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Z,max --symbols 0,0,0,0,4 --out-degree 50 --good 55181 --bad 73000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 64090...
-: Trying 59635...
-: Trying 61862...
-: Trying 62976...
-: Trying 62419...
-: Trying 62697...
-: Trying 62558...
-: Trying 62627...
-: Trying 62662...
-: Trying 62679...
-: Trying 62670...
-: Trying 62666...
-: Trying 62664...
-: Trying 62665...
-: First bad state count: 62666
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Z,max_0,0,0,0,4_*_62665_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: zmax_4_file
-#+RESULTS:
-| 136M | bench/wta_Z.max_0.0.0.0.4_t3133250_50_62665_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Z,max --symbols 0,0,0,0,4 --out-degree 50 --different-values 50 --states 62665
-./bench.py run ../../copar/bin/copar --monoid Z,max --symbols 0,0,0,0,4 --out-degree 50 --different-values 50 --states 62665 --indiv --header
-#+END_SRC
-
-#+NAME: zmax_4
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 3195915 | 15666250 | 397 | 3195915 | 62665 | 11627719 | 154.99753875 | 48.473165 | 98.758726744 | 26.2347205 | 66.312403669 |
-
-*** 4xX^5 mit out-degree und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Z,max --symbols 0,0,0,0,0,4 --out-degree 50 --good 31000 --bad 55000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 43000...
-: Trying 49000...
-: Trying 52000...
-: Trying 50500...
-: Trying 49750...
-: Trying 50125...
-: Trying 49937...
-: Trying 49843...
-: Trying 49890...
-: Trying 49913...
-: Trying 49925...
-: Trying 49931...
-: Trying 49928...
-: Trying 49926...
-: Trying 49927...
-: First bad state count: 49927
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Z,max_0,0,0,0,0,4_*_49926_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: zmax_5_file
-#+RESULTS:
-| 127M | bench/wta_Z.max_0.0.0.0.0.4_t2496300_50_49926_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Z,max --symbols 0,0,0,0,0,4 --out-degree 50 --different-values 50 --states 49926
-./bench.py run ../../copar/bin/copar --monoid Z,max --symbols 0,0,0,0,0,4 --out-degree 50 --different-values 50 --states 49926 --indiv --header
-#+END_SRC
-
-#+NAME: zmax_5
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 2546226 | 14977800 | 376 | 2546226 | 49926 | 11841832 | 129.05925319 | 44.887706321 | 78.246885415 | 20.851582498 | 52.50648174 |
-
-** Word,or
-*** 4xX mit out-degree und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Word,or --symbols 0,4 --out-degree 50 --good 100000 --bad 130000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 115000...
-: Trying 107500...
-: Trying 111250...
-: Trying 113125...
-: Trying 114062...
-: Trying 113593...
-: Trying 113827...
-: Trying 113944...
-: Trying 114003...
-: Trying 113973...
-: Trying 113958...
-: Trying 113951...
-: Trying 113954...
-: Trying 113956...
-: Trying 113957...
-: First bad state count: 113958
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Word,or_0,4_*_113957_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: wordor_1_file
-#+RESULTS:
-| 131M | bench/wta_Word.or_0.4_t5697850_50_113957_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Word,or --symbols 0,4 --out-degree 50 --different-values 50 --states 113957
-./bench.py run ../../copar/bin/copar --monoid Word,or --symbols 0,4 --out-degree 50 --different-values 50 --states 113957 --indiv --header
-#+END_SRC
-
-#+NAME: wordor_1
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 5811807 | 11395700 | 54 | 569783 | 113957 | 11076343 | 223.763784071 | 61.587433127 | 152.359932717 | 32.126284469 | 109.721586786 |
-
-*** 4xX^2 mit out-degree und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Word,or --symbols 0,0,4 --out-degree 50 --bad 93001 --good 80000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 86500...
-: Trying 89750...
-: Trying 91375...
-: Trying 92188...
-: Trying 92594...
-: Trying 92391...
-: Trying 92492...
-: Trying 92441...
-: Trying 92416...
-: Trying 92428...
-: Trying 92434...
-: Trying 92437...
-: Trying 92435...
-: First bad state count: 92435
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Word,or_0,0,4_*_92434_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: wordor_2_file
-#+RESULTS:
-| 141M | bench/wta_Word.or_0.0.4_t4621700_50_92434_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Word,or --symbols 0,0,4 --out-degree 50 --different-values 50 --states 92434
-./bench.py run ../../copar/bin/copar --monoid Word,or --symbols 0,0,4 --out-degree 50 --different-values 50 --states 92434 --indiv --header
-#+END_SRC
-
-#+NAME: wordor_2
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 4714134 | 13865100 | 54 | 4713833 | 92434 | 13286924 | 258.308924373 | 55.730150301 | 184.86496102 | 31.15297126 | 144.089898137 |
-
-*** 4xX^3 mit out-degree und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Word,or --symbols 0,0,0,4 --out-degree 50 --good 67000 --bad 73000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 70000...
-: Trying 68500...
-: Trying 69250...
-: Trying 69625...
-: Trying 69437...
-: Trying 69531...
-: Trying 69578...
-: Trying 69601...
-: Trying 69613...
-: Trying 69619...
-: Trying 69622...
-: Trying 69623...
-: Trying 69624...
-: First bad state count: 69624
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Word,or_0,0,0,4_*_69623_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: wordor_3_file
-#+RESULTS:
-| 132M | bench/wta_Word.or_0.0.0.4_t3481150_50_69623_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Word,or --symbols 0,0,0,4 --out-degree 50 --different-values 50 --states 69623
-./bench.py run ../../copar/bin/copar --monoid Word,or --symbols 0,0,0,4 --out-degree 50 --different-values 50 --states 69623 --indiv --header
-#+END_SRC
-
-#+NAME: wordor_3
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 3550773 | 13924600 | 54 | 3550773 | 69623 | 13715478 | 217.684389256 | 49.986173123 | 159.050504019 | 25.229488197 | 127.060520493 |
-
-*** 4xX^4 mit out-degree und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Word,or --symbols 0,0,0,0,4 --out-degree 50 --bad 60001 --good 55000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 57500...
-: Trying 56250...
-: Trying 56875...
-: Trying 57187...
-: Trying 57343...
-: Trying 57265...
-: Trying 57304...
-: Trying 57323...
-: Trying 57313...
-: Trying 57318...
-: Trying 57320...
-: Trying 57319...
-: First bad state count: 57320
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Word,or_0,0,0,0,4_*_57319_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: wordor_4_file
-#+RESULTS:
-| 130M | bench/wta_Word.or_0.0.0.0.4_t2865950_50_57319_0.coalgebra |
-
-**** Automaton Size
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Word,or --symbols 0,0,0,0,4 --out-degree 50 --different-values 50 --states 57319
-./bench.py run ../../copar/bin/copar --monoid Word,or --symbols 0,0,0,0,4 --out-degree 50 --different-values 50 --states 57319 --indiv --header
-#+END_SRC
-
-#+NAME: wordor_4
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 2923269 | 14329750 | 54 | 2923269 | 57319 | 15469967 | 199.956928585 | 47.318311842 | 146.418279061 | 25.807646931 | 115.015762137 |
-
-*** 4xX^5 mit out-degree und different-values 50
-
-#+BEGIN_SRC shell
-./bench.py bisect ../../copar/bin/random-wta ../../copar/bin/copar --monoid Word,or --symbols 0,0,0,0,0,4 --out-degree 50 --good 48000 --bad 52000 --different-values 50
-#+END_SRC
-
-#+RESULTS:
-: Trying 50000...
-: Trying 49000...
-: Trying 48500...
-: Trying 48750...
-: Trying 48875...
-: Trying 48937...
-: Trying 48968...
-: Trying 48952...
-: Trying 48960...
-: Trying 48964...
-: Trying 48962...
-: Trying 48963...
-: First bad state count: 48964
-
-**** File Size
-
-#+BEGIN_SRC sh :results output table
-ls -sh bench/wta_Word,or_0,0,0,0,0,4_*_48963_* | tr " ," ",."
-#+END_SRC
-
-#+NAME: wordor_5_file
-#+RESULTS:
-| 129M | bench/wta_Word.or_0.0.0.0.0.4_t2448150_50_48963_0.coalgebra |
-
-**** Automaton Size
-
-Aus irgendeinem Grund läuft jetzt 48963 auch schon ins memory limit, deswegen 48962
-
-#+BEGIN_SRC shell
-./bench.py generate ../../copar/bin/random-wta --monoid Word,or --symbols 0,0,0,0,0,4 --out-degree 50 --different-values 50 --states 48962
-./bench.py run ../../copar/bin/copar --monoid Word,or --symbols 0,0,0,0,0,4 --out-degree 50 --different-values 50 --states 48962 --indiv --header
-#+END_SRC
-
-#+NAME: wordor_5
-#+RESULTS:
-| i | states | edges | initial-partition-size | final-partition-size | explicit-final-partition-size | size1-skipped | overall-duration | parse-duration | algorithm-duration | initialize-duration | refine-duration |
-| 0 | 2497062 | 14688600 | 54 | 2497062 | 48962 | 18284567 | 167.692238761 | 45.179623229 | 117.531354091 | 20.079035336 | 92.723851956 |
-** Summary
-*** 4xX
-
-| Monoid | States | States+Interm. | Edges | File Size | init part | final part | t_p | t_i | t_r |
-|----------+--------+----------------+----------+-----------+-----------+------------+-----------+-----------+-----------|
-| powerset | 132177 | 6741027 | 13217700 | 117 M | 6 | 132177 | 53.024256 | 32.711091 | 156.04908 |
-| Z,max | 114888 | 5859288 | 11488800 | 122 M | 416 | 114888 | 58.957254 | 34.573327 | 66.278453 |
-| Word,or | 113957 | 5811807 | 11395700 | 131 M | 54 | 113957 | 61.587433 | 32.126284 | 109.72159 |
-#+TBLFM: $2=$3/51::@2$3=remote(power_1,@2$2)::@2$4=remote(power_1,@2$3)::@2$5=remote(power_1_file,@1$1)::@2$6=remote(power_1, @2$4)::@2$7=remote(power_1, @2$6)::@2$8=remote(power_1,@2$9)::@2$9=remote(power_1,@2$11)::@2$10=remote(power_1,@2$12)::@3$3=remote(zmax_1,@2$2)::@3$4=remote(zmax_1,@2$3)::@3$5=remote(zmax_1_file,@1$1)::@3$6=remote(zmax_1, @2$4)::@3$7=remote(zmax_1, @2$6)::@3$8=remote(zmax_1,@2$9)::@3$9=remote(zmax_1,@2$11)::@3$10=remote(zmax_1,@2$12)::@4$3=remote(wordor_1,@2$2)::@4$4=remote(wordor_1,@2$3)::@4$5=remote(wordor_1_file,@1$1)::@4$6=remote(wordor_1, @2$4)::@4$7=remote(wordor_1, @2$6)::@4$8=remote(wordor_1,@2$9)::@4$9=remote(wordor_1,@2$11)::@4$10=remote(wordor_1,@2$12)
-
-*** 4xX^2
-
-| Monoid | States | States+Interm. | Edges | File Size | init part | final part | t_p | t_i | t_r |
-|----------+--------+----------------+----------+-----------+-----------+------------+-----------+-----------+-----------|
-| powerset | 98670 | 5032170 | 14800500 | 123 M | 6 | 98670 | 46.231443 | 31.400596 | 212.32322 |
-| Z,max | 95287 | 4859637 | 14293050 | 136 M | 404 | 95287 | 54.223932 | 30.811424 | 108.36404 |
-| Word,or | 92434 | 4714134 | 13865100 | 141 M | 54 | 92434 | 55.730150 | 31.152971 | 144.08990 |
-#+TBLFM: $2=$3/51::@2$3=remote(power_2,@2$2)::@2$4=remote(power_2,@2$3)::@2$5=remote(power_2_file,@1$1)::@2$6=remote(power_2, @2$4)::@2$7=remote(power_2, @2$6)::@2$8=remote(power_2,@2$9)::@2$9=remote(power_2,@2$11)::@2$10=remote(power_2,@2$12)::@3$3=remote(zmax_2,@2$2)::@3$4=remote(zmax_2,@2$3)::@3$5=remote(zmax_2_file,@1$1)::@3$6=remote(zmax_2, @2$4)::@3$7=remote(zmax_2, @2$6)::@3$8=remote(zmax_2,@2$9)::@3$9=remote(zmax_2,@2$11)::@3$10=remote(zmax_2,@2$12)::@4$3=remote(wordor_2,@2$2)::@4$4=remote(wordor_2,@2$3)::@4$5=remote(wordor_2_file,@1$1)::@4$6=remote(wordor_2, @2$4)::@4$7=remote(wordor_2, @2$6)::@4$8=remote(wordor_2,@2$9)::@4$9=remote(wordor_2,@2$11)::@4$10=remote(wordor_2,@2$12)
-
-*** 4xx^3
-
-| Monoid | States | States+Interm. | Edges | File Size | init part | final part | t_p | t_i | t_r |
-|----------+--------+----------------+----------+-----------+-----------+------------+-----------+-----------+-----------|
-| powerset | 85016 | 4335816 | 17003200 | 138 M | 6 | 85016 | 47.614393 | 20.446586 | 167.58715 |
-| Z,max | 70660 | 3603660 | 14132000 | 127 M | 397 | 70660 | 49.287429 | 25.886319 | 82.245760 |
-| Word,or | 69623 | 3550773 | 13924600 | 132 M | 54 | 69623 | 49.986173 | 25.229488 | 127.06052 |
-#+TBLFM: $2=$3/51::@2$3=remote(power_3,@2$2)::@2$4=remote(power_3,@2$3)::@2$5=remote(power_3_file,@1$1)::@2$6=remote(power_3, @2$4)::@2$7=remote(power_3, @2$6)::@2$8=remote(power_3,@2$9)::@2$9=remote(power_3,@2$11)::@2$10=remote(power_3,@2$12)::@3$3=remote(zmax_3,@2$2)::@3$4=remote(zmax_3,@2$3)::@3$5=remote(zmax_3_file,@1$1)::@3$6=remote(zmax_3, @2$4)::@3$7=remote(zmax_3, @2$6)::@3$8=remote(zmax_3,@2$9)::@3$9=remote(zmax_3,@2$11)::@3$10=remote(zmax_3,@2$12)::@4$3=remote(wordor_3,@2$2)::@4$4=remote(wordor_3,@2$3)::@4$5=remote(wordor_3_file,@1$1)::@4$6=remote(wordor_3, @2$4)::@4$7=remote(wordor_3, @2$6)::@4$8=remote(wordor_3,@2$9)::@4$9=remote(wordor_3,@2$11)::@4$10=remote(wordor_3,@2$12)
-
-*** 4xx^4
-
-| Monoid | States | States+Interm. | Edges | File Size | init part | final part | t_p | t_i | t_r |
-|----------+--------+----------------+----------+-----------+-----------+------------+-----------+-----------+-----------|
-| powerset | 59596 | 3039396 | 14899000 | 119 M | 6 | 59596 | 41.065065 | 25.383502 | 121.31631 |
-| Z,max | 62665 | 3195915 | 15666250 | 136 M | 397 | 62665 | 48.473165 | 26.234721 | 66.312404 |
-| Word,or | 57319 | 2923269 | 14329750 | 130 M | 54 | 57319 | 47.318312 | 25.807647 | 115.01576 |
-#+TBLFM: $2=$3/51::@2$3=remote(power_4,@2$2)::@2$4=remote(power_4,@2$3)::@2$5=remote(power_4_file,@1$1)::@2$6=remote(power_4, @2$4)::@2$7=remote(power_4, @2$6)::@2$8=remote(power_4,@2$9)::@2$9=remote(power_4,@2$11)::@2$10=remote(power_4,@2$12)::@3$3=remote(zmax_4,@2$2)::@3$4=remote(zmax_4,@2$3)::@3$5=remote(zmax_4_file,@1$1)::@3$6=remote(zmax_4, @2$4)::@3$7=remote(zmax_4, @2$6)::@3$8=remote(zmax_4,@2$9)::@3$9=remote(zmax_4,@2$11)::@3$10=remote(zmax_4,@2$12)::@4$3=remote(wordor_4,@2$2)::@4$4=remote(wordor_4,@2$3)::@4$5=remote(wordor_4_file,@1$1)::@4$6=remote(wordor_4, @2$4)::@4$7=remote(wordor_4, @2$6)::@4$8=remote(wordor_4,@2$9)::@4$9=remote(wordor_4,@2$11)::@4$10=remote(wordor_4,@2$12)
-
-*** 4xx^5
-
-| Monoid | States | States+Interm. | Edges | File Size | init part | final part | t_p | t_i | t_r |
-|----------+--------+----------------+----------+-----------+-----------+------------+-----------+-----------+-----------|
-| powerset | 49375 | 2518125 | 14812500 | 116 M | 6 | 49375 | 38.769441 | 24.415686 | 90.703822 |
-| Z,max | 49926 | 2546226 | 14977800 | 127 M | 376 | 49926 | 44.887706 | 20.851582 | 52.506482 |
-| Word,or | 48962 | 2497062 | 14688600 | 129 M | 54 | 48962 | 45.179623 | 20.079035 | 92.723852 |
-#+TBLFM: $2=$3/51::@2$3=remote(power_5,@2$2)::@2$4=remote(power_5,@2$3)::@2$5=remote(power_5_file,@1$1)::@2$6=remote(power_5, @2$4)::@2$7=remote(power_5, @2$6)::@2$8=remote(power_5,@2$9)::@2$9=remote(power_5,@2$11)::@2$10=remote(power_5,@2$12)::@3$3=remote(zmax_5,@2$2)::@3$4=remote(zmax_5,@2$3)::@3$5=remote(zmax_5_file,@1$1)::@3$6=remote(zmax_5, @2$4)::@3$7=remote(zmax_5, @2$6)::@3$8=remote(zmax_5,@2$9)::@3$9=remote(zmax_5,@2$11)::@3$10=remote(zmax_5,@2$12)::@4$3=remote(wordor_5,@2$2)::@4$4=remote(wordor_5,@2$3)::@4$5=remote(wordor_5_file,@1$1)::@4$6=remote(wordor_5, @2$4)::@4$7=remote(wordor_5, @2$6)::@4$8=remote(wordor_5,@2$9)::@4$9=remote(wordor_5,@2$11)::@4$10=remote(wordor_5,@2$12)