Skip to content
Snippets Groups Projects
Commit 0810ac8b authored by Hans-Peter Deifel's avatar Hans-Peter Deifel
Browse files

dfa: Add more results

parent bf273d1f
Branches
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
dfa/benchmark.org
+ 255
− 55
View file @ 0810ac8b
......@@ -25,6 +25,8 @@
- N :: Number of states of the automaton
- M :: Size of the input alphabet of the automaton
- i :: Number of sample for individual benchmark or sample count for aggregated
values.
- States :: Number of states of the coalgebra
- Edges :: Number of edges in the coalgebra
- I :: Size of initial partition
......@@ -38,6 +40,9 @@
- t_v :: Overall time for Valmari in seconds
- Q_v :: Size of final partition that valmari's tool computes
For aggregated benchmarks, each timing result also has an additional column with
the standard derivation of the samples
Important correctness checks are:
- The number of states should be the same in the coalgebra and automaton
......@@ -45,35 +50,210 @@
- The size of the initial partition must be 2
- The size of the final partition should be the same as for valmari
** Plotting Code
The following python code can plot the tables below and produce an inline image
into this document.
#+NAME: plot
#+BEGIN_SRC python :results file :exports results :var data="" :var filename=""
import matplotlib.pyplot as plt
'''Turn the table data into x and y data'''
x = [a[0] for a in data]
t = [a[9] for a in data]
t_s = [a[10] for a in data]
tp = [a[11] for a in data]
tp_s = [a[12] for a in data]
ta = [a[13] for a in data]
ta_s = [a[14] for a in data]
ti = [a[15] for a in data]
ti_s = [a[16] for a in data]
tr = [a[17] for a in data]
tr_s = [a[18] for a in data]
tv = [a[19] for a in data]
tv_s = [a[20] for a in data]
''' Plot the x and y data'''
plt.errorbar(x, t, yerr=t_s, label="t")
plt.errorbar(x, tp, yerr=tp_s, label="tp")
plt.errorbar(x, ta, yerr=ta_s, label="ta")
plt.errorbar(x, tr, yerr=tr_s, label="tr")
plt.errorbar(x, ti, yerr=ti_s, label="ti")
plt.errorbar(x, tv, yerr=tv_s, label="tv")
''' Set the x and y labels on the graph '''
plt.xlabel("Number of states")
plt.ylabel("Seconds")
''' Create the legend '''
plt.legend(loc="upper left")
''' Save the PNG file '''
plt.savefig(filename)
''' Return the PNG file path to OrgMode '''
return(filename)
#+END_SRC
** TODO Test
#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Q_s1" "Opt" "t (s)" "t_p (s)" "t_a (s)" "t_i (s)" "t_r (s)" "t_v (s)" "Q_v")
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 10 10 --indiv
#+end_src
#+RESULTS:
| N | M | i | States | Edges | I | Q | Q_s1 | Opt | t (s) | t_p (s) | t_a (s) | t_i (s) | t_r (s) | t_v (s) | Q_v |
|----+----+---+--------+-------+---+----+------+-----+-------------+-------------+-------------+------------+------------+-------------+-----|
| 10 | 10 | 0 | 10 | 100 | 2 | 10 | 10 | 83 | 0.001046352 | 0.00092647 | 8.813e-05 | 2.5603e-05 | 4.8467e-05 | 0.0013978 | 10 |
| 10 | 10 | 1 | 10 | 100 | 2 | 10 | 10 | 81 | 0.001332556 | 0.00117165 | 0.000121423 | 4.205e-05 | 6.1448e-05 | 0.001185295 | 10 |
| 10 | 10 | 2 | 10 | 100 | 2 | 10 | 10 | 86 | 0.001131453 | 0.001002346 | 9.7326e-05 | 2.4585e-05 | 5.9035e-05 | 0.001117232 | 10 |
| 10 | 10 | 3 | 10 | 100 | 2 | 10 | 10 | 76 | 0.001090468 | 0.000967138 | 9.1729e-05 | 2.4952e-05 | 5.2869e-05 | 0.001147177 | 10 |
| 10 | 10 | 4 | 10 | 100 | 2 | 10 | 10 | 70 | 0.001085314 | 0.000961097 | 9.1407e-05 | 2.6115e-05 | 5.0625e-05 | 0.001228354 | 10 |
| 10 | 10 | 5 | 10 | 100 | 2 | 10 | 10 | 77 | 0.001098224 | 0.000950174 | 9.715e-05 | 2.6022e-05 | 5.0376e-05 | 0.001149149 | 10 |
| 10 | 10 | 6 | 10 | 100 | 2 | 10 | 10 | 75 | 0.001075821 | 0.000940202 | 0.000101156 | 2.7531e-05 | 5.9542e-05 | 0.001141678 | 10 |
| 10 | 10 | 7 | 10 | 100 | 2 | 10 | 10 | 81 | 0.001051644 | 0.000930808 | 8.7346e-05 | 2.6147e-05 | 4.7219e-05 | 0.001227956 | 10 |
| 10 | 10 | 8 | 10 | 100 | 2 | 10 | 10 | 78 | 0.001059422 | 0.000935757 | 9.0775e-05 | 2.6297e-05 | 4.999e-05 | 0.001297999 | 10 |
| 10 | 10 | 9 | 10 | 100 | 2 | 10 | 10 | 79 | 0.001121328 | 0.000993801 | 9.2128e-05 | 2.6818e-05 | 5.0163e-05 | 0.001188937 | 10 |
** For m = 1:
*** Generating some benchmarks
#+begin_src sh :results output silent
for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000 200000 200000 210000 220000 230000 240000 250000 260000 270000 280000 290000 300000; do
python bench.py generate ../../copar/bin/random-dfa $n 1
done
#+end_src
*** And running them
#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Q_s1" "Opts" "t (s)" "σ" "t_p (s)" "σ" "t_a (s)" "σ" "t_i (s)" "σ" "t_r (s)" "σ" "t_v (s)" "σ" "Q_v")
for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000 200000 210000 220000 230000 240000 250000 260000 270000 280000 290000 300000; do
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 1 --stddev
done
#+end_src
#+NAME: results1
#+RESULTS:
| N | M | i | States | Edges | I | Q | Q_s1 | Opts | t (s) | σ | t_p (s) | σ | t_a (s) | σ | t_i (s) | σ | t_r (s) | σ | t_v (s) | σ | Q_v |
|--------+---+----+--------+--------+---+--------+--------+-------+--------------------+---------------------+--------------------+----------------------+--------------------+---------------------+--------------------+----------------------+--------------------+---------------------+---------------------+----------------------+--------|
| 100000 | 1 | 10 | 100000 | 100000 | 2 | 65407 | 65407 | 8066 | 4.5290850337 | 0.11864934189220436 | 1.2670609352 | 0.008980379315232729 | 2.8846108683000002 | 0.10597994596851053 | 0.523533524 | 0.012535445952974214 | 2.3008021567999997 | 0.1055769208893637 | 0.13519612100000003 | 0.013846578756400505 | 65407 |
| 110000 | 1 | 10 | 110000 | 110000 | 2 | 72036 | 72036 | 8623 | 5.0699149869 | 0.07183568383526712 | 1.550735742 | 0.012781696128692933 | 3.2605090468 | 0.07796323372066957 | 0.5889638794 | 0.010617361192314432 | 2.5968446309 | 0.06931852161488808 | 0.1568809575 | 0.00901868889061136 | 72036 |
| 120000 | 1 | 10 | 120000 | 120000 | 2 | 78722 | 78722 | 10038 | 5.6652413574 | 0.20637898857877984 | 1.523577259 | 0.0084005891907344 | 3.6769752172 | 0.20220102182563818 | 0.6413489954 | 0.01061210230379197 | 2.9675998923 | 0.20755563430437704 | 0.1708820727 | 0.006333066836038465 | 78722 |
| 130000 | 1 | 10 | 130000 | 130000 | 2 | 84540 | 84540 | 10711 | 6.155189409099999 | 0.2498730904358588 | 1.6801251087 | 0.011079371947232368 | 3.9624965937 | 0.24717276999830637 | 0.7186280466 | 0.01787129650816384 | 3.1551785867999995 | 0.25247397197890625 | 0.1844166179 | 0.006399117434715656 | 84540 |
| 140000 | 1 | 10 | 140000 | 140000 | 2 | 91314 | 91314 | 11292 | 6.8928926471 | 0.3869763935650305 | 2.0056164452 | 0.024803607728986293 | 4.5533410205000004 | 0.38060859032445055 | 0.7760390503 | 0.020530754890119275 | 3.6849088641 | 0.3603464592212336 | 0.22152070480000002 | 0.011713410540611872 | 91314 |
| 150000 | 1 | 10 | 150000 | 150000 | 2 | 98130 | 98130 | 12103 | 7.1892871133 | 0.26882752258808607 | 2.0664406011999996 | 0.018577575378630386 | 4.761524056200001 | 0.27522281997278325 | 1.0023196130999998 | 0.0168395383007438 | 3.6536833853 | 0.26955296888379165 | 0.24513165289999997 | 0.0098519607601366 | 98130 |
| 160000 | 1 | 10 | 160000 | 160000 | 2 | 104667 | 104667 | 13301 | 7.7042121983000005 | 0.3368186877504424 | 2.1893636440999997 | 0.016775070908835853 | 5.1237405769 | 0.3259827229869275 | 0.9438437005999999 | 0.013052837572290204 | 4.065045275399999 | 0.3293773664450977 | 0.26880464979999996 | 0.010019774112164618 | 104667 |
| 170000 | 1 | 10 | 170000 | 170000 | 2 | 111441 | 111441 | 13610 | 8.523356352999999 | 0.22551547496172486 | 2.2702832193 | 0.015462354590002336 | 5.527408397299999 | 0.21641631918257642 | 0.9606208465 | 0.014418824957385735 | 4.4693703110000005 | 0.21927683619903873 | 0.3066240502 | 0.011562740597278725 | 111441 |
| 180000 | 1 | 10 | 180000 | 180000 | 2 | 117527 | 117527 | 13716 | 9.0342931981 | 0.30171583923750933 | 2.508549816 | 0.14337287438998608 | 5.965023520499999 | 0.2990616866838848 | 1.0055327957 | 0.03247895472066015 | 4.841517208299999 | 0.2741765672999685 | 0.3435794639 | 0.02234378875361103 | 117527 |
| 190000 | 1 | 10 | 190000 | 190000 | 2 | 125139 | 125139 | 15821 | 10.0954592225 | 0.3085596430961634 | 2.754129648 | 0.017515158610812293 | 6.876505441499999 | 0.31830539193676316 | 1.0902934869999998 | 0.011161747915795436 | 5.6548687162 | 0.31358564770564884 | 0.3690856953 | 0.014175738685518798 | 125139 |
| 200000 | 1 | 10 | 200000 | 200000 | 2 | 130918 | 130918 | 15986 | 10.4757496425 | 0.22533439540768077 | 2.9252740033999993 | 0.02415745502875049 | 7.0661489696 | 0.23829972487490086 | 1.1495982269999998 | 0.023579909192671944 | 5.7629047678 | 0.2231899416219809 | 0.4043948325000001 | 0.01765757044883643 | 130918 |
| 210000 | 1 | 10 | 210000 | 210000 | 2 | 137411 | 137411 | 16862 | 11.2745130532 | 0.24223686396523397 | 3.0470118383 | 0.04214982428360531 | 7.6967156052000005 | 0.22455158375302342 | 1.2213759546 | 0.0423074851512136 | 6.3267563017 | 0.2348502350646554 | 0.45049089979999996 | 0.020270958542198007 | 137411 |
| 220000 | 1 | 10 | 220000 | 220000 | 2 | 144548 | 144548 | 18010 | 11.6568367058 | 0.37507557316666945 | 3.1978298028 | 0.015785463131040848 | 7.905704345100001 | 0.3892413697998992 | 1.2761892973 | 0.01205896715850773 | 6.4696442804 | 0.4029621478464203 | 0.4754177706999999 | 0.01785750755479795 | 144548 |
| 230000 | 1 | 10 | 230000 | 230000 | 2 | 151039 | 151039 | 18722 | 12.332978446999999 | 0.45949278720338216 | 3.3296963674999995 | 0.01931156495026591 | 8.3912996697 | 0.4554112922834242 | 1.3118217757 | 0.02529730289936623 | 6.9276620626 | 0.4562822937482532 | 0.503946619 | 0.01707564691121278 | 151039 |
| 240000 | 1 | 10 | 240000 | 240000 | 2 | 156612 | 156612 | 19180 | 12.885918998800003 | 0.46077805653697573 | 3.5158021212 | 0.03533533475267622 | 8.745929537 | 0.47164831525537076 | 1.3837022755000001 | 0.024756923385713905 | 7.1968549207 | 0.4967341049962083 | 0.5362665576000001 | 0.019188094888293705 | 156612 |
| 250000 | 1 | 10 | 250000 | 250000 | 2 | 163138 | 163138 | 20345 | 13.754044441100001 | 0.5900368494068087 | 3.6878678812999994 | 0.02561377686830973 | 9.4097558968 | 0.5916504156293916 | 1.442515364 | 0.035527587392385866 | 7.7940542921 | 0.5961807595021335 | 0.5768693020999999 | 0.02478708703422776 | 163138 |
| 260000 | 1 | 10 | 260000 | 260000 | 2 | 169815 | 169815 | 20812 | 14.318762113 | 0.43641933794007054 | 3.7940942547 | 0.02219417756155604 | 9.843946485 | 0.4319134695294516 | 1.5023389576 | 0.02639470246730954 | 8.1595752043 | 0.4695835501093594 | 0.6256955605 | 0.019737146399913738 | 169815 |
| 270000 | 1 | 10 | 270000 | 270000 | 2 | 177075 | 177075 | 21866 | 15.132494756700002 | 0.4692937308602713 | 3.9927350449000003 | 0.06157568499841667 | 10.4383791474 | 0.43073889658118064 | 1.5723795329 | 0.028845182692196804 | 8.6743628688 | 0.42903404302592746 | 0.6545056504 | 0.025625836629098682 | 177075 |
| 280000 | 1 | 10 | 280000 | 280000 | 2 | 183454 | 183454 | 20874 | 15.149952985999999 | 0.6248524708978799 | 4.041756857699999 | 0.029096827314996253 | 10.3739228112 | 0.6305485372230583 | 1.9820843660999998 | 0.01832347864488208 | 8.1631790438 | 0.6481129465916208 | 0.6787503918 | 0.030115951119949136 | 183454 |
| 290000 | 1 | 10 | 290000 | 290000 | 2 | 189403 | 189403 | 23451 | 15.909151233500001 | 0.4952063500620966 | 4.167462127399999 | 0.0801315481312222 | 10.9693956524 | 0.4722572332137746 | 1.9568616338000002 | 0.034775908835633834 | 8.8057878826 | 0.4463810026376683 | 0.7275396431000001 | 0.051833540165174545 | 189403 |
| 300000 | 1 | 10 | 300000 | 300000 | 2 | 196856 | 196856 | 23274 | 16.2512560443 | 0.6518491559101164 | 4.2287443279 | 0.04452343591811638 | 11.1972012534 | 0.6098903858382411 | 1.6720893186999999 | 0.02512268266760566 | 9.2872437867 | 0.592680455018582 | 0.7656593273 | 0.03068347134425448 | 196856 |
*** And plotting
#+CALL: plot(results1, "results1.png")
#+RESULTS:
[[file:results1.png]]
** For m = 2
*** Generating some benchmarks
#+begin_src sh :results output silent
for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000 200000 200000 210000 220000 230000 240000 250000 260000 270000 280000 290000 300000; do
python bench.py generate ../../copar/bin/random-dfa $n 2
done
#+end_src
*** And running them
#+begin_src sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Q_s1" "Opts" "t (s)" "σ" "t_p (s)" "σ" "t_a (s)" "σ" "t_i (s)" "σ" "t_r (s)" "σ" "t_v (s)" "σ" "Q_v")
for n in 100000 110000 120000 130000 140000 150000 160000 170000 180000 190000 200000 210000 220000 230000 240000 250000 260000 270000 280000 290000 300000; do
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin $n 2 --stddev
done
#+end_src
#+NAME: results2
#+RESULTS:
| N | M | i | States | Edges | I | Q | Q_s1 | Opts | t (s) | σ | t_p (s) | σ | t_a (s) | σ | t_i (s) | σ | t_r (s) | σ | t_v (s) | σ | Q_v |
|--------+---+----+--------+--------+---+--------+--------+--------+--------------------+----------------------+--------------------+----------------------+--------------------+----------------------+--------------------+----------------------+--------------------+----------------------+---------------------+-----------------------+--------|
| 100000 | 2 | 10 | 100000 | 200000 | 2 | 100000 | 100000 | 181160 | 4.6682598973 | 0.08297578258883191 | 1.7466071907 | 0.046301398085246505 | 2.6960031217 | 0.05678813781082613 | 0.6481005733 | 0.014042033012002532 | 1.9517191918 | 0.06471030997185793 | 0.26132676630000007 | 0.0069860096592970315 | 100000 |
| 110000 | 2 | 10 | 110000 | 220000 | 2 | 110000 | 110000 | 193895 | 5.4192957357000004 | 0.04358637576199741 | 2.0576445523 | 0.011675123216284503 | 3.1132264053 | 0.04179021345785444 | 0.7028545399 | 0.013187307245525014 | 2.3127195961 | 0.03735404238917737 | 0.3032304626 | 0.007008853475927248 | 110000 |
| 120000 | 2 | 10 | 120000 | 240000 | 2 | 119999 | 119999 | 215222 | 5.824416037 | 0.043706854467898265 | 2.1721261753 | 0.007263653574060782 | 3.3699483518 | 0.04570006332247481 | 0.9949342781 | 0.01655469322974228 | 2.2635461306000004 | 0.037319022080389716 | 0.34543054649999994 | 0.006311243247813619 | 119999 |
| 130000 | 2 | 10 | 130000 | 260000 | 2 | 130000 | 130000 | 231896 | 6.5872843755999995 | 0.060125954184014674 | 2.2585897419 | 0.016314558314201432 | 3.8955227494 | 0.09800864208953726 | 0.8475540828 | 0.019342967906861516 | 2.7888025453000003 | 0.04491313776325702 | 0.3878487445 | 0.005603817112682797 | 130000 |
| 140000 | 2 | 10 | 140000 | 280000 | 2 | 140000 | 140000 | 255274 | 7.1925595201000005 | 0.08004527888260175 | 2.4523406517999997 | 0.014677380694934156 | 4.3307462788999995 | 0.0819236107217384 | 0.9227721466000001 | 0.02383398470669321 | 3.0765555106 | 0.04254030915313634 | 0.43550701309999995 | 0.004735207750745585 | 140000 |
| 150000 | 2 | 10 | 150000 | 300000 | 2 | 150000 | 150000 | 269398 | 7.726262699199999 | 0.1018891510396078 | 2.6483071139 | 0.04212872036689742 | 4.6985100744 | 0.06611934352551581 | 0.9622648845 | 0.044662353434898536 | 3.3125214767999998 | 0.051973152271675385 | 0.4825729966000001 | 0.010104925148532968 | 150000 |
| 160000 | 2 | 10 | 160000 | 320000 | 2 | 160000 | 160000 | 282723 | 8.088284542699999 | 0.11056324275020642 | 2.8255136208 | 0.025362129263477914 | 4.8569707722 | 0.1080933056243718 | 0.7956835299 | 0.12196022196505889 | 3.5324954796 | 0.042719272635174936 | 0.5242104568 | 0.00841945000432359 | 160000 |
| 170000 | 2 | 10 | 170000 | 340000 | 2 | 170000 | 170000 | 307395 | 8.748016488199998 | 0.1404946673671456 | 3.0206189285000002 | 0.01754018035172234 | 5.292091254400001 | 0.13747202097694242 | 0.9155214081 | 0.16116275593489873 | 3.8326219171000004 | 0.05449780608019239 | 0.5777620275 | 0.01117342333969392 | 170000 |
| 180000 | 2 | 10 | 180000 | 360000 | 2 | 180000 | 180000 | 324587 | 9.2185365968 | 0.06285179687085718 | 3.0791284150000005 | 0.03392480617279444 | 5.3440163704 | 0.048937608592863474 | 1.1860047516 | 0.006881053367292836 | 3.9879859847 | 0.050225650773681366 | 0.6022483225999999 | 0.004710815984534462 | 180000 |
| 190000 | 2 | 10 | 190000 | 380000 | 2 | 190000 | 190000 | 342255 | 9.7680467295 | 0.13333256355334527 | 3.2486095897 | 0.022399876469984786 | 5.673361809799999 | 0.11381808849368587 | 1.2485236468999998 | 0.0378496943272731 | 4.2545025142999995 | 0.06821252118063109 | 0.650282204 | 0.005512470855261596 | 190000 |
| 200000 | 2 | 10 | 200000 | 400000 | 2 | 199999 | 199999 | 362610 | 10.3407087587 | 0.0767417288167444 | 3.4257339304 | 0.016129665711161565 | 6.0176680523999995 | 0.07760961922622171 | 1.3121143358 | 0.02253292975224462 | 4.5255908624 | 0.05374582752001877 | 0.6963170164000001 | 0.006721465053675577 | 199999 |
| 210000 | 2 | 10 | 210000 | 420000 | 2 | 210000 | 210000 | 377585 | 10.8548011165 | 0.08390739276286706 | 3.9481587426000004 | 0.007551257929878198 | 6.4041871301 | 0.08415521859573528 | 1.3709562733 | 0.023059428483544635 | 4.8434002994 | 0.049450725386594535 | 0.748728885 | 0.006811984754493087 | 210000 |
| 220000 | 2 | 10 | 220000 | 440000 | 2 | 220000 | 220000 | 400128 | 11.3527286721 | 0.13221984244838347 | 4.064920820299999 | 0.058000442572941326 | 6.663397006100001 | 0.09144077241600213 | 1.4458479548 | 0.026848100783522136 | 5.006428469 | 0.07451281472824153 | 0.8035932044 | 0.02525788636476248 | 220000 |
| 230000 | 2 | 10 | 230000 | 460000 | 2 | 229997 | 229997 | 417891 | 11.7633009699 | 0.07954013529620141 | 4.1434973527 | 0.022464517953739698 | 6.957498742 | 0.06464777411724817 | 1.9125527732999998 | 0.02735809426371808 | 4.8281356973000005 | 0.050886701528692374 | 0.8466457965 | 0.007033819689874407 | 229997 |
| 240000 | 2 | 10 | 240000 | 480000 | 2 | 240000 | 240000 | 427250 | 12.0412631762 | 0.09993691701336598 | 4.253491416899999 | 0.01717788575686945 | 7.184492855099999 | 0.10631652161662215 | 1.6463971609 | 0.016231270436728576 | 5.2882176574 | 0.14348630379376012 | 0.9009443936 | 0.0059634086305427715 | 240000 |
| 250000 | 2 | 10 | 250000 | 500000 | 2 | 250000 | 250000 | 455445 | 13.114036148099999 | 0.11508418852903363 | 4.361433761200001 | 0.01985874289541744 | 8.1151224604 | 0.10507993503517742 | 1.6105995995 | 0.04483188751587975 | 5.8376086756 | 0.08158785664921114 | 0.9537266898999999 | 0.012186339009724753 | 250000 |
| 260000 | 2 | 10 | 260000 | 520000 | 2 | 260000 | 260000 | 471100 | 14.3339631114 | 0.42678160354617156 | 4.6227287907 | 0.07753566996398639 | 8.478462721100001 | 0.31683084692120705 | 1.8203560963999998 | 0.08252011104911182 | 6.3941789987 | 0.20718159954774876 | 1.0417980003 | 0.02528434688872553 | 260000 |
| 270000 | 2 | 10 | 270000 | 540000 | 2 | 270000 | 270000 | 478399 | 15.2119843532 | 0.15917364284730137 | 4.8707966894 | 0.027172284273673825 | 9.0250704653 | 0.13003128025594105 | 1.9117913729000002 | 0.0657849307774362 | 6.862785855 | 0.0708436557820075 | 1.1095665595999997 | 0.011132099434290043 | 270000 |
| 280000 | 2 | 10 | 280000 | 560000 | 2 | 280000 | 280000 | 506543 | 15.782168719500001 | 0.11160026362665203 | 5.0754277744 | 0.030023250503622735 | 9.355177391000002 | 0.1328650460885607 | 1.9901339717000002 | 0.04867972695098818 | 7.0925399357000005 | 0.09140784029819397 | 1.1586345641999998 | 0.009941345579413565 | 280000 |
| 290000 | 2 | 10 | 290000 | 580000 | 2 | 289999 | 289999 | 523664 | 16.2064557565 | 0.31135465530203854 | 5.7941623523 | 0.09530698259843859 | 9.6633844364 | 0.2132433521796015 | 2.0475469749 | 0.04726162003024136 | 7.327850477800001 | 0.16892351000284792 | 1.2185979878 | 0.04532561130641692 | 289999 |
| 300000 | 2 | 10 | 300000 | 600000 | 2 | 299999 | 299999 | 541099 | 16.160194264 | 0.08196305876244549 | 5.7780438354 | 0.030898334771084817 | 9.636941188200002 | 0.0782324951530313 | 2.0359088467000004 | 0.019969054547938955 | 7.3043917502 | 0.0695271958751679 | 1.2261407546999998 | 0.015730889175256346 | 299999 |
*** Plot
#+CALL: plot(results2, "results2.png")
#+RESULTS:
[[file:results2.png]]
** For m = 1000:
#+BEGIN_SRC sh :hlines yes :colnames '("N" "M" "States" "Edges" "I" "Q" "Q_s1" "t (s)" "t_p (s)" "t_a (s)" "t_i (s)" "t_r (s)" "t_v (s)" "Q_v")
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 500 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 600 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 700 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 800 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 900 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 2000 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 3000 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 4000 1000
python bench.py run ../../copar/bin/copar ../../valmari_cpp/mdpmin 5000 1000
#+BEGIN_SRC sh :results output silent
for n in 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000; do
python bench.py generate ../../copar/bin/random-dfa $n 1000
done
#+END_SRC
#+BEGIN_SRC sh :hlines yes :colnames '("N" "M" "i" "States" "Edges" "I" "Q" "Q_s1" "t (s)" "σ" "t_p (s)" "σ" "t_a (s)" "σ" "t_i (s)" "σ" "t_r (s)" "σ" "t_v (s)" "σ" "Q_v")
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1000 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1100 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1200 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1300 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1400 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1500 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1600 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1700 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1800 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 1900 1000
python bench.py run --stddev ../../copar/bin/copar ../../valmari_cpp/mdpmin 2000 1000
#+END_SRC
#+NAME: results1000
#+RESULTS:
| N | M | States | Edges | I | Q | Q_s1 | t (s) | t_p (s) | t_a (s) | t_i (s) | t_r (s) | t_v (s) | Q_v |
|------+------+--------+---------+---+------+------+--------------+--------------+-------------+-------------+-------------+-------------+------|
| 500 | 1000 | 500 | 500000 | 2 | 500 | 500 | 1.759578161 | 1.215453013 | 0.5404687 | 0.173693248 | 0.309516028 | 0.368668478 | 500 |
| 600 | 1000 | 600 | 600000 | 2 | 600 | 600 | 2.168463236 | 1.454811198 | 0.709336242 | 0.443648498 | 0.196915142 | 0.480878517 | 600 |
| 700 | 1000 | 700 | 700000 | 2 | 700 | 700 | 2.521342058 | 1.685213915 | 0.831152724 | 0.517186374 | 0.232874282 | 0.603964899 | 700 |
| 800 | 1000 | 800 | 800000 | 2 | 800 | 800 | 2.841151969 | 1.904895555 | 0.930575509 | 0.51245229 | 0.32482546 | 0.759576573 | 800 |
| 900 | 1000 | 900 | 900000 | 2 | 900 | 900 | 3.172877915 | 2.32506566 | 0.841497627 | 0.330241886 | 0.406627262 | 0.887383093 | 900 |
| 1000 | 1000 | 1000 | 1000000 | 2 | 1000 | 1000 | 3.639089449 | 2.403538541 | 1.228517129 | 0.757942098 | 0.355330986 | 1.033265804 | 1000 |
| 2000 | 1000 | 2000 | 2000000 | 2 | 2000 | 2000 | 7.534527759 | 4.960457103 | 2.560123205 | 1.577616423 | 0.74379816 | 2.522043799 | 2000 |
| 3000 | 1000 | 3000 | 3000000 | 2 | 3000 | 3000 | 11.285467267 | 7.390417741 | 3.873770401 | 2.11428741 | 1.404249329 | 4.325322341 | 3000 |
| 4000 | 1000 | 4000 | 4000000 | 2 | 4000 | 4000 | 15.579233962 | 10.20142485 | 5.349548751 | 3.20335082 | 1.669701412 | 5.740334473 | 4000 |
| 5000 | 1000 | 5000 | 5000000 | 2 | 5000 | 5000 | 19.856799093 | 13.059423322 | 6.761123164 | 4.054875991 | 2.097659672 | 7.446535743 | 5000 |
| N | M | i | States | Edges | I | Q | Q_s1 | t (s) | σ | t_p (s) | σ | t_a (s) | σ | t_i (s) | σ | t_r (s) | σ | t_v (s) | σ | Q_v |
|------+------+----+--------+---------+---+------+------+--------------------+----------------------+--------------------+----------------------+--------------------+----------------------+--------------------+-----------------------+--------------------+----------------------+--------------------+----------------------+------|
| 1000 | 1000 | 10 | 1000 | 1000000 | 2 | 1000 | 1000 | 3.9310659747 | 0.08814522097255945 | 2.6619831237 | 0.055274922278595574 | 1.2615084707 | 0.03296793653488977 | 0.6944962514 | 0.01687407887910384 | 0.4457377868 | 0.013997283312452657 | 1.0652697109 | 0.03344509703654639 | 1000 |
| 1100 | 1000 | 10 | 1100 | 1100000 | 2 | 1100 | 1100 | 4.415969773900001 | 0.014060774244470062 | 2.9951420233 | 0.010950712046698105 | 1.4126832495000001 | 0.006914149208955129 | 0.7695829744999999 | 0.005935086168028538 | 0.5062192168999999 | 0.005052405925752206 | 1.2477546128 | 0.01071710076218558 | 1100 |
| 1200 | 1000 | 10 | 1200 | 1200000 | 2 | 1200 | 1200 | 4.8798212744 | 0.019258057016650766 | 3.2991537361 | 0.011860524966871172 | 1.5716358595 | 0.00900004107443779 | 0.8709621137999999 | 0.0036260209472211455 | 0.5517556958000001 | 0.005590352113769007 | 1.4176830762 | 0.018272214867454915 | 1200 |
| 1300 | 1000 | 10 | 1300 | 1300000 | 2 | 1300 | 1300 | 5.3283804624 | 0.0679017917052581 | 3.5886389821 | 0.050536878841655496 | 1.7299176397 | 0.045774169299274595 | 0.9464958896999999 | 0.007628430159047956 | 0.6220662668 | 0.043184464362744986 | 1.5641666131 | 0.016673611716160202 | 1300 |
| 1400 | 1000 | 10 | 1400 | 1400000 | 2 | 1400 | 1400 | 5.7016754482 | 0.031159398409859783 | 3.8447879026000003 | 0.019545430020354814 | 1.8457900890999999 | 0.014757246162394037 | 1.0075971078000001 | 0.006982825946204876 | 0.6617439326 | 0.007445983182010403 | 1.7205272282000004 | 0.009353775824724379 | 1400 |
| 1500 | 1000 | 10 | 1500 | 1500000 | 2 | 1500 | 1500 | 6.0356375616 | 0.0346956113544192 | 4.091011503300001 | 0.02069114545219446 | 1.9333955015999997 | 0.018828514385584867 | 0.5915615061 | 0.007678249481893585 | 0.7181443530999999 | 0.010455137514288866 | 1.8831240636 | 0.018712393477125337 | 1500 |
| 1600 | 1000 | 10 | 1600 | 1600000 | 2 | 1600 | 1600 | 6.2998617715 | 0.12335625087504232 | 4.668728570499999 | 0.10379316514447516 | 1.6191710851 | 0.02446575920621599 | 0.6278956022 | 0.010408651798497795 | 0.7991996886999999 | 0.014438199058374845 | 1.9808607018 | 0.04856700950206196 | 1600 |
| 1700 | 1000 | 10 | 1700 | 1700000 | 2 | 1700 | 1700 | 6.5246626490000015 | 0.017177425535351714 | 4.7919763139 | 0.011461787878394511 | 1.7203131365000002 | 0.008386434342815717 | 0.6658121586 | 0.004604784929576732 | 0.8514301593999999 | 0.009140177364965883 | 2.1240381013000005 | 0.02381032019896135 | 1700 |
| 1800 | 1000 | 10 | 1800 | 1800000 | 2 | 1800 | 1800 | 7.322528412499999 | 0.055849724790596245 | 4.9209584380999996 | 0.055366548709381055 | 2.3885675387999994 | 0.030059184173780385 | 1.4623097175 | 0.01867129183982534 | 0.709102204 | 0.009120696984374153 | 2.2660720953999998 | 0.013897116299011266 | 1800 |
| 1900 | 1000 | 10 | 1900 | 1900000 | 2 | 1900 | 1900 | 7.5982130497 | 0.031621770072322856 | 5.0914218181 | 0.026027182075443755 | 2.4930455598999997 | 0.011183495218059345 | 1.3836263962 | 0.008605595802593416 | 0.8814277291 | 0.00902711118459059 | 2.4351398513999998 | 0.011179909535069346 | 1900 |
| 2000 | 1000 | 10 | 2000 | 2000000 | 2 | 2000 | 2000 | 7.9732384284 | 0.01755021721874407 | 5.343933378099999 | 0.010836338916567953 | 2.6147934439 | 0.009539986444614498 | 1.4413895606 | 0.009829430904089706 | 0.9309159122 | 0.009335978052347366 | 2.6014849412000003 | 0.019299718485336178 | 2000 |
#+BEGIN_SRC python :results file :exports results :var data=results1000
......@@ -82,23 +262,31 @@
'''Turn the table data into x and y data'''
x = [a[0] for a in data]
t = [a[7] for a in data]
tp = [a[8] for a in data]
tr = [a[9] for a in data]
tv = [a[10] for a in data]
t = [a[8] for a in data]
t_s = [a[9]*10 for a in data]
tp = [a[10] for a in data]
tp_s = [a[11]*10 for a in data]
ta = [a[12] for a in data]
ta_s = [a[13]*10 for a in data]
ti = [a[14] for a in data]
ti_s = [a[15]*10 for a in data]
tr = [a[16] for a in data]
tr_s = [a[17]*10 for a in data]
tv = [a[18] for a in data]
tv_s = [a[19]*10 for a in data]
''' Plot the x and y data'''
a, = plt.plot(x, t, label="t", marker='v')
b, = plt.plot(x, tp, label="tp", marker='o')
c, = plt.plot(x, tr, label="tr", marker='x')
d, = plt.plot(x, tv, label="tv", marker='s')
plt.errorbar(x, t, yerr=t_s, label="t")
plt.errorbar(x, tp, yerr=tp_s, label="tp")
plt.errorbar(x, ta, yerr=ta_s, label="ta")
plt.errorbar(x, tv, yerr=tv_s, label="tv")
''' Set the x and y labels on the graph '''
plt.xlabel("Number of states")
plt.ylabel("Seconds")
''' Create the legend '''
plt.legend(handles=[a,b,c,d],loc="upper left")
plt.legend(loc="upper left")
''' Save the PNG file '''
filename = "results1000.png"
......@@ -169,3 +357,15 @@
#+RESULTS:
[[file:results10000.png]]
* Colophone
#+begin_src elisp
(require 'ob-shell)
(require 'ob-python)
#+end_src
#+RESULTS:
: ob-shell
dfa/results1.png 0 → 100644
+ 0
− 0
View file @ 0810ac8b
dfa/results1.png

40.4 KiB

dfa/results1000.png
+ 0
− 0
View replaced file @ bf273d1f View file @ 0810ac8b
dfa/results1000.png

37 KiB | W: | H:

dfa/results1000.png

29.2 KiB | W: | H:

dfa/results1000.png
dfa/results1000.png
dfa/results1000.png
dfa/results1000.png
  • 2-up
  • Swipe
  • Onion skin
dfa/results2.png 0 → 100644
+ 0
− 0
View file @ 0810ac8b
dfa/results2.png

38.9 KiB

0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment