Fun with an AArch64 NAS
If you’re anything like me, managing the data that is produced by our modern lifestyles is a chore. I’m the designated archival person in the family and as such I’m always looking for better ways to manage the huge volumes of data, from family photos and video to all of my music which I’ve digitized from my huge personal CD library.
My NAS over approximately the last 4 years has been my Turris Omnia router, which I’ve equipped with both an external USB disk and an internal mSATA drive. The Turris Omnia for those who aren’t familiar with it, is a high performance router that runs open source software based on OpenWrt. It’s served me very well without ever missing a beatiand will continue as my home router for the foreseeable future. Although a NAS enclosure is available for the Turris Omnia, I never purchased one. That’s because the Turris Omnia NAS option only has 2 drive bays. About a week ago, I started to ponder building a NAS which would take me to the next level of capacity and performance.
As luck would have it, some acquaintances have also been looking at looking at modernizing their NAS setups. So this fact has also provide some motivation. Most are looking at turnkey NAS solutions through, of which many shapes and sizes exist. When it comes to IT, I’m never one to take the easy path. I like to push the envelope and try to build using non-X86 architectures whenever possible. And it’s no different when it comes to a NAS for my household.
A device which recently caught my attention is the Traverse Ten64, which is a high-performance Arm based networking platform. Certainly the fact that it’s Arm based got my attention. Furthermore, it looks quite similar to the Turris Omnia but more powerful. What really piqued my interest was the writeup Building a NAS with Ten64 and Rockstor. Certainly I’ve come across Rockstor before, which is a well known NAS and cloud solution. But I was not aware of the beta Rockstor 4 recipe for aarch64. This really got me thinking! As the Traverse Ten64 is not shipping yet, how could I try out Rockstor on aarch64? Of course, there was the teasing of my acquaintances that I should just get a brand X NAS and be done with it. With that the (NAS) gauntlet was laid down.
Lurking in my basement is a very well used SolidRun MACCHIATObin system. It ticked a number of important boxes: it’s Arm (aarch64) based, has 3 SATA ports, doesn’t consume tons of power and is quiet. It struck me that this would be a great platform to try out Rockstor on aarch64.
The plan
Much like what was described in the Building a NAS with Ten64 and Rockstor article, I decided to run Rockstor on my MACCHIATObin system under KVM. This would prevent me from having to change the host OS to OpenSUSE (from Fedora). The Rockstor 4 ARM64EFI installation recipe is based on OpenSUSE. As a first step, I had to make certain that it was possible to use KVM on MACCHIATObin. After a bit of reading I had my answer and proceeded to get OpenSUSE Leap 15.2 running under KVM with minimal fuss.
Next, I followed the detailed instructions at the Rockstor site for setting up the version 4 beta on top of an existing OpenSUSE 15.2 installation. For this, I followed instructions from the following pages:
https://forum.rockstor.com/t/rockstor-4-installer-recipe-call-for-beta-testers/7237
https://github.com/rockstor/rockstor-installer/blob/master/README.md
https://forum.rockstor.com/t/rockstor-4-pi-build-first-observations/7367/2
For storage, I relied on spinning rust. I attached 2 x 4TB Seagate IronWolf SATA drives to the MACCHIATObin. Overall the installation of Rockstar went very smoothly and with no real hiccups to speak of. At the end of the installation I simply pointed my web browser to the Rockstor node and logged in. Via the Rockstor UI, I parepared the 2 Seagate disks and created a storage pool configured with RAID1 (mirroring). Finally, I defined NFS, SMB and SFTP shares against the storage pool in order to start using the system - in anger.
Below are a few photos of what the rig looks like.
For performance testing, I reliped upon some tried and trusted tools: iperf, dd and iozone. I NFS mounted the Rockstor server from a dual socket POWER9 system running CentOS Stream 8. And of course, my switch in this case connecting it all together is the Turris Omnia (Gigabit Ethernet throughout)
Some data from the sample runs follows:
A quick bandwidth test with iperf
a. Client side output
]$ ./iperf3 -c 192.168.1.YYY
Connecting to host 192.168.1.YYY, port 5201
[ 4] local 192.168.1.XXX port 33738 connected to 192.168.1.YYY port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 444 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 465 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 488 KBytes
[ 4] 3.00-4.00 sec 113 MBytes 948 Mbits/sec 0 667 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 942 Mbits/sec 0 700 KBytes
[ 4] 5.00-6.00 sec 112 MBytes 940 Mbits/sec 0 732 KBytes
[ 4] 6.00-7.00 sec 112 MBytes 937 Mbits/sec 0 882 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 926 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 970 KBytes
[ 4] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 0 1017 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver
iperf Done.b. Server side output
# ./iperf3 -s
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.XXX, port 33736
[ 5] local 192.168.1.YYY port 5201 connected to 192.168.1.XXX port 33738
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 108 MBytes 901 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 938 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 938 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 936 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec
[ 5] 10.00-10.05 sec 4.96 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.05 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.05 sec 1.09 GBytes 936 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------Write 8GB to mountpoint with dd
$ time dd if=/dev/zero of=/mnt/raktar/test_dd bs=8K count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 76.8675 s, 109 MB/s
real 1m16.872s
user 0m0.925s
sys 0m2.584sWrite with 4GB to mountpoint with dd
$ time dd if=/dev/zero of=/mnt/raktar/test_dd bs=4K count=1024000
1024000+0 records in
1024000+0 records out
4194304000 bytes (4.2 GB, 3.9 GiB) copied, 39.5062 s, 106 MB/s
real 0m39.509s
user 0m0.815s
sys 0m2.093sThrashing the spinning rust with iozone
$ time ./iozone -a -R -c -f /mnt/raktar/testfile
Iozone: Performance Test of File I/O
Version $Revision: 3.489 $
Compiled for 64 bit mode.
Build: linux-powerpc64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
Vangel Bojaxhi, Ben England, Vikentsi Lapa,
Alexey Skidanov, Sudhir Kumar.
Run began: Sat Mar 6 15:06:52 2021
Auto Mode
Excel chart generation enabled
Include close in write timing
Command line used: ./iozone -a -R -c -f /mnt/raktar/testfile
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
64 4 766 512 161629 160374 177677 1239 161629 417 59867 1201 512 163699 170672
64 8 1318 665 162019 168847 165823 1612 111531 473 112892 1028 551 102074 103891
64 16 1425 558 87901 104742 100620 1271 101764 557 102229 1556 550 177324 168002
64 32 601 602 171108 180788 175355 1272 101113 563 111300 1224 604 91684 94960
64 64 1480 601 163301 172982 173429 1551 94692 565 105941 1542 548 103053 181276
128 4 3441 1191 297797 295665 302495 3197 188520 1236 186555 2446 1337 347758 199668
128 8 2800 392 193899 342216 268999 1559 302495 1123 198708 3049 1101 199372 288205
128 16 1246 1224 219148 333293 338761 2072 222971 1123 211966 3126 1201 326403 350711
128 32 3066 1192 200939 356533 340264 3225 207141 1348 202913 2506 1324 228961 349797
128 64 2307 1192 231529 344190 334331 3208 229353 1112 226164 3069 1198 242839 320748
128 128 1021 1103 330828 343091 329812 3123 227796 1127 224932 3033 1198 343091 259633
256 4 5550 2182 561637 580461 552960 5128 381597 2449 323630 3618 2407 367241 587127
256 8 2831 2038 375328 582983 621448 4151 585845 2472 641497 6170 1371 603630 609455
256 16 5321 2419 735537 750970 561343 6295 707430 2225 624339 6049 2382 632429 688381
256 32 4422 2200 658411 668246 718796 6395 688381 2413 595594 4987 2400 773140 807442
256 64 6770 2399 785010 800221 686620 5007 717355 2429 609455 4842 995 654798 738065
256 128 6178 2180 663292 654399 684432 5100 658008 2247 668246 5947 2047 635047 678807
256 256 4612 2035 670332 748875 644965 6361 727070 2051 810490 7763 2052 686182 697776
512 4 8478 4432 1155924 1160923 911184 10157 966113 2508 936209 5425 3468 982467 994294
512 8 12369 4387 1099128 1169140 1135150 10039 1073309 3805 1053821 9847 4367 1140576 1152822
512 16 10894 4782 1355045 1380303 1166599 10031 1276891 4449 1227269 7313 4324 1383861 1518938
512 32 7773 3488 1248678 1329871 1308798 12665 1391032 5389 1199171 7036 4065 1422357 1471074
512 64 10716 4810 1375881 1458090 1276891 9979 1286067 4868 1216149 7310 3990 1537424 1528669
512 128 13507 4776 1417662 1429934 1242896 8310 1347393 4543 1115688 7989 3484 1127995 1289930
512 256 5408 4314 1369738 1382969 1296159 12632 1333173 4149 1458090 7946 4821 1245780 1368865
512 512 5106 3765 1163439 1225169 1296159 10144 1387437 4428 1273861 9715 3742 1355045 1413929
1024 4 11637 8251 1600327 1609925 1340164 14150 1326505 8366 1220575 13853 7632 1434138 1594978
1024 8 13016 8906 1874871 1875690 1762539 14240 1633188 8954 1695065 16636 7555 1790460 2004366
1024 16 15004 8720 2102484 2120127 1976691 12715 1968537 7704 1614767 19160 6546 2052253 2206161
1024 32 14118 9301 2236023 2360124 2321847 13136 2106609 9533 2169388 19593 8130 2198257 2338280
1024 64 15127 7499 2326879 2409105 2155236 14981 2286009 8050 2391665 19566 9624 2120127 2516377
1024 128 13110 8832 2360124 2409105 2503178 14925 2265510 8879 2321847 15910 7608 2165014 2478620
1024 256 14024 9611 2432298 2450338 2485792 12700 2286009 8327 2316837 15960 8814 2514903 2546213
1024 512 15271 8842 1783768 2169388 2290886 12649 2337008 5856 2003431 12456 8124 2573677 2632033
1024 1024 14946 9055 2592318 2751762 2767722 13950 2426801 8293 2503178 13786 8807 2516377 2111788
2048 4 20893 14967 2214011 2240576 1747411 22968 1739978 16843 1762469 27893 13966 2124220 2194777
2048 8 23866 15022 2752473 2592166 2386884 22881 2359349 16556 2434922 20044 14155 2760434 2809182
2048 16 15582 16221 3346546 3384788 3061485 22716 2938934 16490 3074635 28567 12449 3374151 3229534
2048 32 22680 14979 3777778 3841986 3717289 25448 3561619 18210 3334853 27811 16144 3901313 3929871
2048 64 21076 15019 3999400 3922692 3806235 20393 3749743 15463 3799501 32083 14991 4038890 4240255
2048 128 19838 16169 4229816 4250747 4164199 16931 3864455 17908 4162181 22159 15187 4248644 4453499
2048 256 12745 9084 4259178 4196751 4293237 20211 3812993 17887 4054139 28122 13249 4154130 4441984
2048 512 26016 15202 4267642 4229816 4453499 24891 4154130 16492 4444282 28216 15017 4188565 4405533
2048 1024 23457 19229 3917325 4130162 4154130 20590 4196751 11737 3650932 24566 15136 2482779 2659582
2048 2048 16566 16249 2494314 2635106 2653011 23273 1933883 15579 2176427 25411 16236 2133188 1906837
4096 4 38048 24611 2298370 2162402 1788674 35265 1640426 32323 1618939 40241 24720 2269226 2391139
4096 8 39802 26380 3503374 3519162 3327622 37629 2294686 34059 2233238 46115 19740 2363506 2585847
4096 16 23628 27804 2970339 3013586 2756362 35645 3821310 33360 3817065 45585 28058 4149921 4280194
4096 32 38047 26565 3290024 3543113 3117487 38176 5761565 32992 5818150 44147 22494 4934183 6847827
4096 64 34621 26347 114155 5007531 4791076 33686 4270618 33444 3706707 45574 26252 3558525 3621537
4096 128 29910 27839 3533639 3580776 3586757 38303 3489144 40310 4735608 37164 18835 3717133 3680502
4096 256 33432 26663 4934183 5101205 5424983 40428 3531460 22529 4906002 40634 20792 3343814 3433360
4096 512 32295 20977 5883908 5424983 5468151 37088 3664018 27316 5332372 36376 15273 5217395 5558378
4096 1024 40430 20977 3682869 3385325 3647681 41777 3771811 31407 3792628 36666 27891 4157956 3706707
4096 2048 21802 26450 3486312 3911800 3719547 38135 4982841 33183 5375755 36398 23427 4904602 5020702
4096 4096 32822 26575 5382492 5190597 5263753 37140 3404782 27528 4965558 39529 26352 14427096 14271297
8192 4 41208 34571 2528346 2205119 1642685 37514 1790147 61064 1945315 49482 29458 3030545 3136499
8192 8 45073 31839 4171488 4498640 3852936 49560 3365696 63898 3840876 42399 31969 3944043 4164914
8192 16 47033 29557 4500997 4373801 3995873 42362 3912159 63290 4067778 26672 44661 111474 5271672
8192 32 53750 34202 2863357 5287898 2123356 30104 4662231 62863 5000831 58402 30695 6480761 6809564
8192 64 42802 34921 5901904 5971654 5602613 37601 5313247 67584 5723946 40963 38040 5712526 5871647
8192 128 43970 34463 5817959 5988306 6071906 46729 5454094 67699 6140266 42889 37885 5785631 5975808
8192 256 43887 33123 5910025 5940679 6054786 43865 5745001 68143 5988306 43016 29500 5814021 6196743
8192 512 41983 34669 5944791 5940679 6239504 44618 5696426 66238 6947247 38716 31906 6145757 5997714
8192 1024 39435 37870 6399884 5830794 6067617 41948 6094523 62955 7937463 40642 28386 6924845 7735533
8192 2048 39275 25258 6072979 7427838 7440706 43609 6374949 35618 8743368 40711 31933 5336352 5142283
8192 4096 46473 28679 4946832 5126937 4876623 33948 5097274 47802 8086916 39041 37522 5010310 4876623
8192 8192 40918 29668 4998648 5056765 4979812 40625 5088216 29293 5688881 51283 33280 4929091 4710810
16384 4 58065 45957 3120803 3071013 2228850 55784 2116542 103626 2184496 57936 42028 2900803 3032391
16384 8 57669 47322 4914218 4948188 2688155 51293 3717820 131949 3654745 51302 43187 4729877 4761338
16384 16 52989 44720 5923681 5845080 4901250 60184 4996397 139152 5012066 58686 47250 5811959 5964297
16384 32 59565 55140 7817198 7717122 6889995 41780 6452650 125794 6537361 60210 39198 8233027 7481050
16384 64 48498 47273 7724930 7853828 7350615 59972 8312700 147326 8484089 60358 47194 7765083 7827884
16384 128 52363 46150 8257760 7933620 7692072 51845 7877236 126376 9277883 58286 50090 7926299 8270680
16384 256 57480 47199 9199633 8225143 8003847 53723 7406867 151085 8459025 54447 36649 9729770 9786581
16384 512 56439 58500 8533606 8240925 8673628 53771 94848 49413 9267872 60466 51694 8623560 8351087
16384 1024 56339 45925 7182365 8291637 8291637 67496 10304615 85962 19068585 46184 53250 8351087 8659420
16384 2048 69858 50186 11009550 10290727 10272267 55174 8168437 91274 12859651 45328 45915 8286637 8467363
16384 4096 57811 48674 8919040 8696680 8692280 67235 9334595 81668 10659461 60520 47228 8359214 8346016
16384 8192 54791 39197 8240925 9984224 9056567 58229 7628034 63197 8440323 56615 46231 7460745 7904418
16384 16384 52309 48775 7239873 7606082 7692072 59469 6798654 47149 7010184 66211 50302 7135381 7331011
32768 64 68452 63438 10767866 11145983 10587860 66188 8553152 196836 10262914 70797 63662 10445431 12477655
32768 128 74330 56587 10937536 11203222 11976409 68123 10425622 270919 12794771 69441 63307 10560200 11665355
32768 256 73302 59555 10796626 11473510 12383220 65037 10701629 164278 12277028 68121 58698 10958466 12068958
32768 512 72445 60667 11631788 12359834 13567777 72396 10879527 171307 12724879 69853 61717 9849567 11800570
32768 1024 71159 67179 9858045 10186091 11123431 65485 11148695 275460 12961285 69705 61881 10151482 10709134
32768 2048 73047 60915 11582774 12755585 13128427 72514 11222432 219878 20532259 68973 62478 10807663 12337643
32768 4096 68780 64220 10252961 10835782 11594500 62992 10227021 132334 18702216 69636 63082 9683705 12305608
32768 8192 70705 63127 9825626 10875222 11311097 70012 10246845 118346 10694135 64358 64404 9902793 12516290
32768 16384 72368 59947 9375862 9728947 9881434 67853 8419541 89947 8839196 60092 58554 112629 9533907
65536 64 82509 79055 10401087 11421453 11023864 77645 9652016 440101 10654675 77948 77407 10149535 11630216
65536 128 84080 78150 10842965 12055622 12293394 82366 10710305 535691 12327576 79992 77452 11395884 13320434
65536 256 80498 72047 10597577 12559643 12197385 80466 11233809 290491 11908856 81132 76306 10447339 12629467
65536 512 81152 74496 10827162 12076278 13295950 75815 11061126 460655 12509344 78097 76545 11636124 14444598
65536 1024 83013 71312 11366197 12867728 13396389 80359 11770154 247265 12900943 81124 76239 11403448 13896193
65536 2048 83236 76272 119806 11270196 10942676 82650 11677648 247548 13306893 82129 78141 12022929 13579712
65536 4096 82357 76272 11948197 12920955 14283968 82385 11696531 258841 22230669 79791 78366 12113531 14039564
65536 8192 85060 77279 13008396 13655947 14105847 78461 12654468 222259 14807195 78973 77165 11548130 12850283
65536 16384 82554 77219 11929012 11872336 11831964 79076 11770154 176494 12284603 81136 76313 11644504 11700514
131072 64 88544 83386 11835991 12145325 11670659 87840 11391587 419979 11686040 88820 82872 11811578 12481740
131072 128 88630 81787 12860445 13091354 13302644 86476 12881840 467063 13230293 88445 79831 12765186 13511230
131072 256 90497 84498 12881840 13012339 12985292 87772 13108523 414247 13319081 89824 83386 12600753 13263809
131072 512 88596 82448 12769930 12958357 13536514 85905 13670482 416191 14118093 83436 84523 13137656 13840848
131072 1024 89407 78835 13194095 13263809 13855499 84908 13827619 442800 13946536 82538 78850 13371887 14115193
131072 2048 88707 84477 13946536 14042363 14785216 86420 14674318 457513 14565072 86827 83928 12639283 13414303
131072 4096 89182 83355 13417250 13516877 13986987 85565 13788430 497857 14016943 83531 83394 13105710 13740526
131072 8192 88133 82921 13332647 13313597 13712081 86749 13703194 415595 1591452 90140 82009 13259651 13324246
131072 16384 88919 83384 12569350 12383894 12445289 87123 12239423 273752 12745062 87518 81834 11830642 11918875
262144 64 92046 89238 12412246 12322798 11471432 90962 11729024 910557 11846678 92929 85955 12535646 12852148
262144 128 93163 87728 13213567 13137311 12926184 90851 12826360 900170 13110368 90907 83813 106392 11100705
262144 256 89530 87436 13288465 13291356 13391725 89797 12979437 924856 13410345 91811 88369 13264418 13639078
262144 512 88940 83024 13982505 13753728 14127128 91268 13858259 863287 13773196 91523 87486 13691728 14047711
262144 1024 94344 87151 14090737 13934836 14349852 91634 14453412 884871 14068741 87858 87861 14205245 14639112
262144 2048 93323 83784 14630931 14471865 15098659 89320 14816258 837400 14496670 91384 87290 14246105 14644962
262144 4096 91123 85736 14333390 14121866 14464820 86524 14602367 831004 14610711 92440 87452 14302812 14770479
262144 8192 93882 88322 14379692 14047711 14591902 89510 14455122 681831 14176855 89511 87001 13901010 14244628
262144 16384 89497 85428 12697054 12364092 12420940 91501 12849594 568459 13704015 90625 80203 12796951 12740160
524288 64 94358 91054 13009600 12589457 11779147 91311 11947459 1255265 11876481 89799 87753 12691250 12857393
524288 128 92317 88896 13905388 13609686 13317259 92937 13277137 1300817 13449123 92321 86529 13869692 14155027
524288 256 95697 90554 13770592 13504875 13470626 92289 13213552 1065322 13137375 93384 92184 13450851 13637369
524288 512 92883 88220 14063505 13708355 13953121 92841 14061617 1341596 14121485 91142 90098 14090720 14200640
524288 1024 95296 89887 14497129 14125204 14520774 91158 14661640 1315825 14669954 93129 88311 14578534 14875572
524288 2048 95723 88312 14628576 14222223 14656851 90120 14689160 1008900 14199907 93524 90571 14814641 15050689
524288 4096 96008 89645 14907236 14518761 15025186 93152 14806661 1239205 14519144 92925 90417 14841838 15099157
524288 8192 94085 90818 14516748 14098760 14655484 93151 14690436 888697 14513490 88415 88908 14614285 14856578
524288 16384 93354 90226 12920169 12536489 12795299 92156 12896679 798495 12860025 92265 89190 12914478 13190646
iozone test complete.
Excel output is below:
"Writer report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 766 1318 1425 601 1480
"128" 3441 2800 1246 3066 2307 1021
"256" 5550 2831 5321 4422 6770 6178 4612
"512" 8478 12369 10894 7773 10716 13507 5408 5106
"1024" 11637 13016 15004 14118 15127 13110 14024 15271 14946
"2048" 20893 23866 15582 22680 21076 19838 12745 26016 23457 16566
"4096" 38048 39802 23628 38047 34621 29910 33432 32295 40430 21802 32822
"8192" 41208 45073 47033 53750 42802 43970 43887 41983 39435 39275 46473 40918
"16384" 58065 57669 52989 59565 48498 52363 57480 56439 56339 69858 57811 54791 52309
"32768" 0 0 0 0 68452 74330 73302 72445 71159 73047 68780 70705 72368
"65536" 0 0 0 0 82509 84080 80498 81152 83013 83236 82357 85060 82554
"131072" 0 0 0 0 88544 88630 90497 88596 89407 88707 89182 88133 88919
"262144" 0 0 0 0 92046 93163 89530 88940 94344 93323 91123 93882 89497
"524288" 0 0 0 0 94358 92317 95697 92883 95296 95723 96008 94085 93354
"Re-writer report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 512 665 558 602 601
"128" 1191 392 1224 1192 1192 1103
"256" 2182 2038 2419 2200 2399 2180 2035
"512" 4432 4387 4782 3488 4810 4776 4314 3765
"1024" 8251 8906 8720 9301 7499 8832 9611 8842 9055
"2048" 14967 15022 16221 14979 15019 16169 9084 15202 19229 16249
"4096" 24611 26380 27804 26565 26347 27839 26663 20977 20977 26450 26575
"8192" 34571 31839 29557 34202 34921 34463 33123 34669 37870 25258 28679 29668
"16384" 45957 47322 44720 55140 47273 46150 47199 58500 45925 50186 48674 39197 48775
"32768" 0 0 0 0 63438 56587 59555 60667 67179 60915 64220 63127 59947
"65536" 0 0 0 0 79055 78150 72047 74496 71312 76272 76272 77279 77219
"131072" 0 0 0 0 83386 81787 84498 82448 78835 84477 83355 82921 83384
"262144" 0 0 0 0 89238 87728 87436 83024 87151 83784 85736 88322 85428
"524288" 0 0 0 0 91054 88896 90554 88220 89887 88312 89645 90818 90226
"Reader report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 161629 162019 87901 171108 163301
"128" 297797 193899 219148 200939 231529 330828
"256" 561637 375328 735537 658411 785010 663292 670332
"512" 1155924 1099128 1355045 1248678 1375881 1417662 1369738 1163439
"1024" 1600327 1874871 2102484 2236023 2326879 2360124 2432298 1783768 2592318
"2048" 2214011 2752473 3346546 3777778 3999400 4229816 4259178 4267642 3917325 2494314
"4096" 2298370 3503374 2970339 3290024 114155 3533639 4934183 5883908 3682869 3486312 5382492
"8192" 2528346 4171488 4500997 2863357 5901904 5817959 5910025 5944791 6399884 6072979 4946832 4998648
"16384" 3120803 4914218 5923681 7817198 7724930 8257760 9199633 8533606 7182365 11009550 8919040 8240925 7239873
"32768" 0 0 0 0 10767866 10937536 10796626 11631788 9858045 11582774 10252961 9825626 9375862
"65536" 0 0 0 0 10401087 10842965 10597577 10827162 11366197 119806 11948197 13008396 11929012
"131072" 0 0 0 0 11835991 12860445 12881840 12769930 13194095 13946536 13417250 13332647 12569350
"262144" 0 0 0 0 12412246 13213567 13288465 13982505 14090737 14630931 14333390 14379692 12697054
"524288" 0 0 0 0 13009600 13905388 13770592 14063505 14497129 14628576 14907236 14516748 12920169
"Re-Reader report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 160374 168847 104742 180788 172982
"128" 295665 342216 333293 356533 344190 343091
"256" 580461 582983 750970 668246 800221 654399 748875
"512" 1160923 1169140 1380303 1329871 1458090 1429934 1382969 1225169
"1024" 1609925 1875690 2120127 2360124 2409105 2409105 2450338 2169388 2751762
"2048" 2240576 2592166 3384788 3841986 3922692 4250747 4196751 4229816 4130162 2635106
"4096" 2162402 3519162 3013586 3543113 5007531 3580776 5101205 5424983 3385325 3911800 5190597
"8192" 2205119 4498640 4373801 5287898 5971654 5988306 5940679 5940679 5830794 7427838 5126937 5056765
"16384" 3071013 4948188 5845080 7717122 7853828 7933620 8225143 8240925 8291637 10290727 8696680 9984224 7606082
"32768" 0 0 0 0 11145983 11203222 11473510 12359834 10186091 12755585 10835782 10875222 9728947
"65536" 0 0 0 0 11421453 12055622 12559643 12076278 12867728 11270196 12920955 13655947 11872336
"131072" 0 0 0 0 12145325 13091354 13012339 12958357 13263809 14042363 13516877 13313597 12383894
"262144" 0 0 0 0 12322798 13137311 13291356 13753728 13934836 14471865 14121866 14047711 12364092
"524288" 0 0 0 0 12589457 13609686 13504875 13708355 14125204 14222223 14518761 14098760 12536489
"Random read report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 177677 165823 100620 175355 173429
"128" 302495 268999 338761 340264 334331 329812
"256" 552960 621448 561343 718796 686620 684432 644965
"512" 911184 1135150 1166599 1308798 1276891 1242896 1296159 1296159
"1024" 1340164 1762539 1976691 2321847 2155236 2503178 2485792 2290886 2767722
"2048" 1747411 2386884 3061485 3717289 3806235 4164199 4293237 4453499 4154130 2653011
"4096" 1788674 3327622 2756362 3117487 4791076 3586757 5424983 5468151 3647681 3719547 5263753
"8192" 1642685 3852936 3995873 2123356 5602613 6071906 6054786 6239504 6067617 7440706 4876623 4979812
"16384" 2228850 2688155 4901250 6889995 7350615 7692072 8003847 8673628 8291637 10272267 8692280 9056567 7692072
"32768" 0 0 0 0 10587860 11976409 12383220 13567777 11123431 13128427 11594500 11311097 9881434
"65536" 0 0 0 0 11023864 12293394 12197385 13295950 13396389 10942676 14283968 14105847 11831964
"131072" 0 0 0 0 11670659 13302644 12985292 13536514 13855499 14785216 13986987 13712081 12445289
"262144" 0 0 0 0 11471432 12926184 13391725 14127128 14349852 15098659 14464820 14591902 12420940
"524288" 0 0 0 0 11779147 13317259 13470626 13953121 14520774 14656851 15025186 14655484 12795299
"Random write report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1239 1612 1271 1272 1551
"128" 3197 1559 2072 3225 3208 3123
"256" 5128 4151 6295 6395 5007 5100 6361
"512" 10157 10039 10031 12665 9979 8310 12632 10144
"1024" 14150 14240 12715 13136 14981 14925 12700 12649 13950
"2048" 22968 22881 22716 25448 20393 16931 20211 24891 20590 23273
"4096" 35265 37629 35645 38176 33686 38303 40428 37088 41777 38135 37140
"8192" 37514 49560 42362 30104 37601 46729 43865 44618 41948 43609 33948 40625
"16384" 55784 51293 60184 41780 59972 51845 53723 53771 67496 55174 67235 58229 59469
"32768" 0 0 0 0 66188 68123 65037 72396 65485 72514 62992 70012 67853
"65536" 0 0 0 0 77645 82366 80466 75815 80359 82650 82385 78461 79076
"131072" 0 0 0 0 87840 86476 87772 85905 84908 86420 85565 86749 87123
"262144" 0 0 0 0 90962 90851 89797 91268 91634 89320 86524 89510 91501
"524288" 0 0 0 0 91311 92937 92289 92841 91158 90120 93152 93151 92156
"Backward read report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 161629 111531 101764 101113 94692
"128" 188520 302495 222971 207141 229353 227796
"256" 381597 585845 707430 688381 717355 658008 727070
"512" 966113 1073309 1276891 1391032 1286067 1347393 1333173 1387437
"1024" 1326505 1633188 1968537 2106609 2286009 2265510 2286009 2337008 2426801
"2048" 1739978 2359349 2938934 3561619 3749743 3864455 3812993 4154130 4196751 1933883
"4096" 1640426 2294686 3821310 5761565 4270618 3489144 3531460 3664018 3771811 4982841 3404782
"8192" 1790147 3365696 3912159 4662231 5313247 5454094 5745001 5696426 6094523 6374949 5097274 5088216
"16384" 2116542 3717820 4996397 6452650 8312700 7877236 7406867 94848 10304615 8168437 9334595 7628034 6798654
"32768" 0 0 0 0 8553152 10425622 10701629 10879527 11148695 11222432 10227021 10246845 8419541
"65536" 0 0 0 0 9652016 10710305 11233809 11061126 11770154 11677648 11696531 12654468 11770154
"131072" 0 0 0 0 11391587 12881840 13108523 13670482 13827619 14674318 13788430 13703194 12239423
"262144" 0 0 0 0 11729024 12826360 12979437 13858259 14453412 14816258 14602367 14455122 12849594
"524288" 0 0 0 0 11947459 13277137 13213552 14061617 14661640 14689160 14806661 14690436 12896679
"Record rewrite report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 417 473 557 563 565
"128" 1236 1123 1123 1348 1112 1127
"256" 2449 2472 2225 2413 2429 2247 2051
"512" 2508 3805 4449 5389 4868 4543 4149 4428
"1024" 8366 8954 7704 9533 8050 8879 8327 5856 8293
"2048" 16843 16556 16490 18210 15463 17908 17887 16492 11737 15579
"4096" 32323 34059 33360 32992 33444 40310 22529 27316 31407 33183 27528
"8192" 61064 63898 63290 62863 67584 67699 68143 66238 62955 35618 47802 29293
"16384" 103626 131949 139152 125794 147326 126376 151085 49413 85962 91274 81668 63197 47149
"32768" 0 0 0 0 196836 270919 164278 171307 275460 219878 132334 118346 89947
"65536" 0 0 0 0 440101 535691 290491 460655 247265 247548 258841 222259 176494
"131072" 0 0 0 0 419979 467063 414247 416191 442800 457513 497857 415595 273752
"262144" 0 0 0 0 910557 900170 924856 863287 884871 837400 831004 681831 568459
"524288" 0 0 0 0 1255265 1300817 1065322 1341596 1315825 1008900 1239205 888697 798495
"Stride read report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 59867 112892 102229 111300 105941
"128" 186555 198708 211966 202913 226164 224932
"256" 323630 641497 624339 595594 609455 668246 810490
"512" 936209 1053821 1227269 1199171 1216149 1115688 1458090 1273861
"1024" 1220575 1695065 1614767 2169388 2391665 2321847 2316837 2003431 2503178
"2048" 1762469 2434922 3074635 3334853 3799501 4162181 4054139 4444282 3650932 2176427
"4096" 1618939 2233238 3817065 5818150 3706707 4735608 4906002 5332372 3792628 5375755 4965558
"8192" 1945315 3840876 4067778 5000831 5723946 6140266 5988306 6947247 7937463 8743368 8086916 5688881
"16384" 2184496 3654745 5012066 6537361 8484089 9277883 8459025 9267872 19068585 12859651 10659461 8440323 7010184
"32768" 0 0 0 0 10262914 12794771 12277028 12724879 12961285 20532259 18702216 10694135 8839196
"65536" 0 0 0 0 10654675 12327576 11908856 12509344 12900943 13306893 22230669 14807195 12284603
"131072" 0 0 0 0 11686040 13230293 13319081 14118093 13946536 14565072 14016943 1591452 12745062
"262144" 0 0 0 0 11846678 13110368 13410345 13773196 14068741 14496670 14610711 14176855 13704015
"524288" 0 0 0 0 11876481 13449123 13137375 14121485 14669954 14199907 14519144 14513490 12860025
"Fwrite report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1201 1028 1556 1224 1542
"128" 2446 3049 3126 2506 3069 3033
"256" 3618 6170 6049 4987 4842 5947 7763
"512" 5425 9847 7313 7036 7310 7989 7946 9715
"1024" 13853 16636 19160 19593 19566 15910 15960 12456 13786
"2048" 27893 20044 28567 27811 32083 22159 28122 28216 24566 25411
"4096" 40241 46115 45585 44147 45574 37164 40634 36376 36666 36398 39529
"8192" 49482 42399 26672 58402 40963 42889 43016 38716 40642 40711 39041 51283
"16384" 57936 51302 58686 60210 60358 58286 54447 60466 46184 45328 60520 56615 66211
"32768" 0 0 0 0 70797 69441 68121 69853 69705 68973 69636 64358 60092
"65536" 0 0 0 0 77948 79992 81132 78097 81124 82129 79791 78973 81136
"131072" 0 0 0 0 88820 88445 89824 83436 82538 86827 83531 90140 87518
"262144" 0 0 0 0 92929 90907 91811 91523 87858 91384 92440 89511 90625
"524288" 0 0 0 0 89799 92321 93384 91142 93129 93524 92925 88415 92265
"Re-Fwrite report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 512 551 550 604 548
"128" 1337 1101 1201 1324 1198 1198
"256" 2407 1371 2382 2400 995 2047 2052
"512" 3468 4367 4324 4065 3990 3484 4821 3742
"1024" 7632 7555 6546 8130 9624 7608 8814 8124 8807
"2048" 13966 14155 12449 16144 14991 15187 13249 15017 15136 16236
"4096" 24720 19740 28058 22494 26252 18835 20792 15273 27891 23427 26352
"8192" 29458 31969 44661 30695 38040 37885 29500 31906 28386 31933 37522 33280
"16384" 42028 43187 47250 39198 47194 50090 36649 51694 53250 45915 47228 46231 50302
"32768" 0 0 0 0 63662 63307 58698 61717 61881 62478 63082 64404 58554
"65536" 0 0 0 0 77407 77452 76306 76545 76239 78141 78366 77165 76313
"131072" 0 0 0 0 82872 79831 83386 84523 78850 83928 83394 82009 81834
"262144" 0 0 0 0 85955 83813 88369 87486 87861 87290 87452 87001 80203
"524288" 0 0 0 0 87753 86529 92184 90098 88311 90571 90417 88908 89190
"Fread report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 163699 102074 177324 91684 103053
"128" 347758 199372 326403 228961 242839 343091
"256" 367241 603630 632429 773140 654798 635047 686182
"512" 982467 1140576 1383861 1422357 1537424 1127995 1245780 1355045
"1024" 1434138 1790460 2052253 2198257 2120127 2165014 2514903 2573677 2516377
"2048" 2124220 2760434 3374151 3901313 4038890 4248644 4154130 4188565 2482779 2133188
"4096" 2269226 2363506 4149921 4934183 3558525 3717133 3343814 5217395 4157956 4904602 14427096
"8192" 3030545 3944043 111474 6480761 5712526 5785631 5814021 6145757 6924845 5336352 5010310 4929091
"16384" 2900803 4729877 5811959 8233027 7765083 7926299 9729770 8623560 8351087 8286637 8359214 7460745 7135381
"32768" 0 0 0 0 10445431 10560200 10958466 9849567 10151482 10807663 9683705 9902793 112629
"65536" 0 0 0 0 10149535 11395884 10447339 11636124 11403448 12022929 12113531 11548130 11644504
"131072" 0 0 0 0 11811578 12765186 12600753 13137656 13371887 12639283 13105710 13259651 11830642
"262144" 0 0 0 0 12535646 106392 13264418 13691728 14205245 14246105 14302812 13901010 12796951
"524288" 0 0 0 0 12691250 13869692 13450851 14090720 14578534 14814641 14841838 14614285 12914478
"Re-Fread report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 170672 103891 168002 94960 181276
"128" 199668 288205 350711 349797 320748 259633
"256" 587127 609455 688381 807442 738065 678807 697776
"512" 994294 1152822 1518938 1471074 1528669 1289930 1368865 1413929
"1024" 1594978 2004366 2206161 2338280 2516377 2478620 2546213 2632033 2111788
"2048" 2194777 2809182 3229534 3929871 4240255 4453499 4441984 4405533 2659582 1906837
"4096" 2391139 2585847 4280194 6847827 3621537 3680502 3433360 5558378 3706707 5020702 14271297
"8192" 3136499 4164914 5271672 6809564 5871647 5975808 6196743 5997714 7735533 5142283 4876623 4710810
"16384" 3032391 4761338 5964297 7481050 7827884 8270680 9786581 8351087 8659420 8467363 8346016 7904418 7331011
"32768" 0 0 0 0 12477655 11665355 12068958 11800570 10709134 12337643 12305608 12516290 9533907
"65536" 0 0 0 0 11630216 13320434 12629467 14444598 13896193 13579712 14039564 12850283 11700514
"131072" 0 0 0 0 12481740 13511230 13263809 13840848 14115193 13414303 13740526 13324246 11918875
"262144" 0 0 0 0 12852148 11100705 13639078 14047711 14639112 14644962 14770479 14244628 12740160
"524288" 0 0 0 0 12857393 14155027 13637369 14200640 14875572 15050689 15099157 14856578 13190646
real 10m25.537s
user 0m1.316s
sys 0m14.994s