ZFS pool extremne pomaly scrub

Chris

kde tu vidis 13GB volnej ram ?
mas 34.2GB alokovanej, 13G zobral system a kedze vyzral aj swap 4.62G, tak tej volnej je malo

+ este raz swap je smrt

tak mozno tomu nerozumiem, ale 34.2G / 47.1G mi evokuje ze je tam 13 Giga volnych.

obcas aj viacej, vid vcera pocas synchornizacie ZFS datastorov

htop2.png

Chris

1. sprav testy:

fio --name=seqwritetest --ioengine=libaio --rw=write --bs=1M --direct=1 --size=10G --numjobs=1 --runtime=60 --time_based --group_reporting
fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 --time_based --group_reporting

2. potom zmen co som ti pisal:

3. sprav testy znova

test na povodny dataset s 128k recordsize, lz4 compression:

seq write:

Kód: Vybrať všetko

/StoragePool/Storj$ fio --name=seqwritetest --ioengine=libaio --rw=write --bs=1M --direct=1 --size=10G --numjobs=1 --runtime=60 --time_based --group_reporting
seqwritetest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
seqwritetest: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=188MiB/s][w=188 IOPS][eta 00m:00s]
seqwritetest: (groupid=0, jobs=1): err= 0: pid=173316: Wed Dec 18 09:36:26 2024
  write: IOPS=471, BW=471MiB/s (494MB/s)(27.6GiB/60003msec); 0 zone resets
    slat (usec): min=86, max=88376, avg=2111.86, stdev=2282.68
    clat (nsec): min=1051, max=1959.1k, avg=5991.77, stdev=20432.60
     lat (usec): min=87, max=88384, avg=2117.85, stdev=2283.78
    clat percentiles (nsec):
     |  1.00th=[  1448],  5.00th=[  1944], 10.00th=[  2512], 20.00th=[  2832],
     | 30.00th=[  3344], 40.00th=[  3920], 50.00th=[  4640], 60.00th=[  5344],
     | 70.00th=[  6048], 80.00th=[  6752], 90.00th=[  7776], 95.00th=[  9536],
     | 99.00th=[ 25984], 99.50th=[ 41216], 99.90th=[197632], 99.95th=[313344],
     | 99.99th=[888832]
   bw (  KiB/s): min=53248, max=3891200, per=100.00%, avg=483955.69, stdev=423295.49, samples=119
   iops        : min=   52, max= 3800, avg=472.53, stdev=413.39, samples=119
  lat (usec)   : 2=5.54%, 4=35.49%, 10=54.45%, 20=2.94%, 50=1.17%
  lat (usec)   : 100=0.18%, 250=0.14%, 500=0.07%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=1.73%, sys=9.25%, ctx=37019, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,28281,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=471MiB/s (494MB/s), 471MiB/s-471MiB/s (494MB/s-494MB/s), io=27.6GiB (29.7GB), run=60003-60003msec

rand write:

Kód: Vybrať všetko

/StoragePool/Storj$ fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 --time_based --group_reporting
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.33
Starting 4 processes
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [w(4)][100.0%][w=38.5MiB/s][w=9866 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=4): err= 0: pid=176271: Wed Dec 18 09:39:05 2024
  write: IOPS=22.7k, BW=88.7MiB/s (93.0MB/s)(5322MiB/60031msec); 0 zone resets
    slat (usec): min=4, max=434147, avg=169.74, stdev=1486.49
    clat (nsec): min=801, max=150947k, avg=4219.68, stdev=252918.26
     lat (usec): min=5, max=434150, avg=173.96, stdev=1511.10
    clat percentiles (nsec):
     |  1.00th=[    844],  5.00th=[    860], 10.00th=[    876],
     | 20.00th=[    900], 30.00th=[    996], 40.00th=[   1096],
     | 50.00th=[   1224], 60.00th=[   1432], 70.00th=[   1720],
     | 80.00th=[   2008], 90.00th=[   2416], 95.00th=[   2832],
     | 99.00th=[   8768], 99.50th=[  17024], 99.90th=[ 252928],
     | 99.95th=[ 905216], 99.99th=[4947968]
   bw (  KiB/s): min= 8558, max=518872, per=100.00%, avg=91536.43, stdev=21392.42, samples=472
   iops        : min= 2139, max=129718, avg=22883.42, stdev=5348.17, samples=472
  lat (nsec)   : 1000=30.85%
  lat (usec)   : 2=48.69%, 4=18.70%, 10=0.84%, 20=0.49%, 50=0.20%
  lat (usec)   : 100=0.08%, 250=0.06%, 500=0.03%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=1.69%, sys=16.62%, ctx=481370, majf=0, minf=52
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1362474,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=88.7MiB/s (93.0MB/s), 88.7MiB/s-88.7MiB/s (93.0MB/s-93.0MB/s), io=5322MiB (5581MB), run=60031-60031msec

znizeny ARC cache na 4 GB, nastaveny swappiness na 1 (nieco zo swapu ubudlo, ale vacsinou je tam stalo to co bolo predtym). vytvoreny novy dataset s 64k recordsize a zstd compression. dedup som nemal ani predtym, write cache na diskoch bol vypnuty a raid karta bola v IT mode

seq write

Kód: Vybrať všetko

/StoragePool/Test$ sudo fio --name=seqwritetest --ioengine=libaio --rw=write --bs=1M --direct=1 --size=10G
--numjobs=1 --runtime=60 --time_based --group_reporting
[sudo] password for molnart:
seqwritetest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s]
seqwritetest: (groupid=0, jobs=1): err= 0: pid=1354790: Wed Dec 18 23:10:58 2024
  write: IOPS=362, BW=363MiB/s (380MB/s)(21.3GiB/60001msec); 0 zone resets
    slat (usec): min=89, max=4635.4k, avg=2751.00, stdev=88232.09
    clat (nsec): min=1093, max=465161, avg=3339.53, stdev=10258.30
     lat (usec): min=91, max=4635.4k, avg=2754.34, stdev=88232.25
    clat percentiles (nsec):
     |  1.00th=[  1304],  5.00th=[  1496], 10.00th=[  1640], 20.00th=[  1864],
     | 30.00th=[  2024], 40.00th=[  2192], 50.00th=[  2352], 60.00th=[  2544],
     | 70.00th=[  2800], 80.00th=[  3184], 90.00th=[  3952], 95.00th=[  5216],
     | 99.00th=[ 16768], 99.50th=[ 38144], 99.90th=[146432], 99.95th=[238592],
     | 99.99th=[387072]
   bw (  MiB/s): min=    6, max= 2787, per=100.00%, avg=1343.41, stdev=691.78, samples=32
   iops        : min=    6, max= 2787, avg=1343.12, stdev=691.73, samples=32
  lat (usec)   : 2=28.53%, 4=61.96%, 10=7.76%, 20=0.94%, 50=0.39%
  lat (usec)   : 100=0.18%, 250=0.20%, 500=0.04%
  cpu          : usr=0.71%, sys=5.25%, ctx=161294, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,21769,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=363MiB/s (380MB/s), 363MiB/s-363MiB/s (380MB/s-380MB/s), io=21.3GiB (22.8GB), run=60001-60001msec

rand write:

Kód: Vybrať všetko

/StoragePool/Test$ sudo fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G
--numjobs=4 --runtime=60 --time_based --group_reporting
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.33
Starting 4 processes
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [w(4)][100.0%][w=1693KiB/s][w=423 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=4): err= 0: pid=1358297: Wed Dec 18 23:13:45 2024
  write: IOPS=1290, BW=5163KiB/s (5287kB/s)(303MiB/60118msec); 0 zone resets
    slat (usec): min=5, max=211920, avg=3090.53, stdev=6897.92
    clat (nsec): min=806, max=11556k, avg=2873.17, stdev=45488.70
     lat (usec): min=5, max=211924, avg=3093.41, stdev=6899.00
    clat percentiles (nsec):
     |  1.00th=[   860],  5.00th=[   916], 10.00th=[   964], 20.00th=[  1012],
     | 30.00th=[  1080], 40.00th=[  1288], 50.00th=[  1896], 60.00th=[  2544],
     | 70.00th=[  2832], 80.00th=[  3440], 90.00th=[  4448], 95.00th=[  5856],
     | 99.00th=[ 15552], 99.50th=[ 18816], 99.90th=[ 47360], 99.95th=[ 80384],
     | 99.99th=[378880]
   bw (  KiB/s): min=  768, max=111272, per=100.00%, avg=5196.18, stdev=2861.22, samples=477
   iops        : min=  192, max=27818, avg=1299.03, stdev=715.31, samples=477
  lat (nsec)   : 1000=17.94%
  lat (usec)   : 2=33.11%, 4=35.76%, 10=11.61%, 20=1.16%, 50=0.33%
  lat (usec)   : 100=0.06%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=0.19%, sys=1.47%, ctx=41770, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,77594,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5163KiB/s (5287kB/s), 5163KiB/s-5163KiB/s (5287kB/s-5287kB/s), io=303MiB (318MB), run=60118-60118msec

TLDR: pri seq write rychlosti nizsie o 30%, pri rand je to pomalsie az 20x

zoom

Tak, teraz staci uz len podakovat

. Skusil som aj ja tie prikazy. Random write je podla ocakavania biednejsi nez u teba pred zmenou nastaveni (RAID-Z2, 6x 16TB IronWolf Pro, 128K recordsize, 2% fragmentacia). Latenciami som na tom trosicku lepsie ale nic zavratne - ja obsluzim 94% poziadaviek do 2 mikrosekund, ty do 4 mikrosekund.

Spoiler: ukázať

Kód: Vybrať všetko

root@nas[~]# fio --name=seqwritetest --ioengine=libaio --rw=write --bs=1M --direct=1 --size=10G --numjobs=1 --runtime=60 --time_based --group_reporting --directory=/mnt/StoragePool/StorageMain/Temp
seqwritetest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
seqwritetest: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=832MiB/s][w=832 IOPS][eta 00m:00s]
seqwritetest: (groupid=0, jobs=1): err= 0: pid=7206: Thu Dec 19 03:20:31 2024
  write: IOPS=642, BW=642MiB/s (673MB/s)(37.6GiB/60001msec); 0 zone resets
    slat (usec): min=114, max=50436, avg=1554.83, stdev=876.48
    clat (nsec): min=661, max=1848.3k, avg=1435.65, stdev=9549.37
     lat (usec): min=114, max=50456, avg=1556.27, stdev=877.35
    clat percentiles (nsec):
     |  1.00th=[  740],  5.00th=[  764], 10.00th=[  788], 20.00th=[  860],
     | 30.00th=[ 1004], 40.00th=[ 1128], 50.00th=[ 1176], 60.00th=[ 1240],
     | 70.00th=[ 1320], 80.00th=[ 1448], 90.00th=[ 1768], 95.00th=[ 2160],
     | 99.00th=[ 8096], 99.50th=[13504], 99.90th=[21376], 99.95th=[23680],
     | 99.99th=[45312]
   bw (  KiB/s): min=192512, max=5720064, per=99.77%, avg=656065.61, stdev=516023.43, samples=119
   iops        : min=  188, max= 5586, avg=640.69, stdev=503.93, samples=119
  lat (nsec)   : 750=1.84%, 1000=27.74%
  lat (usec)   : 2=64.05%, 4=4.67%, 10=0.96%, 20=0.56%, 50=0.17%
  lat (usec)   : 100=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=0.54%, sys=10.94%, ctx=41960, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,38531,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=642MiB/s (673MB/s), 642MiB/s-642MiB/s (673MB/s-673MB/s), io=37.6GiB (40.4GB), run=60001-60001msec

Kód: Vybrať všetko

root@nas[~]# fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 --time_based --group_reporting --directory=/mnt/StoragePool/StorageMain/Temp
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.33
Starting 4 processes
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
writetest: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [w(4)][100.0%][w=27.4MiB/s][w=7007 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=4): err= 0: pid=7284: Thu Dec 19 03:23:53 2024
  write: IOPS=8839, BW=34.5MiB/s (36.2MB/s)(2072MiB/60001msec); 0 zone resets
    slat (usec): min=7, max=24140, avg=450.30, stdev=294.43
    clat (nsec): min=481, max=2259.7k, avg=1183.16, stdev=7399.29
     lat (usec): min=8, max=24255, avg=451.48, stdev=295.26
    clat percentiles (nsec):
     |  1.00th=[   532],  5.00th=[   620], 10.00th=[   652], 20.00th=[   700],
     | 30.00th=[   764], 40.00th=[   844], 50.00th=[   900], 60.00th=[   940],
     | 70.00th=[  1012], 80.00th=[  1080], 90.00th=[  1400], 95.00th=[  2192],
     | 99.00th=[  7072], 99.50th=[  9536], 99.90th=[ 17280], 99.95th=[ 21888],
     | 99.99th=[113152]
   bw (  KiB/s): min=16445, max=296120, per=100.00%, avg=35394.56, stdev=6364.84, samples=476
   iops        : min= 4109, max=74030, avg=8848.61, stdev=1591.22, samples=476
  lat (nsec)   : 500=0.04%, 750=28.09%, 1000=40.18%
  lat (usec)   : 2=26.17%, 4=3.01%, 10=2.07%, 20=0.38%, 50=0.05%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.53%, sys=7.22%, ctx=495735, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,530400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=34.5MiB/s (36.2MB/s), 34.5MiB/s-34.5MiB/s (36.2MB/s-36.2MB/s), io=2072MiB (2173MB), run=60001-60001msec

Napriek tej nizsej rychlosti skoncil scrubbing pred par dnami podla ocakavania:

Kód: Vybrať všetko

root@nas[~]# zpool status
  pool: StoragePool
 state: ONLINE
  scan: scrub repaired 0B in 03:57:48 with 0 errors on Sun Dec 15 03:57:49 2024

Zopar assorted veci, co som ponachadzal a vyskumal:

Utilita ioztat - je to ako iostat, ale zobrazuje I/O pre jednotlive datasety. Pri scrubingu mozes pozorovat, ze ked sa spomali, ci je to vtedy ked ide na nejaky konkretny dataset (napr. ten s milionmi suborov).
Zistil som, ze existuju prikazy arcstat a arc_summary. Mozno sa tam bude dat vycitat niekde, ci prilisne zapratanie ARC nesposobuje narast swapu na disku.
Tuto typek pise preco vypnut prefetch pri scrubbingu a pozit starsi algoritmus. Mozes vyskusat, akurat na Linuxe sa to asi aktivuje trochu inak. Prikaz modinfo zfs zobrazi vsetky pritomne, ich popis je tu. Zrovna tie dve nastavenia som videl na viacerych miestach po nete spomenute, ze pomohli ludom.

Vikendovy projekt pre odvaznych: TrueNAS Scale aj OMV su zalozene na Linuxe. Vypnes pole, vyberies boot disk s OMV, strcis tam hoci aj USB kluc s TrueNAS Scale, nastartujes s minimalnou konfiguraciou a naimportujes existujuce disky, das scrubovat. Takto porovnas, ci s terajsim zaplnenim diskov nebude situacia zasadne odlisna v inom systeme.

Chris

Rebootol si systém po zmene swapiness?

restart som nerobil, aplikoval nastavenia swapu cez sudo sysctl -p nejaky efekt to ma, lebo vyuzivanost swapu postupne klesa, momentalne je na 3.04/4.65

zoom napísal: Št 19. Dec, 2024, 07:03 Utilita ioztat - je to ako iostat, ale zobrazuje I/O pre jednotlive datasety.

znie dobre, vyskusam. ale na 95% viem ze spomalenie bude na pbs a storagenode datasetoch.

zoom napísal: Št 19. Dec, 2024, 07:03 Vypnes pole, vyberies boot disk s OMV, strcis tam hoci aj USB kluc s TrueNAS Scale, nastartujes s minimalnou konfiguraciou a naimportujes existujuce disky, das scrubovat.

toto nie je problem spravit, kedze mi to bezi vo virtualke, takze staci prehodit passtrough raid karty na druhe VM, naimportovat tam konfiguraciu z TrueNas kde som to pole povodne vytvaral a idem. problemom se prave ten storagenode kde som penalizovany za kazdy cas ked to nie je online a rozbehnut pod truenas vanilla docker je o drzku. nehovoriac o tom, ze priebeh scrubu moze byt diametralne odlisny ak pocas toho mi na pole nezapisuje storagenode svojich 150-200 GB denne.

ako ten fio pracuje/nepracuje s cache ? ked to pustim ihned po sebe je normalne ze dostavam takto diametralne odlisne vysledky?

WRITE: bw=16.8MiB/s (17.6MB/s), 16.8MiB/s-16.8MiB/s (17.6MB/s-17.6MB/s), io=1024MiB (1074MB), run=61118-61118msec
WRITE: bw=376MiB/s (394MB/s), 376MiB/s-376MiB/s (394MB/s-394MB/s), io=1024MiB (1074MB), run=2727-2727msec
WRITE: bw=399MiB/s (418MB/s), 399MiB/s-399MiB/s (418MB/s-418MB/s), io=1024MiB (1074MB), run=2567-2567msec

tu som vypol vsetky torrenty, storagenody apod. veci zapisujuce na pool a pustil test okamzite po sebe:
WRITE: bw=21.1MiB/s (22.2MB/s), 21.1MiB/s-21.1MiB/s (22.2MB/s-22.2MB/s), io=1024MiB (1074MB), run=48467-48467msec
WRITE: bw=12.8MiB/s (13.5MB/s), 12.8MiB/s-12.8MiB/s (13.5MB/s-13.5MB/s), io=1024MiB (1074MB), run=79735-79735msec

priebeh je tiez zaujimavy , obcas tam vidim aj ETA cez 30 minut a rychlosti pod 1 MB/s. tu som zapol nahravanie ked uz bol celkom rozbehnuty a dovtedy to uz bezalo nejako minutu:
https://www.youtube.com/watch?v=cjOifHCThww

Kód: Vybrať všetko

molnart@omv6:/StoragePool/Storj$ sudo fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [w(1)][95.4%][w=348MiB/s][w=89.2k IOPS][eta 00m:03s] 
writetest: (groupid=0, jobs=1): err= 0: pid=841725: Fri Dec 20 17:48:58 2024
  write: IOPS=4289, BW=16.8MiB/s (17.6MB/s)(1024MiB/61118msec); 0 zone resets
    slat (usec): min=4, max=414375, avg=231.06, stdev=2054.89
    clat (nsec): min=766, max=19628k, avg=1382.33, stdev=51522.92
     lat (usec): min=5, max=414381, avg=232.44, stdev=2056.12
    clat percentiles (nsec):
     |  1.00th=[   796],  5.00th=[   812], 10.00th=[   812], 20.00th=[   820],
     | 30.00th=[   836], 40.00th=[   844], 50.00th=[   852], 60.00th=[   884],
     | 70.00th=[   972], 80.00th=[  1240], 90.00th=[  1432], 95.00th=[  1784],
     | 99.00th=[  6880], 99.50th=[ 10816], 99.90th=[ 22912], 99.95th=[ 31616],
     | 99.99th=[197632]
   bw (  KiB/s): min=  136, max=397815, per=96.73%, avg=16596.49, stdev=65989.98, samples=122
   iops        : min=   34, max=99453, avg=4149.10, stdev=16497.42, samples=122
  lat (nsec)   : 1000=72.70%
  lat (usec)   : 2=22.92%, 4=1.92%, 10=1.94%, 20=0.40%, 50=0.10%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 20=0.01%
  cpu          : usr=0.90%, sys=4.95%, ctx=10643, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=16.8MiB/s (17.6MB/s), 16.8MiB/s-16.8MiB/s (17.6MB/s-17.6MB/s), io=1024MiB (1074MB), run=61118-61118msec

--------------------
molnart@omv6:/StoragePool/Storj$ sudo fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=359MiB/s][w=91.9k IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=843084: Fri Dec 20 17:49:21 2024
  write: IOPS=82.9k, BW=324MiB/s (340MB/s)(1024MiB/3161msec); 0 zone resets
    slat (usec): min=4, max=44464, avg=10.28, stdev=94.92
    clat (nsec): min=769, max=1295.2k, avg=1181.26, stdev=8552.87
     lat (usec): min=5, max=44472, avg=11.46, stdev=95.39
    clat percentiles (nsec):
     |  1.00th=[   796],  5.00th=[   804], 10.00th=[   812], 20.00th=[   820],
     | 30.00th=[   828], 40.00th=[   844], 50.00th=[   860], 60.00th=[   900],
     | 70.00th=[  1064], 80.00th=[  1160], 90.00th=[  1320], 95.00th=[  1480],
     | 99.00th=[  2160], 99.50th=[  6496], 99.90th=[ 20864], 99.95th=[ 48384],
     | 99.99th=[432128]
   bw (  KiB/s): min=191328, max=522392, per=96.29%, avg=319423.17, stdev=120797.88, samples=6
   iops        : min=47832, max=130598, avg=79856.00, stdev=30199.55, samples=6
  lat (nsec)   : 1000=66.37%
  lat (usec)   : 2=32.41%, 4=0.66%, 10=0.17%, 20=0.29%, 50=0.05%
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=14.46%, sys=72.66%, ctx=1978, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=324MiB/s (340MB/s), 324MiB/s-324MiB/s (340MB/s-340MB/s), io=1024MiB (1074MB), run=3161-3161msec

--------------------
molnart@omv6:/StoragePool/Storj$ sudo fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [w(1)][75.0%][w=358MiB/s][w=91.7k IOPS][eta 00m:01s]
writetest: (groupid=0, jobs=1): err= 0: pid=843179: Fri Dec 20 17:49:31 2024
  write: IOPS=96.1k, BW=376MiB/s (394MB/s)(1024MiB/2727msec); 0 zone resets
    slat (usec): min=4, max=2128, avg= 8.75, stdev=20.87
    clat (nsec): min=784, max=1284.1k, avg=1107.85, stdev=6259.16
     lat (usec): min=5, max=2136, avg= 9.86, stdev=22.01
    clat percentiles (nsec):
     |  1.00th=[   812],  5.00th=[   828], 10.00th=[   828], 20.00th=[   836],
     | 30.00th=[   844], 40.00th=[   852], 50.00th=[   860], 60.00th=[   876],
     | 70.00th=[   908], 80.00th=[   988], 90.00th=[  1208], 95.00th=[  1432],
     | 99.00th=[  2320], 99.50th=[  5920], 99.90th=[ 23680], 99.95th=[ 51968],
     | 99.99th=[264192]
   bw (  KiB/s): min=184744, max=512712, per=99.71%, avg=383410.80, stdev=130015.57, samples=5
   iops        : min=46186, max=128178, avg=95852.60, stdev=32503.94, samples=5
  lat (nsec)   : 1000=81.90%
  lat (usec)   : 2=16.73%, 4=0.76%, 10=0.23%, 20=0.26%, 50=0.07%
  lat (usec)   : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=15.88%, sys=76.05%, ctx=1633, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=376MiB/s (394MB/s), 376MiB/s-376MiB/s (394MB/s-394MB/s), io=1024MiB (1074MB), run=2727-2727msec

--------------------
molnart@omv6:/StoragePool/Storj$ sync; sudo fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [w(1)][-.-%][w=357MiB/s][w=91.3k IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=843543: Fri Dec 20 17:49:58 2024
  write: IOPS=102k, BW=399MiB/s (418MB/s)(1024MiB/2567msec); 0 zone resets
    slat (usec): min=4, max=3662, avg= 8.23, stdev=21.32
    clat (nsec): min=771, max=1190.0k, avg=1053.94, stdev=5319.98
     lat (usec): min=5, max=3669, avg= 9.29, stdev=22.14
    clat percentiles (nsec):
     |  1.00th=[   796],  5.00th=[   804], 10.00th=[   812], 20.00th=[   820],
     | 30.00th=[   820], 40.00th=[   828], 50.00th=[   836], 60.00th=[   844],
     | 70.00th=[   860], 80.00th=[   948], 90.00th=[  1208], 95.00th=[  1432],
     | 99.00th=[  2096], 99.50th=[  6688], 99.90th=[ 22656], 99.95th=[ 41216],
     | 99.99th=[181248]
   bw (  KiB/s): min=165372, max=538784, per=99.59%, avg=406807.20, stdev=155799.14, samples=5
   iops        : min=41343, max=134696, avg=101701.80, stdev=38949.78, samples=5
  lat (nsec)   : 1000=84.70%
  lat (usec)   : 2=14.17%, 4=0.58%, 10=0.14%, 20=0.28%, 50=0.08%
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=15.08%, sys=78.64%, ctx=1370, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=399MiB/s (418MB/s), 399MiB/s-399MiB/s (418MB/s-418MB/s), io=1024MiB (1074MB), run=2567-2567msec

-----------
molnart@omv6:/StoragePool/Test$ sudo fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
writetest: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][96.1%][w=28.8MiB/s][w=7360 IOPS][eta 00m:02s] 
writetest: (groupid=0, jobs=1): err= 0: pid=850664: Fri Dec 20 17:57:59 2024
  write: IOPS=5408, BW=21.1MiB/s (22.2MB/s)(1024MiB/48467msec); 0 zone resets
    slat (usec): min=4, max=280564, avg=182.60, stdev=1655.04
    clat (nsec): min=809, max=826375, avg=1405.34, stdev=3415.91
     lat (usec): min=5, max=280571, avg=184.01, stdev=1655.42
    clat percentiles (nsec):
     |  1.00th=[  844],  5.00th=[  860], 10.00th=[  868], 20.00th=[  876],
     | 30.00th=[  892], 40.00th=[  908], 50.00th=[  972], 60.00th=[ 1020],
     | 70.00th=[ 1144], 80.00th=[ 1464], 90.00th=[ 2480], 95.00th=[ 2704],
     | 99.00th=[ 6752], 99.50th=[13504], 99.90th=[22144], 99.95th=[26752],
     | 99.99th=[63744]
   bw (  KiB/s): min= 3032, max=108976, per=96.73%, avg=20928.11, stdev=15783.22, samples=96
   iops        : min=  758, max=27244, avg=5232.01, stdev=3945.79, samples=96
  lat (nsec)   : 1000=56.06%
  lat (usec)   : 2=29.19%, 4=12.62%, 10=1.54%, 20=0.47%, 50=0.12%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.52%, sys=9.62%, ctx=42021, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=21.1MiB/s (22.2MB/s), 21.1MiB/s-21.1MiB/s (22.2MB/s-22.2MB/s), io=1024MiB (1074MB), run=48467-48467msec
  
-----------
molnart@omv6:/StoragePool/Test$ sudo fio --name=writetest --ioengine=libaio --rw=randwrite --bs=4k --direct=1 --size=1G
writetest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [w(1)][90.9%][w=32.8MiB/s][w=8406 IOPS][eta 00m:08s]
writetest: (groupid=0, jobs=1): err= 0: pid=851444: Fri Dec 20 17:59:23 2024
  write: IOPS=3287, BW=12.8MiB/s (13.5MB/s)(1024MiB/79735msec); 0 zone resets
    slat (usec): min=4, max=321111, avg=301.77, stdev=2383.22
    clat (nsec): min=789, max=1595.3k, avg=1487.11, stdev=5063.03
     lat (usec): min=5, max=321117, avg=303.26, stdev=2383.70
    clat percentiles (nsec):
     |  1.00th=[  828],  5.00th=[  844], 10.00th=[  852], 20.00th=[  868],
     | 30.00th=[  884], 40.00th=[  900], 50.00th=[  956], 60.00th=[  996],
     | 70.00th=[ 1128], 80.00th=[ 1496], 90.00th=[ 2576], 95.00th=[ 3056],
     | 99.00th=[ 7584], 99.50th=[14016], 99.90th=[24192], 99.95th=[32640],
     | 99.99th=[67072]
   bw (  KiB/s): min= 1016, max=114624, per=97.07%, avg=12765.28, stdev=11326.78, samples=159
   iops        : min=  254, max=28656, avg=3191.30, stdev=2831.68, samples=159
  lat (nsec)   : 1000=60.12%
  lat (usec)   : 2=24.12%, 4=12.17%, 10=2.89%, 20=0.52%, 50=0.15%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=1.00%, sys=6.33%, ctx=46951, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=12.8MiB/s (13.5MB/s), 12.8MiB/s-12.8MiB/s (13.5MB/s-13.5MB/s), io=1024MiB (1074MB), run=79735-79735msec

tak nastal cas dalsieho scrubu, situacia zatial rovnaka - read wait times na jednom z vdevov je az 25 s, na druhom je to milisekundach

Kód: Vybrať všetko

                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim  rebuild
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
StoragePool                               40.5T  23.6T    690      0  4.68M      0    19s      -   16ms      -   67us      -    2ms      -    20s      -      -
  raidz1-0                                18.2T  13.8T    147      0  1.65M      0  142ms      -   16ms      -  154us      -    2ms      -  139ms      -      -
    a755e11b-566a-4e0d-9e1b-ad0fe75c569b      -      -     42      0   490K      0   18ms      -   15ms      -  420us      -    2ms      -    3ms      -      -
    7038290b-70d1-43c5-9116-052cc493b97f      -      -     48      0   560K      0   18ms      -   16ms      -  658ns      -    1ms      -    2ms      -      -
    678a9f0c-0786-4616-90f5-6852ee56d286      -      -     56      0   636K      0  343ms      -   17ms      -  736ns      -    3ms      -  352ms      -      -
  raidz1-1                                22.3T  9.78T    543      0  3.03M      0    25s      -   16ms      -    3us      -    3ms      -    25s      -      -
    93e98116-7a8c-489d-89d9-d5a2deb600d4      -      -    178      0   938K      0    25s      -   16ms      -    3us      -    2ms      -    25s      -      -
    c056dab7-7c01-43b6-a920-5356b76a64cc      -      -    184      0  1.05M      0    25s      -   15ms      -    3us      -    1us      -    25s      -      -
    ce6b997b-2d4f-4e88-bf78-759895aae5a0      -      -    180      0  1.07M      0    25s      -   16ms      -    2us      -    4ms      -    25s      -      -

medzitym som updatol firmware na diskoch (to bol tiez porod, okrem toho ze sa nikde nedali stiahnut a musel mi ich poslat chlapik, trebalo ich este manualne upravovat) a zmenil som recordsize na datasetoch s malymi subormi (ale neprepisoval som znova data, takze realne sa zmenil len na datach pripisanych za posledny mesiac - +/- rovnako mnoztvsto dat bolo aj premazanych)

posledny scrubbing trval 8 dni, ak to tentoraz bude cokolvek pod nim tak to bude progres. pride mi ze bezne operacie, hlavne PBS garbage collection sa vyrazne zrychlili odkedy som zmenil recordsize (z 10+ hodin na 2-3) - ticho poznamenam ze ked rovnaka operacia bezala na standalone ext4 disku tak skoncila do 10 min

zoom napísal: Št 19. Dec, 2024, 07:03 Utilita ioztat - je to ako iostat, ale zobrazuje I/O pre jednotlive datasety. Pri scrubingu mozes pozorovat, ze ked sa spomali, ci je to vtedy ked ide na nejaky konkretny dataset (napr. ten s milionmi suborov).

vyzera ze tento nastroj neidentifukuje operace procas scrubingu, vidim len tych klasickych cca 1 MB/s co tam mam vzdy

tak dam sem nejaky update, asi je problem vyrieseny:
* zaciatkom roka vydal StorJ novy sposob ukladania dat ako techpreview, tak som na to premigroval moj storagenode, plus som nejake cold storage archivne veci zbalil to klasickeho gzipu. takto som zredukoval pocet suborov asi o 50 milionov
* objavil som ze moje disky pouzivaju "Type 2 protection", teda namiesto 512b sektorov maju 520 a tychto zvysnych 8 pouzivaju na nejaku chybovu korekciu s podporovanym radicom. neviem teda ci to ma aj nejaky speed impact okrem ukrajovania priestoru, ale TrueNAS kvoli tomu krici. S tymto som zatial nic neurobil, lebo musim data vykopirovat mimo pola, sformatovat disky a dat data spat
* ja jednom z diskov sa objavili 2 verify smart errory, co neviem do akej miery je kriticke, ale evidentne disky nie su uplne ok

kazdopadne posledny scrub uz trval len nieco cez den. stale celkom vela, ale zasa mam tam nejakych 30 TB dat.

Kód: Vybrať všetko

class: scrub_finish
  host: omv6
  time: 2025-03-10 01:50:07+0100
  pool: StoragePool
state: ONLINE
  scan: scrub repaired 0B in 1 days 01:26:05 with 0 errors on Mon Mar 10 01:50:07 2025
config:

rychlosti sa vacsinou drzali nad 500 MB/s

Obrázok

mp3turbo

na 520bajtove disky pozor, to mate z nejakych diskovych poli (ibm, emc, dell...) a tam byvaju strasne specificke firmware, ktore robia strasne specificke radosti.

https://forums.servethehome.com/index.p ... size.4968/
a kopec dalsich rovnakych tem.

no asi 520b nie je to iste ako to protection, ktore sa stale javi ako 512b ale tie zvysne byty pouziva pre nieco. mam aj ssdcka ktore prisli ako 520b a tie kym som ich nepreformatoval cez sg3_utils tak ani nesli a na dellackom perc ani formatovat nesli. tieto disky sli v pohode naformatovat a neukazovali ziadne specifikum, az od urcitej verzie TrueNAS ktora sa na to zacala stazovat.

mp3turbo

no lebo tie starsie Truenasy s takouto hovadinou neratali - na tejto platforme sa pocita s pouzitim klasickych diskov, nie specialit z ebayu, vyrvanych z velkych diskovych poli ktore maju svoje specifika.

Pretaktovanie.sk

ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub

Re: ZFS pool extremne pomaly scrub