file1: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=4 file2: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32 file3: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=8 file4: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=16 fio-3.28 Starting 4 processes file1: Laying out IO file (1 file / 512MiB) file2: Laying out IO file (1 file / 512MiB) file3: Laying out IO file (1 file / 512MiB) file4: Laying out IO file (1 file / 512MiB) file1: (groupid=0, jobs=1): err= 0: pid=3874: Fri Feb 3 11:57:34 2023 read: IOPS=3052, BW=382MiB/s (400MB/s)(512MiB/1342msec) slat (usec): min=43, max=1734, avg=324.61, stdev=124.89 clat (usec): min=3, max=3235, avg=981.87, stdev=279.37 lat (usec): min=316, max=4163, avg=1306.80, stdev=357.46 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 523], 10.00th=[ 578], 20.00th=[ 709], | 30.00th=[ 881], 40.00th=[ 971], 50.00th=[ 1029], 60.00th=[ 1074], | 70.00th=[ 1106], 80.00th=[ 1172], 90.00th=[ 1254], 95.00th=[ 1385], | 99.00th=[ 1811], 99.50th=[ 1991], 99.90th=[ 2343], 99.95th=[ 2835], | 99.99th=[ 3228] bw ( KiB/s): min=387328, max=414720, per=25.76%, avg=401024.00, stdev=19369.07, samples=2 iops : min= 3026, max= 3240, avg=3133.00, stdev=151.32, samples=2 lat (usec) : 4=0.02%, 500=3.17%, 750=19.21%, 1000=22.22% lat (msec) : 2=54.91%, 4=0.46% cpu : usr=0.67%, sys=4.55%, ctx=4108, majf=0, minf=140 IO depths : 1=0.1%, 2=0.1%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=4 file2: (groupid=0, jobs=1): err= 0: pid=3875: Fri Feb 3 11:57:34 2023 read: IOPS=3056, BW=382MiB/s (401MB/s)(512MiB/1340msec) slat (usec): min=51, max=2498, avg=324.12, stdev=123.68 clat (usec): min=2, max=15906, avg=10073.77, stdev=2307.32 lat (usec): min=168, max=16259, avg=10398.19, stdev=2369.33 clat percentiles (usec): | 1.00th=[ 5145], 5.00th=[ 5604], 10.00th=[ 6194], 20.00th=[ 7767], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10814], 60.00th=[11076], | 70.00th=[11338], 80.00th=[11731], 90.00th=[12256], 95.00th=[12780], | 99.00th=[14746], 99.50th=[15270], 99.90th=[15795], 99.95th=[15795], | 99.99th=[15926] bw ( KiB/s): min=379648, max=417792, per=25.61%, avg=398720.00, stdev=26971.88, samples=2 iops : min= 2966, max= 3264, avg=3115.00, stdev=210.72, samples=2 lat (usec) : 4=0.02%, 250=0.02%, 500=0.02%, 1000=0.02% lat (msec) : 2=0.07%, 4=0.15%, 10=33.42%, 20=66.26% cpu : usr=0.97%, sys=4.56%, ctx=4105, majf=0, minf=1037 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 file3: (groupid=0, jobs=1): err= 0: pid=3876: Fri Feb 3 11:57:34 2023 read: IOPS=3049, BW=381MiB/s (400MB/s)(512MiB/1343msec) slat (usec): min=43, max=1783, avg=325.19, stdev=126.68 clat (nsec): min=1051, max=4486.4k, avg=2288024.69, stdev=598959.87 lat (usec): min=146, max=6232, avg=2613.47, stdev=677.26 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1205], 10.00th=[ 1319], 20.00th=[ 1680], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2540], | 70.00th=[ 2606], 80.00th=[ 2704], 90.00th=[ 2868], 95.00th=[ 3097], | 99.00th=[ 3720], 99.50th=[ 3916], 99.90th=[ 4359], 99.95th=[ 4424], | 99.99th=[ 4490] bw ( KiB/s): min=386816, max=413696, per=25.71%, avg=400256.00, stdev=19007.03, samples=2 iops : min= 3022, max= 3232, avg=3127.00, stdev=148.49, samples=2 lat (usec) : 2=0.02%, 250=0.02%, 500=0.05%, 750=0.05%, 1000=0.27% lat (msec) : 2=26.73%, 4=72.51%, 10=0.34% cpu : usr=0.52%, sys=4.25%, ctx=4107, majf=0, minf=269 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 file4: (groupid=0, jobs=1): err= 0: pid=3877: Fri Feb 3 11:57:34 2023 read: IOPS=3040, BW=380MiB/s (399MB/s)(512MiB/1347msec) slat (usec): min=45, max=2816, avg=325.94, stdev=134.44 clat (nsec): min=1649, max=9579.1k, avg=4916548.15, stdev=1269810.51 lat (usec): min=73, max=10187, avg=5242.78, stdev=1345.21 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 2638], 10.00th=[ 2868], 20.00th=[ 3621], | 30.00th=[ 4686], 40.00th=[ 5080], 50.00th=[ 5211], 60.00th=[ 5407], | 70.00th=[ 5604], 80.00th=[ 5800], 90.00th=[ 6128], 95.00th=[ 6521], | 99.00th=[ 8160], 99.50th=[ 8586], 99.90th=[ 9110], 99.95th=[ 9372], | 99.99th=[ 9634] bw ( KiB/s): min=376320, max=414464, per=25.40%, avg=395392.00, stdev=26971.88, samples=2 iops : min= 2940, max= 3238, avg=3089.00, stdev=210.72, samples=2 lat (usec) : 2=0.02%, 100=0.02%, 250=0.02%, 500=0.07%, 750=0.07% lat (usec) : 1000=0.02% lat (msec) : 2=0.88%, 4=22.46%, 10=76.42% cpu : usr=1.78%, sys=3.19%, ctx=4105, majf=0, minf=525 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1520MiB/s (1594MB/s), 380MiB/s-382MiB/s (399MB/s-401MB/s), io=2048MiB (2147MB), run=1340-1347msec