file1: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=4
file2: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
file3: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=8
file4: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=16
fio-3.28
Starting 4 processes

file1: (groupid=0, jobs=1): err= 0: pid=35039: Fri Jan 13 12:28:41 2023
  read: IOPS=13.9k, BW=1741MiB/s (1826MB/s)(512MiB/294msec)
    slat (usec): min=2, max=1218, avg=15.89, stdev=92.74
    clat (usec): min=57, max=3374, avg=268.99, stdev=358.49
     lat (usec): min=61, max=3379, avg=284.92, stdev=389.27
    clat percentiles (usec):
     |  1.00th=[   59],  5.00th=[   60], 10.00th=[   60], 20.00th=[   61],
     | 30.00th=[   61], 40.00th=[   62], 50.00th=[   65], 60.00th=[  200],
     | 70.00th=[  204], 80.00th=[  474], 90.00th=[  766], 95.00th=[ 1123],
     | 99.00th=[ 1221], 99.50th=[ 2245], 99.90th=[ 2343], 99.95th=[ 2573],
     | 99.99th=[ 3359]
  lat (usec)   : 100=50.46%, 250=25.56%, 500=9.99%, 750=3.93%, 1000=0.95%
  lat (msec)   : 2=8.40%, 4=0.71%
  cpu          : usr=0.68%, sys=10.24%, ctx=3521, majf=0, minf=139
  IO depths    : 1=0.1%, 2=0.1%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
file2: (groupid=0, jobs=1): err= 0: pid=35040: Fri Jan 13 12:28:41 2023
  read: IOPS=26.9k, BW=3368MiB/s (3532MB/s)(512MiB/152msec)
    slat (usec): min=2, max=1067, avg=11.06, stdev=55.64
    clat (usec): min=248, max=8011, avg=1129.81, stdev=408.47
     lat (usec): min=265, max=8458, avg=1140.90, stdev=434.74
    clat percentiles (usec):
     |  1.00th=[  519],  5.00th=[  832], 10.00th=[  996], 20.00th=[ 1012],
     | 30.00th=[ 1057], 40.00th=[ 1057], 50.00th=[ 1057], 60.00th=[ 1074],
     | 70.00th=[ 1074], 80.00th=[ 1074], 90.00th=[ 1205], 95.00th=[ 1975],
     | 99.00th=[ 2671], 99.50th=[ 2900], 99.90th=[ 5604], 99.95th=[ 6849],
     | 99.99th=[ 8029]
  lat (usec)   : 250=0.02%, 500=0.88%, 750=1.76%, 1000=12.30%
  lat (msec)   : 2=80.10%, 4=4.66%, 10=0.27%
  cpu          : usr=0.00%, sys=16.56%, ctx=3740, majf=0, minf=1037
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
file3: (groupid=0, jobs=1): err= 0: pid=35041: Fri Jan 13 12:28:41 2023
  read: IOPS=15.8k, BW=1969MiB/s (2065MB/s)(512MiB/260msec)
    slat (usec): min=2, max=1121, avg=12.29, stdev=65.83
    clat (usec): min=121, max=3613, avg=490.92, stdev=366.25
     lat (usec): min=138, max=4080, avg=503.25, stdev=381.76
    clat percentiles (usec):
     |  1.00th=[  196],  5.00th=[  198], 10.00th=[  198], 20.00th=[  200],
     | 30.00th=[  200], 40.00th=[  208], 50.00th=[  429], 60.00th=[  494],
     | 70.00th=[  498], 80.00th=[  996], 90.00th=[ 1057], 95.00th=[ 1074],
     | 99.00th=[ 1287], 99.50th=[ 1991], 99.90th=[ 2376], 99.95th=[ 3064],
     | 99.99th=[ 3621]
  lat (usec)   : 250=47.83%, 500=24.71%, 750=3.96%, 1000=3.71%
  lat (msec)   : 2=19.31%, 4=0.49%
  cpu          : usr=0.00%, sys=10.04%, ctx=3725, majf=0, minf=269
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
file4: (groupid=0, jobs=1): err= 0: pid=35042: Fri Jan 13 12:28:41 2023
  read: IOPS=19.4k, BW=2427MiB/s (2544MB/s)(512MiB/211msec)
    slat (usec): min=2, max=1059, avg=11.56, stdev=59.43
    clat (usec): min=121, max=5625, avg=795.54, stdev=361.69
     lat (usec): min=138, max=6129, avg=807.13, stdev=380.13
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  486], 10.00th=[  486], 20.00th=[  494],
     | 30.00th=[  494], 40.00th=[  498], 50.00th=[  832], 60.00th=[ 1020],
     | 70.00th=[ 1057], 80.00th=[ 1074], 90.00th=[ 1074], 95.00th=[ 1106],
     | 99.00th=[ 1811], 99.50th=[ 2147], 99.90th=[ 4047], 99.95th=[ 4883],
     | 99.99th=[ 5604]
  lat (usec)   : 250=0.27%, 500=45.68%, 750=2.78%, 1000=7.40%
  lat (msec)   : 2=43.19%, 4=0.56%, 10=0.12%
  cpu          : usr=0.00%, sys=12.38%, ctx=3747, majf=0, minf=524
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6966MiB/s (7304MB/s), 1741MiB/s-3368MiB/s (1826MB/s-3532MB/s), io=2048MiB (2147MB), run=152-294msec