file1: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=4
file2: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
file3: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=8
file4: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=16
fio-3.28
Starting 4 processes
file1: Laying out IO file (1 file / 512MiB)
file2: Laying out IO file (1 file / 512MiB)
file3: Laying out IO file (1 file / 512MiB)
file4: Laying out IO file (1 file / 512MiB)

file1: (groupid=0, jobs=1): err= 0: pid=108150: Tue Feb  7 18:53:46 2023
  read: IOPS=15.5k, BW=1939MiB/s (2034MB/s)(512MiB/264msec)
    slat (usec): min=2, max=121, avg= 4.46, stdev= 5.31
    clat (usec): min=17, max=2145, avg=252.31, stdev=309.02
     lat (usec): min=23, max=2155, avg=256.81, stdev=310.87
    clat percentiles (usec):
     |  1.00th=[   51],  5.00th=[   54], 10.00th=[   55], 20.00th=[   56],
     | 30.00th=[   57], 40.00th=[   58], 50.00th=[  145], 60.00th=[  182],
     | 70.00th=[  188], 80.00th=[  433], 90.00th=[  922], 95.00th=[  947],
     | 99.00th=[ 1090], 99.50th=[ 1336], 99.90th=[ 1958], 99.95th=[ 2147],
     | 99.99th=[ 2147]
  lat (usec)   : 20=0.27%, 50=0.71%, 100=48.19%, 250=25.61%, 500=12.40%
  lat (usec)   : 750=0.29%, 1000=10.52%
  lat (msec)   : 2=1.90%, 4=0.10%
  cpu          : usr=1.90%, sys=8.75%, ctx=3787, majf=0, minf=139
  IO depths    : 1=0.1%, 2=0.1%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
file2: (groupid=0, jobs=1): err= 0: pid=108151: Tue Feb  7 18:53:46 2023
  read: IOPS=31.8k, BW=3969MiB/s (4162MB/s)(512MiB/129msec)
    slat (nsec): min=2260, max=90257, avg=4079.06, stdev=4877.99
    clat (usec): min=441, max=2550, avg=989.05, stdev=186.88
     lat (usec): min=449, max=2604, avg=993.16, stdev=189.54
    clat percentiles (usec):
     |  1.00th=[  873],  5.00th=[  898], 10.00th=[  906], 20.00th=[  922],
     | 30.00th=[  930], 40.00th=[  938], 50.00th=[  947], 60.00th=[  947],
     | 70.00th=[  955], 80.00th=[  979], 90.00th=[ 1074], 95.00th=[ 1270],
     | 99.00th=[ 2147], 99.50th=[ 2147], 99.90th=[ 2442], 99.95th=[ 2507],
     | 99.99th=[ 2540]
  lat (usec)   : 500=0.07%, 750=0.24%, 1000=82.52%
  lat (msec)   : 2=15.99%, 4=1.17%
  cpu          : usr=0.00%, sys=19.53%, ctx=3856, majf=0, minf=1036
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
file3: (groupid=0, jobs=1): err= 0: pid=108152: Tue Feb  7 18:53:46 2023
  read: IOPS=17.6k, BW=2197MiB/s (2304MB/s)(512MiB/233msec)
    slat (nsec): min=2411, max=92956, avg=4069.30, stdev=4766.44
    clat (usec): min=96, max=2437, avg=449.22, stdev=339.94
     lat (usec): min=107, max=2484, avg=453.32, stdev=340.81
    clat percentiles (usec):
     |  1.00th=[  153],  5.00th=[  180], 10.00th=[  182], 20.00th=[  182],
     | 30.00th=[  184], 40.00th=[  186], 50.00th=[  355], 60.00th=[  433],
     | 70.00th=[  449], 80.00th=[  914], 90.00th=[  947], 95.00th=[  979],
     | 99.00th=[ 1336], 99.50th=[ 1729], 99.90th=[ 2114], 99.95th=[ 2147],
     | 99.99th=[ 2442]
  lat (usec)   : 100=0.05%, 250=49.15%, 500=25.12%, 750=0.66%, 1000=21.04%
  lat (msec)   : 2=3.59%, 4=0.39%
  cpu          : usr=5.17%, sys=5.60%, ctx=3712, majf=0, minf=267
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
file4: (groupid=0, jobs=1): err= 0: pid=108153: Tue Feb  7 18:53:46 2023
  read: IOPS=22.1k, BW=2768MiB/s (2902MB/s)(512MiB/185msec)
    slat (nsec): min=2393, max=74730, avg=3706.28, stdev=4347.69
    clat (usec): min=365, max=2545, avg=714.48, stdev=304.70
     lat (usec): min=369, max=2589, avg=718.22, stdev=306.10
    clat percentiles (usec):
     |  1.00th=[  424],  5.00th=[  429], 10.00th=[  433], 20.00th=[  433],
     | 30.00th=[  437], 40.00th=[  445], 50.00th=[  832], 60.00th=[  922],
     | 70.00th=[  938], 80.00th=[  947], 90.00th=[  979], 95.00th=[ 1057],
     | 99.00th=[ 1811], 99.50th=[ 2089], 99.90th=[ 2409], 99.95th=[ 2507],
     | 99.99th=[ 2540]
  lat (usec)   : 500=48.71%, 750=1.07%, 1000=42.24%
  lat (msec)   : 2=7.35%, 4=0.63%
  cpu          : usr=0.00%, sys=12.50%, ctx=3826, majf=0, minf=522
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7758MiB/s (8134MB/s), 1939MiB/s-3969MiB/s (2034MB/s-4162MB/s), io=2048MiB (2147MB), run=129-264msec