file1: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=4
file2: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
file3: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=8
file4: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=16
fio-3.28
Starting 4 processes
file1: Laying out IO file (1 file / 512MiB)
file2: Laying out IO file (1 file / 512MiB)
file3: Laying out IO file (1 file / 512MiB)
file4: Laying out IO file (1 file / 512MiB)

file1: (groupid=0, jobs=1): err= 0: pid=61922: Thu Jan 12 16:58:20 2023
  read: IOPS=9660, BW=1208MiB/s (1266MB/s)(512MiB/424msec)
    slat (usec): min=8, max=213, avg=14.34, stdev= 7.81
    clat (usec): min=23, max=9258, avg=397.47, stdev=608.81
     lat (usec): min=95, max=9473, avg=411.90, stdev=613.36
    clat percentiles (usec):
     |  1.00th=[   91],  5.00th=[   96], 10.00th=[  104], 20.00th=[  115],
     | 30.00th=[  128], 40.00th=[  143], 50.00th=[  165], 60.00th=[  198],
     | 70.00th=[  255], 80.00th=[  449], 90.00th=[ 1188], 95.00th=[ 1418],
     | 99.00th=[ 3032], 99.50th=[ 3654], 99.90th=[ 4621], 99.95th=[ 8848],
     | 99.99th=[ 9241]
  lat (usec)   : 50=0.02%, 100=7.71%, 250=61.62%, 500=11.79%, 750=3.08%
  lat (usec)   : 1000=2.54%
  lat (msec)   : 2=11.23%, 4=1.64%, 10=0.37%
  cpu          : usr=3.55%, sys=15.84%, ctx=3273, majf=0, minf=140
  IO depths    : 1=0.1%, 2=0.1%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
file2: (groupid=0, jobs=1): err= 0: pid=61923: Thu Jan 12 16:58:20 2023
  read: IOPS=21.6k, BW=2695MiB/s (2826MB/s)(512MiB/190msec)
    slat (usec): min=9, max=152, avg=14.97, stdev= 6.72
    clat (usec): min=113, max=9051, avg=1453.20, stdev=1052.30
     lat (usec): min=127, max=9205, avg=1468.26, stdev=1055.87
    clat percentiles (usec):
     |  1.00th=[  137],  5.00th=[  204], 10.00th=[  416], 20.00th=[  979],
     | 30.00th=[ 1139], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303],
     | 70.00th=[ 1385], 80.00th=[ 1663], 90.00th=[ 2737], 95.00th=[ 3523],
     | 99.00th=[ 5276], 99.50th=[ 8029], 99.90th=[ 8848], 99.95th=[ 8848],
     | 99.99th=[ 9110]
  lat (usec)   : 250=6.64%, 500=4.88%, 750=5.10%, 1000=3.54%
  lat (msec)   : 2=63.65%, 4=13.40%, 10=2.78%
  cpu          : usr=0.00%, sys=42.33%, ctx=2534, majf=0, minf=1037
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
file3: (groupid=0, jobs=1): err= 0: pid=61924: Thu Jan 12 16:58:20 2023
  read: IOPS=11.7k, BW=1467MiB/s (1538MB/s)(512MiB/349msec)
    slat (usec): min=7, max=190, avg=13.69, stdev= 5.58
    clat (usec): min=89, max=8823, avg=665.52, stdev=768.51
     lat (usec): min=100, max=8926, avg=679.29, stdev=771.14
    clat percentiles (usec):
     |  1.00th=[  113],  5.00th=[  135], 10.00th=[  151], 20.00th=[  184],
     | 30.00th=[  212], 40.00th=[  255], 50.00th=[  318], 60.00th=[  445],
     | 70.00th=[  799], 80.00th=[ 1205], 90.00th=[ 1434], 95.00th=[ 1844],
     | 99.00th=[ 3752], 99.50th=[ 3982], 99.90th=[ 8586], 99.95th=[ 8717],
     | 99.99th=[ 8848]
  lat (usec)   : 100=0.17%, 250=38.92%, 500=23.44%, 750=6.62%, 1000=3.91%
  lat (msec)   : 2=22.75%, 4=3.76%, 10=0.44%
  cpu          : usr=1.44%, sys=20.40%, ctx=3004, majf=0, minf=269
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
file4: (groupid=0, jobs=1): err= 0: pid=61925: Thu Jan 12 16:58:20 2023
  read: IOPS=14.5k, BW=1809MiB/s (1897MB/s)(512MiB/283msec)
    slat (usec): min=9, max=154, avg=15.19, stdev= 6.46
    clat (usec): min=96, max=8808, avg=1083.36, stdev=891.55
     lat (usec): min=110, max=8951, avg=1098.64, stdev=894.61
    clat percentiles (usec):
     |  1.00th=[  121],  5.00th=[  151], 10.00th=[  186], 20.00th=[  347],
     | 30.00th=[  578], 40.00th=[  824], 50.00th=[ 1106], 60.00th=[ 1188],
     | 70.00th=[ 1270], 80.00th=[ 1418], 90.00th=[ 1811], 95.00th=[ 2737],
     | 99.00th=[ 4047], 99.50th=[ 4424], 99.90th=[ 8717], 99.95th=[ 8717],
     | 99.99th=[ 8848]
  lat (usec)   : 100=0.02%, 250=15.19%, 500=11.52%, 750=10.38%, 1000=8.84%
  lat (msec)   : 2=45.70%, 4=7.20%, 10=1.15%
  cpu          : usr=0.00%, sys=29.79%, ctx=2911, majf=0, minf=525
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4830MiB/s (5065MB/s), 1208MiB/s-2695MiB/s (1266MB/s-2826MB/s), io=2048MiB (2147MB), run=190-424msec

Disk stats (read/write):
  nvme0n1: ios=8204/0, merge=0/0, ticks=11464/0, in_queue=11464, util=67.97%