file1: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=4
file2: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
file3: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=8
file4: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=16
fio-3.28
Starting 4 processes
file1: Laying out IO file (1 file / 512MiB)
file2: Laying out IO file (1 file / 512MiB)
file3: Laying out IO file (1 file / 512MiB)
file4: Laying out IO file (1 file / 512MiB)

file1: (groupid=0, jobs=1): err= 0: pid=3126: Tue Feb  7 20:58:28 2023
  read: IOPS=9799, BW=1225MiB/s (1284MB/s)(512MiB/418msec)
    slat (usec): min=8, max=202, avg=14.65, stdev= 7.08
    clat (usec): min=85, max=9239, avg=391.96, stdev=575.39
     lat (usec): min=97, max=9417, avg=406.72, stdev=578.82
    clat percentiles (usec):
     |  1.00th=[   91],  5.00th=[   97], 10.00th=[  106], 20.00th=[  116],
     | 30.00th=[  128], 40.00th=[  143], 50.00th=[  167], 60.00th=[  204],
     | 70.00th=[  269], 80.00th=[  490], 90.00th=[ 1172], 95.00th=[ 1385],
     | 99.00th=[ 2376], 99.50th=[ 3097], 99.90th=[ 5604], 99.95th=[ 8979],
     | 99.99th=[ 9241]
  lat (usec)   : 100=7.23%, 250=60.84%, 500=12.08%, 750=3.66%, 1000=3.27%
  lat (msec)   : 2=11.45%, 4=1.20%, 10=0.27%
  cpu          : usr=2.40%, sys=17.75%, ctx=3192, majf=0, minf=140
  IO depths    : 1=0.1%, 2=0.1%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
file2: (groupid=0, jobs=1): err= 0: pid=3127: Tue Feb  7 20:58:28 2023
  read: IOPS=22.3k, BW=2783MiB/s (2918MB/s)(512MiB/184msec)
    slat (usec): min=8, max=146, avg=16.16, stdev= 5.45
    clat (usec): min=308, max=9220, avg=1405.84, stdev=861.72
     lat (usec): min=324, max=9279, avg=1422.10, stdev=864.98
    clat percentiles (usec):
     |  1.00th=[  490],  5.00th=[  685], 10.00th=[  807], 20.00th=[ 1045],
     | 30.00th=[ 1139], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303],
     | 70.00th=[ 1385], 80.00th=[ 1532], 90.00th=[ 2089], 95.00th=[ 2638],
     | 99.00th=[ 4015], 99.50th=[ 8848], 99.90th=[ 9110], 99.95th=[ 9110],
     | 99.99th=[ 9241]
  lat (usec)   : 500=1.10%, 750=6.62%, 1000=10.84%
  lat (msec)   : 2=70.75%, 4=9.67%, 10=1.03%
  cpu          : usr=7.65%, sys=39.34%, ctx=2352, majf=0, minf=1037
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
file3: (groupid=0, jobs=1): err= 0: pid=3128: Tue Feb  7 20:58:28 2023
  read: IOPS=11.9k, BW=1493MiB/s (1565MB/s)(512MiB/343msec)
    slat (usec): min=8, max=151, avg=16.34, stdev= 6.39
    clat (usec): min=88, max=9095, avg=650.03, stdev=713.51
     lat (usec): min=107, max=9166, avg=666.48, stdev=715.83
    clat percentiles (usec):
     |  1.00th=[  116],  5.00th=[  135], 10.00th=[  153], 20.00th=[  184],
     | 30.00th=[  217], 40.00th=[  262], 50.00th=[  338], 60.00th=[  494],
     | 70.00th=[  857], 80.00th=[ 1188], 90.00th=[ 1401], 95.00th=[ 1729],
     | 99.00th=[ 2737], 99.50th=[ 3523], 99.90th=[ 8848], 99.95th=[ 8979],
     | 99.99th=[ 9110]
  lat (usec)   : 100=0.12%, 250=37.35%, 500=22.80%, 750=7.20%, 1000=5.59%
  lat (msec)   : 2=24.05%, 4=2.44%, 10=0.44%
  cpu          : usr=2.92%, sys=23.68%, ctx=2770, majf=0, minf=271
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
file4: (groupid=0, jobs=1): err= 0: pid=3129: Tue Feb  7 20:58:28 2023
  read: IOPS=14.8k, BW=1848MiB/s (1938MB/s)(512MiB/277msec)
    slat (usec): min=8, max=170, avg=14.52, stdev= 5.71
    clat (usec): min=111, max=9129, avg=1062.17, stdev=831.27
     lat (usec): min=122, max=9287, avg=1076.80, stdev=833.63
    clat percentiles (usec):
     |  1.00th=[  127],  5.00th=[  163], 10.00th=[  229], 20.00th=[  437],
     | 30.00th=[  652], 40.00th=[  840], 50.00th=[ 1074], 60.00th=[ 1172],
     | 70.00th=[ 1270], 80.00th=[ 1401], 90.00th=[ 1729], 95.00th=[ 2089],
     | 99.00th=[ 3982], 99.50th=[ 5997], 99.90th=[ 8979], 99.95th=[ 9110],
     | 99.99th=[ 9110]
  lat (usec)   : 250=11.55%, 500=10.86%, 750=12.92%, 1000=11.77%
  lat (msec)   : 2=47.09%, 4=4.86%, 10=0.95%
  cpu          : usr=4.71%, sys=25.36%, ctx=2851, majf=0, minf=525
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4900MiB/s (5138MB/s), 1225MiB/s-2783MiB/s (1284MB/s-2918MB/s), io=2048MiB (2147MB), run=184-418msec

Disk stats (read/write):
  nvme0n1: ios=10344/0, merge=0/0, ticks=12643/0, in_queue=12642, util=73.24%