使用fio对volume的性能进行了测试，对以下几种情况进行了测试和对比：

1. k8s with rbd volume(k8s running on kvm VM)

2. kvm VM accessing rbd through linux kernal

3. kvm VM accessing through librbd

4. host accessing RBD(linux kernal, rbd map)

5. k8s PVC

使用fio进行测试

fio -name=/root/tmpfile -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=16k -size=16G -numjobs=1  -group_reporting=1

fio -name=/root/tmpfile -direct=1 -iodepth=32 -rw=randread-ioengine=libaio -bs=16k -size=16G -numjobs=1  -group_reporting=1

以下是测试结果和对比, 只是一个大概的测试， 仅仅使用16k block

item	kubernetes rbd volume	kvm VM by linux kernal	kvm VM by librbd	host accesing RBD	kubernates by Persistent Volumes(RBD)
16k randread IOPS	8566	9628	9685	相近	相近
16k randwrite IOPS	2741	2754	3171	相近	相近
16k randread bw	137058KB/s	154052KB/s	154975KB/s	相近	相近
16k randwrite bw	43857KB/s	44078KB/s	50744KB/s	相近	相近

结论：

1. ceph rdb在guest VM和host上性能差不多

2. ceph rbd kernal和librbd性能相差不大，librbd性能相对好一些。

3. docker上的rdb volume和vm,还有host上的性能也差不多，docker上的rbd volume性能相当不错。

详细的测试结果：

================================

test1: kubernetes rbd volume:

================================
< -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=16k -size=16G -numj>
/mnt/rbd/tmpfile: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
Jobs: 1 (f=1): [w] [100.0% done] [0KB/17952KB/0KB /s] [0/1122/0 iops] [eta 00m:00s]
/mnt/rbd/tmpfile: (groupid=0, jobs=1): err= 0: pid=1751: Tue May  9 05:06:27 2017
  write: io=16384MB, bw=43857KB/s, iops=2741, runt=382544msec
    slat (usec): min=4, max=1920.6K, avg=35.92, stdev=2042.96
    clat (msec): min=2, max=4803, avg=11.63, stdev=54.37
     lat (msec): min=2, max=4803, avg=11.67, stdev=54.41
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    9], 60.00th=[   10],
     | 70.00th=[   11], 80.00th=[   13], 90.00th=[   17], 95.00th=[   21],
     | 99.00th=[   39], 99.50th=[   57], 99.90th=[  229], 99.95th=[  783],
     | 99.99th=[ 2966]
    bw (KB  /s): min=    6, max=55648, per=100.00%, avg=46518.63, stdev=10622.23
    lat (msec) : 4=0.49%, 10=63.86%, 20=30.01%, 50=5.04%, 100=0.40%
    lat (msec) : 250=0.11%, 500=0.03%, 750=0.01%, 1000=0.01%, 2000=0.01%
    lat (msec) : >=2000=0.03%
  cpu          : usr=1.94%, sys=7.92%, ctx=709934, majf=0, minf=25
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=16384MB, aggrb=43856KB/s, minb=43856KB/s, maxb=43856KB/s, mint=382544msec, maxt=382544msec

Disk stats (read/write):
  rbd0: ios=0/1053645, merge=0/148156, ticks=0/12241232, in_queue=12241570, util=100.00%

  <=randread -ioengine=libaio -bs=16k -size=16G -numjobs=1  -group_reporting=1
/mnt/rbd/tmpfile: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [160.2MB/0KB/0KB /s] [10.4K/0/0 iops] [eta 00m:00s]
/mnt/rbd/tmpfile: (groupid=0, jobs=1): err= 0: pid=1754: Tue May  9 05:09:41 2017
  read : io=16384MB, bw=137058KB/s, iops=8566, runt=122410msec
    slat (usec): min=2, max=6772, avg= 7.94, stdev=26.85
    clat (usec): min=549, max=952927, avg=3725.35, stdev=9155.47
     lat (usec): min=559, max=952936, avg=3733.60, stdev=9155.65
    clat percentiles (usec):
     |  1.00th=[ 1032],  5.00th=[ 1336], 10.00th=[ 1560], 20.00th=[ 1896],
     | 30.00th=[ 2160], 40.00th=[ 2416], 50.00th=[ 2704], 60.00th=[ 2992],
     | 70.00th=[ 3376], 80.00th=[ 3824], 90.00th=[ 4704], 95.00th=[ 5856],
     | 99.00th=[25216], 99.50th=[44800], 99.90th=[126464], 99.95th=[166912],
     | 99.99th=[374784]
    bw (KB  /s): min=30272, max=187360, per=100.00%, avg=137147.92, stdev=33606.47
    lat (usec) : 750=0.01%, 1000=0.69%
    lat (msec) : 2=23.18%, 4=58.73%, 10=14.74%, 20=1.33%, 50=0.87%
    lat (msec) : 100=0.28%, 250=0.14%, 500=0.02%, 750=0.01%, 1000=0.01%
  cpu          : usr=2.70%, sys=8.61%, ctx=383200, majf=0, minf=153
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=16384MB, aggrb=137057KB/s, minb=137057KB/s, maxb=137057KB/s, mint=122410msec, maxt=122410msec

Disk stats (read/write):
  rbd0: ios=1047827/4, merge=0/1, ticks=3754197/67, in_queue=3754965, util=100.00%

 ================================================================
 
test2:  kvm instance:
================================================================
 [root@localhost fio-2.1]# fio -name=/root/jzhao/tmpfile -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=16k -size=16G -numjobs=1  -group_reporting=1
/root/jzhao/tmpfile: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
/root/jzhao/tmpfile: Laying out IO file(s) (1 file(s) / 16384MB)
Jobs: 1 (f=1): [w] [100.0% done] [0KB/17968KB/0KB /s] [0/1123/0 iops] [eta 00m:00s]
/root/jzhao/tmpfile: (groupid=0, jobs=1): err= 0: pid=4062: Tue May  9 05:45:22 2017
  write: io=16384MB, bw=44078KB/s, iops=2754, runt=380623msec
    slat (usec): min=4, max=178921, avg=36.39, stdev=804.74
    clat (msec): min=2, max=3347, avg=11.58, stdev=32.03
     lat (msec): min=2, max=3347, avg=11.61, stdev=32.06
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    9], 60.00th=[   10],
     | 70.00th=[   12], 80.00th=[   14], 90.00th=[   18], 95.00th=[   24],
     | 99.00th=[   45], 99.50th=[   65], 99.90th=[  161], 99.95th=[  338],
     | 99.99th=[ 1434]
    bw (KB  /s): min=  519, max=54464, per=100.00%, avg=45126.18, stdev=8457.41
    lat (msec) : 4=0.47%, 10=61.20%, 20=30.95%, 50=6.61%, 100=0.51%
    lat (msec) : 250=0.18%, 500=0.04%, 750=0.01%, 1000=0.01%, 2000=0.01%
    lat (msec) : >=2000=0.01%
  cpu          : usr=2.01%, sys=8.31%, ctx=737590, majf=0, minf=24
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=16384MB, aggrb=44078KB/s, minb=44078KB/s, maxb=44078KB/s, mint=380623msec, maxt=380623msec

Disk stats (read/write):
  rbd1: ios=0/1055672, merge=0/155683, ticks=0/12304677, in_queue=12304483, util=100.00%

  [root@localhost fio-2.1]# fio -name=/root/jzhao/tmpfile -direct=1 -iodepth=32 -rw=randread -ioengine=libaio -bs=16k -size=16G -numjobs=1  -group_reporting=1
/root/jzhao/tmpfile: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [165.1MB/0KB/0KB /s] [10.6K/0/0 iops] [eta 00m:00s]
/root/jzhao/tmpfile: (groupid=0, jobs=1): err= 0: pid=9642: Tue May  9 05:49:13 2017
  read : io=16384MB, bw=154052KB/s, iops=9628, runt=108906msec
    slat (usec): min=2, max=6237, avg= 7.41, stdev=21.18
    clat (usec): min=564, max=738661, avg=3313.94, stdev=5667.71
     lat (usec): min=569, max=738664, avg=3321.62, stdev=5667.82
    clat percentiles (usec):
     |  1.00th=[ 1080],  5.00th=[ 1384], 10.00th=[ 1608], 20.00th=[ 1928],
     | 30.00th=[ 2192], 40.00th=[ 2480], 50.00th=[ 2736], 60.00th=[ 3056],
     | 70.00th=[ 3408], 80.00th=[ 3888], 90.00th=[ 4640], 95.00th=[ 5536],
     | 99.00th=[13632], 99.50th=[22400], 99.90th=[72192], 99.95th=[108032],
     | 99.99th=[211968]
    bw (KB  /s): min=44960, max=185984, per=100.00%, avg=154205.67, stdev=22074.34
    lat (usec) : 750=0.01%, 1000=0.47%
    lat (msec) : 2=21.98%, 4=59.65%, 10=16.48%, 20=0.81%, 50=0.42%
    lat (msec) : 100=0.11%, 250=0.05%, 500=0.01%, 750=0.01%
  cpu          : usr=3.01%, sys=9.16%, ctx=391490, majf=0, minf=152
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=16384MB, aggrb=154052KB/s, minb=154052KB/s, maxb=154052KB/s, mint=108906msec, maxt=108906msec

Disk stats (read/write):
  rbd1: ios=1047621/4, merge=0/1, ticks=3322911/31, in_queue=3323436, util=100.00%

 
==========================================================

test3: kvm vm with librbd:
===========================================================
[root@localhost ~]# fio -name=/jzhao/tmpfile -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=16k -size=10G -numjobs=1  -group_reporting=1
/jzhao/tmpfile: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
/jzhao/tmpfile: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 1 (f=1): [w] [100.0% done] [0KB/80303KB/0KB /s] [0/5018/0 iops] [eta 00m:00s]
/jzhao/tmpfile: (groupid=0, jobs=1): err= 0: pid=30341: Wed May 10 05:11:08 2017
  write: io=10240MB, bw=50744KB/s, iops=3171, runt=206640msec
    slat (usec): min=5, max=2780.6K, avg=30.84, stdev=3443.02
    clat (usec): min=118, max=2861.7K, avg=10055.79, stdev=30333.27
     lat (usec): min=149, max=2861.7K, avg=10087.06, stdev=30526.98
    clat percentiles (usec):
     |  1.00th=[  652],  5.00th=[  908], 10.00th=[ 1496], 20.00th=[ 4448],
     | 30.00th=[ 6432], 40.00th=[ 7776], 50.00th=[ 8896], 60.00th=[10048],
     | 70.00th=[11328], 80.00th=[12992], 90.00th=[15808], 95.00th=[19328],
     | 99.00th=[34560], 99.50th=[46848], 99.90th=[232448], 99.95th=[325632],
     | 99.99th=[477184]
    bw (KB  /s): min= 3501, max=131206, per=100.00%, avg=52221.12, stdev=14813.34
    lat (usec) : 250=0.01%, 500=0.02%, 750=2.63%, 1000=3.53%
    lat (msec) : 2=5.57%, 4=6.50%, 10=41.52%, 20=35.77%, 50=4.01%
    lat (msec) : 100=0.24%, 250=0.12%, 500=0.08%, >=2000=0.01%
  cpu          : usr=2.01%, sys=8.14%, ctx=241327, majf=0, minf=24
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=655360/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=10240MB, aggrb=50744KB/s, minb=50744KB/s, maxb=50744KB/s, mint=206640msec, maxt=206640msec

Disk stats (read/write):
  vda: ios=0/655668, merge=0/19280, ticks=0/6453652, in_queue=6468591, util=100.00%

  [root@localhost ~]# fio -name=/jzhao/tmpfile -direct=1 -iodepth=32 -rw=randread -ioengine=libaio -bs=16k -size=10G -numjobs=1  -group_reporting=1
/jzhao/tmpfile: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [200.4MB/0KB/0KB /s] [12.9K/0/0 iops] [eta 00m:00s]
/jzhao/tmpfile: (groupid=0, jobs=1): err= 0: pid=666: Wed May 10 05:13:06 2017
  read : io=10240MB, bw=154975KB/s, iops=9685, runt= 67661msec
    slat (usec): min=2, max=3715, avg=12.52, stdev=14.56
    clat (usec): min=137, max=601820, avg=3288.13, stdev=12099.75
     lat (usec): min=437, max=601824, avg=3301.01, stdev=12099.73
    clat percentiles (usec):
     |  1.00th=[  636],  5.00th=[  884], 10.00th=[ 1080], 20.00th=[ 1368],
     | 30.00th=[ 1608], 40.00th=[ 1832], 50.00th=[ 2040], 60.00th=[ 2256],
     | 70.00th=[ 2512], 80.00th=[ 2864], 90.00th=[ 3568], 95.00th=[ 4704],
     | 99.00th=[31872], 99.50th=[61696], 99.90th=[191488], 99.95th=[268288],
     | 99.99th=[407552]
    bw (KB  /s): min=16192, max=219680, per=100.00%, avg=155079.54, stdev=49649.27
    lat (usec) : 250=0.01%, 500=0.01%, 750=2.55%, 1000=5.21%
    lat (msec) : 2=40.27%, 4=44.68%, 10=4.94%, 20=0.87%, 50=0.83%
    lat (msec) : 100=0.34%, 250=0.23%, 500=0.06%, 750=0.01%
  cpu          : usr=5.63%, sys=17.22%, ctx=477444, majf=0, minf=152
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=655360/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=10240MB, aggrb=154974KB/s, minb=154974KB/s, maxb=154974KB/s, mint=67661msec, maxt=67661msec

Disk stats (read/write):
  vda: ios=653810/742, merge=0/136, ticks=2136381/17532, in_queue=2154169, util=99.96%

===========================================