分析virtio-blk+qemu+spdk环境中virtio-blk不用执行virtqueue_kick操作通知后端处理IO的原因_virtqueue kick

分析virtio-blk+qemu+spdk场景下不用virtqueue_kick通知后端的原因

        当VM向virtio-blk的vring中添加IO vector之后，代码会进入到virtqueue_kick函数，该函数的实现如下

        virtqueue_kick_prepare函数中如果vq->event不存在，则判断virtio-blk设备是否设置了【VRING_USED_F_NO_NOTIFY】feature，如果virtio-blk设备设置了此feature，则needs_kick一直为false，则virtqueue_kick_prepare函数一直返回false，从而virtqueue_kick函数不会执行virtqueue_notify函数通知virtio-blk后端来处理IO。

        而virtqueue_kick_prepare函数中vq->event的值是在__vring_new_virtqueue函数中赋值，赋值的代码如下
        
                vq->event = virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX);
        
        也就是说virtio-blk设备不支持VIRTIO_RING_F_EVENT_IDX feature的话，那么vq->event就为NULL，此时我们看看spdk在进行virtio-blk设备feature协商的时候都设置了哪些feature。下面是spdk enable的feature
                #define SPDK_VHOST_FEATURES ((1ULL << VHOST_F_LOG_ALL) |
                        (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |
                        (1ULL << VIRTIO_F_VERSION_1) |
                        (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
                        (1ULL << VIRTIO_RING_F_EVENT_IDX) |
                        (1ULL << VIRTIO_RING_F_INDIRECT_DESC))

        
        spdk_vhost_dev_register函数设置feature的代码如下

        
        rte_vhost_driver_set_features函数直接设置path对应的vsocket->feature

        
        rte_vhost_driver_disable_features函数去掉vsocket->feature中想要disable的feature。

        通过跟踪rte_vhost_driver_disable_features函数中的features参数发现，此features = vhost_blk_device_backend ->disabled_features ,而vhost_blk_device_backend->disabled_features 数据结构的值如下：

                static const struct spdk_vhost_dev_backend vhost_blk_device_backend =
                {
                        …
                        .disabled_features = SPDK_VHOST_DISABLED_FEATURES |
                                                (1ULL << VIRTIO_BLK_F_GEOMETRY) |
                                                (1ULL << VIRTIO_BLK_F_RO) |
                                                (1ULL << VIRTIO_BLK_F_FLUSH) |
                                                (1ULL << VIRTIO_BLK_F_CONFIG_WCE) |
                                                (1ULL << VIRTIO_BLK_F_BARRIER) |
                                                (1ULL << VIRTIO_BLK_F_SCSI),
                        …
                }

        而 SPDK_VHOST_DISABLED_FEATURES 宏定义如下：
                        #define SPDK_VHOST_DISABLED_FEATURES ((1ULL << VIRTIO_RING_F_EVENT_IDX) |
(1ULL << VIRTIO_F_NOTIFY_ON_EMPTY))

        这样的话virtio-blk设备就不存在feature【VIRTIO_RING_F_EVENT_IDX】。这样vm的driver代码执行到__vring_new_virtqueue函数下面代码时，vq->event为false。
        
                        vq->event = virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX);
        
        这样的话，virtqueue_kick_prepare函数将执行下图中红框部分的代码。

        下面我们看下virtio-blk设备是否支持feature【VRING_USED_F_NO_NOTIFY】，我们期望是支持的，这样need_kicks就为false，从而不用通知后端来处理IO，我们再从spdk代码中分析此feature。
        spdk在rte_vhost_enable_guest_notification函数设置VRING_USED_F_NO_NOTIFY
        
                        dev->virtqueue[queue_id]->used->flags = VRING_USED_F_NO_NOTIFY;
        
        rte_vhost_enable_guest_notification函数的调用栈如下

(gdb) bt
#0 rte_vhost_enable_guest_notification (vid=vid@entry=0, queue_id=queue_id@entry=0, enable=enable@entry=0) at vhost.c:572
#1 0x0000000000442bd2 in start_device (vid=0) at vhost.c:1655
#2 0x000000000049707c in vhost_user_msg_handler (vid=, fd=fd@entry=60) at vhost_user.c:2373
#3 0x00000000004941ec in vhost_user_read_cb (connfd=60, dat=0x7ffecc0008c0, remove=0x7ffedb911f98) at socket.c:572
#4 0x000000000049a231 in fdset_event_dispatch (arg=0x80f940 <vhost_user+8192>) at fd_man.c:314
#5 0x00007ffff5ce8e25 in start_thread () from /lib64/libpthread.so.0
#6 0x00007ffff4a7835d in clone () from /lib64/libc.so.6
        
        start_device函数中调用rte_vhost_enable_guest_notification函数的代码如下

        从上图可以得出，rte_vhost_enable_guest_notification函数的第三个参数是0，rte_vhost_enable_guest_notification函数实现如下，当第三个参数enable为false，则设置feature VRING_USED_F_NO_NOTIFY

        start_device是spdk对virtio-blk后端设备初始化过程中调用的，当设备开始处理IO时，feature都已经设置完毕。
        至此，当virtio-blk处理IO时，就会执行到下面的代码中，virtqueue_kick操作不会通知后端，避免vm-exit，从而提升IO处理性能。