[ovs-discuss] can not ping host in VM when using dpdkvhostuser
Mauricio Vasquez
mauricio.vasquez at polito.it
Fri Apr 22 13:26:02 UTC 2016
Hi Lui Chao,
There is a known bug in ovs v2.5.0 that prevents vhost user receiving
packets when a version of QEMU lower than 2.5.0 is used. (details at
http://openvswitch.org/pipermail/dev/2016-March/068282.html). It has
been fixed in the repository, then you should clone the ovs repo and use
the branch-2.5.
Mauricio Vasquez,
On 04/22/2016 03:14 PM, 刘超 wrote:
> Hi, I am following the
> guide: http://openvswitch.org/support/dist-docs-2.5/INSTALL.DPDK.md.html
> to deploy ovs + dpdk + qemu
>
> At last I find that, I can not ping br0 on host in the VM
>
> if I dump the packages with ovs-ofctl
>
> * port 3: rx pkts=127, bytes=5646, drop=?, errs=0, frame=?, over=?,
> crc=?*
> * tx pkts=0, bytes=0, drop=194, errs=?, coll=?*
>
> I found that vhost-user1 port, it can receive packages from VM, so rx
> is not zero.
>
> But it can not send packages to VM, so tx is zero, all packages are droped
>
> So did you guys face this problem? how to fix that?
>
> -----------------------detailed steps--------------------------
> 1. set the environment
>
> [root at localhost ~]# cat dpdkrc
> export DPDK_DIR=/root/dpdk-2.2.0
> export RTE_SDK=/root/dpdk-2.2.0
> export RTE_TARGET=x86_64-native-linuxapp-gcc
> export DESTDIR=/usr/local/bin/dpdk
> export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc
> export OVS_DIR=/root/openvswitch-2.5.0
>
> 2. build the dpdk
>
> cd to /root/dpdk-2.2.0
>
> |sed
> ||'s/CONFIG_RTE_BUILD_COMBINE_LIBS=n/CONFIG_RTE_BUILD_COMBINE_LIBS=y/'||-||i
> config||/||common_linuxapp|
> |
> |
> ||make install T||=||x86_64||-||native||-||linuxapp||-||gcc||
> ||
> ||
> 3. build the ovs
>
> cd $OVS_DIR
> ./boot.sh
> ./configure --with-dpdk=$DPDK_BUILD
> make
>
> 4. modify kernel paramters and reboot
>
> [root at localhost ~]# cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-4.0.4-301.fc22.x86_64 root=/dev/mapper/centos-root
> ro rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16
> rd.lvm.lv=centos/root crashkernel=auto vconsole.keymap=us rhgb quiet
> nouveau.modeset=0 rd.driver.blacklist=nouveau default_hugepagesz=1G
> hugepagesz=1G hugepages=8
>
> 5. install uio
>
> modprobe uio
>
> insmod $DPDK_BUILD/kmod/igb_uio.ko
>
> 6. bind the eth card
>
> [root at localhost ~]# dpdk_nic_bind.py --status
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:15:00.1 'I350 Gigabit Network Connection' drv=igb_uio unused=
> 0000:15:00.3 'I350 Gigabit Network Connection' drv=igb_uio unused=
>
> Network devices using kernel driver
> ===================================
> 0000:0b:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp11s0f0
> drv=bnx2 unused=igb_uio
> 0000:0b:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp11s0f1
> drv=bnx2 unused=igb_uio *Active*
> 0000:10:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp16s0f0
> drv=bnx2 unused=igb_uio
> 0000:10:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp16s0f1
> drv=bnx2 unused=igb_uio
> 0000:15:00.0 'I350 Gigabit Network Connection' if=ens2f0 drv=igb
> unused=igb_uio
> 0000:15:00.2 'I350 Gigabit Network Connection' if=ens2f2 drv=igb
> unused=igb_uio
> |
> |
> |7. mount hugepage|
> |
> |
> |mount -t hugetlbfs -o pagesize=1G none /dev/hugepages|
> |
> |
> |
> [root at localhost ~]# mount
> ...
> none on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=1G)
>
> 8. setup and run db
>
> $OVS_DIR/ovsdb/ovsdb-server
> --remote=punix:/usr/local/var/run/openvswitch/db.sock
> --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile
> --detach --log-file
> |
>
> 9. start ovs-vswitchd
>
> [root at localhost ~]# $OVS_DIR/vswitchd/ovs-vswitchd --dpdk -c 0x1 -n 4
> --socket-mem 1024,0
> 2016-04-22T13:03:04Z|00001|dpdk|INFO|No -vhost_sock_dir provided -
> defaulting to /usr/local/var/run/openvswitch
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 9 on socket 0
> EAL: Detected lcore 3 as core 10 on socket 0
> EAL: Detected lcore 4 as core 0 on socket 1
> EAL: Detected lcore 5 as core 1 on socket 1
> EAL: Detected lcore 6 as core 9 on socket 1
> EAL: Detected lcore 7 as core 10 on socket 1
> EAL: Detected lcore 8 as core 0 on socket 0
> EAL: Detected lcore 9 as core 1 on socket 0
> EAL: Detected lcore 10 as core 9 on socket 0
> EAL: Detected lcore 11 as core 10 on socket 0
> EAL: Detected lcore 12 as core 0 on socket 1
> EAL: Detected lcore 13 as core 1 on socket 1
> EAL: Detected lcore 14 as core 9 on socket 1
> EAL: Detected lcore 15 as core 10 on socket 1
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 16 lcore(s)
> EAL: VFIO modules not all loaded, skip VFIO support...
> EAL: Setting up physically contiguous memory...
> EAL: Ask a virtual area of 0x100000000 bytes
> EAL: Virtual area found at 0x7fa9c0000000 (size = 0x100000000)
> EAL: Ask a virtual area of 0x80000000 bytes
> EAL: Virtual area found at 0x7fa900000000 (size = 0x80000000)
> EAL: Ask a virtual area of 0x40000000 bytes
> EAL: Virtual area found at 0x7fa880000000 (size = 0x40000000)
> EAL: Requesting 1 pages of size 1024MB from socket 0
> EAL: TSC frequency is ~2400082 KHz
> EAL: Master lcore 0 is ready (tid=55128c00;cpuset=[0])
> EAL: PCI device 0000:15:00.0 on NUMA socket -1
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:15:00.1 on NUMA socket -1
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: PCI memory mapped at 0x7faa00000000
> EAL: PCI memory mapped at 0x7faa00100000
> PMD: eth_igb_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x1521
> EAL: PCI device 0000:15:00.2 on NUMA socket -1
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:15:00.3 on NUMA socket -1
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: PCI memory mapped at 0x7faa00104000
> EAL: PCI memory mapped at 0x7faa00204000
> PMD: eth_igb_dev_init(): port_id 1 vendorID=0x8086 deviceID=0x1521
> Zone 0: name:<RG_MP_log_history>, phys:0x17fffdec0, len:0x2080,
> virt:0x7fa9ffffdec0, socket_id:0, flags:0
> Zone 1: name:<MP_log_history>, phys:0x17fd73d40, len:0x28a0c0,
> virt:0x7fa9ffd73d40, socket_id:0, flags:0
> Zone 2: name:<rte_eth_dev_data>, phys:0x17fd43380, len:0x2f700,
> virt:0x7fa9ffd43380, socket_id:0, flags:0
> 2016-04-22T13:03:08Z|00002|ovs_numa|INFO|Discovered 8 CPU cores on
> NUMA node 0
> 2016-04-22T13:03:08Z|00003|ovs_numa|INFO|Discovered 8 CPU cores on
> NUMA node 1
> 2016-04-22T13:03:08Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and
> 16 CPU cores
> 2016-04-22T13:03:08Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
> connecting...
> 2016-04-22T13:03:08Z|00006|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
> connected
> 2016-04-22T13:03:08Z|00007|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath supports recirculation
> 2016-04-22T13:03:08Z|00008|ofproto_dpif|INFO|netdev at ovs-netdev: MPLS
> label stack length probed as 3
> 2016-04-22T13:03:08Z|00009|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath supports unique flow ids
> 2016-04-22T13:03:08Z|00010|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath does not support ct_state
> 2016-04-22T13:03:08Z|00011|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath does not support ct_zone
> 2016-04-22T13:03:08Z|00012|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath does not support ct_mark
> 2016-04-22T13:03:08Z|00013|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath does not support ct_label
> VHOST_CONFIG: socket created, fd:32
> VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/vhost-user2
> 2016-04-22T13:03:08Z|00014|dpdk|INFO|Socket
> /usr/local/var/run/openvswitch/vhost-user2 created for vhost-user port
> vhost-user2
> 2016-04-22T13:03:08Z|00015|dpif_netdev|INFO|Created 1 pmd threads on
> numa node 0
> 2016-04-22T13:03:08Z|00016|bridge|INFO|bridge br0: added interface
> vhost-user2 on port 1
> 2016-04-22T13:03:08Z|00017|bridge|INFO|bridge br0: added interface br0
> on port 65534
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da4940c0
> hw_ring=0x7fa9da49c100 dma_addr=0x15a49c100
> PMD: eth_igb_rx_queue_setup(): sw_ring=0x7fa9da47ff40
> hw_ring=0x7fa9da483f80 dma_addr=0x15a483f80
> 2016-04-22T13:03:08Z|00001|dpif_netdev(pmd12)|INFO|Core 0 processing
> port 'vhost-user2'
> PMD: eth_igb_start(): <<
> 2016-04-22T13:03:08Z|00018|dpdk|INFO|Port 0: a0:36:9f:a1:dc:f9
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da4940c0
> hw_ring=0x7fa9da49c100 dma_addr=0x15a49c100
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da467ec0
> hw_ring=0x7fa9da46ff00 dma_addr=0x15a46ff00
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da44fe40
> hw_ring=0x7fa9da457e80 dma_addr=0x15a457e80
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da437dc0
> hw_ring=0x7fa9da43fe00 dma_addr=0x15a43fe00
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da41fd40
> hw_ring=0x7fa9da427d80 dma_addr=0x15a427d80
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da407cc0
> hw_ring=0x7fa9da40fd00 dma_addr=0x15a40fd00
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da3efc40
> hw_ring=0x7fa9da3f7c80 dma_addr=0x15a3f7c80
> PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
> consider setting the TX WTHRESH value to 4, 8, or 16.
> PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da3d7bc0
> hw_ring=0x7fa9da3dfc00 dma_addr=0x15a3dfc00
> PMD: eth_igb_rx_queue_setup(): sw_ring=0x7fa9da47ff40
> hw_ring=0x7fa9da483f80 dma_addr=0x15a483f80
> PMD: eth_igb_start(): <<
> 2016-04-22T13:03:08Z|00019|dpdk|INFO|Port 0: a0:36:9f:a1:dc:f9
> 2016-04-22T13:03:08Z|00002|dpif_netdev(pmd12)|INFO|Core 0 processing
> port 'vhost-user2'
> 2016-04-22T13:03:08Z|00003|dpif_netdev(pmd12)|INFO|Core 0 processing
> port 'dpdk0'
> 2016-04-22T13:03:08Z|00020|bridge|INFO|bridge br0: added interface
> dpdk0 on port 2
> VHOST_CONFIG: socket created, fd:43
> VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/vhost-user1
> 2016-04-22T13:03:08Z|00021|dpdk|INFO|Socket
> /usr/local/var/run/openvswitch/vhost-user1 created for vhost-user port
> vhost-user1
> 2016-04-22T13:03:08Z|00004|dpif_netdev(pmd12)|INFO|Core 0 processing
> port 'vhost-user2'
> 2016-04-22T13:03:08Z|00005|dpif_netdev(pmd12)|INFO|Core 0 processing
> port 'dpdk0'
> 2016-04-22T13:03:08Z|00006|dpif_netdev(pmd12)|INFO|Core 0 processing
> port 'vhost-user1'
> 2016-04-22T13:03:08Z|00022|bridge|INFO|bridge br0: added interface
> vhost-user1 on port 3
> 2016-04-22T13:03:08Z|00023|bridge|INFO|bridge br0: using datapath ID
> 0000a0369fa1dcf9
> 2016-04-22T13:03:08Z|00024|connmgr|INFO|br0: added service controller
> "punix:/usr/local/var/run/openvswitch/br0.mgmt"
> 2016-04-22T13:03:08Z|00025|dpif_netdev|INFO|Created 1 pmd threads on
> numa node 0
> 2016-04-22T13:03:08Z|00026|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.5.0
> 2016-04-22T13:03:08Z|00001|dpif_netdev(pmd20)|INFO|Core 0 processing
> port 'vhost-user2'
> 2016-04-22T13:03:08Z|00002|dpif_netdev(pmd20)|INFO|Core 0 processing
> port 'dpdk0'
> 2016-04-22T13:03:08Z|00003|dpif_netdev(pmd20)|INFO|Core 0 processing
> port 'vhost-user1'
> 2016-04-22T13:03:15Z|00027|memory|INFO|17240 kB peak resident set size
> after 10.2 seconds
> 2016-04-22T13:03:15Z|00028|memory|INFO|handlers:5 ports:4
> revalidators:3 rules:5
>
> 10. create bridges and port
>
> $OVS_DIR/utilities/ovs-vsctl add-br br0 -- set bridge br0
> datapath_type=netdev
> $OVS_DIR/utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0
> type=dpdk
>
> ovs-vsctl add-port br0 vhost-user-2 -- set Interface vhost-user-2
> type=dpdkvhostuser
> ovs-vsctl add-port br0 vhost-user-2 -- set Interface vhost-user-2
> type=dpdkvhostuser
>
> 11. run qemu
>
> [root at localhost ~]# qemu-system-x86_64 -enable-kvm -name ubuntutest
> -smp 4 -cpu host -m 1024 -hda /home/kvm/dpdknode1.qcow2 -vnc :19 -net
> user,hostfwd=tcp::10022-:22 -net nic -chardev
> socket,id=char1,path=/usr/local/var/run/openvswitch/vhost-user1
> -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device
> virtio-net-pci,mac=52:54:00:00:00:01,netdev=mynet1 -object
> memory-backend-file,id=mem,size=1024M,mem-path=/dev/hugepages,share=on
> -numa node,memdev=mem -mem-prealloc
> qemu-system-x86_64: -netdev
> type=vhost-user,id=mynet1,chardev=char1,vhostforce: chardev "char1"
> went up
>
> [root at localhost ~]# qemu-system-x86_64 --version
> QEMU emulator version 2.4.1 (qemu-2.4.1-8.fc23), Copyright (c)
> 2003-2008 Fabrice Bellard
>
> 12. config IP of br0 and eth0 in VM
>
> br0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel state
> UNKNOWN qlen 500
> link/ether a0:36:9f:a1:dc:f9 brd ff:ff:ff:ff:ff:ff
> inet 192.168.201.2/24 brd 192.168.201.255 scope global br0
> valid_lft forever preferred_lft forever
> inet6 fe80::a236:9fff:fea1:dcf9/64 scope link
> valid_lft forever preferred_lft forever
>
> eth0 in VM is 192.168.201.3/24
>
> 13. ping br0 in VM
>
> I can not ping, do tcpdump on br0
>
> 21:11:00.564650 ARP, Request who-has 192.168.201.2 tell 192.168.201.3,
> length 28
> 21:11:00.564659 ARP, Reply 192.168.201.2 is-at a0:36:9f:a1:dc:f9,
> length 28
>
> br0 can get arp and response but VM dose not get the response
>
> 14. ovs-ofctl dump
>
> [root at localhost ~]# ovs-ofctl show br0
> OFPT_FEATURES_REPLY (xid=0x2): dpid:0000a0369fa1dcf9
> n_tables:254, n_buffers:256
> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
> actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan
> mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src
> mod_tp_dst
> 1(vhost-user2): addr:00:00:00:00:00:00
> config: PORT_DOWN
> state: LINK_DOWN
> speed: 0 Mbps now, 0 Mbps max
> 2(dpdk0): addr:a0:36:9f:a1:dc:f9
> config: 0
> state: 0
> current: 1GB-FD
> speed: 1000 Mbps now, 0 Mbps max
> 3(vhost-user1): addr:00:00:00:00:00:00
> config: PORT_DOWN
> state: LINK_DOWN
> speed: 0 Mbps now, 0 Mbps max
> LOCAL(br0): addr:a0:36:9f:a1:dc:f9
> config: 0
> state: 0
> current: 10MB-FD COPPER
> speed: 10 Mbps now, 0 Mbps max
> OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
> [root at localhost ~]# ovs-ofctl dump-ports br0
> OFPST_PORT reply (xid=0x2): 4 ports
> port 1: rx pkts=0, bytes=0, drop=?, errs=0, frame=?, over=?, crc=?
> tx pkts=0, bytes=0, drop=202, errs=?, coll=?
> port LOCAL: rx pkts=127, bytes=5646, drop=0, errs=0, frame=0,
> over=0, crc=0
> tx pkts=194, bytes=18000, drop=0, errs=0, coll=0
> port 2: rx pkts=67, bytes=12354, drop=0, errs=0, frame=?, over=?, crc=?
> tx pkts=135, bytes=8436, drop=0, errs=0, coll=?
> * port 3: rx pkts=127, bytes=5646, drop=?, errs=0, frame=?, over=?,
> crc=?*
> * tx pkts=0, bytes=0, drop=194, errs=?, coll=?*
>
> I found that vhost-user1 port, it can receive packages from VM, so rx
> is not zero.
>
> But it can not send packages to VM, so tx is zero, all packages are droped
>
> So did you guys face this problem? how to fix that?
>
> Thanks
>
> Yours
>
> Liu Chao
>
>
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20160422/360770d6/attachment-0002.html>
More information about the discuss
mailing list