[ovs-discuss] Phy-VM over OVS-DPDK no connectivity

[ppnaik]

Hi All,

Our setup is as follows:

We have two servers which are connected peer to peer over 40G 
interfaces.

On one server we have setup OVS and added the physical 40G interface as 
a DPDK interface to the ovs bridge.

We created another dpdkvhostuser interface for the VM. We added this 
interface to the VM (by editing the XML). We are able to see this 
interface inside the VM and have configure IP to the interface.

We want to communicate between the other server and VM inside this 
server through the OVS interface created for the VM.

The steps we followed (on the server with OVS) are:

modprobe uio

cd /usr/src/dpdk-18.11/x86_64-native-linuxapp-gcc/kmod/

insmod igb_uio.ko

cd /usr/src/dpdk-18.11/usertools/
./dpdk-devbind.py --bind=igb_uio 0000:81:00.1

  export PATH=$PATH:/usr/local/share/openvswitch/scripts
  export DB_SOCK=/usr/local/var/run/openvswitch/db.sock

  ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock 
--remote=db:Open_vSwitch,Open_vSwitch,manager_options 
--private-key=db:Open_vSwitch,SSL,private_key 
--certificate=db:Open_vSwitch,SSL,certificate 
--bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach 
--log-file

  ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
  ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start

ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk 
options:dpdk-devargs=0000:81:00.1
ovs-vsctl add-port br0 dpdkvhostuser0     -- set Interface 
dpdkvhostuser0 type=dpdkvhostuser ofport_request=3

ovs-ofctl add-flow br0 in_port=1,action=output:3
ovs-ofctl add-flow br0 in_port=3,action=output:1

echo 'vm.nr_hugepages=2048' > /etc/sysctl.d/hugepages.conf
grep HugePages_ /proc/meminfo

edit VM XML to add this interface:

first line:
<domain type='kvm' 
xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

add before </domain> tag:

<qemu:commandline>
     <qemu:arg value='-chardev'/>
     <qemu:arg 
value='socket,id=char1,path=/usr/local/var/run/openvswitch/dpdkvhostuser0'/>
     <qemu:arg value='-netdev'/>
     <qemu:arg 
value='vhost-user,id=mynet1,chardev=char1,vhostforce=on,queues=1'/>
     <qemu:arg value='-device'/>
     <qemu:arg 
value='virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet1,mq=on,vectors=4'/>
     <qemu:arg value='-m'/>
     <qemu:arg value='4096'/>
     <qemu:arg value='-object'/>
     <qemu:arg 
value='memory-backend-file,id=mem1,size=4096M,mem-path=/dev/hugepages,share=on'/>
     <qemu:arg value='-mem-prealloc'/>
     <qemu:arg value='-numa'/>
     <qemu:arg value='node,memdev=mem1'/>
   </qemu:commandline>

Please help us resolve this issue. I assumed ping would work between 
the other server and the VM. But it is not working in our case. Also, 
let us know if we are missing some setup step or if there is some 
misconfiguration. If ping would not work can you let us know a way to 
verify the connectivity?

Thanks,
Priyanka