[ovs-discuss] vhost communication between virtio drivers in vms

[David Christensen]

> I'm trying to achieve communication across an OVS switch between to VMs 
> connected together via an ovs bridge.  I was able to do this with a ovs 
> bridge with to generic libvirt 'network' types.  I'm failing to 
> replicate using DPDK vhost.
> 
> In libvirt 'network' scenario vnet[0,1] ports are added to the bridge 
> that also has a port for the physical NIC on the system.  The switch 
> defaults to its L2 function and I can access the system's IP addresses.
> 
> In the DPDK vhost scenario, the client server connections get made, but 
> the host running the switch doesn't put the ports on the IP stack.  I 
> haven't been able to route a packet between the VMs.  Do I need to 
> create "flows" to mimic a single wire type of connection?  I don't need 
> L2 function, but I did think it might be possible to move IP packets 
> through DPDK.  Any pointers appreciated.
> 
> 
> [root at cn03 openvswitch]# ovs-vsctl show
> d78a22bb-84c7-46d4-9625-6c9c38aeeff7
>      Bridge br0
>          Port br0
>              Interface br0
>                  type: internal
>          Port vnet1
>              Interface vnet1
>          Port vnet0
>              Interface vnet0
>          Port enP5p1s0f0
>              Interface enP5p1s0f0
>      Bridge ovs-br0
>          datapath_type: netdev
>          Port dpdkvhostclient0
>              Interface dpdkvhostclient0
>                  type: dpdkvhostuserclient
>                  options: 
> {vhost-server-path="/usr/local/var/run/openvswitch/dpdkvhostclient0"}
>          Port dpdkvhostclient1
>              Interface dpdkvhostclient1
>                  type: dpdkvhostuserclient
>                  options: 
> {vhost-server-path="/usr/local/var/run/openvswitch/dpdkvhostclient1"}
>          Port ovs-br0
>              Interface ovs-br0
>                  type: internal
>      ovs_version: "2.13.1"

Yes, you definitely need to setup flow rules once you connect the VMs 
with OVS/DPDK.  Here's the script I use to setup a learning switch which 
should get you base functionality you're looking for:

#!/bin/bash
PATH=/usr/share/openvswitch/scripts:$PATH
MAX_PORT=2

# Make sure we're running as root
if [[ $EUID -ne 0 ]]; then
    echo "This script must be run as root"
    exit 1
fi

# Check command line arguments for the bridge name
BRIDGE=$1
[ -z "$BRIDGE" ] && BRIDGE=ovsbr0

# Verify that OVS is running and the bridge exists
ovs-vsctl br-exists $BRIDGE > /dev/null 2>&1
RETVAL=$?
if [ $RETVAL -ne 0 ]; then
	echo "Can't find OVS bridge $BRIDGE, exiting ..."
	exit 1
fi

# For example, "0-3,7" becomes "0,1,2,3,7".
# Requires two arguments:
# $1 - The string to be expanded
# $2 - The list of CPUs being returned
function parse_range {
         arr=()
   IFS=', ' read -a ranges <<< $1
   for range in "${ranges[@]}"; do
     IFS=- read start end <<< "$range"
     [ -z "$start" ] && continue
     [ -z "$end" ] && end=$start
     for ((i = start; i <= end; i++)); do
                         arr+=($i)
     done
   done
   eval $2=$(IFS=,;printf "%s" "${arr[*]}")
}

# Set the OFPORTS flows for the learning switch
PORT_RANGE="1-${MAX_PORT}"
PORT_LIST=""
parse_range $PORT_RANGE PORT_LIST
echo "Configuring learning switch for ports $PORT_RANGE ..."

#######################################################################
# Bridge is setup, program the flow configuration
#######################################################################
echo "Clearing current flow setup for $BRIDGE..."
ovs-ofctl del-flows $BRIDGE

echo "Configuring flows on $BRIDGE for learning switch..."

#=======================================================================
# Setup table0, admission control
#=======================================================================

# Drop packets with a multicast source address (illegal packet)
ovs-ofctl add-flow $BRIDGE \
     "table=0, dl_src=01:00:00:00:00:00/01:00:00:00:00:00, actions=drop"
# Drop spanning tree packets
ovs-ofctl add-flow $BRIDGE \
     "table=0, dl_dst=01:80:c2:00:00:00/ff:ff:ff:ff:ff:f0, actions=drop"
# Pass all other frames onto table1
ovs-ofctl add-flow $BRIDGE "table=0, priority=0, actions=resubmit(,1)"

#=======================================================================
# Setup table1, MAC learning
#=======================================================================
ovs-ofctl add-flow $BRIDGE \
     "table=1 actions=learn(table=10, NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[], \
                            load:NXM_OF_IN_PORT[]->NXM_NX_REG0[0..15]), \
                      resubmit(,2)"

#=======================================================================
# Setup table2, destination lookup
#=======================================================================

# Always forward multicast/broadcast packets
ovs-ofctl add-flow $BRIDGE \
     "table=2 priority=99 dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 \
       actions=resubmit(,3)"

# Lookup the destination in table 10, move the packet on to table 3
ovs-ofctl add-flow $BRIDGE \
     "table=2 priority=50 actions=resubmit(,10), resubmit(,3)"

#=======================================================================
# Setup table3, output processing
#=======================================================================
# Handle broadcast/multicast packets (OpenFlow will prevent
# the packet from being re-transmitted out the ingress port)
ovs-ofctl add-flow $BRIDGE "table=3 reg0=0 actions=${BRIDGE},${PORT_LIST}"

# Setup the unicast flows for the vhost interfaces
for i in $(seq 1 $MAX_PORT); do
	ovs-ofctl add-flow $BRIDGE "table=3 reg0=$i actions=$i"
done

# Don't forget the bridge's tap interface to the host
ovs-ofctl add-flow $BRIDGE "table=3 reg0=65534 actions=local"

echo "New flow configuration for $BRIDGE..."
ovs-ofctl dump-flows $BRIDGE