[ovs-discuss] ovs-vswitchd with crashes when adding a dpdk port

Daniel Badea daniel.badea at windriver.com
Fri Sep 26 13:37:55 UTC 2014


I finished writing the environment setup script following instructions from
https://raw.githubusercontent.com/openvswitch/ovs/master/INSTALL.DPDK and
the results are:
  - ovs still not building against dpdk head (version 1.7.1) because: const
struct rte_mbuf' has no member named 'pkt'
  - ovs builds against dpdk v1.7.0 
  - ovs-vswitchd starts and creates dpdk ports without crashing
  - able to run iperf through the dpdk port between the ovs with dpdk and
another Ubuntu machine where the NIC is handled by the kernel (ixgbe
module). Results: 550 Mbits/sec on a 10G link

So I was not able to reproduce the crash after following the new set of
instructions. I was probably repeatedly doing something wrong, but can't
pinpoint the problem. I thought it was an issue caused by the number of
cores or memory channels but it's not, I've tried with multiple combinations
and everything is ok. Sorry for the false alarm.

What is the expected release date for OVS 2.4? (I'm interested in the dpdk
vhost support)

Thanks,
Daniel

Here's the output of ovs-vsctl show after starting ovs-vswitchd:

ovs-vsctl show
#12f96467-8814-4814-a482-6fad87e8281b
#    Bridge "br0"
#        Port "br0"
#            Interface "br0"
#                type: internal
#        Port "dpdk0"
#            Interface "dpdk0"
#                type: dpdk
#        Port "dpdk1"
#            Interface "dpdk1"
#                type: dpdk

and here's the setup script I'm using to build dpdk and ovs, `a la DevStack:
  - creates a screen session and runs commands in parallel in multiple
windows
  - uses my specific NIC PCI configuration

#!/bin/bash

SELF=$(cd $( dirname ${BASH_SOURCE[0]} ) && pwd)/$(basename
${BASH_SOURCE[0]})
TMP=/tmp/run-ovs-dpdk
ROOT=/opt/ovs-dpdk
NIC_PCI="0000:47:00.0 0000:47:00.1"
HUGETLBFS_MOUNTPOINT=/mnt/huge

env_setup() {
    export DPDK_DIR=${ROOT}/dpdk
    export RTE_SDK=${DPDK_DIR}
    export RTE_TARGET=x86_64-native-linuxapp-gcc
    export DPDK_BUILD=${DPDK_DIR}/${RTE_TARGET}
    export OVS_DIR=${ROOT}/ovs
    export OVS_DB_SOCK=/usr/local/var/run/openvswitch/db.sock
}

setup() {
    screen -S dpdk -X quit
    rm -r ${TMP}
    mkdir -p ${TMP}
    screen -dmS dpdk
    screen -S dpdk -X screen
    screen -S dpdk -X screen
    screen -S dpdk -X screen
    screen -S dpdk -X caption always '%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%<'
    screen -S dpdk -p 0 -X title shell
    screen -S dpdk -p 1 -X title dpdk_build
    screen -S dpdk -p 1 -X stuff "${SELF} dpdk-build"
    screen -S dpdk -p 1 -X stuff $'\r'
    screen -S dpdk -p 2 -X title ovsdb-server
    screen -S dpdk -p 2 -X stuff "${SELF} ovsdb-server"
    screen -S dpdk -p 2 -X stuff $'\r'
    screen -S dpdk -p 3 -X title ovs-vswitchd
    screen -S dpdk -p 3 -X stuff "${SELF} ovs-vswitchd"
    screen -S dpdk -p 3 -X stuff $'\r'
}

dpdk_build() {
    rm ${TMP}/dpdk.*
    echo "Remove existing DPDK folder ${DPDK_DIR}"
    rm -rf ${DPDK_DIR}
    git clone git://dpdk.org/dpdk ${DPDK_DIR}
    cd ${DPDK_DIR}
    git checkout tags/v1.7.0
    patch -p1 <<EOF
diff --git a/config/common_linuxapp b/config/common_linuxapp
index 5bee910..4413620 100644
--- a/config/common_linuxapp
+++ b/config/common_linuxapp
@@ -81,7 +81,7 @@ CONFIG_RTE_BUILD_SHARED_LIB=n
 #
 # Combine to one single library
 #
-CONFIG_RTE_BUILD_COMBINE_LIBS=n
+CONFIG_RTE_BUILD_COMBINE_LIBS=y
 CONFIG_RTE_LIBNAME="intel_dpdk"
 
 #
diff --git a/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
b/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
index 5a06383..66ae116 100644
--- a/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
+++ b/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
@@ -3859,7 +3859,7 @@ static inline struct sk_buff
*__kc__vlan_hwaccel_put_tag(struct sk_buff *skb,
 #define HAVE_ENCAP_TSO_OFFLOAD
 #endif /* >= 3.10.0 */
 
-#if ( LINUX_VERSION_CODE < KERNEL_VERSION(3,14,0) )
+#if ( LINUX_VERSION_CODE < KERNEL_VERSION(3,13,0) )
 #if (!(RHEL_RELEASE_CODE && RHEL_RELEASE_CODE >=
RHEL_RELEASE_VERSION(7,0)))
 #if (!(UBUNTU_RELEASE_CODE == UBUNTU_RELEASE_VERSION(14,4) &&
UBUNTU_KERNEL_CODE >= UBUNTU_KERNEL_VERSION(3,13,0,3
 #ifdef NETIF_F_RXHASH
EOF
    make T=${RTE_TARGET} config
    make T=${RTE_TARGET} install -j 16 || return
    touch ${TMP}/dpdk.build
    eval $(sed 's/ /\n/g' /proc/cmdline | grep hugepage)
    if [[ ${hugepagesz} != "1G" ]] || (( hugepages < 1 )); then
        echo "Error: unsupported huge pages configuration"
        return
    fi
    rmmod igb_uio
    rmmod uio
    modprobe uio
    insmod ${DPDK_BUILD}/kmod/igb_uio.ko
    $DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio ${NIC_PCI}
    $DPDK_DIR/tools/dpdk_nic_bind.py --status
    umount ${HUGETLBFS_MOUNTPOINT}
    mkdir -p ${HUGETLBFS_MOUNTPOINT}
    mount -t hugetlbfs -o pagesize=1G none ${HUGETLBFS_MOUNTPOINT}
    touch ${TMP}/dpdk.ready
}

ovs_build() {
    rm ${TMP}/ovs.build
    pkill -9 ovsdb-server
    pkill -9 ovs-vswitchd
    echo "Remove existing OVS folder"
    rm -rf ${OVS_DIR}
    git clone https://github.com/openvswitch/ovs.git ${OVS_DIR}
    echo "wait for dpdk.build"
    while [[ ! -f ${TMP}/dpdk.build ]]; do
        sleep 1
    done
    cd ${OVS_DIR}
    ./boot.sh
    ./configure --with-dpdk=${DPDK_BUILD}
    make -j 16
    if [[ $? -eq 0 ]]; then
        touch ${TMP}/ovs.build
        return 0
    else
        return 1
    fi
}

ovsdb_server() {
    ovs_build || return
    rm ${TMP}/ovsdb-server.*
    pkill ovsdb-server
    sleep 1
    pkill -9 ovsdb-server
    mkdir -p /usr/local/etc/openvswitch
    mkdir -p /usr/local/var/run/openvswitch
    rm /usr/local/etc/openvswitch/conf.db
    cd ${OVS_DIR}
    ./ovsdb/ovsdb-tool create \
        /usr/local/etc/openvswitch/conf.db \
        ./vswitchd/vswitch.ovsschema
    ./ovsdb/ovsdb-server --remote=punix:${OVS_DB_SOCK} \
        --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
        --private-key=db:Open_vSwitch,SSL,private_key \
        --certificate=Open_vSwitch,SSL,certificate \
        --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
        --pidfile 2>&1 \
        | tee >(awk "(found==0 && /peak resident set size/){system(\"touch
${TMP}/ovsdb-server.running\");found=1;}") &
    while [[ ! -f ${TMP}/ovsdb-server.running ]]; do
        sleep 1
    done
    ./utilities/ovs-vsctl --no-wait init 
    touch ${TMP}/ovsdb-server.ready
    echo "ovsdb-server running in background"
}

ovs_vswitchd() {
    rm ${TMP}/ovs-vswitchd.*
    pkill ovs-vswitchd
    sleep 1
    pkill -9 ovs-vswitchd
    echo "wait for dpdk.ready"
    while [[ ! -f ${TMP}/dpdk.ready ]]; do
        sleep 1
    done
    while [[ ! -f ${TMP}/ovsdb-server.ready ]]; do
        sleep 1
    done
    cd ${OVS_DIR}
    rmmod openvswitch
    modprobe openvswitch
    ./vswitchd/ovs-vswitchd --dpdk -c 0x3 -n 4 -- unix:${OVS_DB_SOCK}
--pidfile 2>&1 \
        | tee >(awk "(found==0 && /ovs-vswitchd \(Open
vSwitch\)/){system(\"touch ${TMP}/ovs-vswitchd.running\");found=1;}") &
     while [[ ! -f ${TMP}/ovs-vswitchd.running ]]; do
         sleep 1
     done
    ./utilities/ovs-vsctl add-br br0
    ./utilities/ovs-vsctl set bridge br0 datapath_type=netdev
    ./utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0
type=dpdk
    ./utilities/ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1
type=dpdk
    ./utilities/ovs-vsctl show
     touch ${TMP}/ovs-vswitchd.ready
     echo "ovsdb-server running in background"
}

env_setup
if [[ $# -eq 0 ]]; then
    setup
    screen -x dpdk -p 1
else
    if [[ $1 =~ ^(dpdk-build|ovsdb-server|ovs-vswitchd)$ ]]; then
        cmd=${1/-/_}
        shift
        ${cmd} $*
    else
        echo "error: unknown command"
    fi
fi



-----Original Message-----
From: Daniel Badea [mailto:daniel.badea at windriver.com] 
Sent: Friday, September 26, 2014 1:40 PM
To: 'Daniele Di Proietto'
Cc: 'discuss at openvswitch.org'; 'Alex Wang'
Subject: RE: [ovs-discuss] ovs-vswitchd with crashes when adding a dpdk port

Here the output of building OVS master vs DPDK master (make -j 16).
I'm working on an environment setup script to build everything from scratch
in a reproducible way and will post it when it's done.

Thanks,
Daniel

cd dpdk
grep url .git/config
# url = git://dpdk.org/dpdk
git log -n1 --pretty=format:%H%d
# 23fcffe8ffaccf8a2901050e7daa4979597141ed (HEAD, origin/master,
origin/HEAD, master)

cd ../ovs
grep url .git/config
# url = https://github.com/openvswitch/ovs.git
git log -n1 --pretty=format:%H%d
# 260f76b7e3cf6bd52a2a80d5186f2a68b0123017 (HEAD, origin/master,
origin/HEAD, master)

...
lib/ofpbuf.h:363:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_data':
lib/ofpbuf.h:368:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h: In function 'ofpbuf_size':
lib/ofpbuf.h:383:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_size':
lib/ofpbuf.h:388:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h:389:12: error: 'struct rte_mbuf' has no member named 'pkt'
In file included from
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_ring.h:98:0,
                 from
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:74,
                 from
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:56,
                 from
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:178,
                 from lib/netdev-dpdk.h:13,
                 from lib/ofpbuf.h:25,
                 from lib/dpif.h:392,
                 from lib/dpif-provider.h:26,
                 from lib/dpif.c:18:
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_common.h: In
function 'rte_is_aligned':
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_common.h:176:9:
warning: cast from function call of type 'uintptr_t' to non-matching type
'void *' [-Wbad-function-cast] In file included from lib/dpif.h:392:0,
                 from lib/dpctl.c:33:
lib/ofpbuf.h: In function 'ofpbuf_data':
lib/ofpbuf.h:363:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_data':
lib/ofpbuf.h:368:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h: In function 'ofpbuf_size':
lib/ofpbuf.h:383:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_size':
lib/ofpbuf.h:388:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h:389:12: error: 'struct rte_mbuf' has no member named 'pkt'
In file included from lib/dpif.h:392:0,
                 from lib/dpif-provider.h:26,
                 from lib/dpif.c:18:
lib/ofpbuf.h: In function 'ofpbuf_data':
lib/ofpbuf.h:363:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_data':
lib/ofpbuf.h:368:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h: In function 'ofpbuf_size':
lib/ofpbuf.h:383:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_size':
lib/ofpbuf.h:388:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h:389:12: error: 'struct rte_mbuf' has no member named 'pkt'
In file included from lib/dpif-netdev.c:57:0:
lib/packet-dpif.h: In function 'dpif_packet_get_dp_hash':
lib/packet-dpif.h:52:26: error: 'struct rte_mbuf' has no member named 'pkt'
lib/packet-dpif.h: In function 'dpif_packet_set_dp_hash':
lib/packet-dpif.h:62:19: error: 'struct rte_mbuf' has no member named 'pkt'
lib/dpctl.c: In function 'ofpbuf_size':
lib/ofpbuf.h:384:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpctl.c: In function 'ofpbuf_data':
lib/ofpbuf.h:364:1: warning: control reaches end of non-void function
[-Wreturn-type]
make[2]: *** [lib/dpctl.lo] Error 1
In file included from lib/dpif.c:38:0:
lib/packet-dpif.h: In function 'dpif_packet_get_dp_hash':
lib/packet-dpif.h:52:26: error: 'struct rte_mbuf' has no member named 'pkt'
lib/packet-dpif.h: In function 'dpif_packet_set_dp_hash':
lib/packet-dpif.h:62:19: error: 'struct rte_mbuf' has no member named 'pkt'
lib/dpif-netdev.c: In function 'ofpbuf_size':
lib/ofpbuf.h:384:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpif-netdev.c: In function 'dpif_packet_get_dp_hash':
lib/packet-dpif.h:56:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpif-netdev.c: In function 'ofpbuf_data':
lib/ofpbuf.h:364:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpif.c: In function 'ofpbuf_size':
lib/ofpbuf.h:384:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpif.c: In function 'ofpbuf_data':
lib/ofpbuf.h:364:1: warning: control reaches end of non-void function
[-Wreturn-type]
make[2]: *** [lib/dpif.lo] Error 1
make[2]: *** [lib/dpif-netdev.lo] Error 1
make[2]: Leaving directory `/opt/ovs-dpdk/ovs'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/opt/ovs-dpdk/ovs'
make: *** [all] Error 2

-----Original Message-----
From: Daniele Di Proietto [mailto:ddiproietto at vmware.com]
Sent: Thursday, September 25, 2014 11:14 PM
To: Daniel Badea
Cc: discuss at openvswitch.org; Alex Wang
Subject: Re: [ovs-discuss] ovs-vswitchd with crashes when adding a dpdk port

Hi Daniel,

On 9/25/14, 10:34 AM, "Daniel Badea" <daniel.badea at windriver.com> wrote:

>I encountered the following issue: ovs-vswitchd with crashes when 
>adding a dpdk port.
> 
>Building ovs with dpdk support is not straightforward because:
>  - ovs doest not build with the latest dpdk version 1.7.1

We're regularly building with dpdk 1.7.1 (dpdk 1.7.0 is also supported).
Can you please tell us the error that you're experiencing, so that we can
try to fix it?

>  - the document here
>http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plai
>n;f =INSTALL.DPDK;hb=HEAD refers to dpdk 1.6.0r2 but that doesn't work 
>either.

That's an old version of the file. Please read INSTALL.DPDK in the source
tree (or
https://raw.githubusercontent.com/openvswitch/ovs/master/INSTALL.DPDK)

> 
>So I checked out dpdk 1.7.0 did some minor changes (combine libs and 
>kcompat fix for kernel 3.13.0) Now ovs builds ok but:
>  - if the database is empty then ovs-vswitchd starts but crashes on 
>ovs-vsctl add-port ... type=dpdk
>  - if the database already contains dpdk port configuration then 
>ovs-vswitchd crashes at startup
> 
>Logs seem to indicate a numa socket = -1 problem.
> 
>Any ideea how to fix his?

We've managed to reproduce the crash on a single node numa machine. Alex is
about to post a fix to the mailing list.


Thank you for the detailed report!

Daniele





More information about the discuss mailing list