[ovs-dev] [PATCH v2 3/3] netdev-dpdk: vHost client mode and reconnect

Loftus, Ciara ciara.loftus at intel.com
Thu Aug 11 16:29:40 UTC 2016


> 
> 
> 2016-08-08 7:18 GMT-07:00 Loftus, Ciara <ciara.loftus at intel.com>:
> >
> > The patch mostly looks good to me, thanks.
> > I'm not 100% sure about the interface.  Can we make the flag interface
> > specific?
> 
> I'm not 100% sure about making the flag interface specific :) Do you think
> there's a use case for both client and server mode ports to be used in
> conjunction with each other?
> 
> Well, I don't have any specific use case in mind :-).  I just think it's cleaner
> making it per interface for two reasons:
> * I'd like to provide the user with the maximum flexibility that the API
> allows.  I don't like adding artificial limitations, especially in user interfaces,
> since those are supposed to be stable.
> * The behavior of an interface depends on the status of the switch.  It's like
> having a global variable that influences the behavior of all the functions.
> 
> 
> > If I'm not mistaken we currently limit vhost-sock-dir to be under OVS
> > rundir.  With client mode this is not necessary anymore.
> 
> Correct I've fixed this in the next version. Thanks.
> 
> > I hope that client will be made the default mode at some point, I think we
> > should keep that in mind when considering the interface.
> 
> I agree. I think we should wait until at least the QEMU v2.7.0 release though.
> 
> > Since we're planning to break compatibility with the dpdk phy naming
> > change, maybe we can break compatibility also with vhost ports and add a
> > path option.
> 
> Ok. So something like this?
> 
> ovs-vsctl add-port vhost0
> ovs-vsctl set Interface vhost0 options:vhost-path=/tmp/v0.sock
> 
> Maybe we can rely on the presence of this attribute to discern between
> client and server ports (I would call it vhost-server-path).

I've added this in the next version. Do you think the user should be able to switch between modes? For the first pass I've only allowed the change from server -> client but not back the other way. Let me know what you think.

I also assumed that vhost-server-path should point to the directory in which the client socket (whose name is the name of the port) should be found. Let me know if this isn’t what you intended.

Thanks,
Ciara

> 
> Maybe something for a separate standalone patch?
> 
> If we're going with the per-interface configuration I think this should be done
> at the same time.
> Thanks,
> Daniele
> 
> Thanks,
> Ciara
> 
> >
> > Thoughts?
> > Daniele
> >
> > 2016-08-04 7:09 GMT-07:00 Ciara Loftus <ciara.loftus at intel.com>:
> > A new other_config DB option has been added called 'vhost-driver-mode'.
> > By default this is set to 'server' which is the mode of operation OVS
> > with DPDK has used up until this point - whereby OVS creates and manages
> > vHost user sockets.
> >
> > If set to 'client', OVS will act as the vHost client and connect to
> > sockets created and managed by QEMU which acts as the server. This
> mode
> > allows for reconnect capability, which allows vHost ports to resume
> > normal connectivity in event of switch reset.
> >
> > QEMU v2.7.0+ is required when using OVS in client mode and QEMU in
> > server mode.
> >
> > Signed-off-by: Ciara Loftus <ciara.loftus at intel.com>
> > ---
> > v2
> > - Updated comments in vhost construct & destruct
> > - Add check for server-mode before printing error when destruct is called
> >   on a running VM
> > - Fixed coding style/standards issues
> > - Use strcmp instead of strncmp when processing 'vhost-driver-mode'
> >
> >  INSTALL.DPDK-ADVANCED.md | 27 +++++++++++++++++++++++++++
> >  NEWS                     |  1 +
> >  lib/netdev-dpdk.c        | 31 +++++++++++++++++++++++--------
> >  vswitchd/vswitch.xml     | 13 +++++++++++++
> >  4 files changed, 64 insertions(+), 8 deletions(-)
> >
> > diff --git a/INSTALL.DPDK-ADVANCED.md b/INSTALL.DPDK-ADVANCED.md
> > index f9587b5..a773533 100755
> > --- a/INSTALL.DPDK-ADVANCED.md
> > +++ b/INSTALL.DPDK-ADVANCED.md
> > @@ -483,6 +483,33 @@ For users wanting to do packet forwarding using
> > kernel stack below are the steps
> >         where `-L`: Changes the numbers of channels of the specified network
> > device
> >         and `combined`: Changes the number of multi-purpose channels.
> >
> > +    4. Enable OVS vHost client-mode & vHost reconnect (OPTIONAL)
> > +
> > +       By default, OVS DPDK acts as the vHost socket server and QEMU the
> > +       client. In QEMU v2.7 the option is available for QEMU to act as the
> > +       server. In order for this to work, OVS DPDK must be switched to 'client'
> > +       mode. This is possible by setting the 'vhost-driver-mode' DB entry to
> > +       'client' like so:
> > +
> > +       ```
> > +       ovs-vsctl set Open_vSwitch . other_config:vhost-driver-mode="client"
> > +       ```
> > +
> > +       This must be done before the switch is launched. It cannot sucessfully
> > +       be changed after switch has launched.
> > +
> > +       One must also append ',server' to the 'chardev' arguments on the
> > QEMU
> > +       command line, to instruct QEMU to use vHost server mode, like so:
> > +
> > +       ````
> > +       -chardev
> > socket,id=char0,path=/usr/local/var/run/openvswitch/vhost0,server
> > +       ````
> > +
> > +       One benefit of using this mode is the ability for vHost ports to
> > +       'reconnect' in event of the switch crashing or being brought down.
> Once
> > +       it is brought back up, the vHost ports will reconnect automatically and
> > +       normal service will resume.
> > +
> >    - VM Configuration with libvirt
> >
> >      * change the user/group, access control policty and restart libvirtd.
> > diff --git a/NEWS b/NEWS
> > index 9f09e1c..99412ba 100644
> > --- a/NEWS
> > +++ b/NEWS
> > @@ -70,6 +70,7 @@ Post-v2.5.0
> >         fragmentation or NAT support yet)
> >       * Support for DPDK 16.07
> >       * Remove dpdkvhostcuse port type.
> > +     * OVS client mode for vHost and vHost reconnect (Requires QEMU 2.7)
> >     - Increase number of registers to 16.
> >     - ovs-benchmark: This utility has been removed due to lack of use and
> >       bitrot.
> > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
> > index 7692cc8..39c448b 100644
> > --- a/lib/netdev-dpdk.c
> > +++ b/lib/netdev-dpdk.c
> > @@ -136,7 +136,8 @@ BUILD_ASSERT_DECL((MAX_NB_MBUF /
> > ROUND_DOWN_POW2(MAX_NB_MBUF/MIN_NB_MBUF))
> >  #define OVS_VHOST_QUEUE_DISABLED    (-2) /* Queue was disabled by
> > guest and not
> >                                            * yet mapped to another queue. */
> >
> > -static char *vhost_sock_dir = NULL;   /* Location of vhost-user sockets */
> > +static char *vhost_sock_dir = NULL;     /* Location of vhost-user sockets */
> > +static uint64_t vhost_driver_flags = 0; /* Denote whether client/server
> > mode */
> >
> >  #define VHOST_ENQ_RETRY_NUM 8
> >  #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ)
> > @@ -833,7 +834,6 @@ netdev_dpdk_vhost_user_construct(struct netdev
> > *netdev)
> >      struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
> >      const char *name = netdev->name;
> >      int err;
> > -    uint64_t flags = 0;
> >
> >      /* 'name' is appended to 'vhost_sock_dir' and used to create a socket in
> >       * the file system. '/' or '\' would traverse directories, so they're not
> > @@ -856,14 +856,17 @@ netdev_dpdk_vhost_user_construct(struct
> netdev
> > *netdev)
> >      snprintf(dev->vhost_id, sizeof(dev->vhost_id), "%s/%s",
> >               vhost_sock_dir, name);
> >
> > -    err = rte_vhost_driver_register(dev->vhost_id, flags);
> > +    err = rte_vhost_driver_register(dev->vhost_id, vhost_driver_flags);
> >      if (err) {
> >          VLOG_ERR("vhost-user socket device setup failure for socket %s\n",
> >                   dev->vhost_id);
> >      } else {
> > -        fatal_signal_add_file_to_unlink(dev->vhost_id);
> > -        VLOG_INFO("Socket %s created for vhost-user port %s\n",
> > -                  dev->vhost_id, name);
> > +        if (!(vhost_driver_flags & RTE_VHOST_USER_CLIENT)) {
> > +            /* OVS server mode - add this socket to list for deletion */
> > +            fatal_signal_add_file_to_unlink(dev->vhost_id);
> > +            VLOG_INFO("Socket %s created for vhost-user port %s\n",
> > +                      dev->vhost_id, name);
> > +        }
> >          err = netdev_dpdk_init(netdev, -1, DPDK_DEV_VHOST);
> >      }
> >
> > @@ -917,7 +920,8 @@ netdev_dpdk_vhost_destruct(struct netdev
> > *netdev)
> >      struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
> >
> >      /* Guest becomes an orphan if still attached. */
> > -    if (netdev_dpdk_get_vid(dev) >= 0) {
> > +    if (netdev_dpdk_get_vid(dev) >= 0
> > +        && !(vhost_driver_flags & RTE_VHOST_USER_CLIENT)) {
> >          VLOG_ERR("Removing port '%s' while vhost device still attached.",
> >                   netdev->name);
> >          VLOG_ERR("To restore connectivity after re-adding of port, VM on
> > socket"
> > @@ -927,7 +931,8 @@ netdev_dpdk_vhost_destruct(struct netdev
> > *netdev)
> >
> >      if (rte_vhost_driver_unregister(dev->vhost_id)) {
> >          VLOG_ERR("Unable to remove vhost-user socket %s", dev->vhost_id);
> > -    } else {
> > +    } else if (!(vhost_driver_flags & RTE_VHOST_USER_CLIENT)) {
> > +        /* OVS server mode - remove this socket from list for deletion */
> >          fatal_signal_remove_file_to_unlink(dev->vhost_id);
> >      }
> >
> > @@ -3181,6 +3186,7 @@ dpdk_init__(const struct smap
> *ovs_other_config)
> >      int err = 0;
> >      cpu_set_t cpuset;
> >      char *sock_dir_subcomponent;
> > +    const char *val;
> >
> >      if (!smap_get_bool(ovs_other_config, "dpdk-init", false)) {
> >          VLOG_INFO("DPDK Disabled - to change this requires a restart.\n");
> > @@ -3211,6 +3217,15 @@ dpdk_init__(const struct smap
> > *ovs_other_config)
> >      } else {
> >          vhost_sock_dir = sock_dir_subcomponent;
> >      }
> > +    val = smap_get(ovs_other_config, "vhost-driver-mode");
> > +    if (!val || strcmp(val, "client")) {
> > +        /* default case */
> > +        vhost_driver_flags &= ~RTE_VHOST_USER_CLIENT;
> > +        VLOG_INFO("OVS server mode (QEMU client mode) selected for
> > vHost");
> > +    } else  {
> > +        vhost_driver_flags |= RTE_VHOST_USER_CLIENT;
> > +        VLOG_INFO("OVS client mode (QEMU server mode) selected for
> > vHost");
> > +    }
> >
> >      argv = grow_argv(&argv, 0, 1);
> >      argc = 1;
> > diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml
> > index 63f0d89..41807b3 100644
> > --- a/vswitchd/vswitch.xml
> > +++ b/vswitchd/vswitch.xml
> > @@ -299,6 +299,19 @@
> >          </p>
> >        </column>
> >
> > +      <column name="other_config" key="vhost-driver-mode"
> > +              type='{"type": "string", "enum": ["set", ["server", "client"]]}'>
> > +        <p>
> > +          Specifies which mode OVS will use for vHost. In 'server' mode, OVS
> > +          creates and destroys the vHost User sockets. In 'client' mode, OVS
> > +          attaches to sockets created by QEMU.
> > +        </p>
> > +        <p>
> > +          Defaults to 'server' mode. Changing this value requires restarting
> > +          the daemon.
> > +        </p>
> > +      </column>
> > +
> >        <column name="other_config" key="n-handler-threads"
> >                type='{"type": "integer", "minInteger": 1}'>
> >          <p>
> > --
> > 2.4.3
> >
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > http://openvswitch.org/mailman/listinfo/dev



More information about the dev mailing list