[ovs-dev] [PATCH v2 3/3] netdev-dpdk: vHost client mode and reconnect
Daniele Di Proietto
diproiettod at ovn.org
Wed Aug 10 01:33:48 UTC 2016
2016-08-08 7:18 GMT-07:00 Loftus, Ciara <ciara.loftus at intel.com>:
> >
> > The patch mostly looks good to me, thanks.
> > I'm not 100% sure about the interface. Can we make the flag interface
> > specific?
>
> I'm not 100% sure about making the flag interface specific :) Do you think
> there's a use case for both client and server mode ports to be used in
> conjunction with each other?
>
Well, I don't have any specific use case in mind :-). I just think it's
cleaner making it per interface for two reasons:
* I'd like to provide the user with the maximum flexibility that the API
allows. I don't like adding artificial limitations, especially in user
interfaces, since those are supposed to be stable.
* The behavior of an interface depends on the status of the switch. It's
like having a global variable that influences the behavior of all the
functions.
>
> > If I'm not mistaken we currently limit vhost-sock-dir to be under OVS
> > rundir. With client mode this is not necessary anymore.
>
> Correct I've fixed this in the next version. Thanks.
>
> > I hope that client will be made the default mode at some point, I think
> we
> > should keep that in mind when considering the interface.
>
> I agree. I think we should wait until at least the QEMU v2.7.0 release
> though.
>
> > Since we're planning to break compatibility with the dpdk phy naming
> > change, maybe we can break compatibility also with vhost ports and add a
> > path option.
>
> Ok. So something like this?
>
> ovs-vsctl add-port vhost0
> ovs-vsctl set Interface vhost0 options:vhost-path=/tmp/v0.sock
Maybe we can rely on the presence of this attribute to discern between
client and server ports (I would call it vhost-server-path).
> Maybe something for a separate standalone patch?
>
If we're going with the per-interface configuration I think this should be
done at the same time.
Thanks,
Daniele
> Thanks,
> Ciara
>
> >
> > Thoughts?
> > Daniele
> >
> > 2016-08-04 7:09 GMT-07:00 Ciara Loftus <ciara.loftus at intel.com>:
> > A new other_config DB option has been added called 'vhost-driver-mode'.
> > By default this is set to 'server' which is the mode of operation OVS
> > with DPDK has used up until this point - whereby OVS creates and manages
> > vHost user sockets.
> >
> > If set to 'client', OVS will act as the vHost client and connect to
> > sockets created and managed by QEMU which acts as the server. This mode
> > allows for reconnect capability, which allows vHost ports to resume
> > normal connectivity in event of switch reset.
> >
> > QEMU v2.7.0+ is required when using OVS in client mode and QEMU in
> > server mode.
> >
> > Signed-off-by: Ciara Loftus <ciara.loftus at intel.com>
> > ---
> > v2
> > - Updated comments in vhost construct & destruct
> > - Add check for server-mode before printing error when destruct is called
> > on a running VM
> > - Fixed coding style/standards issues
> > - Use strcmp instead of strncmp when processing 'vhost-driver-mode'
> >
> > INSTALL.DPDK-ADVANCED.md | 27 +++++++++++++++++++++++++++
> > NEWS | 1 +
> > lib/netdev-dpdk.c | 31 +++++++++++++++++++++++--------
> > vswitchd/vswitch.xml | 13 +++++++++++++
> > 4 files changed, 64 insertions(+), 8 deletions(-)
> >
> > diff --git a/INSTALL.DPDK-ADVANCED.md b/INSTALL.DPDK-ADVANCED.md
> > index f9587b5..a773533 100755
> > --- a/INSTALL.DPDK-ADVANCED.md
> > +++ b/INSTALL.DPDK-ADVANCED.md
> > @@ -483,6 +483,33 @@ For users wanting to do packet forwarding using
> > kernel stack below are the steps
> > where `-L`: Changes the numbers of channels of the specified
> network
> > device
> > and `combined`: Changes the number of multi-purpose channels.
> >
> > + 4. Enable OVS vHost client-mode & vHost reconnect (OPTIONAL)
> > +
> > + By default, OVS DPDK acts as the vHost socket server and QEMU the
> > + client. In QEMU v2.7 the option is available for QEMU to act as
> the
> > + server. In order for this to work, OVS DPDK must be switched to
> 'client'
> > + mode. This is possible by setting the 'vhost-driver-mode' DB
> entry to
> > + 'client' like so:
> > +
> > + ```
> > + ovs-vsctl set Open_vSwitch . other_config:vhost-driver-
> mode="client"
> > + ```
> > +
> > + This must be done before the switch is launched. It cannot
> sucessfully
> > + be changed after switch has launched.
> > +
> > + One must also append ',server' to the 'chardev' arguments on the
> > QEMU
> > + command line, to instruct QEMU to use vHost server mode, like so:
> > +
> > + ````
> > + -chardev
> > socket,id=char0,path=/usr/local/var/run/openvswitch/vhost0,server
> > + ````
> > +
> > + One benefit of using this mode is the ability for vHost ports to
> > + 'reconnect' in event of the switch crashing or being brought
> down. Once
> > + it is brought back up, the vHost ports will reconnect
> automatically and
> > + normal service will resume.
> > +
> > - VM Configuration with libvirt
> >
> > * change the user/group, access control policty and restart
> libvirtd.
> > diff --git a/NEWS b/NEWS
> > index 9f09e1c..99412ba 100644
> > --- a/NEWS
> > +++ b/NEWS
> > @@ -70,6 +70,7 @@ Post-v2.5.0
> > fragmentation or NAT support yet)
> > * Support for DPDK 16.07
> > * Remove dpdkvhostcuse port type.
> > + * OVS client mode for vHost and vHost reconnect (Requires QEMU 2.7)
> > - Increase number of registers to 16.
> > - ovs-benchmark: This utility has been removed due to lack of use and
> > bitrot.
> > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
> > index 7692cc8..39c448b 100644
> > --- a/lib/netdev-dpdk.c
> > +++ b/lib/netdev-dpdk.c
> > @@ -136,7 +136,8 @@ BUILD_ASSERT_DECL((MAX_NB_MBUF /
> > ROUND_DOWN_POW2(MAX_NB_MBUF/MIN_NB_MBUF))
> > #define OVS_VHOST_QUEUE_DISABLED (-2) /* Queue was disabled by
> > guest and not
> > * yet mapped to another
> queue. */
> >
> > -static char *vhost_sock_dir = NULL; /* Location of vhost-user sockets
> */
> > +static char *vhost_sock_dir = NULL; /* Location of vhost-user
> sockets */
> > +static uint64_t vhost_driver_flags = 0; /* Denote whether client/server
> > mode */
> >
> > #define VHOST_ENQ_RETRY_NUM 8
> > #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ)
> > @@ -833,7 +834,6 @@ netdev_dpdk_vhost_user_construct(struct netdev
> > *netdev)
> > struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
> > const char *name = netdev->name;
> > int err;
> > - uint64_t flags = 0;
> >
> > /* 'name' is appended to 'vhost_sock_dir' and used to create a
> socket in
> > * the file system. '/' or '\' would traverse directories, so
> they're not
> > @@ -856,14 +856,17 @@ netdev_dpdk_vhost_user_construct(struct netdev
> > *netdev)
> > snprintf(dev->vhost_id, sizeof(dev->vhost_id), "%s/%s",
> > vhost_sock_dir, name);
> >
> > - err = rte_vhost_driver_register(dev->vhost_id, flags);
> > + err = rte_vhost_driver_register(dev->vhost_id, vhost_driver_flags);
> > if (err) {
> > VLOG_ERR("vhost-user socket device setup failure for socket
> %s\n",
> > dev->vhost_id);
> > } else {
> > - fatal_signal_add_file_to_unlink(dev->vhost_id);
> > - VLOG_INFO("Socket %s created for vhost-user port %s\n",
> > - dev->vhost_id, name);
> > + if (!(vhost_driver_flags & RTE_VHOST_USER_CLIENT)) {
> > + /* OVS server mode - add this socket to list for deletion */
> > + fatal_signal_add_file_to_unlink(dev->vhost_id);
> > + VLOG_INFO("Socket %s created for vhost-user port %s\n",
> > + dev->vhost_id, name);
> > + }
> > err = netdev_dpdk_init(netdev, -1, DPDK_DEV_VHOST);
> > }
> >
> > @@ -917,7 +920,8 @@ netdev_dpdk_vhost_destruct(struct netdev
> > *netdev)
> > struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
> >
> > /* Guest becomes an orphan if still attached. */
> > - if (netdev_dpdk_get_vid(dev) >= 0) {
> > + if (netdev_dpdk_get_vid(dev) >= 0
> > + && !(vhost_driver_flags & RTE_VHOST_USER_CLIENT)) {
> > VLOG_ERR("Removing port '%s' while vhost device still
> attached.",
> > netdev->name);
> > VLOG_ERR("To restore connectivity after re-adding of port, VM on
> > socket"
> > @@ -927,7 +931,8 @@ netdev_dpdk_vhost_destruct(struct netdev
> > *netdev)
> >
> > if (rte_vhost_driver_unregister(dev->vhost_id)) {
> > VLOG_ERR("Unable to remove vhost-user socket %s",
> dev->vhost_id);
> > - } else {
> > + } else if (!(vhost_driver_flags & RTE_VHOST_USER_CLIENT)) {
> > + /* OVS server mode - remove this socket from list for deletion
> */
> > fatal_signal_remove_file_to_unlink(dev->vhost_id);
> > }
> >
> > @@ -3181,6 +3186,7 @@ dpdk_init__(const struct smap *ovs_other_config)
> > int err = 0;
> > cpu_set_t cpuset;
> > char *sock_dir_subcomponent;
> > + const char *val;
> >
> > if (!smap_get_bool(ovs_other_config, "dpdk-init", false)) {
> > VLOG_INFO("DPDK Disabled - to change this requires a
> restart.\n");
> > @@ -3211,6 +3217,15 @@ dpdk_init__(const struct smap
> > *ovs_other_config)
> > } else {
> > vhost_sock_dir = sock_dir_subcomponent;
> > }
> > + val = smap_get(ovs_other_config, "vhost-driver-mode");
> > + if (!val || strcmp(val, "client")) {
> > + /* default case */
> > + vhost_driver_flags &= ~RTE_VHOST_USER_CLIENT;
> > + VLOG_INFO("OVS server mode (QEMU client mode) selected for
> > vHost");
> > + } else {
> > + vhost_driver_flags |= RTE_VHOST_USER_CLIENT;
> > + VLOG_INFO("OVS client mode (QEMU server mode) selected for
> > vHost");
> > + }
> >
> > argv = grow_argv(&argv, 0, 1);
> > argc = 1;
> > diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml
> > index 63f0d89..41807b3 100644
> > --- a/vswitchd/vswitch.xml
> > +++ b/vswitchd/vswitch.xml
> > @@ -299,6 +299,19 @@
> > </p>
> > </column>
> >
> > + <column name="other_config" key="vhost-driver-mode"
> > + type='{"type": "string", "enum": ["set", ["server",
> "client"]]}'>
> > + <p>
> > + Specifies which mode OVS will use for vHost. In 'server'
> mode, OVS
> > + creates and destroys the vHost User sockets. In 'client'
> mode, OVS
> > + attaches to sockets created by QEMU.
> > + </p>
> > + <p>
> > + Defaults to 'server' mode. Changing this value requires
> restarting
> > + the daemon.
> > + </p>
> > + </column>
> > +
> > <column name="other_config" key="n-handler-threads"
> > type='{"type": "integer", "minInteger": 1}'>
> > <p>
> > --
> > 2.4.3
> >
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > http://openvswitch.org/mailman/listinfo/dev
>
>
More information about the dev
mailing list