[ovs-dev] [PATCH] netdev-dpdk: Post-copy Live Migration support for vhost-user-client.
Maxime Coquelin
maxime.coquelin at redhat.com
Tue May 14 11:36:48 UTC 2019
On 5/14/19 12:58 PM, Ilya Maximets wrote:
> On 14.05.2019 10:44, Maxime Coquelin wrote:
>> On 5/8/19 3:54 PM, Ilya Maximets wrote:
>>> From: Liliia Butorina <l.butorina at partner.samsung.com>
>>>
>>> Post-copy Live Migration for vHost supported since DPDK 18.11 and
>>> QEMU 2.12. New global config option 'vhost-postcopy-support' added
>>> to control this feature. Ex.:
>>>
>>> ovs-vsctl set Open_vSwitch . other_config:vhost-postcopy-support=true
>>>
>>> Changing this value requires restarting the daemon. It's safe to
>>> enable this knob even if QEMU doesn't support post-copy LM.
>>>
>>> Feature marked as experimental and disabled by default because it may
>>> cause PMD thread hang on destination host on page fault for the time
>>> of page downloading from the source.
>>>
>>> Feature is not compatible with 'mlockall' and 'dequeue zero-copy'.
>>> Support added only for vhost-user-client.
>>>
>>> Signed-off-by: Liliia Butorina <l.butorina at partner.samsung.com>
>>> Co-authored-by: Ilya Maximets <i.maximets at samsung.com>
>>> Signed-off-by: Ilya Maximets <i.maximets at samsung.com>
>>
>> Thanks Ilya & Liliia for taking care of that.
>>
>>
>>> ---
>>> Documentation/topics/dpdk/vhost-user.rst | 53 +++++++++++++++++++++++-
>>> NEWS | 1 +
>>> lib/dpdk-stub.c | 6 +++
>>> lib/dpdk.c | 13 ++++++
>>> lib/dpdk.h | 1 +
>>> lib/netdev-dpdk.c | 5 +++
>>> vswitchd/vswitch.xml | 16 +++++++
>>> 7 files changed, 94 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/Documentation/topics/dpdk/vhost-user.rst b/Documentation/topics/dpdk/vhost-user.rst
>>> index 993797de5..6bec8c1f7 100644
>>> --- a/Documentation/topics/dpdk/vhost-user.rst
>>> +++ b/Documentation/topics/dpdk/vhost-user.rst
>>> @@ -111,7 +111,8 @@ the guest. There are two ways to do this: using QEMU directly, or using
>>> libvirt.
>>> .. note::
>>> - IOMMU is not supported with vhost-user ports.
>>> +
>>> + IOMMU and Post-copy Live Migration are not supported with vhost-user ports.
>>> Adding vhost-user ports to the guest (QEMU)
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> @@ -301,6 +302,52 @@ The default value is false.
>>> QEMU). Starting with QEMU v2.9.1, vhost-iommu-support can safely be
>>> enabled, even without having an IOMMU device, with no performance penalty.
>>> +vhost-user-client Post-copy Live Migration Support (experimental)
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +``Post-copy`` migration is the migration mode where the destination CPUs are
>>> +started before all the memory has been transferred. The main advantage is the
>>> +predictable migration time. Mostly used as a second phase after the normal
>>> +'pre-copy' migration in case it takes too long to converge.
>>> +
>>> +More information can be found in QEMU `docs`_.
>>> +
>>> +.. _`docs`: https://git.qemu.org/?p=qemu.git;a=blob;f=docs/devel/migration.rst
>>> +
>>> +Post-copy support may be enabled via a global config value
>>> +``vhost-postcopy-support``. Setting this to ``true`` enables Post-copy support
>>> +for all vhost-user-client ports::
>>> +
>>> + $ ovs-vsctl set Open_vSwitch . other_config:vhost-postcopy-support=true
>>> +
>>> +The default value is ``false``.
>>> +
>>> +.. important::
>>> +
>>> + Changing this value requires restarting the daemon.
>>> +
>>> +.. important::
>>> +
>>> + DPDK Post-copy migration mode uses userfaultfd syscall to communicate with
>>> + the kernel about page fault handling and uses shared memory based on huge
>>> + pages. So destination host linux kernel should support userfaultfd over
>>> + shared hugetlbfs. This feature only introduced in kernel upstream version
>>> + 4.11.
>>> +
>>> + Post-copy feature supported in DPDK since 18.11.0 version and in QEMU
>>> + since 2.12.0 version. But it's suggested to use QEMU >= 3.0.1 because
>>> + migration recovery was fixed for post-copy in 3.0 and few additional bug
>>> + fixes (like userfaulfd leak) was released in 3.0.1.
>>> +
>>> + DPDK Post-copy feature requires avoiding to populate the guest memory
>>> + (application must not call mlock* syscall). So enabling mlockall and
>>> + dequeue zero-copy features is mis-compatible with post-copy feature.
>>> +
>>> + Note that during migration of vhost-user device, PMD threads hang for the
>>> + time of faulted pages download from source host. Transferring 1GB hugepage
>>> + across a 10Gbps link possibly unacceptably slow. So recommended hugepage
>>> + size is 2MB.
>>> +
>>> .. _dpdk-testpmd:
>>> DPDK in the Guest
>>> @@ -500,6 +547,10 @@ QEMU versions v2.10 and greater). This value can be set like so::
>>> Because of this limitation, this feature is considered 'experimental'.
>>> +.. note::
>>> +
>>> + Post-copy Live Migration is not compatible with dequeue zero copy.
>>> +
>>> Further information can be found in the
>>> `DPDK documentation
>>> <https://doc.dpdk.org/guides-18.11/prog_guide/vhost_lib.html>`__
>>> diff --git a/NEWS b/NEWS
>>> index 293531db0..f1f6f074e 100644
>>> --- a/NEWS
>>> +++ b/NEWS
>>> @@ -3,6 +3,7 @@ Post-v2.11.0
>>> - DPDK:
>>> * New option 'other_config:dpdk-socket-limit' to limit amount of
>>> hugepage memory that can be used by DPDK.
>>> + * Add support for vHost Post-copy Live Migration (experimental).
>>> - OpenFlow:
>>> * Removed support for OpenFlow 1.6 (draft), which ONF abandoned.
>>> * New action "check_pkt_larger".
>>> diff --git a/lib/dpdk-stub.c b/lib/dpdk-stub.c
>>> index 1e0f46101..e55be5750 100644
>>> --- a/lib/dpdk-stub.c
>>> +++ b/lib/dpdk-stub.c
>>> @@ -56,6 +56,12 @@ dpdk_vhost_iommu_enabled(void)
>>> return false;
>>> }
>>> +bool
>>> +dpdk_vhost_postcopy_enabled(void)
>>> +{
>>> + return false;
>>> +}
>>> +
>>> bool
>>> dpdk_per_port_memory(void)
>>> {
>>> diff --git a/lib/dpdk.c b/lib/dpdk.c
>>> index dc6171546..d9ec3cf64 100644
>>> --- a/lib/dpdk.c
>>> +++ b/lib/dpdk.c
>>> @@ -47,6 +47,8 @@ static FILE *log_stream = NULL; /* Stream for DPDK log redirection */
>>> static char *vhost_sock_dir = NULL; /* Location of vhost-user sockets */
>>> static bool vhost_iommu_enabled = false; /* Status of vHost IOMMU support */
>>> +static bool vhost_postcopy_enabled = false; /* Status of vHost POSTCOPY
>>> + * support. */
>>> static bool dpdk_initialized = false; /* Indicates successful initialization
>>> * of DPDK. */
>>> static bool per_port_memory = false; /* Status of per port memory support */
>>> @@ -316,6 +318,11 @@ dpdk_init__(const struct smap *ovs_other_config)
>>> VLOG_INFO("Per port memory for DPDK devices %s.",
>>> per_port_memory ? "enabled" : "disabled");
>>> + vhost_postcopy_enabled = smap_get_bool(ovs_other_config,
>>> + "vhost-postcopy-support", false);
>>> + VLOG_INFO("POSTCOPY support for vhost-user-client %s.",
>>> + vhost_postcopy_enabled ? "enabled" : "disabled");
>>> +
>>> svec_add(&args, ovs_get_program_name());
>>> construct_dpdk_args(ovs_other_config, &args);
>>> @@ -492,6 +499,12 @@ dpdk_vhost_iommu_enabled(void)
>>> return vhost_iommu_enabled;
>>> }
>>> +bool
>>> +dpdk_vhost_postcopy_enabled(void)
>>> +{
>>> + return vhost_postcopy_enabled;
>>> +}
>>> +
>>> bool
>>> dpdk_per_port_memory(void)
>>> {
>>> diff --git a/lib/dpdk.h b/lib/dpdk.h
>>> index bbb89d4e6..7dab83775 100644
>>> --- a/lib/dpdk.h
>>> +++ b/lib/dpdk.h
>>> @@ -39,6 +39,7 @@ void dpdk_init(const struct smap *ovs_other_config);
>>> void dpdk_set_lcore_id(unsigned cpu);
>>> const char *dpdk_get_vhost_sock_dir(void);
>>> bool dpdk_vhost_iommu_enabled(void);
>>> +bool dpdk_vhost_postcopy_enabled(void);
>>> bool dpdk_per_port_memory(void);
>>> void print_dpdk_version(void);
>>> void dpdk_status(const struct ovsrec_open_vswitch *);
>>> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
>>> index 47153dc60..c06f46931 100644
>>> --- a/lib/netdev-dpdk.c
>>> +++ b/lib/netdev-dpdk.c
>>> @@ -4147,6 +4147,11 @@ netdev_dpdk_vhost_client_reconfigure(struct netdev *netdev)
>>> vhost_flags |= RTE_VHOST_USER_IOMMU_SUPPORT;
>>> }
>>> + /* Enable POSTCOPY support, if explicitly requested. */
>>> + if (dpdk_vhost_postcopy_enabled()) {
>>> + vhost_flags |= RTE_VHOST_USER_POSTCOPY_SUPPORT;
>>> + }
>>
>> Couldn't we also enforce postcopy disablement in the case --mlockall option is passed?
>
> Sure. This could be done on 'dpdk_init' stage. I'll take care of this.
>
> OTOH, this will block the case where source has mlock, but destination doesn't.
I would have thought source doesn't need to advertise
VHOST_USER_PROTOCOL_F_PAGEFAULT to have the postcopy migration to work.
I haven't tested it though, but in theory it shouldn't be needed.
> In theory, it's a valid case. Not sure how practical it is. What do you think?
Anyway, I don't think it would problematic in practice.
Thanks,
Maxime
> Best regards, Ilya Maximets.
>
More information about the dev
mailing list