[ovs-dev] [PATCH 2/2] netdev-dpdk: add support for vhost IOMMU feature

[Mooney, Sean K]

> -----Original Message-----
> From: Jan Scheurich [mailto:jan.scheurich at ericsson.com]
> Sent: Monday, November 27, 2017 2:15 PM
> To: Kavanagh, Mark B <mark.b.kavanagh at intel.com>; Mooney, Sean K
> <sean.k.mooney at intel.com>; Kevin Traynor <ktraynor at redhat.com>;
> dev at openvswitch.org
> Cc: maxime.coquelin at redhat.com; Flavio Leitner <fbl at redhat.com>; Franck
> Baudin <fbaudin at redhat.com>; Ilya Maximets <i.maximets at samsung.com>;
> Stokes, Ian <ian.stokes at intel.com>; Loftus, Ciara
> <ciara.loftus at intel.com>; Darrell Ball <dball at vmware.com>; Aaron Conole
> <aconole at redhat.com>
> Subject: RE: [ovs-dev] [PATCH 2/2] netdev-dpdk: add support for vhost
> IOMMU feature
> 
> > >> >> Hi Mark, All,
> > >> >>
> > >> >> I'm thinking about this and whether the current approach
> > >> >> provides more than what is actually needed by users at the cost
> > >> >> of making the user interface more complex.
> > >[Mooney, Sean K] I am personally split on this. To enable iommu
> > >support in openstack with the above interface we would have to do
> two
> > >things. 1 extend the image metatdata or flavor extra specs to allow
> > >requesting a vIOMMU. Second we would have to modify os-vif to
> produce
> > >the add-port command above. Using this interfaces gives us a nice
> > >parity in our api as we only enable iommu support in ovs if we
> enable
> > >for qemu. E.g. if the falvor/image does not request a virtualized
> > >iommu we wont declare its support in the options.
> > >> >>
> > >> >> As an alternative, how about having a global other_config (to
> be
> > >> >> set like vhost-socket-dir can be) for this instead of having to
> > >> >> set it for each individual interface. It would mean that it
> > >> >> would only have to be set once, instead of having this (ugly?!)
> > >> >> option every time a vhost port is added, so it's a less
> > >> >> intrusive change and I can't really think that a user would
> > >> >> require to do this per vhostclient
> > >[Mooney, Sean K]  well one taught that instantly comes to mind is if
> > >I set The global other_config option what happens if I do not
> request
> > >a iommu in qemu Or I have an old qemu.  If it would have any
> negative
> > >effect on network connectivity Or prevent the vm from functioning,
> > >that would require the nova scheduler to be Aware of which node had
> > >this option set and take that into account when placing vms.
> > >I assume if it had no negative effects  then we would not need a
> > >option, global or per Interface.
> > >> >> interface??? It's pain to have to add this at all for a bug in
> > >> >> QEMU and I'm sure in 1/2/3 years time someone will say that
> > >> >> users could still be using QEMU < 2.9.1 and we can't remove it,
> > >> >> so it would be nice to keep it as discreet as possible as we're
> > >> >> going to be stuck
> > >> with it for a while.
> > >> >>
> > >> >> I assume that a user would only use one version of QEMU at a
> > >> >> time
> > >> and
> > >> >> would either want or not want this feature. In the worst case,
> > >> >> if that proved completely wrong in the future, then a per
> > >> >> interface override could easily be added. Once there's a way to
> > >> >> maintain backwards compatibility (which there would be) I'd
> > >> >> rather err on the side of introducing just enough enough
> > >> >> functionality over increasing complexity for the user.
> > >> >>
> > >> >> What do you think?
> > >[Mooney, Sean K] I'm not sure that a single qemu version is a valid
> > >assumption to make.
> > >At least in an OpenStack context where rolling upgrades are a thing.
> > >But you are right That it will be less common however if it was no
> > >existent we would not have the issue with Live migration between
> > >nodes with different feature sets that is also trying to be
> addressed
> > >this Cycle. If we add a global config option for iommu support that
> > >is yet another thing that needs To be accounted for during live
> > >migration.
> > >> >>
> >
> > Hi Kevin, Jan,
> >
> > Any thoughts on Sean's concerns?
> >
> > I'll hold off on implementation until we have consensus.
> >
> > Thanks,
> > Mark
> 
> Here are my 2cts:
> 
> As I see it, vIOMMU for vhostuser is a pure infrastructure feature
> negotiated between guest driver,  Qemu and OVS. If both Qemu and OVS
> support it and the driver requests it, it should be used, otherwise
> not.
> 
> My understanding is that the vhostuser library in DPDK 17.11 supports
> vIOMMU, such that OVS could always signal support for this feature. The
> only reason not to do so is to work around the problem that Qemu claims
> vIOMMU support in certain releases but the vhostuser backend
> implementation is broken (prior to 2.9.1). For these cases it should be
> sufficient to globally disable vIOMMU support in OVS.
> 
> I don't see the need for one OVS instance to interwork with multiple
> different Qemu versions on the same host. And even if that were the
> case in an upgrade scenario, one could keep vIOMMU disabled until the
> old Qemu is removed.
> 
> The specific enabling of vIOMMU per tenant VM port is done by supplying
> an iommu device to the VM in the Libvirt XML and enabling iommu for the
> device in the driver element of the interface section. These are the
> places that Nova/OpenStack would use to control the vIOMMU per VM based
> on image properties or flavor extra specs. I do not see any value to
> additionally configure the iommu support per vhostuser port through OS-
> VIF.
> 
> Conclusion: A global other_config parameter to enable/disable vhostuser
> IOMMU is sufficient. By default this could be OFF for now and changed
> to ON when broken Qemu versions are largely gone.
[Mooney, Sean K] hi yes I just responded to this before I saw your reply.
A global option will be fine if we can confirm that enabling the iommu feature negotiation
In ovs will not impact vms that do not have a virtualized iommu enabled in the Libvirt/ qemu commandline.
I would personally consider it a driver bug if ovs advertised support for using a virtualized iommu and
The driver in the guest requested it with our have an actual iommu present in the vm.
> 
> BR, Jan