[ovs-dev] Issues configuring OVS-DPDK in openstack queens

Manojawa Paritala manojawapv at biarca.com
Thu Oct 25 09:32:47 UTC 2018


Hi Billy,

The issue of br-flat1 getting deleted is fixed now with the key-value pair
of "datapath_types = netdev,system" in OVS neutron-openvswitch-agent.ini in
the compute node.

Earlier, we used the key-value pair as "datapath_type = netdev". Please
note the new key name we used is "datapath_types" and in the the value we
mentioned "netdev,system".

So, now our updated OVS section in neutron-openvswitch-agent.ini is as
below.

[OVS]
bridge_mappings =
flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0,vlan2:br1
local_ip = 192.168.5.14
datapath_types = netdev,system


One issue that we see now is, when we add "vlan1:br0,vlan2:br1" to
bridge_mappings and restart the neutron-openvswitch-agent.ini and
openvsiwtch-agent services, in the "ovs-vsctl show" output, the ports in
the bridges br0 & br1 have the below error. This issue is getting resolved,
if I change the datapath_type of those bridges to netdev.

Bridge "br1"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "phy-br1"
            Interface "phy-br1"
                type: patch
                options: {peer="int-br1"}
        Port "vhost-user-5"
            Interface "vhost-user-5"
                type: dpdkvhostuser
                error: "could not add network device vhost-user-5 to
ofproto (Invalid argument)"
        Port "port1"
            Interface "port1"
                type: dpdk
                options: {dpdk-devargs="0000:b1:00.1"}
                error: "Error attaching device '0000:b1:00.1' to DPDK"
        Port "vhost-user-6"
            Interface "vhost-user-6"
                type: dpdkvhostuser
                error: "could not add network device vhost-user-6 to
ofproto (Invalid argument)"
        Port "br1"
            Interface "br1"
                type: internal
    Bridge "br0"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "vhost-user-1"
            Interface "vhost-user-1"
                type: dpdkvhostuser
                error: "could not add network device vhost-user-1 to
ofproto (Invalid argument)"
        Port "br0"
            Interface "br0"
                type: internal
        Port "port0"
            Interface "port0"
                type: dpdk
                options: {dpdk-devargs="0000:af:00.1"}
                error: "could not add network device port0 to ofproto
(Invalid argument)"
        Port "phy-br0"
            Interface "phy-br0"
                type: patch
                options: {peer="int-br0"}
        Port "vhost-user-3"
            Interface "vhost-user-3"
                type: dpdkvhostuser
                error: "could not add network device vhost-user-3 to
ofproto (Invalid argument)"
        Port "vhost-user-2"
            Interface "vhost-user-2"
                type: dpdkvhostuser
                error: "could not add network device vhost-user-2 to
ofproto (Invalid argument)"
        Port "vhost-user-4"
            Interface "vhost-user-4"
                type: dpdkvhostuser
                error: "could not add network device vhost-user-4 to
ofproto (Invalid argument)"


Thanks & Regards,
PVMJ

On Wed, Oct 24, 2018 at 7:47 PM Manojawa Paritala <manojawapv at biarca.com>
wrote:

> Hi Billy,
>
> The reason for br-vlan getting deleted in my previous attempt was due to a
> small misconfiguration in the openstack-agent.ini. I corrected it and
> re-done the test. Attaching the new logs with this email. Please ignore the
> earlier logs.
>
> Apologies for the confusion.
>
> Thanks & Regards,
> PVMJ
>
> On Wed, Oct 24, 2018 at 7:26 PM Manojawa Paritala <manojawapv at biarca.com>
> wrote:
>
>> Hi Billy,
>>
>> Firstly, thank you for taking a look at the logs I sent yesterday.
>>
>> Please see in-line. Please let  me know if I miss anything.
>>
>> Thanks & Regards,
>> PVMJ
>>
>> On Wed, Oct 24, 2018 at 3:36 PM O Mahony, Billy <billy.o.mahony at intel.com>
>> wrote:
>>
>>>
>>>
>>>
>>>
>>> *From:* Manojawa Paritala [mailto:manojawapv at biarca.com]
>>> *Sent:* Tuesday, October 23, 2018 5:37 PM
>>> *To:* O Mahony, Billy <billy.o.mahony at intel.com>
>>> *Cc:* ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org; Subba Rao
>>> Kodavalla <srk at biarca.com>; Song, Kee SangX <kee.sangx.song at intel.com>;
>>> Srinivasa Goda <srinig at biarca.com>; Kris Rajana <krisr at biarca.com>
>>> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>>>
>>>
>>>
>>> Hi Billy,
>>>
>>>
>>>
>>> There are no br-flat1 entries in ovsdb.
>>>
>>>
>>>
>>> As suggested I increased the log level to debug and then tried the same
>>> scenario again. Though the result was same (br-flat1 getting deleted), I
>>> observed the below 2 issues (i assume).
>>>
>>> *[[BO'M]] You can just increase the log level for the bridge module (all
>>> the extra revalidator debug is making it hard to read the logs.):*
>>>
>>> ovs-appctl vlog/set bridge:file:dbg
>>>
>>>
>>>
>> *Manoj :- Added the log level of debug to only bridge module and
>> collected the logs. Attached zip file has the logs.*
>>
>>
>>> *[[BO'M]] Also is there logging from the neutron agent. OvS should not
>>> be removing the br-flat1 of it’s own accord. Something external is updating
>>> ovsdb to remove it’s record. Is ovn in the loop here? Is there and
>>> ovn-controller or similar process running on the host?*
>>>
>>> *I think this is important. As far as I know the decision to delete the
>>> bridge will not be made by vswitchd. It will be something external that
>>> will remove the record in ovsdb bridge table. If the neutron agent log does
>>> not mention it is doing this then maybe check the ovsdb .log file (it may
>>> not exist if it’s not configured on the ovsdb command line) – you’ll have
>>> to check the ovsdb man pages and also set. The ovsdb-tool is showing just
>>> the delete transaction so we need to find out where that delete request is
>>> coming from.*
>>>
>>>
>>>
>> *Manoj :- I do not see any changes in the OVSDB logs after the
>> configuration change. The attached zip file also contains neutron-ovs-agent
>> logs.*
>>
>>
>>> Issue-1 :-
>>>
>>> 1. Everything is up and running. That is all the bridges are displayed
>>> in OVS and no issues in the logs.
>>>
>>>
>>>
>>> *[[BO'M]] can you add the output from vsctl show at this point so I can
>>> see the desired post agent restart state.*
>>>
>> *Manoj :- Attached zip file contains the output of the ovs-vsctl show
>> command before making changes to the neutron-ovs-agent and restarting the
>> service.*
>>
>>
>>> 2. I add the below entries in the OVS section of neutron's
>>> openvswitch-agent.ini file and restart the respective service.
>>>
>>>
>>>
>>> datapath_type=netdev
>>>
>>> vhostuser_socket_dir=/var/run/openvswitch
>>>
>>>
>>>
>>> 3. As mentioned earlier, bridge br-flat1 is deleted. At this point of
>>> time, observed the below.
>>>
>>>
>>>
>>> 3.1 br-int dtapath type changed from "system" to "netdev".
>>>
>>> 3.2. Not sure if it is an expected behavour, but there were MAC address
>>> changed prints only for both br-flat1 & br-int.
>>>
>>>
>>>
>>> 2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address
>>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>>>
>>> 2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address
>>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>>>
>>> 2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address
>>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>>>
>>> 2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address
>>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>>>
>>> *[[BO'M]] The datapath type change is expected. The MAC address changes
>>> I’m not sure.*
>>>
>> *Manoj :- Ok. Thank you for the clarification.*
>>
>>>
>>>
>>> 3.3 interface state of br-int & eth8 (attached to br-flat1) are down.
>>>
>>> *[[BO'M]] Can you copy the o/p from vsctl show again at this point.*
>>>
>> *Manoj :- Attached zip file contains the ovs-vsctl output after changing
>> the nutron-ovs-agent parameters and restartong the service. Please note
>> that this time we also observed that 2 bridges got deleted. br-flat1 &
>> br-vlan. *
>>
>>>
>>>
>>> Attaching the debug logs of ovs-vswitch.
>>>
>>>
>>>
>>>
>>>
>>> Issue-2 :-
>>>
>>> 1. Everything is up and running. That is all the bridges are displayed
>>> in OVS and no issues in the logs. I have one bridge named br0, which is of
>>> netdev type and I have attached a dpdk port, vhost-user port. Everthing is
>>> file.
>>>
>>>
>>>
>>>
>>>
>>> *[[BO'M]] When you say everything is fine br-flat1 is still missing
>>> right? If that is the case lets stick with the missing bridge issue for
>>> now. (There is actually quite a few issues in the vsctl output below that
>>> we deal with after we figure out who deleted  br-flat1).*
>>>
>> *Manoj :- Ok Sure. *
>>
>>>
>>>
>>> 2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to
>>> the existing "bridge_mappings" key, I added an extra value "vlan1:br0". The
>>> new key-value pair now is as below. I wanted to create a new network and
>>> then map the bridge br0. So, I added this entry.
>>>
>>>
>>>
>>> bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0
>>>
>>>
>>>
>>> 3. Now, when I restart the openvswitch-agent service, I observed that
>>> the datapath type of br0 changed from netdev to system. In the ovs-vsctl
>>> show output, I see the below.
>>>
>>>
>>>
>>>    Bridge "br0"
>>>
>>>         Controller "tcp:127.0.0.1:6633"
>>>
>>>             is_connected: true
>>>
>>>         fail_mode: secure
>>>
>>>         Port "vhost-user-1"
>>>
>>>             Interface "vhost-user-1"
>>>
>>>                 type: dpdkvhostuser
>>>
>>>                 error: "could not add network device vhost-user-1 to
>>> ofproto (Invalid argument)"
>>>
>>>         Port "br0"
>>>
>>>             Interface "br0"
>>>
>>>                 type: internal
>>>
>>>         Port "port0"
>>>
>>>             Interface "port0"
>>>
>>>                 type: dpdk
>>>
>>>                 options: {dpdk-devargs="0000:af:00.1"}
>>>
>>>                 error: "could not add network device port0 to ofproto
>>> (Invalid argument)"
>>>
>>>         Port "phy-br0"
>>>
>>>             Interface "phy-br0"
>>>
>>>                 type: patch
>>>
>>>                 options: {peer="int-br0"}
>>>
>>>
>>>
>>> Attaching the logs for this issue as well.
>>>
>>>
>>>
>>>
>>>
>>> Thanks & Regards,
>>>
>>> PVMJ
>>>
>>>
>>>
>>> On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy <
>>> billy.o.mahony at intel.com> wrote:
>>>
>>> Hi Manojawa,
>>>
>>>
>>>
>>> So is there any remaining entry br-flat entry in the ovsdb? Does it give
>>> any clue to the reason – there may be a free-form ‘status’ or ‘info’ field
>>> for that purpose.
>>>
>>>
>>>
>>> I can understand the situation where a bridge might get incorrectly
>>> configured but I can’t understand why it is deleted by something other than
>>> the agent.
>>>
>>>
>>>
>>> Maybe it tries to create the bridge, there is some error so it decides
>>> to delete it. Are there more detailed log levels available for the agent?
>>> You may be able to turn on more detailed logging for the bridge logging in
>>> OvS too.
>>>
>>>
>>>
>>> /Billy.
>>>
>>>
>>>
>>>
>>>
>>> *From:* Manojawa Paritala [mailto:manojawapv at biarca.com]
>>> *Sent:* Tuesday, October 23, 2018 12:16 PM
>>> *To:* O Mahony, Billy <billy.o.mahony at intel.com>
>>> *Cc:* ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org
>>> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>>>
>>>
>>>
>>> Hi Billy,
>>>
>>>
>>>
>>> Thank you for your reply.
>>>
>>>
>>>
>>> 1. Huge pages are properly set. Based on the dpdk configuration
>>> dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages.
>>>
>>> 2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as
>>> br-flat1.
>>>
>>> 3. Yes,  'ovs-vsctl show'  does not show br-flat1. As soon as I add the
>>> below entries in openvswitch-agent.ini and restart the
>>> neutron-openmvswitch-agent service, br-flat1 is getting deleted. I can see
>>> that in the ovs-vswitch logs and also in the output of "ovsdb-tool -mmm
>>> show-log"
>>>
>>>
>>>
>>> datapath_type=netdev
>>>
>>> vhostuser_socket_dir=/var/run/openvswitch
>>>
>>>
>>>
>>> 4. I do not see any errors in then neutron-openvswitch-agent logs,
>>> except for the below which are displayed after the bridge is deleted.
>>>
>>>
>>>
>>> ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
>>> [req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for
>>> physical network flat does not exist. Agent terminated!
>>>
>>>
>>>
>>> Thanks & Regards,
>>>
>>> PVMJ
>>>
>>>
>>>
>>> On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy <
>>> billy.o.mahony at intel.com> wrote:
>>>
>>> Hi,
>>>
>>> I don't see any errors relating to the dpdk interfaces. But it is also
>>> not clear where the user-space drivers are bound and the hugepage memory is
>>> set up. So double check those two items.
>>>
>>> Is the dpdk-p0 interface being attached to br-flat? Even if there are
>>> issues with the dpdk port the bridge should not be deleted (at least not
>>> automatically by OvS).
>>>
>>> Can you confirm with 'ovs-vsctl show' that the br-flat is actually not
>>> present after the agent is restarted. And that the dpdk-p0 is not reporting
>>> an error.
>>>
>>> What does the neutron-openmvswitch-agent logs say?
>>>
>>> Also run ovsdb-tool -mmm show-log which might give a clue as to when and
>>> how br-flat is being modified.
>>>
>>> Regards,
>>> Billy
>>>
>>> > -----Original Message-----
>>> > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
>>> > bounces at openvswitch.org] On Behalf Of Manojawa Paritala
>>> > Sent: Monday, October 22, 2018 3:31 PM
>>> > To: ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org
>>> > Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>>> >
>>> > Hello All,
>>> >
>>> > On a 3 node (one controller + 2 compute), we configured Openstack
>>> Queens
>>> > using OSA with OVS. On all the nodes, we defined br-mgmt as linux
>>> bridge, br-
>>> > tun as private network and br-flat as external.
>>> > Installation was successful and we could create networks and instances
>>> on
>>> > Openstack.
>>> >
>>> > Below are the versions of the OVS packages used on each node.
>>> >
>>> > Controller :- openstack-vswitch - 2.9.0
>>> > Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure
>>> dpdk on
>>> > the compute hosts)
>>> >
>>> > The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk
>>> version
>>> > 17.11.3. When we tried to enable DPDK it failed with the below error.
>>> >
>>> > dpdk|ERR|DPDK not supported in this copy of Open vSwitch
>>> >
>>> > So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2,
>>> built
>>> > openvswitch with dpdk as suggested in the below official link.
>>> > No issues on Openstack or OVS.
>>> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/
>>> >
>>> > Then, we added the below parameters to OVS and everything looked ok.
>>> > No issues in Openstack or OVS.
>>> >
>>> > $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2",
>>> dpdk-init="true",
>>> > dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd-
>>> > cpu-mask="0xf00003c0000", vhost-iommu-support="true"}
>>> >
>>> > Then on the compute node, in openvswitch_agent.ini file - OVS section,
>>> I added
>>> > the below (based on the link
>>> >
>>> https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h
>>> > tml
>>> > )
>>> > and restarted neutron-openmvswitch-agent service.
>>> >
>>> > datapath_type=netdev
>>> > vhostuser_socket_dir=/var/run/openvswitch
>>> >
>>> > After the above change, bridge br-flat is getting deleted from OVS.
>>> > Attached are the logs after I restart the neutron-openmvswitch-agent
>>> service on
>>> > the compute now. Not sure what the issue is.
>>> >
>>> > Can any of you please let me know if we are missing anything?
>>> >
>>> > Best Regards,
>>> > PVMJ
>>>
>>>


More information about the dev mailing list