[ovs-dev] Issues configuring OVS-DPDK in openstack queens

O Mahony, Billy billy.o.mahony at intel.com
Wed Oct 24 10:06:30 UTC 2018



From: Manojawa Paritala [mailto:manojawapv at biarca.com]
Sent: Tuesday, October 23, 2018 5:37 PM
To: O Mahony, Billy <billy.o.mahony at intel.com>
Cc: ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org; Subba Rao Kodavalla <srk at biarca.com>; Song, Kee SangX <kee.sangx.song at intel.com>; Srinivasa Goda <srinig at biarca.com>; Kris Rajana <krisr at biarca.com>
Subject: Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens

Hi Billy,

There are no br-flat1 entries in ovsdb.

As suggested I increased the log level to debug and then tried the same scenario again. Though the result was same (br-flat1 getting deleted), I observed the below 2 issues (i assume).
[[BO'M]] You can just increase the log level for the bridge module (all the extra revalidator debug is making it hard to read the logs.):
ovs-appctl vlog/set bridge:file:dbg

[[BO'M]] Also is there logging from the neutron agent. OvS should not be removing the br-flat1 of it’s own accord. Something external is updating ovsdb to remove it’s record. Is ovn in the loop here? Is there and ovn-controller or similar process running on the host?
I think this is important. As far as I know the decision to delete the bridge will not be made by vswitchd. It will be something external that will remove the record in ovsdb bridge table. If the neutron agent log does not mention it is doing this then maybe check the ovsdb .log file (it may not exist if it’s not configured on the ovsdb command line) – you’ll have to check the ovsdb man pages and also set. The ovsdb-tool is showing just the delete transaction so we need to find out where that delete request is coming from.

Issue-1 :-
1. Everything is up and running. That is all the bridges are displayed in OVS and no issues in the logs.

[[BO'M]] can you add the output from vsctl show at this point so I can see the desired post agent restart state.
2. I add the below entries in the OVS section of neutron's openvswitch-agent.ini file and restart the respective service.

datapath_type=netdev
vhostuser_socket_dir=/var/run/openvswitch

3. As mentioned earlier, bridge br-flat1 is deleted. At this point of time, observed the below.

3.1 br-int dtapath type changed from "system" to "netdev".
3.2. Not sure if it is an expected behavour, but there were MAC address changed prints only for both br-flat1 & br-int.

2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
[[BO'M]] The datapath type change is expected. The MAC address changes I’m not sure.

3.3 interface state of br-int & eth8 (attached to br-flat1) are down.
[[BO'M]] Can you copy the o/p from vsctl show again at this point.

Attaching the debug logs of ovs-vswitch.


Issue-2 :-
1. Everything is up and running. That is all the bridges are displayed in OVS and no issues in the logs. I have one bridge named br0, which is of netdev type and I have attached a dpdk port, vhost-user port. Everthing is file.


[[BO'M]] When you say everything is fine br-flat1 is still missing right? If that is the case lets stick with the missing bridge issue for now. (There is actually quite a few issues in the vsctl output below that we deal with after we figure out who deleted  br-flat1).

2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to the existing "bridge_mappings" key, I added an extra value "vlan1:br0". The new key-value pair now is as below. I wanted to create a new network and then map the bridge br0. So, I added this entry.

bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0

3. Now, when I restart the openvswitch-agent service, I observed that the datapath type of br0 changed from netdev to system. In the ovs-vsctl show output, I see the below.

   Bridge "br0"
        Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>"
            is_connected: true
        fail_mode: secure
        Port "vhost-user-1"
            Interface "vhost-user-1"
                type: dpdkvhostuser
                error: "could not add network device vhost-user-1 to ofproto (Invalid argument)"
        Port "br0"
            Interface "br0"
                type: internal
        Port "port0"
            Interface "port0"
                type: dpdk
                options: {dpdk-devargs="0000:af:00.1"}
                error: "could not add network device port0 to ofproto (Invalid argument)"
        Port "phy-br0"
            Interface "phy-br0"
                type: patch
                options: {peer="int-br0"}

Attaching the logs for this issue as well.


Thanks & Regards,
PVMJ

On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy <billy.o.mahony at intel.com<mailto:billy.o.mahony at intel.com>> wrote:
Hi Manojawa,

So is there any remaining entry br-flat entry in the ovsdb? Does it give any clue to the reason – there may be a free-form ‘status’ or ‘info’ field for that purpose.

I can understand the situation where a bridge might get incorrectly configured but I can’t understand why it is deleted by something other than the agent.

Maybe it tries to create the bridge, there is some error so it decides to delete it. Are there more detailed log levels available for the agent? You may be able to turn on more detailed logging for the bridge logging in OvS too.

/Billy.


From: Manojawa Paritala [mailto:manojawapv at biarca.com<mailto:manojawapv at biarca.com>]
Sent: Tuesday, October 23, 2018 12:16 PM
To: O Mahony, Billy <billy.o.mahony at intel.com<mailto:billy.o.mahony at intel.com>>
Cc: ovs-discuss at openvswitch.org<mailto:ovs-discuss at openvswitch.org>; ovs-dev at openvswitch.org<mailto:ovs-dev at openvswitch.org>
Subject: Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens

Hi Billy,

Thank you for your reply.

1. Huge pages are properly set. Based on the dpdk configuration dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages.
2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as br-flat1.
3. Yes,  'ovs-vsctl show'  does not show br-flat1. As soon as I add the below entries in openvswitch-agent.ini and restart the neutron-openmvswitch-agent service, br-flat1 is getting deleted. I can see that in the ovs-vswitch logs and also in the output of "ovsdb-tool -mmm show-log"

datapath_type=netdev
vhostuser_socket_dir=/var/run/openvswitch

4. I do not see any errors in then neutron-openvswitch-agent logs, except for the below which are displayed after the bridge is deleted.

ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for physical network flat does not exist. Agent terminated!

Thanks & Regards,
PVMJ

On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy <billy.o.mahony at intel.com<mailto:billy.o.mahony at intel.com>> wrote:
Hi,

I don't see any errors relating to the dpdk interfaces. But it is also not clear where the user-space drivers are bound and the hugepage memory is set up. So double check those two items.

Is the dpdk-p0 interface being attached to br-flat? Even if there are issues with the dpdk port the bridge should not be deleted (at least not automatically by OvS).

Can you confirm with 'ovs-vsctl show' that the br-flat is actually not present after the agent is restarted. And that the dpdk-p0 is not reporting an error.

What does the neutron-openmvswitch-agent logs say?

Also run ovsdb-tool -mmm show-log which might give a clue as to when and how br-flat is being modified.

Regards,
Billy

> -----Original Message-----
> From: ovs-dev-bounces at openvswitch.org<mailto:ovs-dev-bounces at openvswitch.org> [mailto:ovs-dev-<mailto:ovs-dev->
> bounces at openvswitch.org<mailto:bounces at openvswitch.org>] On Behalf Of Manojawa Paritala
> Sent: Monday, October 22, 2018 3:31 PM
> To: ovs-discuss at openvswitch.org<mailto:ovs-discuss at openvswitch.org>; ovs-dev at openvswitch.org<mailto:ovs-dev at openvswitch.org>
> Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>
> Hello All,
>
> On a 3 node (one controller + 2 compute), we configured Openstack Queens
> using OSA with OVS. On all the nodes, we defined br-mgmt as linux bridge, br-
> tun as private network and br-flat as external.
> Installation was successful and we could create networks and instances on
> Openstack.
>
> Below are the versions of the OVS packages used on each node.
>
> Controller :- openstack-vswitch - 2.9.0
> Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure dpdk on
> the compute hosts)
>
> The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk version
> 17.11.3. When we tried to enable DPDK it failed with the below error.
>
> dpdk|ERR|DPDK not supported in this copy of Open vSwitch
>
> So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2, built
> openvswitch with dpdk as suggested in the below official link.
> No issues on Openstack or OVS.
> http://docs.openvswitch.org/en/latest/intro/install/dpdk/
>
> Then, we added the below parameters to OVS and everything looked ok.
> No issues in Openstack or OVS.
>
> $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2", dpdk-init="true",
> dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd-
> cpu-mask="0xf00003c0000", vhost-iommu-support="true"}
>
> Then on the compute node, in openvswitch_agent.ini file - OVS section, I added
> the below (based on the link
> https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h
> tml
> )
> and restarted neutron-openmvswitch-agent service.
>
> datapath_type=netdev
> vhostuser_socket_dir=/var/run/openvswitch
>
> After the above change, bridge br-flat is getting deleted from OVS.
> Attached are the logs after I restart the neutron-openmvswitch-agent service on
> the compute now. Not sure what the issue is.
>
> Can any of you please let me know if we are missing anything?
>
> Best Regards,
> PVMJ


More information about the dev mailing list