[ovs-dev] Issues configuring OVS-DPDK in openstack queens

Manojawa Paritala manojawapv at biarca.com
Wed Oct 24 14:17:49 UTC 2018


Hi Billy,

The reason for br-vlan getting deleted in my previous attempt was due to a
small misconfiguration in the openstack-agent.ini. I corrected it and
re-done the test. Attaching the new logs with this email. Please ignore the
earlier logs.

Apologies for the confusion.

Thanks & Regards,
PVMJ

On Wed, Oct 24, 2018 at 7:26 PM Manojawa Paritala <manojawapv at biarca.com>
wrote:

> Hi Billy,
>
> Firstly, thank you for taking a look at the logs I sent yesterday.
>
> Please see in-line. Please let  me know if I miss anything.
>
> Thanks & Regards,
> PVMJ
>
> On Wed, Oct 24, 2018 at 3:36 PM O Mahony, Billy <billy.o.mahony at intel.com>
> wrote:
>
>>
>>
>>
>>
>> *From:* Manojawa Paritala [mailto:manojawapv at biarca.com]
>> *Sent:* Tuesday, October 23, 2018 5:37 PM
>> *To:* O Mahony, Billy <billy.o.mahony at intel.com>
>> *Cc:* ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org; Subba Rao
>> Kodavalla <srk at biarca.com>; Song, Kee SangX <kee.sangx.song at intel.com>;
>> Srinivasa Goda <srinig at biarca.com>; Kris Rajana <krisr at biarca.com>
>> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>>
>>
>>
>> Hi Billy,
>>
>>
>>
>> There are no br-flat1 entries in ovsdb.
>>
>>
>>
>> As suggested I increased the log level to debug and then tried the same
>> scenario again. Though the result was same (br-flat1 getting deleted), I
>> observed the below 2 issues (i assume).
>>
>> *[[BO'M]] You can just increase the log level for the bridge module (all
>> the extra revalidator debug is making it hard to read the logs.):*
>>
>> ovs-appctl vlog/set bridge:file:dbg
>>
>>
>>
> *Manoj :- Added the log level of debug to only bridge module and collected
> the logs. Attached zip file has the logs.*
>
>
>> *[[BO'M]] Also is there logging from the neutron agent. OvS should not be
>> removing the br-flat1 of it’s own accord. Something external is updating
>> ovsdb to remove it’s record. Is ovn in the loop here? Is there and
>> ovn-controller or similar process running on the host?*
>>
>> *I think this is important. As far as I know the decision to delete the
>> bridge will not be made by vswitchd. It will be something external that
>> will remove the record in ovsdb bridge table. If the neutron agent log does
>> not mention it is doing this then maybe check the ovsdb .log file (it may
>> not exist if it’s not configured on the ovsdb command line) – you’ll have
>> to check the ovsdb man pages and also set. The ovsdb-tool is showing just
>> the delete transaction so we need to find out where that delete request is
>> coming from.*
>>
>>
>>
> *Manoj :- I do not see any changes in the OVSDB logs after the
> configuration change. The attached zip file also contains neutron-ovs-agent
> logs.*
>
>
>> Issue-1 :-
>>
>> 1. Everything is up and running. That is all the bridges are displayed in
>> OVS and no issues in the logs.
>>
>>
>>
>> *[[BO'M]] can you add the output from vsctl show at this point so I can
>> see the desired post agent restart state.*
>>
> *Manoj :- Attached zip file contains the output of the ovs-vsctl show
> command before making changes to the neutron-ovs-agent and restarting the
> service.*
>
>
>> 2. I add the below entries in the OVS section of neutron's
>> openvswitch-agent.ini file and restart the respective service.
>>
>>
>>
>> datapath_type=netdev
>>
>> vhostuser_socket_dir=/var/run/openvswitch
>>
>>
>>
>> 3. As mentioned earlier, bridge br-flat1 is deleted. At this point of
>> time, observed the below.
>>
>>
>>
>> 3.1 br-int dtapath type changed from "system" to "netdev".
>>
>> 3.2. Not sure if it is an expected behavour, but there were MAC address
>> changed prints only for both br-flat1 & br-int.
>>
>>
>>
>> 2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address
>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>>
>> 2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address
>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>>
>> 2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address
>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>>
>> 2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address
>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>>
>> *[[BO'M]] The datapath type change is expected. The MAC address changes
>> I’m not sure.*
>>
> *Manoj :- Ok. Thank you for the clarification.*
>
>>
>>
>> 3.3 interface state of br-int & eth8 (attached to br-flat1) are down.
>>
>> *[[BO'M]] Can you copy the o/p from vsctl show again at this point.*
>>
> *Manoj :- Attached zip file contains the ovs-vsctl output after changing
> the nutron-ovs-agent parameters and restartong the service. Please note
> that this time we also observed that 2 bridges got deleted. br-flat1 &
> br-vlan. *
>
>>
>>
>> Attaching the debug logs of ovs-vswitch.
>>
>>
>>
>>
>>
>> Issue-2 :-
>>
>> 1. Everything is up and running. That is all the bridges are displayed in
>> OVS and no issues in the logs. I have one bridge named br0, which is of
>> netdev type and I have attached a dpdk port, vhost-user port. Everthing is
>> file.
>>
>>
>>
>>
>>
>> *[[BO'M]] When you say everything is fine br-flat1 is still missing
>> right? If that is the case lets stick with the missing bridge issue for
>> now. (There is actually quite a few issues in the vsctl output below that
>> we deal with after we figure out who deleted  br-flat1).*
>>
> *Manoj :- Ok Sure. *
>
>>
>>
>> 2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to the
>> existing "bridge_mappings" key, I added an extra value "vlan1:br0". The new
>> key-value pair now is as below. I wanted to create a new network and then
>> map the bridge br0. So, I added this entry.
>>
>>
>>
>> bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0
>>
>>
>>
>> 3. Now, when I restart the openvswitch-agent service, I observed that the
>> datapath type of br0 changed from netdev to system. In the ovs-vsctl show
>> output, I see the below.
>>
>>
>>
>>    Bridge "br0"
>>
>>         Controller "tcp:127.0.0.1:6633"
>>
>>             is_connected: true
>>
>>         fail_mode: secure
>>
>>         Port "vhost-user-1"
>>
>>             Interface "vhost-user-1"
>>
>>                 type: dpdkvhostuser
>>
>>                 error: "could not add network device vhost-user-1 to
>> ofproto (Invalid argument)"
>>
>>         Port "br0"
>>
>>             Interface "br0"
>>
>>                 type: internal
>>
>>         Port "port0"
>>
>>             Interface "port0"
>>
>>                 type: dpdk
>>
>>                 options: {dpdk-devargs="0000:af:00.1"}
>>
>>                 error: "could not add network device port0 to ofproto
>> (Invalid argument)"
>>
>>         Port "phy-br0"
>>
>>             Interface "phy-br0"
>>
>>                 type: patch
>>
>>                 options: {peer="int-br0"}
>>
>>
>>
>> Attaching the logs for this issue as well.
>>
>>
>>
>>
>>
>> Thanks & Regards,
>>
>> PVMJ
>>
>>
>>
>> On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy <billy.o.mahony at intel.com>
>> wrote:
>>
>> Hi Manojawa,
>>
>>
>>
>> So is there any remaining entry br-flat entry in the ovsdb? Does it give
>> any clue to the reason – there may be a free-form ‘status’ or ‘info’ field
>> for that purpose.
>>
>>
>>
>> I can understand the situation where a bridge might get incorrectly
>> configured but I can’t understand why it is deleted by something other than
>> the agent.
>>
>>
>>
>> Maybe it tries to create the bridge, there is some error so it decides to
>> delete it. Are there more detailed log levels available for the agent? You
>> may be able to turn on more detailed logging for the bridge logging in OvS
>> too.
>>
>>
>>
>> /Billy.
>>
>>
>>
>>
>>
>> *From:* Manojawa Paritala [mailto:manojawapv at biarca.com]
>> *Sent:* Tuesday, October 23, 2018 12:16 PM
>> *To:* O Mahony, Billy <billy.o.mahony at intel.com>
>> *Cc:* ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org
>> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>>
>>
>>
>> Hi Billy,
>>
>>
>>
>> Thank you for your reply.
>>
>>
>>
>> 1. Huge pages are properly set. Based on the dpdk configuration
>> dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages.
>>
>> 2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as
>> br-flat1.
>>
>> 3. Yes,  'ovs-vsctl show'  does not show br-flat1. As soon as I add the
>> below entries in openvswitch-agent.ini and restart the
>> neutron-openmvswitch-agent service, br-flat1 is getting deleted. I can see
>> that in the ovs-vswitch logs and also in the output of "ovsdb-tool -mmm
>> show-log"
>>
>>
>>
>> datapath_type=netdev
>>
>> vhostuser_socket_dir=/var/run/openvswitch
>>
>>
>>
>> 4. I do not see any errors in then neutron-openvswitch-agent logs, except
>> for the below which are displayed after the bridge is deleted.
>>
>>
>>
>> ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
>> [req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for
>> physical network flat does not exist. Agent terminated!
>>
>>
>>
>> Thanks & Regards,
>>
>> PVMJ
>>
>>
>>
>> On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy <billy.o.mahony at intel.com>
>> wrote:
>>
>> Hi,
>>
>> I don't see any errors relating to the dpdk interfaces. But it is also
>> not clear where the user-space drivers are bound and the hugepage memory is
>> set up. So double check those two items.
>>
>> Is the dpdk-p0 interface being attached to br-flat? Even if there are
>> issues with the dpdk port the bridge should not be deleted (at least not
>> automatically by OvS).
>>
>> Can you confirm with 'ovs-vsctl show' that the br-flat is actually not
>> present after the agent is restarted. And that the dpdk-p0 is not reporting
>> an error.
>>
>> What does the neutron-openmvswitch-agent logs say?
>>
>> Also run ovsdb-tool -mmm show-log which might give a clue as to when and
>> how br-flat is being modified.
>>
>> Regards,
>> Billy
>>
>> > -----Original Message-----
>> > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
>> > bounces at openvswitch.org] On Behalf Of Manojawa Paritala
>> > Sent: Monday, October 22, 2018 3:31 PM
>> > To: ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org
>> > Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>> >
>> > Hello All,
>> >
>> > On a 3 node (one controller + 2 compute), we configured Openstack Queens
>> > using OSA with OVS. On all the nodes, we defined br-mgmt as linux
>> bridge, br-
>> > tun as private network and br-flat as external.
>> > Installation was successful and we could create networks and instances
>> on
>> > Openstack.
>> >
>> > Below are the versions of the OVS packages used on each node.
>> >
>> > Controller :- openstack-vswitch - 2.9.0
>> > Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure
>> dpdk on
>> > the compute hosts)
>> >
>> > The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk
>> version
>> > 17.11.3. When we tried to enable DPDK it failed with the below error.
>> >
>> > dpdk|ERR|DPDK not supported in this copy of Open vSwitch
>> >
>> > So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2,
>> built
>> > openvswitch with dpdk as suggested in the below official link.
>> > No issues on Openstack or OVS.
>> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/
>> >
>> > Then, we added the below parameters to OVS and everything looked ok.
>> > No issues in Openstack or OVS.
>> >
>> > $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2",
>> dpdk-init="true",
>> > dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd-
>> > cpu-mask="0xf00003c0000", vhost-iommu-support="true"}
>> >
>> > Then on the compute node, in openvswitch_agent.ini file - OVS section,
>> I added
>> > the below (based on the link
>> >
>> https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h
>> > tml
>> > )
>> > and restarted neutron-openmvswitch-agent service.
>> >
>> > datapath_type=netdev
>> > vhostuser_socket_dir=/var/run/openvswitch
>> >
>> > After the above change, bridge br-flat is getting deleted from OVS.
>> > Attached are the logs after I restart the neutron-openmvswitch-agent
>> service on
>> > the compute now. Not sure what the issue is.
>> >
>> > Can any of you please let me know if we are missing anything?
>> >
>> > Best Regards,
>> > PVMJ
>>
>>


More information about the dev mailing list