[ovs-dev] Issues configuring OVS-DPDK in openstack queens
Manojawa Paritala
manojawapv at biarca.com
Wed Oct 24 13:56:25 UTC 2018
Hi Billy,
Firstly, thank you for taking a look at the logs I sent yesterday.
Please see in-line. Please let me know if I miss anything.
Thanks & Regards,
PVMJ
On Wed, Oct 24, 2018 at 3:36 PM O Mahony, Billy <billy.o.mahony at intel.com>
wrote:
>
>
>
>
> *From:* Manojawa Paritala [mailto:manojawapv at biarca.com]
> *Sent:* Tuesday, October 23, 2018 5:37 PM
> *To:* O Mahony, Billy <billy.o.mahony at intel.com>
> *Cc:* ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org; Subba Rao
> Kodavalla <srk at biarca.com>; Song, Kee SangX <kee.sangx.song at intel.com>;
> Srinivasa Goda <srinig at biarca.com>; Kris Rajana <krisr at biarca.com>
> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>
>
>
> Hi Billy,
>
>
>
> There are no br-flat1 entries in ovsdb.
>
>
>
> As suggested I increased the log level to debug and then tried the same
> scenario again. Though the result was same (br-flat1 getting deleted), I
> observed the below 2 issues (i assume).
>
> *[[BO'M]] You can just increase the log level for the bridge module (all
> the extra revalidator debug is making it hard to read the logs.):*
>
> ovs-appctl vlog/set bridge:file:dbg
>
>
>
*Manoj :- Added the log level of debug to only bridge module and collected
the logs. Attached zip file has the logs.*
> *[[BO'M]] Also is there logging from the neutron agent. OvS should not be
> removing the br-flat1 of it’s own accord. Something external is updating
> ovsdb to remove it’s record. Is ovn in the loop here? Is there and
> ovn-controller or similar process running on the host?*
>
> *I think this is important. As far as I know the decision to delete the
> bridge will not be made by vswitchd. It will be something external that
> will remove the record in ovsdb bridge table. If the neutron agent log does
> not mention it is doing this then maybe check the ovsdb .log file (it may
> not exist if it’s not configured on the ovsdb command line) – you’ll have
> to check the ovsdb man pages and also set. The ovsdb-tool is showing just
> the delete transaction so we need to find out where that delete request is
> coming from.*
>
>
>
*Manoj :- I do not see any changes in the OVSDB logs after the
configuration change. The attached zip file also contains neutron-ovs-agent
logs.*
> Issue-1 :-
>
> 1. Everything is up and running. That is all the bridges are displayed in
> OVS and no issues in the logs.
>
>
>
> *[[BO'M]] can you add the output from vsctl show at this point so I can
> see the desired post agent restart state.*
>
*Manoj :- Attached zip file contains the output of the ovs-vsctl show
command before making changes to the neutron-ovs-agent and restarting the
service.*
> 2. I add the below entries in the OVS section of neutron's
> openvswitch-agent.ini file and restart the respective service.
>
>
>
> datapath_type=netdev
>
> vhostuser_socket_dir=/var/run/openvswitch
>
>
>
> 3. As mentioned earlier, bridge br-flat1 is deleted. At this point of
> time, observed the below.
>
>
>
> 3.1 br-int dtapath type changed from "system" to "netdev".
>
> 3.2. Not sure if it is an expected behavour, but there were MAC address
> changed prints only for both br-flat1 & br-int.
>
>
>
> 2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address
> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>
> 2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address
> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>
> 2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address
> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>
> 2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address
> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
>
> *[[BO'M]] The datapath type change is expected. The MAC address changes
> I’m not sure.*
>
*Manoj :- Ok. Thank you for the clarification.*
>
>
> 3.3 interface state of br-int & eth8 (attached to br-flat1) are down.
>
> *[[BO'M]] Can you copy the o/p from vsctl show again at this point.*
>
*Manoj :- Attached zip file contains the ovs-vsctl output after changing
the nutron-ovs-agent parameters and restartong the service. Please note
that this time we also observed that 2 bridges got deleted. br-flat1 &
br-vlan. *
>
>
> Attaching the debug logs of ovs-vswitch.
>
>
>
>
>
> Issue-2 :-
>
> 1. Everything is up and running. That is all the bridges are displayed in
> OVS and no issues in the logs. I have one bridge named br0, which is of
> netdev type and I have attached a dpdk port, vhost-user port. Everthing is
> file.
>
>
>
>
>
> *[[BO'M]] When you say everything is fine br-flat1 is still missing right?
> If that is the case lets stick with the missing bridge issue for now.
> (There is actually quite a few issues in the vsctl output below that we
> deal with after we figure out who deleted br-flat1).*
>
*Manoj :- Ok Sure. *
>
>
> 2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to the
> existing "bridge_mappings" key, I added an extra value "vlan1:br0". The new
> key-value pair now is as below. I wanted to create a new network and then
> map the bridge br0. So, I added this entry.
>
>
>
> bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0
>
>
>
> 3. Now, when I restart the openvswitch-agent service, I observed that the
> datapath type of br0 changed from netdev to system. In the ovs-vsctl show
> output, I see the below.
>
>
>
> Bridge "br0"
>
> Controller "tcp:127.0.0.1:6633"
>
> is_connected: true
>
> fail_mode: secure
>
> Port "vhost-user-1"
>
> Interface "vhost-user-1"
>
> type: dpdkvhostuser
>
> error: "could not add network device vhost-user-1 to
> ofproto (Invalid argument)"
>
> Port "br0"
>
> Interface "br0"
>
> type: internal
>
> Port "port0"
>
> Interface "port0"
>
> type: dpdk
>
> options: {dpdk-devargs="0000:af:00.1"}
>
> error: "could not add network device port0 to ofproto
> (Invalid argument)"
>
> Port "phy-br0"
>
> Interface "phy-br0"
>
> type: patch
>
> options: {peer="int-br0"}
>
>
>
> Attaching the logs for this issue as well.
>
>
>
>
>
> Thanks & Regards,
>
> PVMJ
>
>
>
> On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy <billy.o.mahony at intel.com>
> wrote:
>
> Hi Manojawa,
>
>
>
> So is there any remaining entry br-flat entry in the ovsdb? Does it give
> any clue to the reason – there may be a free-form ‘status’ or ‘info’ field
> for that purpose.
>
>
>
> I can understand the situation where a bridge might get incorrectly
> configured but I can’t understand why it is deleted by something other than
> the agent.
>
>
>
> Maybe it tries to create the bridge, there is some error so it decides to
> delete it. Are there more detailed log levels available for the agent? You
> may be able to turn on more detailed logging for the bridge logging in OvS
> too.
>
>
>
> /Billy.
>
>
>
>
>
> *From:* Manojawa Paritala [mailto:manojawapv at biarca.com]
> *Sent:* Tuesday, October 23, 2018 12:16 PM
> *To:* O Mahony, Billy <billy.o.mahony at intel.com>
> *Cc:* ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org
> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>
>
>
> Hi Billy,
>
>
>
> Thank you for your reply.
>
>
>
> 1. Huge pages are properly set. Based on the dpdk configuration
> dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages.
>
> 2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as
> br-flat1.
>
> 3. Yes, 'ovs-vsctl show' does not show br-flat1. As soon as I add the
> below entries in openvswitch-agent.ini and restart the
> neutron-openmvswitch-agent service, br-flat1 is getting deleted. I can see
> that in the ovs-vswitch logs and also in the output of "ovsdb-tool -mmm
> show-log"
>
>
>
> datapath_type=netdev
>
> vhostuser_socket_dir=/var/run/openvswitch
>
>
>
> 4. I do not see any errors in then neutron-openvswitch-agent logs, except
> for the below which are displayed after the bridge is deleted.
>
>
>
> ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
> [req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for
> physical network flat does not exist. Agent terminated!
>
>
>
> Thanks & Regards,
>
> PVMJ
>
>
>
> On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy <billy.o.mahony at intel.com>
> wrote:
>
> Hi,
>
> I don't see any errors relating to the dpdk interfaces. But it is also not
> clear where the user-space drivers are bound and the hugepage memory is set
> up. So double check those two items.
>
> Is the dpdk-p0 interface being attached to br-flat? Even if there are
> issues with the dpdk port the bridge should not be deleted (at least not
> automatically by OvS).
>
> Can you confirm with 'ovs-vsctl show' that the br-flat is actually not
> present after the agent is restarted. And that the dpdk-p0 is not reporting
> an error.
>
> What does the neutron-openmvswitch-agent logs say?
>
> Also run ovsdb-tool -mmm show-log which might give a clue as to when and
> how br-flat is being modified.
>
> Regards,
> Billy
>
> > -----Original Message-----
> > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
> > bounces at openvswitch.org] On Behalf Of Manojawa Paritala
> > Sent: Monday, October 22, 2018 3:31 PM
> > To: ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org
> > Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
> >
> > Hello All,
> >
> > On a 3 node (one controller + 2 compute), we configured Openstack Queens
> > using OSA with OVS. On all the nodes, we defined br-mgmt as linux
> bridge, br-
> > tun as private network and br-flat as external.
> > Installation was successful and we could create networks and instances on
> > Openstack.
> >
> > Below are the versions of the OVS packages used on each node.
> >
> > Controller :- openstack-vswitch - 2.9.0
> > Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure
> dpdk on
> > the compute hosts)
> >
> > The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk
> version
> > 17.11.3. When we tried to enable DPDK it failed with the below error.
> >
> > dpdk|ERR|DPDK not supported in this copy of Open vSwitch
> >
> > So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2,
> built
> > openvswitch with dpdk as suggested in the below official link.
> > No issues on Openstack or OVS.
> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/
> >
> > Then, we added the below parameters to OVS and everything looked ok.
> > No issues in Openstack or OVS.
> >
> > $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2",
> dpdk-init="true",
> > dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd-
> > cpu-mask="0xf00003c0000", vhost-iommu-support="true"}
> >
> > Then on the compute node, in openvswitch_agent.ini file - OVS section, I
> added
> > the below (based on the link
> >
> https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h
> > tml
> > )
> > and restarted neutron-openmvswitch-agent service.
> >
> > datapath_type=netdev
> > vhostuser_socket_dir=/var/run/openvswitch
> >
> > After the above change, bridge br-flat is getting deleted from OVS.
> > Attached are the logs after I restart the neutron-openmvswitch-agent
> service on
> > the compute now. Not sure what the issue is.
> >
> > Can any of you please let me know if we are missing anything?
> >
> > Best Regards,
> > PVMJ
>
>
More information about the dev
mailing list