[ovs-discuss] duplicate option: of_interface

Mooney, Sean K sean.k.mooney at intel.com
Wed Aug 26 11:12:20 UTC 2015


I know the feeling. I think your horizon issue can be fixed by
Running 
sudo iptables -F 
sudo iptables -X
iptables based security groups do not work with vhost-user so this will have no effect on vm security

I think the ovs-ofctl: br-int is not a bridge or a socket error is because the 
datapath_type on the bridge is not currently set to netdev.

Looking at your previous error I noticed the follow command the datapath_type is being set to system.
['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 
> '--may-exist', 'add- br', 'br-int', '--', 'set', 'Bridge', 'br-int', 
> 'datapath_type=system']


I think the reason the bridge does not exists is because it has the wrong datapath type.
You can validate this by running 
sudo ovs-vsctl list-br

when using kernel ovs the output will be similar to this

_uuid               : 50c64fb3-3c0c-40f9-8e45-6384a99687aa
controller          : []
datapath_id         : "0000b24fc650f940"
datapath_type       : system
external_ids        : {bridge-id=br-int}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : []
name                : br-int
netflow             : []
other_config        : {}
ports               : [1e2aeb85-01a2-4f1d-a0ef-f4954acab3f2, b8c7a19e-c510-47c1-874c-9d99ec6085f5, c370d053-6aea-4f16-a6bc-f54f1aabfcb5, e1a7ef91-53a4-4222-9bf7-249888ad48ef, e917d939-44db-4661-9311-bb68a6c132c4]
protocols           : ["OpenFlow10"]
sflow               : []
status              : {}
stp_enable          : false

when using ovs with dpdk the datapath_type  should be netdev.

If it is currently system and assuming you have OVS_DATAPATH_TYPE=netdev set in your local.conf this could be caused,
as one of my patches to neutron merged Saturday and we have not updated our deployment code
To add the appropriate configuration.
https://github.com/openstack/neutron/commit/5b708d5f0e9a5ddb46751489a52665e673a5cb0b


If you add 
[ovs]
datapath_type = netdev 
to the ml2_conf.ini the q-agt will configure the dpdk netdev datapath 
instead of the system kernel datapath.

For your running system you can run the following commands to set the datapath

sudo ovs-vsctl set bridge <bridge name> datapath=netdev.

This should be run for all ovs bridges.

On huge table memory as the system is left running main memory gets fragment.
when you start ovs with our service file it allocates hugempage memory in the kernel from the current free memory.

When you compile dpdk there is a hugepage segment limit of 256 segment.
If the dpdk application cannot allocated its hugepage memory in less segment then this limit the application will exit.
You can adjust this limit by setting OVS_DPDK_MEM_SEGMENTS=<value> in your local.conf
The more often you stack and unstack between reboots the more fragmented your memory will be so for development
I tend to set the OVS_DPDK_MEM_SEGMENTS quite high. 

Regards
Sean. 
-----Original Message-----
From: Gabe Black [mailto:Gabe.Black at viavisolutions.com] 
Sent: Tuesday, August 25, 2015 9:45 PM
To: Gabe Black; Mooney, Sean K; bugs at openvswitch.org
Subject: RE: duplicate option: of_interface

Sorry, that was premature.  Apparently hugetable memory gets fragmented or leaks or something as the ovs-dpdk failed to start.  Rebooting fixed that.

I still get many errors in the q-agt log file similar to this:
ERROR neutron.agent.linux.utils [-]
Command: ['ovs-ofctl', 'add-flows', 'br-int', '-'] Exit code: 1
Stdin: hard_timeout=0,idle_timeout=0,priority=0,table=0,cookie=0x0,actions=normal
Stdout:
Stderr: ovs-ofctl: br-int is not a bridge or a socket

and 

ERROR neutron.agent.common.ovs_lib [req-ab1e1f65-d14c-4209-ac20-d29e450eda38 None None] Unable to execute ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']. Exception:
Command: ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23'] Exit code: 1
Stdin:
Stdout:
Stderr: ovs-ofctl: br-int is not a bridge or a socket

But at least q-agt isn't dying.  I remember why I hate fedora though.. Even though horizon says it is running and we turned the firewall to permissive, I can never access the website.  Seems like something else is protecting or acting as a firewall.



> -----Original Message-----
> From: discuss [mailto:discuss-bounces at openvswitch.org] On Behalf Of 
> Gabe Black
> Sent: Tuesday, August 25, 2015 2:06 PM
> To: Mooney, Sean K; bugs at openvswitch.org
> Subject: Re: [ovs-discuss] duplicate option: of_interface
> 
> Thanks for the reply and hints on changes...  Armed with that I undid 
> the changes from git commit 053bfc5a (neutron) and that allowed the 
> q-agt process to at least not die there with that callstack.  The 
> commit message seemed to indicate it was for better 
> restarting/cleanup; so I hope it is relatively low impact to back out.
> 
> However, it now dies because it is still unable to create the br-int bridge:
> 
> Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', 
> '--format=json', '--', '--may-exist', 'add-br', 'br-int', '--', 'set', 
> 'Bridge', 'br-int', 'datapath_type=system'].
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Traceback 
> (most recent call last):
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl   File
> "/opt/stack/neutron/neutron/agent/ovsdb/impl_vsctl.py", line 63, in 
> run_vsctl
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl
> log_fail_as_error=False).rstrip()
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl   File
> "/opt/stack/neutron/neutron/agent/linux/utils.py", line 153, in execute
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl     raise
> RuntimeError(m)
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl
> RuntimeError:
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Command:
> ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 
> '--may-exist', 'add- br', 'br-int', '--', 'set', 'Bridge', 'br-int', 
> 'datapath_type=system']
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Exit 
> code: -14
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Stdin:
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Stdout:
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Stderr: 
> 2015- 08-25T19:59:44Z|00002|fatal_signal|WARN|terminating with signal 
> 14 (Alarm
> clock)
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl
> 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl
> 
> I noticed a difference between a multi-node setup and a single node, 
> where multi-node's controller has openvswitch and ovsdpdk for the 
> Q_ML2_PLUGIN_MECHANISM_DRIVERS.  Is it perhaps related to that?  Or 
> should it be trying to create br-int as datapath_type netdev instead?  
> Just throwing things out there because I'm ignorant :-)  I'll try 
> variations of that in hopes I get lucky.
> 
> 
> Gabe
> 
> 
> 
> > -----Original Message-----
> > From: Mooney, Sean K [mailto:sean.k.mooney at intel.com]
> > Sent: Tuesday, August 25, 2015 1:30 PM
> > To: Gabe Black; bugs at openvswitch.org
> > Subject: RE: duplicate option: of_interface
> >
> > Hi Gabe
> > We have started to see that message in our ci since this weekend.
> > We are currently investigating it but I belive a change has merged 
> > to neuton that we need to back port to Our agent.
> >
> > A lot of code has merged in the last 2 weeks as the code freeze for 
> > the liberty release is moday.
> > The stable kilo branch should be unaffected but we are actively 
> > looking into this at present.
> >
> > Regards
> > Sean.
> >
> >
> > -----Original Message-----
> > From: Gabe Black [mailto:Gabe.Black at viavisolutions.com]
> > Sent: Tuesday, August 25, 2015 7:50 PM
> > To: bugs at openvswitch.org
> > Cc: Mooney, Sean K
> > Subject: duplicate option: of_interface
> >
> > I have followed the getting started guide
> > (http://git.openstack.org/cgit/stackforge/networking-ovs-
> > dpdk/tree/doc/source/getstarted.rst) on both fedora 21 and Ubuntu
> > 15.04 to get a single-node set up with dpdk ovs.
> >
> > My local.conf file is identical to the one provided as the single 
> > node
> template:
> > http://git.openstack.org/cgit/stackforge/networking-ovs-
> > dpdk/tree/doc/source/_downloads/local.conf.single_node
> >
> > I set HOST_IP_IFACE=eno1, HOST_IP=10.3.73.124, 
> > OVS_BRIDGE_MAPPINGS="default:br-enp4s0f0", and
> > ML2_VLAN_RANGES=default:1000:1010
> >
> > eno1 and associated IP is the interface/ip address of the server 
> > (i.e. what
> we
> > use to ssh to the box).   enp4s0f0 is the 10G intel nic interface that will
> > eventually be used for the data interface in a multi-node setup.
> > Finally the vlan range was just arbitrarily chosen.
> >
> > Other than that, there isn't anything else modified other than 
> > following instructions of the getting started guide.  However for 
> > both Fedora 21, and Ubuntu 15.04 (Ubuntu there were some mods that 
> > needed to take place like disabling apparmor, symlinking 
> > /var/run/openstack, and fixing ovs-dpdk-init
> > script) result in the following error message in q-agt:
> >
> > Traceback (most recent call last):
> >   File "/usr/bin/networking-ovs-dpdk-agent", line 10, in <module>
> >     sys.exit(main())
> >   File "/usr/lib/python2.7/site-
> > packages/networking_ovs_dpdk/eventlet/ovs_dpdk_neutron_agent.py",
> > line 20, in main
> >     agent_main.main()
> >   File "/usr/lib/python2.7/site-
> > packages/networking_ovs_dpdk/agent/main.py", line 43, in main
> >     mod = importutils.import_module(mod_name)
> >   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py",
> > line 57, in import_module
> >     __import__(import_str)
> >   File "/usr/lib/python2.7/site-
> >
> packages/networking_ovs_dpdk/agent/openflow/ovsdpdk_ofctl/main.py",
> > line 17, in <module>
> >     from networking_ovs_dpdk.agent import ovs_dpdk_neutron_agent
> >   File "/usr/lib/python2.7/site-
> > packages/networking_ovs_dpdk/agent/ovs_dpdk_neutron_agent.py",
> line
> > 47, in <module>
> >     from neutron.plugins.ml2.drivers.openvswitch.agent import 
> > ovs_dvr_neutron_agent
> >   File
> >
> "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_
> > dvr_neutron_agent.py", line 29, in <module>
> >     cfg.CONF.import_group('AGENT',
> > 'neutron.plugins.ml2.drivers.openvswitch.'
> >   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 
> > 2088, in import_group
> >     __import__(module_str)
> >   File
> >
> "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/com
> > mon/config.py", line 111, in <module>
> >     cfg.CONF.register_opts(ovs_opts, "OVS")
> >   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 
> > 1824, in __inner
> >     result = f(self, *args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 
> > 1983, in register_opts
> >     self.register_opt(opt, group, clear_cache=False)
> >   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 
> > 1828, in __inner
> >     return f(self, *args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 
> > 1967, in register_opt
> >     return group._register_opt(opt, cli)
> >   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 
> > 1345, in _register_opt
> >     if _is_opt_registered(self._opts, opt):
> >   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 
> > 574, in _is_opt_registered
> >     raise DuplicateOptError(opt.name)
> > oslo_config.cfg.DuplicateOptError: duplicate option: of_interface 
> > q-agt failed to start
> >
> > I thought this error message was just because Ubuntu testing/support 
> > hasn't been fleshed out yet with ovs-dpdk, but then I got the exact 
> > same error on Fedora 21.  I tried editing both
> >
> /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/comm
> > on/config.py and /opt/stack/networking-ovs- 
> > dpdk/networking_ovs_dpdk/common/config.py to get past the error, but 
> > then there are complaints about not finding br-int... So I'm 
> > guessing that isn't the correct workaround.  Anyone have any 
> > suggestions of what I might have misconfigured?
> >
> > Thank you for your help!
> > Gabriel Black
> >
> 
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss


More information about the discuss mailing list