[ovs-discuss] [neutron][networking-ovn] OpenStack Newton using OVN as mechanism driver is giving error.
Numan Siddique
nusiddiq at redhat.com
Thu Feb 2 13:49:15 UTC 2017
On Thu, Feb 2, 2017 at 6:05 PM, pranab boruah <pranabjyotiboruah at gmail.com>
wrote:
> Numan,
> Sorry for making this thread too long.
> Kind of deadlock scenario here:
> The process listening on port 6641 is ovsdb-server(not NB db).
>
This is strange. The default port should be 6640 I suppose if not
specified.
> Details:
> Start OVS:
> # /usr/share/openvswitch/scripts/ovs-ctl start --system-id="random"
> # netstat -putna | grep 6641
> tcp 0 0 192.168.10.10:6641 0.0.0.0:*
> LISTEN 5118/ovsdb-server
> # ps -afx | grep ovsdb
> 5118 ? S< 0:00 \_ ovsdb-server /etc/openvswitch/conf.db
> -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock
> --private-key=db:Open_vSwitch,SSL,private_key
> --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert
> --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log
> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
>
> Start NB db
> # /usr/share/openvswitch/scripts/ovn-ctl restart_northd
> --db-sb-create-insecure-remote=yes --db-nb-create-insecure-remote=yes
>
> # netstat -putna | grep 6641
> tcp 0 0 192.168.10.10:6641 0.0.0.0:*
> LISTEN 5118/ovsdb-server
>
> # ps -afx | grep ovsdb
> 5118 ? S< 0:00 \_ ovsdb-server /etc/openvswitch/conf.db
> -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock
> --private-key=db:Open_vSwitch,SSL,private_key
> --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert
> --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log
> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
>
> 5215 ? S 0:00 \_ ovsdb-server --detach --monitor
> -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-nb.log
> --remote=punix:/var/run/openvswitch/ovnnb_db.sock --pidfile=/var/run/openvswitch/ovnnb_db.pid
> --remote=db:OVN_Northbound,NB_Global,connections --unixctl=ovnnb_db.ctl
> --private-key=db:OVN_Northbound,SSL,private_key --certificate=db:OVN_Northbound,SSL,certificate
> --ca-cert=db:OVN_Northbound,SSL,ca_cert --remote=ptcp:6641:192.168.10.10
> /etc/openvswitch/ovnnb_db.db
>
> 5223 ? S 0:00 \_ ovsdb-server --detach --monitor
> -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-sb.log
> --remote=punix:/var/run/openvswitch/ovnsb_db.sock --pidfile=/var/run/openvswitch/ovnsb_db.pid
> --remote=db:OVN_Southbound,SB_Global,connections --unixctl=ovnsb_db.ctl
> --private-key=db:OVN_Southbound,SSL,private_key --certificate=db:OVN_Southbound,SSL,certificate
> --ca-cert=db:OVN_Southbound,SSL,ca_cert --remote=ptcp:6642:192.168.10.10
> /etc/openvswitch/ovnsb_db.db
>
> Northbound DB logs show the same error.
>
> Well I am not supposed to kill the ovsdb-server process. Am I ?
>
> Also when I kill the ovsdb-server process, the error goes away. But my
> doubt is we need the ovsdb-server process on the Controller node. Right?
>
OVN as such doesn't expect ovs-vswitchd (and ovsdb-server for ovs db)
running in controller node. May be you can stop the ovs-vswitchd service
if you don't need it in the controller.
Also you can configure the NB and SB db ports to some other values if you
wish so and update the new values in appropriate locations.
Glad that you figured out the issue.
Numan
> Thanks for your patience.
>
> Pranab
>
>
>
>
> On 2 February 2017 at 17:45, Numan Siddique <nusiddiq at redhat.com> wrote:
>
>> In the ovsdb logs I see
>>
>> 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10: bind:
>> Address already in use
>> 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10: bind:
>> Address already in use
>> 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10: bind:
>> Address already in use
>> 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10: bind:
>> Address already in use
>>
>> Probably you can use netstat and see which processes are listening on
>> port 6641, kill those processes and restart ovn-northd service and see if
>> it works.
>>
>> Thanks
>> Numan
>>
>>
>> On Thu, Feb 2, 2017 at 5:26 PM, pranab boruah <
>> pranabjyotiboruah at gmail.com> wrote:
>>
>>> NB db is not listening on the port 6641:
>>> # ovn-nbctl --db=tcp:192.168.10.10:6641 show
>>> 2017-02-02T11:35:10Z|00001|ovsdb_idl|ERR|tcp:192.168.10.10:6641:
>>> requested schema not found
>>> ovn-nbctl: tcp:192.168.10.10:6641: database connection failed (No such
>>> file or directory)
>>>
>>> # ovn-sbctl --db=tcp:192.168.10.10:6642 show
>>> Chassis "e8968c35-11db-4b3b-ab24-c20866ac3dc8"
>>> hostname: "compute-447"
>>> Encap geneve
>>> ip: "172.16.10.11"
>>> options: {csum="true"}
>>> Encap vxlan
>>> ip: "172.16.10.11"
>>> options: {csum="true"}
>>> Chassis "9f20e7bd-5e16-4273-8ce8-96d0e0a12047"
>>> hostname: "compute-397"
>>> Encap geneve
>>> ip: "172.16.10.12"
>>> options: {csum="true"}
>>> Encap vxlan
>>> ip: "172.16.10.12"
>>> options: {csum="true"}
>>>
>>> # ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock Connection
>>> Connection table
>>> _uuid external_ids inactivity_probe
>>> is_connected max_backoff other_config status
>>> target
>>> ------------------------------------ ------------ ----------------
>>> ------------ ----------- ------------ ---------------
>>> -------------------------
>>> c110ffff-5ba2-4125-8a51-3c3db583f267 {} []
>>> false [] {} {state=BACKOFF} "ptcp:
>>> 192.168.10.10:6641"
>>>
>>> # ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock Connection
>>> Connection table
>>> _uuid external_ids inactivity_probe
>>> is_connected max_backoff other_config read_only status
>>> target
>>> ------------------------------------ ------------ ----------------
>>> ------------ ----------- ------------ --------- ---------------
>>> -------------------------
>>> 1beb9cba-cc01-4981-8a67-a8b265326df9 {} []
>>> false [] {} false {state=BACKOFF} "ptcp:
>>> 192.168.10.10:6642"
>>>
>>> No change after restarting Controller node and restating Northd as
>>> mentioned here:
>>> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291
>>> 63236d48fa0f9105687149
>>>
>>> Pranab
>>>
>>>
>>>
>>>
>>>
>>> On 2 February 2017 at 17:00, Numan Siddique <nusiddiq at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Feb 2, 2017 at 4:44 PM, pranab boruah <
>>>> pranabjyotiboruah at gmail.com> wrote:
>>>>
>>>>> Numan,
>>>>>
>>>>> Thanks, but, the above commands didn't changed anything. Checked the
>>>>> logs and found this:
>>>>>
>>>>>
>>>>
>>>> Could you run the below two commands and see how it goes
>>>>
>>>> - "ovn-nbctl --db=tcp:192.168.10.10:6641 show" -> See if this command
>>>> is successful or not. If not then ovsdb-server is either not listening on
>>>> this port or it may be possible that you need to add iptables rule to allow
>>>> this port.
>>>>
>>>> -
>>>> "ovn-
>>>> s
>>>> bctl --db=tcp:192.168.10.10:664
>>>> 2
>>>> show"
>>>> -
>>>>
>>>>
>>>> On the controller node where you are running the ovsdb-server's run
>>>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock
>>>> Connection"
>>>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock
>>>> Connection"
>>>>
>>>> If the Connection table is empty, it means ovsdb-server is not
>>>> listening on the tcp ports.
>>>>
>>>> Thanks
>>>> Numan
>>>>
>>>>
>>>> LOG of NB database:
>>>>>
>>>>> 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:07:16.184Z|00024|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:09:55.403Z|00025|ovsdb_jsonrpc_server|INFO|ptcp:6642:
>>>>> 192.168.10.10: remote deconfigured
>>>>> 2017-02-02T11:09:55.403Z|00026|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:09:55.403Z|00027|ovsdb_jsonrpc_server|ERR|Dropped 10
>>>>> log messages in last 176 seconds (most recently, 159 seconds ago) due to
>>>>> excessive rate
>>>>> 2017-02-02T11:09:55.403Z|00028|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>>>> 192.168.10.10: listen failed: Address already in use
>>>>> 2017-02-02T11:09:55.403Z|00029|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:09:55.403Z|00030|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>>>> 192.168.10.10: listen failed: Address already in use
>>>>> 2017-02-02T11:09:57.460Z|00031|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:09:57.460Z|00032|ovsdb_jsonrpc_server|ERR|ptcp:6641:
>>>>> 192.168.10.10: listen failed: Address already in use
>>>>> 2017-02-02T11:09:57.460Z|00033|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:09:59.961Z|00034|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:10:02.464Z|00035|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:10:04.967Z|00036|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:10:07.469Z|00037|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:10:09.972Z|00038|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:10:12.475Z|00039|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:10:14.978Z|00040|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>> 2017-02-02T11:10:17.481Z|00041|socket_util|ERR|6641:192.168.10.10:
>>>>> bind: Address already in use
>>>>>
>>>>> -Pranab
>>>>>
>>>>>
>>>>> On 2 February 2017 at 16:36, Numan Siddique <nusiddiq at redhat.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 2, 2017 at 4:00 PM, pranab boruah <
>>>>>> pranabjyotiboruah at gmail.com> wrote:
>>>>>>
>>>>>>> Numan,
>>>>>>> Details as requested:
>>>>>>>
>>>>>>> ___________
>>>>>>> ml2_conf.ini:|
>>>>>>> ```````````````````
>>>>>>> ovn_nb_connection = tcp:192.168.10.10:6641
>>>>>>> ovn_sb_connection = tcp:192.168.10.10:6642
>>>>>>> ovn_l3_mode = True
>>>>>>> ovn_l3_scheduler = chance
>>>>>>> ovn_l3_admin_net_cidr = 169.254.128.0/30
>>>>>>> ovn_native_dhcp = True
>>>>>>>
>>>>>>> Note: 192.168.10.10 is the management ip of controller. Firewall,
>>>>>>> NetworkManager and SElinux are disabled on all nodes.
>>>>>>> ____________________________
>>>>>>> output of "ps -aef | grep ovsdb" : |
>>>>>>> `````````````````````````````````````````````````
>>>>>>> root 21220 21219 0 Feb01 ? 00:00:44 ovsdb-server
>>>>>>> /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info
>>>>>>> --remote=punix:/var/run/openvswitch/db.sock
>>>>>>> --private-key=db:Open_vSwitch,SSL,private_key
>>>>>>> --certificate=db:Open_vSwitch,SSL,certificate
>>>>>>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir
>>>>>>> --log-file=/var/log/openvswitch/ovsdb-server.log
>>>>>>> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
>>>>>>>
>>>>>>> root 21324 21323 0 Feb01 ? 00:00:03 ovsdb-server
>>>>>>> --detach --monitor -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-nb.log
>>>>>>> --remote=punix:/var/run/openvswitch/ovnnb_db.sock
>>>>>>> --pidfile=/var/run/openvswitch/ovnnb_db.pid
>>>>>>> --remote=db:OVN_Northbound,NB_Global,connections
>>>>>>> --unixctl=ovnnb_db.ctl --private-key=db:OVN_Northbound,SSL,private_key
>>>>>>> --certificate=db:OVN_Northbound,SSL,certificate
>>>>>>> --ca-cert=db:OVN_Northbound,SSL,ca_cert /etc/openvswitch/ovnnb_db.db
>>>>>>>
>>>>>>> root 21332 21331 0 Feb01 ? 00:00:03 ovsdb-server
>>>>>>> --detach --monitor -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-sb.log
>>>>>>> --remote=punix:/var/run/openvswitch/ovnsb_db.sock
>>>>>>> --pidfile=/var/run/openvswitch/ovnsb_db.pid
>>>>>>> --remote=db:OVN_Southbound,SB_Global,connections
>>>>>>> --unixctl=ovnsb_db.ctl --private-key=db:OVN_Southbound,SSL,private_key
>>>>>>> --certificate=db:OVN_Southbound,SSL,certificate
>>>>>>> --ca-cert=db:OVN_Southbound,SSL,ca_cert /etc/openvswitch/ovnsb_db.db
>>>>>>>
>>>>>>> Note: NB and SB databases are supposed to be listening on a tcp
>>>>>>> socket. But it isn't.
>>>>>>>
>>>>>>>
>>>>>> Running the below commands should solve your problem
>>>>>> - ovn-nbctl set-connection ptcp:6641:192.168.10.10
>>>>>>
>>>>>> - ovn-
>>>>>> s
>>>>>> bctl set-connection ptcp:664
>>>>>> 2
>>>>>> :192.168.10.10
>>>>>>
>>>>>> L
>>>>>> et me know if you still face problems.
>>>>>>
>>>>>> The reason for this is this commit -
>>>>>> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291
>>>>>> 63236d48fa0f9105687149
>>>>>>
>>>>>> P
>>>>>> lease have a look at it.
>>>>>>
>>>>>> Thanks
>>>>>> Numan
>>>>>>
>>>>>>
>>>>>>
>>>>>>> _________________________
>>>>>>> output of logs |
>>>>>>> ```````````````````````````````````````````
>>>>>>> Neutron server log :
>>>>>>> - http://paste.openstack.org/show/597320/
>>>>>>> ovs-nothd log:
>>>>>>> - http://paste.openstack.org/show/597322/
>>>>>>> ovs-vswitchd log on Controller:
>>>>>>> - http://paste.openstack.org/show/597324/
>>>>>>> _________________________________________
>>>>>>> # ovn-nbctl and ovn-sbctl show no output. |
>>>>>>> ````````````````````````````````````````````````````````````
>>>>>>> ```````````
>>>>>>> ________________________________________
>>>>>>> OVN RPMS installation and download details: |
>>>>>>> ````````````````````````````````````````````````````````````
>>>>>>> ``````````
>>>>>>> On controller:
>>>>>>> # git clone https://github.com/openvswitch/ovs.git
>>>>>>> # ./boot.sh
>>>>>>> # ./configure
>>>>>>> # make rpm-fedora RPMBUILD_OPT="--without check"
>>>>>>> # make rpm-fedora-kmod
>>>>>>>
>>>>>>> ALL nodes:
>>>>>>>
>>>>>>> # rpm -i openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm# rpm -U openvswitch-2.6.90-1.el7.centos.x86_64.rpm
>>>>>>>
>>>>>>> Only Controller:
>>>>>>>
>>>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i openvswitch-ovn-central-*.x86_64.rpm
>>>>>>>
>>>>>>> Only Compute:
>>>>>>>
>>>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i openvswitch-ovn-host-*.x86_64.rpm
>>>>>>>
>>>>>>> # lsmod | grep open
>>>>>>> - shows all the required modules
>>>>>>>
>>>>>>> Let me know if you need anything else.
>>>>>>>
>>>>>>> Thanks.
>>>>>>> pjb
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2 February 2017 at 15:23, Numan Siddique <nusiddiq at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Adding openstack-dev ML with [networking-ovn] tag.
>>>>>>>>
>>>>>>>> Please see comments inline
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 2, 2017 at 3:01 PM, pranab boruah <
>>>>>>>> pranabjyotiboruah at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi guys,
>>>>>>>>> We have a OpenStack multi-node lab. Everything was working, until
>>>>>>>>> I tried integrating with OVN.
>>>>>>>>> Followed the official guide for integration:
>>>>>>>>>
>>>>>>>>> http://docs.openstack.org/developer/networking-ovn/install.html
>>>>>>>>>
>>>>>>>>> Lab details:
>>>>>>>>> 1 Controller, 2 computes
>>>>>>>>>
>>>>>>>>> CentOS-7.3, OpenStack Newton, separate network for mgmt and
>>>>>>>>> tunnel.
>>>>>>>>>
>>>>>>>>> Error details :
>>>>>>>>> Neutron Server log shows :
>>>>>>>>>
>>>>>>>>> ERROR networking_ovn.ovsdb.impl_idl_ovn [-] OVS database
>>>>>>>>> connection to OVN_Northbound failed with error: '{u'error': u'unknown
>>>>>>>>> database', u'details': u'get_schema request specifies unknown database
>>>>>>>>> OVN_Northbound', u'syntax': u'["OVN_Northbound"]'}'. Verify that the OVS
>>>>>>>>> and OVN services are available and that the 'ovn_nb_connection' and
>>>>>>>>> 'ovn_sb_connection' configuration options are correct.
>>>>>>>>>
>>>>>>>>> Stuck in this for more than 2 week. Starting to hate my life.
>>>>>>>>> Please help. Before I tried installing in physical machines, followed the
>>>>>>>>> same guide and installed in VMs. Everything was working then.
>>>>>>>>>
>>>>>>>>> Solutions I tried:
>>>>>>>>> 1) Completely fresh installing everything.
>>>>>>>>> 2) Rebuilding ovs
>>>>>>>>> 3) Check SB and NB configuration options in plugin.ini are exactly
>>>>>>>>> correct.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> Could you please share
>>>>>>>> - the values of ovn_nb_connection and ovn_sb_connection defined in
>>>>>>>> ml2_conf.ini.
>>>>>>>> - the output of "ps -aef | grep ovsdb" on the node where you are
>>>>>>>> running the ovn-northd server.
>>>>>>>> - If you could share the output of neutron-server logs in some
>>>>>>>> paste link, it would be helpful.
>>>>>>>>
>>>>>>>> Are you able to run "ovn-nbctl show" ? I presume you have installed
>>>>>>>> the appropriate ovn RPMS ? If not so, please let me know how you have
>>>>>>>> installed it.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Numan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Let me know if additional details are required.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -pjb
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> discuss mailing list
>>>>>>>>> discuss at openvswitch.org
>>>>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20170202/a0d960b5/attachment-0001.html>
More information about the discuss
mailing list