[ovs-dev] Clustered DB commits causes sandbox errors

Mark Michelson mmichels at redhat.com
Wed Mar 28 15:06:05 UTC 2018


Thanks for figuring this out! I tested with this patch and all works as 
expected in the sandbox.

On 03/27/2018 05:34 PM, aginwala wrote:
> Hi Mark:
> 
> 
> The thing is clustering db uses new sockets nb1.ovsdb and sb1.ovsdb. 
> However, northd was still trying to use old ovnsb_db.sock and 
> ovnnb_db.sock .
> 
> I was able to fix the issue as per below patch
> 
> diff --git a/tutorial/ovs-sandbox b/tutorial/ovs-sandbox
> index babc032..c3e9f12 100755
> --- a/tutorial/ovs-sandbox
> +++ b/tutorial/ovs-sandbox
> @@ -510,8 +510,8 @@ if $ovn; then
>       fi
>       rungdb $gdb_ovn_northd $gdb_ovn_northd_ex ovn-northd --detach \
>           --no-chdir --pidfile -vconsole:off --log-file \
> -        --ovnsb-db=unix:"$sandbox"/ovnsb_db.sock \
> -        --ovnnb-db=unix:"$sandbox"/ovnnb_db.sock
> +        --ovnsb-db=unix:"$sandbox"/sb1.ovsdb \
> +        --ovnnb-db=unix:"$sandbox"/nb1.ovsdb
> 
> test:
> 
> #/ovs/tutorial/sandbox# ovn-nbctl --wait=hv sync
> #/ovs/tutorial/sandbox#
> #/ovs/tutorial/sandbox#ovn-nbctl --wait=hv ls-add ls1
> #/ovs/tutorial/sandbox# ovn-nbctl --wait=hv ls-del ls1
> #/ovs/tutorial/sandbox
> 
> Will submit the formal patch and add --wait=hv in ovn-setup.sh so that 
> its easier to trace this condition too to ensure northd is always 
> working fine.
> 
> 
> Let me know if someone else has any comments.
> 
> 
> Regards,
> 
> 
> On Tue, Mar 27, 2018 at 10:34 AM, aginwala <aginwala at asu.edu 
> <mailto:aginwala at asu.edu>> wrote:
> 
>     Hi Mark:
> 
>     I did reset to HEAD~6 and the sandbox still crashes . So took
>     commit: cb8cbbbe97b56401c399fa261b9670eb1698bf14 that Han recently
>     used to rebase his patches and it works fine. So the diff is
>     somewhere from this commit to the master.
> 
> 
>     Also, I noticed one thing that if we are using ssl by default for
>     sandbox, it's better to configure northd to use ssl too by generating :
>     e.g. I restarted northd as below in the current master sandbox and
>     it worked fine:
>     ovn-northd --detach --no-chdir --pidfile -vconsole:off --log-file
>     --ovnsb-db=ssl:127.0.0.1:6642 <http://127.0.0.1:6642>
>     --ovnnb-db=ssl:127.0.0.1:6641 <http://127.0.0.1:6641> -p
>     /root/ovs/tutorial/sandbox/chassis-1-privkey.pem -c
>     /root/ovs/tutorial/sandbox/chassis-1-cert.pem -C
>     /root/ovs/tutorial/sandbox/pki/switchca/cacert.pem
> 
>     Will try to see if I can get the fix real quick.  Ofcourse the diff
>     is super big! :)
> 
>     On Mon, Mar 26, 2018 at 2:42 PM, Mark Michelson <mmichels at redhat.com
>     <mailto:mmichels at redhat.com>> wrote:
> 
>         Hi,
> 
>         I'm currently on the master branch of OVS, commit "1b1d2e6da
>         ovsdb: Introduce experimental support for clustered databases."
>         I started the OVS sandbox using `make sandbox
>         SANDBOXFLAGS="--ovn"` . I tried to run some tests to add some
>         logical switch ports to OVN. Running `ovn-nbctl --wait=hv
>         lsp-add ls0 lsp0` blocks forever. I found that ovn-northd.log
>         was peppered with lines like the following:
> 
>         2018-03-26T21:21:06.509Z|00018|reconnect|INFO|unix:/home/putnopvut/ovs/tutorial/sandbox/ovnnb_db.sock:
>         connecting...
>         2018-03-26T21:21:06.509Z|00019|reconnect|INFO|unix:/home/putnopvut/ovs/tutorial/sandbox/ovnnb_db.sock:
>         connection attempt failed (No such file or directory)
>         2018-03-26T21:21:06.509Z|00020|reconnect|INFO|unix:/home/putnopvut/ovs/tutorial/sandbox/ovnnb_db.sock:
>         continuing to reconnect in the background but suppressing
>         further logging
>         2018-03-26T21:21:06.509Z|00021|reconnect|INFO|unix:/home/putnopvut/ovs/tutorial/sandbox/ovnsb_db.sock:
>         connecting...
>         2018-03-26T21:21:06.509Z|00022|reconnect|INFO|unix:/home/putnopvut/ovs/tutorial/sandbox/ovnsb_db.sock:
>         connection attempt failed (No such file or directory)
>         2018-03-26T21:21:06.509Z|00023|reconnect|INFO|unix:/home/putnopvut/ovs/tutorial/sandbox/ovnsb_db.sock:
>         continuing to reconnect in the background but suppressing
>         further logging
> 
>         And ovn-controller.log has lines like:
> 
>         2018-03-26T21:21:00.202Z|00021|rconn|INFO|unix:/home/putnopvut/ovs/tutorial/sandbox/br-int.mgmt:
>         connected
>         2018-03-26T21:21:00.203Z|00022|ovsdb_idl|WARN|transaction error:
>         {"details":"RBAC rules for client \"chassis-1\" role
>         \"ovn-controller\" prohibit row insertion into table
>         \"Encap\".","error":"permission error"}
> 
>         I attempted to bisect to see what commit introduced the problem,
>         but I ran into problems here, too. If I revert to HEAD~6
>         (077f03028 jsonrpc-server: Separate changing read_only status
>         from reconnecting.), then the ovs-sandbox works as expected. If
>         I revert to HEAD~5, HEAD~4, HEAD~3, HEAD~2, or HEAD~, I hit a
>         compilation error:
> 
>         In file included from lib/ovsdb-idl.c:45:0:
>         lib/ovsdb-idl.c: In function ‘ovsdb_idl_send_monitor_request’:
>         lib/ovsdb-idl.c:1638:34: error: ‘struct ovsdb_idl’ has no member
>         named ‘class_’
>                                         idl->class_->database,
>         column->name);
>                                            ^
>         ./include/openvswitch/vlog.h:271:41: note: in definition of
>         macro ‘VLOG’
>                       vlog(&this_module, level__, __VA_ARGS__);   \
>                                                   ^~~~~~~~~~~
>         lib/ovsdb-idl.c:1636:21: note: in expansion of macro ‘VLOG_WARN’
>                               VLOG_WARN("%s table in %s database has
>         synthetic "
>                               ^~~~~~~~~
> 
>         Unfortunately, I have a 6 commit range where the error may have
>         been introduced. I would love to have submitted a patch to fix
>         this, but I don't have much more time left today to work on
>         this, I'm off tomorrow, and the diff between HEAD~6 and HEAD is
>         massive.
> 
>         Mark!
>         _______________________________________________
>         dev mailing list
>         dev at openvswitch.org <mailto:dev at openvswitch.org>
>         https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>         <https://mail.openvswitch.org/mailman/listinfo/ovs-dev>
> 
> 
> 



More information about the dev mailing list