[ovs-discuss] SSL setup issue with OVN

Satish Patel satish.txt at gmail.com
Fri Nov 19 20:47:36 UTC 2021


Update:

Looks like the issue is related to communication between NB and SB
over SSL. looking at logs

root at ovn-lab-infra-1-neutron-ovn-northd-container-cb55f5ef:~# tail -f
/var/log/ovn/ovsdb-server-nb.log
2021-11-19T20:39:05.210Z|00352|reconnect|WARN|ssl:10.62.7.252:46988:
connection dropped (Protocol error)
2021-11-19T20:39:13.218Z|00353|stream_ssl|WARN|SSL_accept: system
error (Success)
2021-11-19T20:39:13.218Z|00354|jsonrpc|WARN|Dropped 1 log messages in
last 8 seconds (most recently, 8 seconds ago) due to excessive rate
2021-11-19T20:39:13.218Z|00355|jsonrpc|WARN|ssl:10.62.7.252:47050:
receive error: Protocol error
2021-11-19T20:39:13.218Z|00356|reconnect|WARN|ssl:10.62.7.252:47050:
connection dropped (Protocol error)
2021-11-19T20:39:21.226Z|00357|stream_ssl|WARN|SSL_accept: system
error (Success)
2021-11-19T20:39:21.227Z|00358|jsonrpc|WARN|ssl:10.62.7.252:47076:
receive error: Protocol error
2021-11-19T20:39:21.227Z|00359|reconnect|WARN|ssl:10.62.7.252:47076:
connection dropped (Protocol error)
2021-11-19T20:39:29.235Z|00360|stream_ssl|WARN|SSL_accept: system
error (Success)
2021-11-19T20:39:29.235Z|00361|reconnect|WARN|ssl:10.62.7.252:47096:
connection dropped (Protocol error)
2021-11-19T20:39:37.243Z|00362|stream_ssl|WARN|SSL_accept: system
error (Success)
2021-11-19T20:39:37.243Z|00363|jsonrpc|WARN|Dropped 1 log messages in
last 8 seconds (most recently, 8 seconds ago) due to excessive rate


root at ovn-lab-infra-1-neutron-ovn-northd-container-cb55f5ef:/etc/default#
tail -f /var/log/ovn/ovn-northd.log
2021-11-19T20:43:45.494Z|00752|stream_ssl|ERR|CA certificate must be
configured to use SSL
2021-11-19T20:43:45.494Z|00753|stream_ssl|ERR|Private key must be
configured to use SSL
2021-11-19T20:43:45.494Z|00754|stream_ssl|ERR|Certificate must be
configured to use SSL
2021-11-19T20:43:45.494Z|00755|stream_ssl|ERR|CA certificate must be
configured to use SSL
2021-11-19T20:43:53.503Z|00756|stream_ssl|ERR|Private key must be
configured to use SSL
2021-11-19T20:43:53.503Z|00757|stream_ssl|ERR|Certificate must be
configured to use SSL
2021-11-19T20:43:53.503Z|00758|stream_ssl|ERR|CA certificate must be
configured to use SSL
2021-11-19T20:43:53.503Z|00759|stream_ssl|ERR|Private key must be
configured to use SSL
2021-11-19T20:43:53.503Z|00760|stream_ssl|ERR|Certificate must be
configured to use SSL
2021-11-19T20:43:53.503Z|00761|stream_ssl|ERR|CA certificate must be
configured to use SSL


I have the following config on ovn-northd central server for my
cluster definition. Currently I have a single node but this is the
place I add more nodes to the scale cluster.  If I delete ovn-central
file then everything works. NB starts talking to SB and all errors
disappear. as per ovn-northd.log its saying configure SSL so i added
SSL options in /etc/default/ovn-central file but that didn't help i am
still seeing error

File: /etc/default/ovn-central

# OVN cluster parameters
OVN_CTL_OPTS=" \
  --db-nb-create-insecure-remote=yes \
  --db-sb-create-insecure-remote=yes \
  --db-nb-addr=10.62.7.252 \
  --db-sb-addr=10.62.7.252 \
  --db-nb-cluster-local-addr=10.62.7.252 \
  --db-sb-cluster-local-addr=10.62.7.252 \
  --ovn-northd-nb-db=ssl:10.62.7.252:6641 \
  --ovn-northd-sb-db=ssl:10.62.7.252:6642 \
"

I tried following to pass ssl options but didn't help

# OVN cluster parameters
OVN_CTL_OPTS=" \
  --db-nb-create-insecure-remote=no \
  --db-sb-create-insecure-remote=no \
  --db-nb-addr=10.62.7.252 \
  --db-sb-addr=10.62.7.252 \
  --db-nb-cluster-local-addr=10.62.7.252 \
  --db-sb-cluster-local-addr=10.62.7.252 \
    --ovn-northd-nb-db=ssl:10.62.7.252:6641 \
  --ovn-northd-sb-db=ssl:10.62.7.252:6642 \
  --ovn-nb-db-ssl-key=/etc/openvswitch/ovnnb-privkey.pem \
  --ovn-nb-db-ssl-cert=/etc/openvswitch/ovnnb-cert.pem \
  --ovn-nb-db-ssl-ca-cert=/etc/openvswitch/cacert.pem \
  --ovn-sb-db-ssl-key=/etc/openvswitch/ovnsb-privkey.pem \
  --ovn-sb-db-ssl-cert=/etc/openvswitch/ovnsb-cert.pem \
  --ovn-sb-db-ssl-ca-cert=/etc/openvswitch/cacert.pem \
"

What does the following error mean and how do I configure SSL with
Raft cluster as i mentioned above?

2021-11-19T20:43:45.494Z|00752|stream_ssl|ERR|CA certificate must be
configured to use SSL
2021-11-19T20:43:45.494Z|00753|stream_ssl|ERR|Private key must be
configured to use SSL
2021-11-19T20:43:45.494Z|00754|stream_ssl|ERR|Certificate must be
configured to use SSL

On Fri, Nov 19, 2021 at 12:27 AM Satish Patel <satish.txt at gmail.com> wrote:
>
> Folks,
>
> Reference doc: https://github.com/ovn-org/ovn-kubernetes/blob/master/docs/INSTALL.SSL.md
>
> I am getting the following error in ovn-controller logs and not sure
> how to debug to get more details.
>
> 2021-11-19T04:59:35.502Z|00014|stream_ssl|WARN|SSL_connect: system
> error (Success)
> 2021-11-19T04:59:35.502Z|00015|reconnect|INFO|ssl:10.62.7.252:6642:
> connection attempt failed (Protocol error)
> 2021-11-19T04:59:35.502Z|00016|reconnect|INFO|ssl:10.62.7.252:6642:
> waiting 4 seconds before reconnect
> 2021-11-19T05:12:12.355Z|00114|stream_ssl|WARN|SSL_connect: system
> error (Success)
> 2021-11-19T05:12:20.363Z|00115|stream_ssl|WARN|SSL_connect: system
> error (Success)
> 2021-11-19T05:12:28.371Z|00116|stream_ssl|WARN|SSL_connect: system
> error (Success)
> 2021-11-19T05:12:36.381Z|00117|stream_ssl|WARN|SSL_connect: system
> error (Success)
> 2021-11-19T05:12:44.390Z|00118|stream_ssl|WARN|SSL_connect: system
> error (Success)
> 2021-11-19T05:12:52.395Z|00119|stream_ssl|WARN|SSL_connect: system
> error (Success)
>
>
> I have doubts on CN common name  of SSL cert which does not match my
> case with ovs hostname. In ovn-controller certificate i have
> "DNS:ovn-lab-comp-gen-1 id:4bbe9550-733f-414b-8602-ff97b4bd6780".
>
> But on ovs i have external_ids        :
> {hostname=ovn-lab-comp-gen-1.example.net,
>
> I have used ovs-pki to generate all certificate
>
> This is what my config looks like
>
> On Master node (my ovn-northd server running on NB/SB central
> services, i have following certs setup)
>
> root at ovn-lab-infra-1-neutron-ovn-northd-container-cb55f5ef:~# ovn-nbctl get-ssl
> Private key: /etc/openvswitch/ovncert/ovnnb-privkey.pem
> Certificate: /etc/openvswitch/ovncert/ovnnb-cert.pem
> CA Certificate: /etc/openvswitch/cacert.pem
> Bootstrap: false
>
> root at ovn-lab-infra-1-neutron-ovn-northd-container-cb55f5ef:~# ovn-sbctl get-ssl
> Private key: /etc/openvswitch/ovncert/ovnsb-privkey.pem
> Certificate: /etc/openvswitch/ovncert/ovnsb-cert.pem
> CA Certificate: /etc/openvswitch/cacert.pem
> Bootstrap: false
>
> Here is my connection info which is set to pssl
>
> root at ovn-lab-infra-1-neutron-ovn-northd-container-cb55f5ef:~#
> ovn-nbctl get-connection
> pssl:6641
> root at ovn-lab-infra-1-neutron-ovn-northd-container-cb55f5ef:~#
> ovn-sbctl get-connection
> read-write role="" ptcp:6642
>
>
> On my compute nodes (ovn-controller)
>
> root at ovn-lab-comp-gen-1:~# ovs-vsctl get-ssl
> Private key: /etc/openvswitch/ovncert/ovn-lab-comp-gen-1-privkey.pem
> Certificate: /etc/openvswitch/ovncert/ovn-lab-comp-gen-1-cert.pem
> CA Certificate: /etc/openvswitch/cacert.pem
> Bootstrap: false
>
> File:  /etc/default/ovn-host
>
> OVN_CTL_OPTS="--ovn-controller-ssl-key=/etc/openvswitch/ovncert/ovn-lab-comp-gen-1-privkey.pem
> --ovn-controller-ssl-cert=/etc/openvswitch/ovncert/ovn-lab-comp-gen-1-cert.pem
> --ovn-controller-ssl-ca-cert=/etc/openvswitch/cacert.pem"
>
> File: /etc/neutron/plugins/ml2/ml2_conf.ini
>
> [ovn]
> ovn_native_dhcp = True
> ovn_nb_connection = ssl:10.62.7.252:6641
> ovn_sb_connection = ssl:10.62.7.252:6642
> ovn_l3_scheduler = leastloaded
> ovn_metadata_enabled = True
> ovn_sb_ca_cert="/etc/openvswitch/cacert.pem"
> ovn_sb_certificate="/etc/openvswitch/ovncert/ovn-lab-comp-gen-1-cert.pem"
> ovn_sb_private_key="/etc/openvswitch/ovncert/ovn-lab-comp-gen-1-privkey.pem"
> ovn_nb_ca_cert="/etc/openvswitch/cacert.pem"
> ovn_nb_certificate="/etc/openvswitch/ovncert/ovn-lab-comp-gen-1-cert.pem"
> ovn_nb_private_key="/etc/openvswitch/ovncert/ovn-lab-comp-gen-1-privkey.pem"


More information about the discuss mailing list