[ovs-dev] [ovn] transaction error in ovn-controller with configured RBAC on branch-20.06
Odintsov Vladislav
VlOdintsov at croc.ru
Thu Dec 3 13:01:32 UTC 2020
But neither IP nor system-id was changed. I've double-checked:
ovn-controller 20.06.2:
Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
hostname: host.local
Encap vxlan
ip: "172.24.33.105"
options: {csum="true"}
Encap stt
ip: "172.24.33.105"
options: {csum="true"}
Port_Binding eni-3E9901E0
Port_Binding eni-35AFCD00
# ovs-vsctl get open . external-ids:system-id
"04540082-b5b5-4ab5-9901-03ed445c772d"
# systemctl stop ovn-controller
Chassis was deleted:
# ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d
ovn-sbctl: no row "04540082-b5b5-4ab5-9901-03ed445c772d" in table Chassis
# yum update ovn-host -y
# systemctl restart ovn-controller
Chassis with same system-id and encap IPs was re-added:
Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
hostname: host.local
Encap vxlan
ip: "172.24.33.105"
options: {csum="true"}
Encap stt
ip: "172.24.33.105"
options: {csum="true"}
But, there are no port_bindings, and in ovn-controller logs again transaction error:
2020-12-03T12:53:54.031Z|00035|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
2020-12-03T12:53:54.031Z|00036|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
2020-12-03T12:53:54.031Z|00037|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
2020-12-03T12:53:54.031Z|00038|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
2020-12-03T12:53:54.041Z|00039|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
2020-12-03T12:53:54.042Z|00040|main|INFO|OVNSB commit failed, force recompute next time.
Moreover, if I forcefully delete chassis, port claim successful, but after restart ovn-controller, promlem appears again:
# ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
2020-12-03T12:56:20.119Z|00045|main|INFO|OVNSB commit failed, force recompute next time.
2020-12-03T12:56:23.803Z|00046|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
2020-12-03T12:56:23.803Z|00047|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
2020-12-03T12:56:23.803Z|00048|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
2020-12-03T12:56:23.803Z|00049|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
# systemctl restart ovn-controller
2020-12-03T12:56:38.590Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller.log
2020-12-03T12:56:38.592Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
2020-12-03T12:56:38.592Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
2020-12-03T12:56:38.596Z|00004|main|INFO|OVS IDL reconnected, force recompute.
2020-12-03T12:56:38.600Z|00005|reconnect|INFO|ssl:x.x.x.x:6642: connecting...
2020-12-03T12:56:38.600Z|00006|main|INFO|OVNSB IDL reconnected, force recompute.
2020-12-03T12:56:38.645Z|00007|reconnect|INFO|ssl:x.x.x.x:6642: connected
2020-12-03T12:56:38.650Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
2020-12-03T12:56:38.650Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-12-03T12:56:38.651Z|00010|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
2020-12-03T12:56:38.654Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
2020-12-03T12:56:38.654Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
2020-12-03T12:56:38.654Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
2020-12-03T12:56:38.654Z|00012|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
2020-12-03T12:56:38.654Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
2020-12-03T12:56:38.654Z|00014|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
2020-12-03T12:56:38.655Z|00015|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
2020-12-03T12:56:38.655Z|00016|main|INFO|OVNSB commit failed, force recompute next time.
Maybe, I just don’t understand your idea...
Regards,
Vladislav Odintsov
On 03.12.2020, 15:38, "Dumitru Ceara" <dceara at redhat.com> wrote:
Sorry, I removed the list by accident, readding ovs-dev.
On 12/3/20 1:23 PM, Odintsov Vladislav wrote:
> Hi Dumitru,
>
> This helped!
>
> Chassis destroyed, and port successfully claimed:
>
> # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
>
> 2020-12-03T12:20:41.222Z|6550427|main|INFO|OVNSB commit failed, force recompute next time.
> 2020-12-03T12:20:42.922Z|6550428|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
> 2020-12-03T12:20:42.922Z|6550429|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
> 2020-12-03T12:20:42.922Z|6550430|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
> 2020-12-03T12:20:42.922Z|6550431|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
>
>
> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
> hostname: host.local
> Encap vxlan
> ip: "X.X.X.X"
> options: {csum="true"}
> Encap stt
> ip: "X.X.X.X"
> options: {csum="true"}
> Port_Binding eni-3E9901E0
> Port_Binding eni-35AFCD00
>
>
> But I don't understand what could go wrong? Problem appears right after fresh deploy.
I guess the system-id of the chassis changed but the IP didn't. This is
an issue that should be documented as the CMS should clear the stale
chassis entries if RBAC is enabled [0].
Regards,
Dumitru
[0]
https://mail.openvswitch.org/pipermail/ovs-dev/2020-September/374653.html
>
>
> Regards,
>
> Vladislav Odintsov
> Lead System Engineer at Croc Cloud Development Team
>
> On 03.12.2020, 15:15, "Dumitru Ceara" <dceara at redhat.com> wrote:
>
> On 12/3/20 11:50 AM, Odintsov Vladislav wrote:
> > Hi,
> >
> > It seems, that I see regression with claim port functionality in my OVN installation between v20.06.2 and latest branch-20.06 (78174ea) on cluster with enabled rbac.
>
> Hi Vladislav,
>
> >
> > On v20.06.2 ovn-controller successfully claims port:
> >
> > # ovn-controller --version
> > ovn-controller 20.06.2
> > Open vSwitch Library 2.13.0
> > OpenFlow versions 0x6:0x6
> >
> >
> > 2020-12-02T18:25:28.787Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
> > 2020-12-02T18:25:28.787Z|00012|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
> > 2020-12-02T18:25:28.787Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
> > 2020-12-02T18:25:28.787Z|00014|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
> >
> >
> > Transaction request:
> >
> > 2020-12-02T18:50:36.128Z|01605|jsonrpc|DBG|ssl:X.X.X.X:6642: send request, method="transact", params=["OVN_Southbound",{"where":[["_uuid","==",["uuid","4e9bd54c-f083-45cd-93d3-a65f4d20d688"]]],"row":{"chassis":["uuid","9d414bfc-da12-487e-80a0-5c1f2a98a05a"]},"op":"update","table":"Port_Binding"}], id=310
> >
> > # ovn-sbctl show | grep 04540082-b5b5-4ab5-9901-03ed445c772d -A 9
> > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
> > hostname: host.local
> > Encap vxlan
> > ip: "Y.Y.Y.Y"
> > options: {csum="true"}
> > Encap stt
> > ip: "Y.Y.Y.Y"
> > options: {csum="true"}
> > Port_Binding eni-3E9901E0
> > Port_Binding eni-35AFCD00
> >
> >
> > Then I run update OVN (doesn’t matter only ovn controller or full ovn installation):
> >
> > # ovn-controller --version
> > ovn-controller 20.06.3
> > Open vSwitch Library 2.13.0
> > OpenFlow versions 0x6:0x6
> >
> > ovn-controller is unable to claim lport:
> >
> > 2020-12-02T18:53:35.309Z|00043|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
> > 2020-12-02T18:53:35.309Z|00044|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
> > 2020-12-02T18:53:35.309Z|00045|binding|INFO|Claiming lport eni-DB28C420 for this chassis.
> > 2020-12-02T18:53:35.309Z|00046|binding|INFO|eni-DB28C420: Claiming 0a:00:db:28:c4:20 192.168.0.6
> > 2020-12-02T18:53:35.309Z|00047|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
> > 2020-12-02T18:53:35.309Z|00048|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
> > 2020-12-02T18:53:35.345Z|00049|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
> > 2020-12-02T18:53:35.345Z|00050|main|INFO|OVNSB commit failed, force recompute next time.
> >
> >
> > Transaction request (added encap table modification comparing to previous version):
> >
> > 2020-12-02T18:57:33.661Z|20500|jsonrpc|DBG|ssl:X.X.X.X:6642: send request, method="transact", params=["OVN_Southbound",{"where":[["_uuid","==",["uuid","9a397740-4072-4853-9b75-9cc120fe4b34"]]],"row":{"chassis":["uuid","e1436af9-4a15-4480-937e-7584e64033a3"]},"op":"update","table":"Port_Binding"},{"where":[["_uuid","==",["uuid","7ec10f55-c89a-4fd3-a2ab-8ac22f845c85"]]],"row":{"chassis_name":"04540082-b5b5-4ab5-9901-03ed445c772d"},"op":"update","table":"Encap"},{"where":[["_uuid","==",["uuid","256d47ca-ef69-4d75-b967-7ab19bd413a7"]]],"row":{"chassis_name":"04540082-b5b5-4ab5-9901-03ed445c772d"},"op":"update","table":"Encap"},{"where":[["_uuid","==",["uuid","34856b67-7f15-44d3-8071-e20ae0f6029f"]]],"row":{"chassis":["uuid","e1436af9-4a15-4480-937e-7584e64033a3"]},"op":"update","table":"Port_Binding"}], id=113
> >
> >
> > I’ve configured rbac consulting with this instruction: https://docs.ovn.org/en/latest/tutorials/ovn-rbac.html
> >
> > Some rbac-related parameters:
> >
> > # ovn-sbctl list connection
> > _uuid : 4940feb2-c4ae-47d9-ade7-6f25c26a2a71
> > external_ids : {}
> > inactivity_probe : []
> > is_connected : false
> > max_backoff : []
> > other_config : {}
> > read_only : false
> > role : ""
> > status : {}
> > target : "pssl:16642"
> >
> > _uuid : ed9366ef-e352-4210-998f-655f648d638d
> > external_ids : {}
> > inactivity_probe : []
> > is_connected : false
> > max_backoff : []
> > other_config : {}
> > read_only : false
> > role : ovn-controller
> > status : {}
> > target : "pssl:6642"
> > # ovn-sbctl list rbac_role
> > _uuid : 91e9fee1-4aff-4d94-93bf-d4c5119a0dd2
> > name : ovn-controller
> > permissions : {Chassis=4a0070bf-1327-4c4d-a7be-83cf91fa1e42, Encap=91da95b4-4eaf-4659-b803-789c72ea3fad, MAC_Binding=660466ef-f0f0-4e58-8be1-a6d16a640ef9, Port_Binding=046836f0-caf1-4d22-88b3-a1d9562d2b58, Service_Monitor=dabca251-6c8e-4953-8769-88f687285a60}
> > # ovn-sbctl list rbac_permission
> > _uuid : 91da95b4-4eaf-4659-b803-789c72ea3fad
> > authorization : [chassis_name]
> > insert_delete : true
> > table : Encap
> > update : [ip, options, type]
> >
> > _uuid : 046836f0-caf1-4d22-88b3-a1d9562d2b58
> > authorization : [""]
> > insert_delete : false
> > table : Port_Binding
> > update : [chassis]
> >
> > _uuid : dabca251-6c8e-4953-8769-88f687285a60
> > authorization : [""]
> > insert_delete : false
> > table : Service_Monitor
> > update : [status]
> >
> > _uuid : 660466ef-f0f0-4e58-8be1-a6d16a640ef9
> > authorization : [""]
> > insert_delete : true
> > table : MAC_Binding
> > update : [datapath, ip, logical_port, mac]
> >
> > _uuid : 4a0070bf-1327-4c4d-a7be-83cf91fa1e42
> > authorization : [name]
> > insert_delete : true
> > table : Chassis
> > update : [encaps, external_ids, nb_cfg, other_config, vtep_logical_switches]
> >
> >
> > # ovs-vsctl get open . external-ids:system-id
> > "04540082-b5b5-4ab5-9901-03ed445c772d"
> > # ovs-vsctl get-ssl
> > Private key: /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-privkey.pem
> > Certificate: /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-cert.pem
> > CA Certificate: /var/lib/openvswitch/pki/host/cacert.pem
> > Bootstrap: false
> > # openssl x509 -noout -subject -in /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-cert.pem
> > subject= /C=US/ST=CA/O=Open vSwitch/OU=Open vSwitch certifier/CN=04540082-b5b5-4ab5-9901-03ed445c772d
> >
> > # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d
> > _uuid : e1436af9-4a15-4480-937e-7584e64033a3
> > encaps : [256d47ca-ef69-4d75-b967-7ab19bd413a7, 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85]
> > external_ids : {datapath-type="", iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options="", ovn-monitor-all="false"}
> > hostname : host.local
> > name : "04540082-b5b5-4ab5-9901-03ed445c772d"
> > nb_cfg : 0
> > other_config : {datapath-type="", iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options="", ovn-monitor-all="false"}
> > transport_zones : []
> > vtep_logical_switches: []
> >
> > # ovn-sbctl list encap 256d47ca-ef69-4d75-b967-7ab19bd413a7
> > _uuid : 256d47ca-ef69-4d75-b967-7ab19bd413a7
> > chassis_name : "04540082-b5b5-4ab5-9901-03ed445c772d"
> > ip : "Y.Y.Y.Y"
> > options : {csum="true"}
> > type : stt
> > # ovn-sbctl list encap 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85
> > _uuid : 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85
> > chassis_name : "04540082-b5b5-4ab5-9901-03ed445c772d"
> > ip : "Y.Y.Y.Y"
> > options : {csum="true"}
> > type : vxlan
> >
> > Can anybody point me what could go wrong?
> > Am I missing something?
> >
>
> Could you please try to force ovn-controller to recreate the
> Chassis/Chassis_private records after the update? Something like the
> following for a chassis with name=04540082-b5b5-4ab5-9901-03ed445c772d:
>
> ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
> ovn-sbctl destroy chassis_private 04540082-b5b5-4ab5-9901-03ed445c772d
>
> Thanks,
> Dumitru
>
> >
> > Regards,
> >
> > Vladislav Odintsov
> >
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
>
>
More information about the dev
mailing list