[ovs-dev] [ovn] transaction error in ovn-controller with configured RBAC on branch-20.06

Tue Dec 8 19:33:10 UTC 2020

On 12/8/20 8:28 PM, Dumitru Ceara wrote:
> On 12/3/20 4:11 PM, Dumitru Ceara wrote:
>> On 12/3/20 2:01 PM, Odintsov Vladislav wrote:
>>> But neither IP nor system-id was changed. I've double-checked:
>>>
>>> ovn-controller 20.06.2:
>>>
>>> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
>>>     hostname: host.local
>>>     Encap vxlan
>>>         ip: "172.24.33.105"
>>>         options: {csum="true"}
>>>     Encap stt
>>>         ip: "172.24.33.105"
>>>         options: {csum="true"}
>>>     Port_Binding eni-3E9901E0
>>>     Port_Binding eni-35AFCD00
>>>
>>> # ovs-vsctl get open . external-ids:system-id
>>> "04540082-b5b5-4ab5-9901-03ed445c772d"
>>>
>>> # systemctl stop ovn-controller
>>>
>>> Chassis was deleted:
>>>
>>> # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d
>>> ovn-sbctl: no row "04540082-b5b5-4ab5-9901-03ed445c772d" in table Chassis
>>>
>>> # yum update ovn-host -y
>>> # systemctl restart ovn-controller
>>>
>>> Chassis with same system-id and encap IPs was re-added:
>>>
>>> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
>>>     hostname: host.local
>>>     Encap vxlan
>>>         ip: "172.24.33.105"
>>>         options: {csum="true"}
>>>     Encap stt
>>>         ip: "172.24.33.105"
>>>         options: {csum="true"}
>>>
>>> But, there are no port_bindings, and in ovn-controller logs again transaction error:
>>>
>>> 2020-12-03T12:53:54.031Z|00035|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
>>> 2020-12-03T12:53:54.031Z|00036|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
>>> 2020-12-03T12:53:54.031Z|00037|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
>>> 2020-12-03T12:53:54.031Z|00038|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
>>> 2020-12-03T12:53:54.041Z|00039|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
>>> 2020-12-03T12:53:54.042Z|00040|main|INFO|OVNSB commit failed, force recompute next time.
>>>
>>>
>>> Moreover, if I forcefully delete chassis, port claim successful, but after restart ovn-controller, promlem appears again:
>>>
>>> # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
>>>
>>> 2020-12-03T12:56:20.119Z|00045|main|INFO|OVNSB commit failed, force recompute next time.
>>> 2020-12-03T12:56:23.803Z|00046|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
>>> 2020-12-03T12:56:23.803Z|00047|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
>>> 2020-12-03T12:56:23.803Z|00048|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
>>> 2020-12-03T12:56:23.803Z|00049|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
>>>
>>> # systemctl restart ovn-controller
>>>
>>> 2020-12-03T12:56:38.590Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller.log
>>> 2020-12-03T12:56:38.592Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
>>> 2020-12-03T12:56:38.592Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
>>> 2020-12-03T12:56:38.596Z|00004|main|INFO|OVS IDL reconnected, force recompute.
>>> 2020-12-03T12:56:38.600Z|00005|reconnect|INFO|ssl:x.x.x.x:6642: connecting...
>>> 2020-12-03T12:56:38.600Z|00006|main|INFO|OVNSB IDL reconnected, force recompute.
>>> 2020-12-03T12:56:38.645Z|00007|reconnect|INFO|ssl:x.x.x.x:6642: connected
>>> 2020-12-03T12:56:38.650Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
>>> 2020-12-03T12:56:38.650Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
>>> 2020-12-03T12:56:38.651Z|00010|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
>>> 2020-12-03T12:56:38.654Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
>>> 2020-12-03T12:56:38.654Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
>>> 2020-12-03T12:56:38.654Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
>>> 2020-12-03T12:56:38.654Z|00012|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
>>> 2020-12-03T12:56:38.654Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
>>> 2020-12-03T12:56:38.654Z|00014|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
>>> 2020-12-03T12:56:38.655Z|00015|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
>>> 2020-12-03T12:56:38.655Z|00016|main|INFO|OVNSB commit failed, force recompute next time.
>>>
>>>
>>> Maybe, I just don’t understand your idea...
>>
>> I see.  I'm pretty sure it's related to this commit that tries to reuse
>> Encaps (and that's wrong because it doesn't work with RBAC):
>>
>> https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3
>>
>> I'll try to fix it and update this thread.
>>
> 
> Hi Vladislav,
> 
> The problem is that branch-20.06 misses the following commit:
> https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3

Oops, this should've been:
https://github.com/ovn-org/ovn/commit/dce1af31b550a9fb57b01cbe0b4139b6768f2521

> 
> However, at Han's suggestion we decided to remove the code that allowed
> ovn-controller to reuse stale chassis records from the SB (because it
> wasn't working properly with RBAC).  At this point I don't think it
> makes sense to backport the missing commit because we'll be just
> reverting it as soon as the new patch is accepted:
> 
> http://patchwork.ozlabs.org/project/ovn/patch/1607455279-21771-1-git-send-email-dceara@redhat.com/
> 
> Once/if the above is accepted, I'll send backport patches for all stable
> branches.
> 
> Thanks,
> Dumitru
>