[ovs-dev] [ovn] transaction error in ovn-controller with configured RBAC on branch-20.06

Dumitru Ceara dceara at redhat.com
Tue Apr 20 10:35:58 UTC 2021


On 4/19/21 7:36 PM, Odintsov Vladislav wrote:
> Hi Dumitru,
> 

Hi Vladislav,

> I've seen your patches have been backported to 20.06 branch and tried it with RBAC-enabled installation. It seems working for ovn-controller, but for ovn-controller-vtep I still see similar errors.
> Should this be fixed in ovn-controller-vtep as well?
> 
> 2021-04-19T17:26:22Z|00824|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"cumulus-01\" role \"ovn-controller\" prohibit row insertion into table \"Encap\".","error":"permission error"}
> 2021-04-19T17:26:22Z|00825|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back
> 2021-04-19T17:26:22Z|00826|gateway|INFO|add Chassis row for VTEP physical switch (cumulus-01)
> 2021-04-19T17:26:27Z|00827|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back
> 2021-04-19T17:26:27Z|00828|gateway|INFO|add Chassis row for VTEP physical switch (cumulus-01)
> 2021-04-19T17:26:32Z|00829|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back
> 
> As workaround, if I switch ovn-controller-vtep to another ovnsbdb port (without rbac engine), ovn-controller-vtep successfully adds chassis record, then I switch it back to rbac socket and continue working well. So, error occurs only on first run of chassis. When chassis exists in DB, things work well.

I sent a patch that should fix the ovn-controller-vtep RBAC too:

http://patchwork.ozlabs.org/project/ovn/patch/20210420103221.1123-1-dceara@redhat.com/

It should apply cleanly on branch-20.06 too, do you mind trying it out
when you get the chance?

Thanks,
Dumitru

> 
> Regards,
>  
> Vladislav Odintsov
> 
> On 09.12.2020, 11:30, "Odintsov Vladislav" <VlOdintsov at croc.ru> wrote:
> 
>     Hi Dumitru,
> 
>     That’s good news, thanks for that!
> 
> 
>     Regards,
> 
>     Vladislav Odintsov
> 
>     On 08.12.2020, 22:33, "Dumitru Ceara" <dceara at redhat.com> wrote:
> 
>         On 12/8/20 8:28 PM, Dumitru Ceara wrote:
>         > On 12/3/20 4:11 PM, Dumitru Ceara wrote:
>         >> On 12/3/20 2:01 PM, Odintsov Vladislav wrote:
>         >>> But neither IP nor system-id was changed. I've double-checked:
>         >>>
>         >>> ovn-controller 20.06.2:
>         >>>
>         >>> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
>         >>>     hostname: host.local
>         >>>     Encap vxlan
>         >>>         ip: "172.24.33.105"
>         >>>         options: {csum="true"}
>         >>>     Encap stt
>         >>>         ip: "172.24.33.105"
>         >>>         options: {csum="true"}
>         >>>     Port_Binding eni-3E9901E0
>         >>>     Port_Binding eni-35AFCD00
>         >>>
>         >>> # ovs-vsctl get open . external-ids:system-id
>         >>> "04540082-b5b5-4ab5-9901-03ed445c772d"
>         >>>
>         >>> # systemctl stop ovn-controller
>         >>>
>         >>> Chassis was deleted:
>         >>>
>         >>> # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d
>         >>> ovn-sbctl: no row "04540082-b5b5-4ab5-9901-03ed445c772d" in table Chassis
>         >>>
>         >>> # yum update ovn-host -y
>         >>> # systemctl restart ovn-controller
>         >>>
>         >>> Chassis with same system-id and encap IPs was re-added:
>         >>>
>         >>> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
>         >>>     hostname: host.local
>         >>>     Encap vxlan
>         >>>         ip: "172.24.33.105"
>         >>>         options: {csum="true"}
>         >>>     Encap stt
>         >>>         ip: "172.24.33.105"
>         >>>         options: {csum="true"}
>         >>>
>         >>> But, there are no port_bindings, and in ovn-controller logs again transaction error:
>         >>>
>         >>> 2020-12-03T12:53:54.031Z|00035|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
>         >>> 2020-12-03T12:53:54.031Z|00036|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
>         >>> 2020-12-03T12:53:54.031Z|00037|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
>         >>> 2020-12-03T12:53:54.031Z|00038|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
>         >>> 2020-12-03T12:53:54.041Z|00039|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
>         >>> 2020-12-03T12:53:54.042Z|00040|main|INFO|OVNSB commit failed, force recompute next time.
>         >>>
>         >>>
>         >>> Moreover, if I forcefully delete chassis, port claim successful, but after restart ovn-controller, promlem appears again:
>         >>>
>         >>> # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
>         >>>
>         >>> 2020-12-03T12:56:20.119Z|00045|main|INFO|OVNSB commit failed, force recompute next time.
>         >>> 2020-12-03T12:56:23.803Z|00046|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
>         >>> 2020-12-03T12:56:23.803Z|00047|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
>         >>> 2020-12-03T12:56:23.803Z|00048|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
>         >>> 2020-12-03T12:56:23.803Z|00049|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
>         >>>
>         >>> # systemctl restart ovn-controller
>         >>>
>         >>> 2020-12-03T12:56:38.590Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller.log
>         >>> 2020-12-03T12:56:38.592Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
>         >>> 2020-12-03T12:56:38.592Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
>         >>> 2020-12-03T12:56:38.596Z|00004|main|INFO|OVS IDL reconnected, force recompute.
>         >>> 2020-12-03T12:56:38.600Z|00005|reconnect|INFO|ssl:x.x.x.x:6642: connecting...
>         >>> 2020-12-03T12:56:38.600Z|00006|main|INFO|OVNSB IDL reconnected, force recompute.
>         >>> 2020-12-03T12:56:38.645Z|00007|reconnect|INFO|ssl:x.x.x.x:6642: connected
>         >>> 2020-12-03T12:56:38.650Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
>         >>> 2020-12-03T12:56:38.650Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
>         >>> 2020-12-03T12:56:38.651Z|00010|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
>         >>> 2020-12-03T12:56:38.654Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
>         >>> 2020-12-03T12:56:38.654Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
>         >>> 2020-12-03T12:56:38.654Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
>         >>> 2020-12-03T12:56:38.654Z|00012|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
>         >>> 2020-12-03T12:56:38.654Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
>         >>> 2020-12-03T12:56:38.654Z|00014|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
>         >>> 2020-12-03T12:56:38.655Z|00015|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
>         >>> 2020-12-03T12:56:38.655Z|00016|main|INFO|OVNSB commit failed, force recompute next time.
>         >>>
>         >>>
>         >>> Maybe, I just don’t understand your idea...
>         >>
>         >> I see.  I'm pretty sure it's related to this commit that tries to reuse
>         >> Encaps (and that's wrong because it doesn't work with RBAC):
>         >>
>         >> https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3
>         >>
>         >> I'll try to fix it and update this thread.
>         >>
>         > 
>         > Hi Vladislav,
>         > 
>         > The problem is that branch-20.06 misses the following commit:
>         > https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3
> 
>         Oops, this should've been:
>         https://github.com/ovn-org/ovn/commit/dce1af31b550a9fb57b01cbe0b4139b6768f2521
> 
>         > 
>         > However, at Han's suggestion we decided to remove the code that allowed
>         > ovn-controller to reuse stale chassis records from the SB (because it
>         > wasn't working properly with RBAC).  At this point I don't think it
>         > makes sense to backport the missing commit because we'll be just
>         > reverting it as soon as the new patch is accepted:
>         > 
>         > http://patchwork.ozlabs.org/project/ovn/patch/1607455279-21771-1-git-send-email-dceara@redhat.com/
>         > 
>         > Once/if the above is accepted, I'll send backport patches for all stable
>         > branches.
>         > 
>         > Thanks,
>         > Dumitru
>         > 
> 
> 
> 
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 



More information about the dev mailing list