[ovs-dev] [ovn] transaction error in ovn-controller with configured RBAC on branch-20.06

Odintsov Vladislav VlOdintsov at croc.ru
Mon Apr 19 17:36:23 UTC 2021


Hi Dumitru,

I've seen your patches have been backported to 20.06 branch and tried it with RBAC-enabled installation. It seems working for ovn-controller, but for ovn-controller-vtep I still see similar errors.
Should this be fixed in ovn-controller-vtep as well?

2021-04-19T17:26:22Z|00824|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"cumulus-01\" role \"ovn-controller\" prohibit row insertion into table \"Encap\".","error":"permission error"}
2021-04-19T17:26:22Z|00825|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back
2021-04-19T17:26:22Z|00826|gateway|INFO|add Chassis row for VTEP physical switch (cumulus-01)
2021-04-19T17:26:27Z|00827|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back
2021-04-19T17:26:27Z|00828|gateway|INFO|add Chassis row for VTEP physical switch (cumulus-01)
2021-04-19T17:26:32Z|00829|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back

As workaround, if I switch ovn-controller-vtep to another ovnsbdb port (without rbac engine), ovn-controller-vtep successfully adds chassis record, then I switch it back to rbac socket and continue working well. So, error occurs only on first run of chassis. When chassis exists in DB, things work well.

Regards,
 
Vladislav Odintsov

On 09.12.2020, 11:30, "Odintsov Vladislav" <VlOdintsov at croc.ru> wrote:

    Hi Dumitru,

    That’s good news, thanks for that!


    Regards,

    Vladislav Odintsov

    On 08.12.2020, 22:33, "Dumitru Ceara" <dceara at redhat.com> wrote:

        On 12/8/20 8:28 PM, Dumitru Ceara wrote:
        > On 12/3/20 4:11 PM, Dumitru Ceara wrote:
        >> On 12/3/20 2:01 PM, Odintsov Vladislav wrote:
        >>> But neither IP nor system-id was changed. I've double-checked:
        >>>
        >>> ovn-controller 20.06.2:
        >>>
        >>> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
        >>>     hostname: host.local
        >>>     Encap vxlan
        >>>         ip: "172.24.33.105"
        >>>         options: {csum="true"}
        >>>     Encap stt
        >>>         ip: "172.24.33.105"
        >>>         options: {csum="true"}
        >>>     Port_Binding eni-3E9901E0
        >>>     Port_Binding eni-35AFCD00
        >>>
        >>> # ovs-vsctl get open . external-ids:system-id
        >>> "04540082-b5b5-4ab5-9901-03ed445c772d"
        >>>
        >>> # systemctl stop ovn-controller
        >>>
        >>> Chassis was deleted:
        >>>
        >>> # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d
        >>> ovn-sbctl: no row "04540082-b5b5-4ab5-9901-03ed445c772d" in table Chassis
        >>>
        >>> # yum update ovn-host -y
        >>> # systemctl restart ovn-controller
        >>>
        >>> Chassis with same system-id and encap IPs was re-added:
        >>>
        >>> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
        >>>     hostname: host.local
        >>>     Encap vxlan
        >>>         ip: "172.24.33.105"
        >>>         options: {csum="true"}
        >>>     Encap stt
        >>>         ip: "172.24.33.105"
        >>>         options: {csum="true"}
        >>>
        >>> But, there are no port_bindings, and in ovn-controller logs again transaction error:
        >>>
        >>> 2020-12-03T12:53:54.031Z|00035|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
        >>> 2020-12-03T12:53:54.031Z|00036|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
        >>> 2020-12-03T12:53:54.031Z|00037|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
        >>> 2020-12-03T12:53:54.031Z|00038|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
        >>> 2020-12-03T12:53:54.041Z|00039|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
        >>> 2020-12-03T12:53:54.042Z|00040|main|INFO|OVNSB commit failed, force recompute next time.
        >>>
        >>>
        >>> Moreover, if I forcefully delete chassis, port claim successful, but after restart ovn-controller, promlem appears again:
        >>>
        >>> # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
        >>>
        >>> 2020-12-03T12:56:20.119Z|00045|main|INFO|OVNSB commit failed, force recompute next time.
        >>> 2020-12-03T12:56:23.803Z|00046|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
        >>> 2020-12-03T12:56:23.803Z|00047|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
        >>> 2020-12-03T12:56:23.803Z|00048|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
        >>> 2020-12-03T12:56:23.803Z|00049|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
        >>>
        >>> # systemctl restart ovn-controller
        >>>
        >>> 2020-12-03T12:56:38.590Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller.log
        >>> 2020-12-03T12:56:38.592Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
        >>> 2020-12-03T12:56:38.592Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
        >>> 2020-12-03T12:56:38.596Z|00004|main|INFO|OVS IDL reconnected, force recompute.
        >>> 2020-12-03T12:56:38.600Z|00005|reconnect|INFO|ssl:x.x.x.x:6642: connecting...
        >>> 2020-12-03T12:56:38.600Z|00006|main|INFO|OVNSB IDL reconnected, force recompute.
        >>> 2020-12-03T12:56:38.645Z|00007|reconnect|INFO|ssl:x.x.x.x:6642: connected
        >>> 2020-12-03T12:56:38.650Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
        >>> 2020-12-03T12:56:38.650Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
        >>> 2020-12-03T12:56:38.651Z|00010|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
        >>> 2020-12-03T12:56:38.654Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
        >>> 2020-12-03T12:56:38.654Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
        >>> 2020-12-03T12:56:38.654Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
        >>> 2020-12-03T12:56:38.654Z|00012|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
        >>> 2020-12-03T12:56:38.654Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
        >>> 2020-12-03T12:56:38.654Z|00014|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
        >>> 2020-12-03T12:56:38.655Z|00015|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
        >>> 2020-12-03T12:56:38.655Z|00016|main|INFO|OVNSB commit failed, force recompute next time.
        >>>
        >>>
        >>> Maybe, I just don’t understand your idea...
        >>
        >> I see.  I'm pretty sure it's related to this commit that tries to reuse
        >> Encaps (and that's wrong because it doesn't work with RBAC):
        >>
        >> https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3
        >>
        >> I'll try to fix it and update this thread.
        >>
        > 
        > Hi Vladislav,
        > 
        > The problem is that branch-20.06 misses the following commit:
        > https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3

        Oops, this should've been:
        https://github.com/ovn-org/ovn/commit/dce1af31b550a9fb57b01cbe0b4139b6768f2521

        > 
        > However, at Han's suggestion we decided to remove the code that allowed
        > ovn-controller to reuse stale chassis records from the SB (because it
        > wasn't working properly with RBAC).  At this point I don't think it
        > makes sense to backport the missing commit because we'll be just
        > reverting it as soon as the new patch is accepted:
        > 
        > http://patchwork.ozlabs.org/project/ovn/patch/1607455279-21771-1-git-send-email-dceara@redhat.com/
        > 
        > Once/if the above is accepted, I'll send backport patches for all stable
        > branches.
        > 
        > Thanks,
        > Dumitru
        > 





More information about the dev mailing list