[ovs-dev] [ovn] transaction error in ovn-controller with configured RBAC on branch-20.06
Dumitru Ceara
dceara at redhat.com
Tue Apr 20 10:35:58 UTC 2021
On 4/19/21 7:36 PM, Odintsov Vladislav wrote:
> Hi Dumitru,
>
Hi Vladislav,
> I've seen your patches have been backported to 20.06 branch and tried it with RBAC-enabled installation. It seems working for ovn-controller, but for ovn-controller-vtep I still see similar errors.
> Should this be fixed in ovn-controller-vtep as well?
>
> 2021-04-19T17:26:22Z|00824|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"cumulus-01\" role \"ovn-controller\" prohibit row insertion into table \"Encap\".","error":"permission error"}
> 2021-04-19T17:26:22Z|00825|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back
> 2021-04-19T17:26:22Z|00826|gateway|INFO|add Chassis row for VTEP physical switch (cumulus-01)
> 2021-04-19T17:26:27Z|00827|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back
> 2021-04-19T17:26:27Z|00828|gateway|INFO|add Chassis row for VTEP physical switch (cumulus-01)
> 2021-04-19T17:26:32Z|00829|gateway|WARN|Chassis for VTEP physical switch (cumulus-01) disappears, maybe deleted by ovn-sbctl, adding it back
>
> As workaround, if I switch ovn-controller-vtep to another ovnsbdb port (without rbac engine), ovn-controller-vtep successfully adds chassis record, then I switch it back to rbac socket and continue working well. So, error occurs only on first run of chassis. When chassis exists in DB, things work well.
I sent a patch that should fix the ovn-controller-vtep RBAC too:
http://patchwork.ozlabs.org/project/ovn/patch/20210420103221.1123-1-dceara@redhat.com/
It should apply cleanly on branch-20.06 too, do you mind trying it out
when you get the chance?
Thanks,
Dumitru
>
> Regards,
>
> Vladislav Odintsov
>
> On 09.12.2020, 11:30, "Odintsov Vladislav" <VlOdintsov at croc.ru> wrote:
>
> Hi Dumitru,
>
> That’s good news, thanks for that!
>
>
> Regards,
>
> Vladislav Odintsov
>
> On 08.12.2020, 22:33, "Dumitru Ceara" <dceara at redhat.com> wrote:
>
> On 12/8/20 8:28 PM, Dumitru Ceara wrote:
> > On 12/3/20 4:11 PM, Dumitru Ceara wrote:
> >> On 12/3/20 2:01 PM, Odintsov Vladislav wrote:
> >>> But neither IP nor system-id was changed. I've double-checked:
> >>>
> >>> ovn-controller 20.06.2:
> >>>
> >>> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
> >>> hostname: host.local
> >>> Encap vxlan
> >>> ip: "172.24.33.105"
> >>> options: {csum="true"}
> >>> Encap stt
> >>> ip: "172.24.33.105"
> >>> options: {csum="true"}
> >>> Port_Binding eni-3E9901E0
> >>> Port_Binding eni-35AFCD00
> >>>
> >>> # ovs-vsctl get open . external-ids:system-id
> >>> "04540082-b5b5-4ab5-9901-03ed445c772d"
> >>>
> >>> # systemctl stop ovn-controller
> >>>
> >>> Chassis was deleted:
> >>>
> >>> # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d
> >>> ovn-sbctl: no row "04540082-b5b5-4ab5-9901-03ed445c772d" in table Chassis
> >>>
> >>> # yum update ovn-host -y
> >>> # systemctl restart ovn-controller
> >>>
> >>> Chassis with same system-id and encap IPs was re-added:
> >>>
> >>> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
> >>> hostname: host.local
> >>> Encap vxlan
> >>> ip: "172.24.33.105"
> >>> options: {csum="true"}
> >>> Encap stt
> >>> ip: "172.24.33.105"
> >>> options: {csum="true"}
> >>>
> >>> But, there are no port_bindings, and in ovn-controller logs again transaction error:
> >>>
> >>> 2020-12-03T12:53:54.031Z|00035|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
> >>> 2020-12-03T12:53:54.031Z|00036|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
> >>> 2020-12-03T12:53:54.031Z|00037|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
> >>> 2020-12-03T12:53:54.031Z|00038|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
> >>> 2020-12-03T12:53:54.041Z|00039|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
> >>> 2020-12-03T12:53:54.042Z|00040|main|INFO|OVNSB commit failed, force recompute next time.
> >>>
> >>>
> >>> Moreover, if I forcefully delete chassis, port claim successful, but after restart ovn-controller, promlem appears again:
> >>>
> >>> # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
> >>>
> >>> 2020-12-03T12:56:20.119Z|00045|main|INFO|OVNSB commit failed, force recompute next time.
> >>> 2020-12-03T12:56:23.803Z|00046|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
> >>> 2020-12-03T12:56:23.803Z|00047|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
> >>> 2020-12-03T12:56:23.803Z|00048|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
> >>> 2020-12-03T12:56:23.803Z|00049|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
> >>>
> >>> # systemctl restart ovn-controller
> >>>
> >>> 2020-12-03T12:56:38.590Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller.log
> >>> 2020-12-03T12:56:38.592Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
> >>> 2020-12-03T12:56:38.592Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
> >>> 2020-12-03T12:56:38.596Z|00004|main|INFO|OVS IDL reconnected, force recompute.
> >>> 2020-12-03T12:56:38.600Z|00005|reconnect|INFO|ssl:x.x.x.x:6642: connecting...
> >>> 2020-12-03T12:56:38.600Z|00006|main|INFO|OVNSB IDL reconnected, force recompute.
> >>> 2020-12-03T12:56:38.645Z|00007|reconnect|INFO|ssl:x.x.x.x:6642: connected
> >>> 2020-12-03T12:56:38.650Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
> >>> 2020-12-03T12:56:38.650Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
> >>> 2020-12-03T12:56:38.651Z|00010|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected
> >>> 2020-12-03T12:56:38.654Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch
> >>> 2020-12-03T12:56:38.654Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting...
> >>> 2020-12-03T12:56:38.654Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for this chassis.
> >>> 2020-12-03T12:56:38.654Z|00012|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5
> >>> 2020-12-03T12:56:38.654Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for this chassis.
> >>> 2020-12-03T12:56:38.654Z|00014|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4
> >>> 2020-12-03T12:56:38.655Z|00015|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"}
> >>> 2020-12-03T12:56:38.655Z|00016|main|INFO|OVNSB commit failed, force recompute next time.
> >>>
> >>>
> >>> Maybe, I just don’t understand your idea...
> >>
> >> I see. I'm pretty sure it's related to this commit that tries to reuse
> >> Encaps (and that's wrong because it doesn't work with RBAC):
> >>
> >> https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3
> >>
> >> I'll try to fix it and update this thread.
> >>
> >
> > Hi Vladislav,
> >
> > The problem is that branch-20.06 misses the following commit:
> > https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3
>
> Oops, this should've been:
> https://github.com/ovn-org/ovn/commit/dce1af31b550a9fb57b01cbe0b4139b6768f2521
>
> >
> > However, at Han's suggestion we decided to remove the code that allowed
> > ovn-controller to reuse stale chassis records from the SB (because it
> > wasn't working properly with RBAC). At this point I don't think it
> > makes sense to backport the missing commit because we'll be just
> > reverting it as soon as the new patch is accepted:
> >
> > http://patchwork.ozlabs.org/project/ovn/patch/1607455279-21771-1-git-send-email-dceara@redhat.com/
> >
> > Once/if the above is accepted, I'll send backport patches for all stable
> > branches.
> >
> > Thanks,
> > Dumitru
> >
>
>
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
More information about the dev
mailing list