[ovs-discuss] OVN Scale with RAFT: how to make raft cluster clients to balanced state again

Han Zhou zhouhan at gmail.com
Thu Aug 6 00:23:30 UTC 2020


On Wed, Aug 5, 2020 at 4:35 PM Girish Moodalbail <gmoodalbail at gmail.com>
wrote:

>
>
> On Wed, Aug 5, 2020 at 3:05 PM Han Zhou <hzhou at ovn.org> wrote:
>
>>
>>
>> On Wed, Aug 5, 2020 at 12:51 PM Winson Wang <windson.wang at gmail.com>
>> wrote:
>>
>>> Hello OVN Experts:
>>>
>>> With large scale ovn-k8s cluster,  there are several conditions that
>>> would make ovn-controller clients connect SB central from a balanced state
>>> to an unbalanced state.
>>> Is there an ongoing project to address this problem?
>>> If not,  I have one proposal not sure if it is doable.
>>> Please share your thoughts.
>>>
>>> The issue:
>>>
>>> OVN SB RAFT 3 node cluster,  at first all the ovn-controller clients
>>> will connect all the 3 nodes in a balanced state.
>>>
>>> The following conditions will make the connections become unbalanced.
>>>
>>>    -
>>>
>>>    One RAFT node restart,  all the ovn-controller clients to reconnect
>>>    to the two remaining cluster nodes.
>>>
>>>
>>>    -
>>>
>>>    Ovn-k8s,  after SB raft pods rolling upgrade, the last raft pod has
>>>    no client connections.
>>>
>>>
>>> RAFT clients in an unbalanced state would trigger more stress to the
>>> raft cluster,  which makes the raft unstable under stress compared to a
>>> balanced state.
>>> The proposal solution:
>>>
>>> Ovn-controller adds next unix commands “reconnect” with argument of
>>> preferred SB node IP.
>>>
>>> When unbalanced state happens,  the UNIX command can trigger
>>> ovn-controller reconnect
>>>
>>> To new SB raft node with fast sync which doesn’t trigger the whole DB
>>> downloading process.
>>>
>>>
>> Thanks Winson. The proposal sounds good to me. Will you implement it?
>>
>
> Han/Winson,
>
> The fast re-sync is for ovsdb-server restart and it will not apply for
> ovn-controller restart, right?
>
>
Right, but the proposal is to provide a command just to reconnect, without
restarting. In that case fast-resync should work.


> If the ovsdb-client (ovn-controller) restarts, then it would have lost all
> its state and when it starts again it will still need to download
> logical_flows, port_bindings , and other tables it cares about. So, fast
> re-sync may not apply to this case.
>
> Also, the ovn-controller should stash the IP address of the SB server to
> which it is connected to in Open_vSwitch table's external_id column. It
> updates this field whenever it re-connects to a different SB server
> (because that ovsdb-server instance failed or restarted). When
> ovn-controller itself restarts it could check for the value in this field
> and try to connect to it first and on failure fallback to connect to
> default connection approach.
>

The imbalance is usually caused by failover on server side. When one server
is down, all clients are expected to connect to the rest of the servers,
and when the server is back, there is no motivation for the clients to
reconnect again (unless you purposely restart the clients, which would
bring 1/3 of the restarted clients back to the old server). So I don't
understand how "stash the IP address" would work in this scenario.

The proposal above by Winson is to purposely trigger a reconnection towards
the desired server without restarting the clients, which I think solves
this problem directly.

Thanks,
Han


>
> Regards,
> ~Girish
>
>
>
>
>>
>> Han
>>
>>
>>
>>>
>>> --
>>> Winson
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "ovn-kubernetes" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to ovn-kubernetes+unsubscribe at googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS--iOW0LxxtkOhJpRT49E-9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS--iOW0LxxtkOhJpRT49E-9bJVy0iXraC1LMDUWeu6kLA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>
> --
> You received this message because you are subscribed to the Google Groups
> "ovn-kubernetes" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ovn-kubernetes+unsubscribe at googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTrZb%2BNo8%2B3%3DOJcMqd6T_1sS5bm-xnF6v_P4%2B2uqKtZAQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTrZb%2BNo8%2B3%3DOJcMqd6T_1sS5bm-xnF6v_P4%2B2uqKtZAQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200805/b021a782/attachment-0001.html>


More information about the discuss mailing list