[ovs-dev] [PATCH RFC 2/2] release-process: LTS transition period and policy for unmaintained branches.

Wed Sep 16 14:57:02 UTC 2020

On 16/09/2020 12:25, Ilya Maximets wrote:
> On 9/8/20 6:41 PM, Kevin Traynor wrote:
>> On 03/09/2020 15:20, Ilya Maximets wrote:
>>> On 8/28/20 3:55 PM, Flavio Leitner wrote:
>>>> On Tue, Aug 25, 2020 at 02:24:11PM +0200, Ilya Maximets wrote:
>>>>> While only 2 branches are formally maintained (LTS and latest release),
>>>>> OVS team usually provides stable releases for other branches too, at
>>>>> least for branches between LTS and latest.
>>>>>
>>>>> While LTS change happens, according to release-process.rst, we're
>>>>> immediately dropping support for the old LTS and, according to
>>>>> backporting-patches.rst could stop backporting bug fixes to branches
>>>>> older than new LTS.  While this might be OK for an upstream project
>>>>> (some upstream projects like QEMU doesn't support anything at all
>>>>> except the last release) it doesn't sound like a user-friendly policy.
>>>>>
>>>>> Below addition to the release process might make the process a bit
>>>>> smoother in terms that we will continue support of branches a little
>>>>> bit longer even after changing current LTS, i.e. providing at least a
>>>>> minimal transition period (1 release frame) for users of old LTS.
>>>>> We will also not drop support for not so old branches even after the
>>>>> transition period if committers will follow the "as far as it goes"
>>>>> backporting policy.
>>>>>
>>>>> Still keeping the room for us to not backport disruptive changes or
>>>>> changes that are hard to maintain or OVN related fixes anywhere but
>>>>> LTS and the latest released branch.
>>>>>
>>>>> After 2 year period (4 releases) committers are still free to backport
>>>>> fixes they think are needed on older branches, however we will likely
>>>>> not provide actual releases on these branches, unless it's specially
>>>>> requested and discussed.
>>>>>
>>>>> Effectively, this change means that we will support branch-2.5 until
>>>>> 2.15 release, i.e. we will provide the last release, if any, on
>>>>> branch-2.5 somewhere around Feb 2021. (I don't actually expect
>>>>> much fixes there)  And, presumably, at the same time we will provide
>>>>> last releases for branch 2.11 and below, if needed.
>>>>>
>>>>> Additionally, "4 releases" policy aligns with the DPDK LTS support
>>>>> policy, i.e. we will be able to validate and release last OVS releases
>>>>> with the last available DPDK LTS, e.g. OVS 2.11 last stable release
>>>>> will likely be released with the 18.11 EOL release validated.
>>>>>
>>>>> Signed-off-by: Ilya Maximets <i.maximets at ovn.org>
>>>>> ---
>>>>>  .../contributing/backporting-patches.rst      |  3 ++-
>>>>>  Documentation/internals/release-process.rst   | 21 ++++++++++++-------
>>>>>  2 files changed, 16 insertions(+), 8 deletions(-)
>>>>>
>>>>> diff --git a/Documentation/internals/contributing/backporting-patches.rst b/Documentation/internals/contributing/backporting-patches.rst
>>>>> index e8f4f271c..162e9d209 100644
>>>>> --- a/Documentation/internals/contributing/backporting-patches.rst
>>>>> +++ b/Documentation/internals/contributing/backporting-patches.rst
>>>>> @@ -69,7 +69,8 @@ targeted to the `master` branch, using the ``Fixes`` tag described in
>>>>>  :doc:`submitting-patches`. The maintainer first applies the patch to `master`,
>>>>>  then backports the patch to each older affected tree, as far back as it goes or
>>>>>  at least to all currently supported branches. This is usually each branch back
>>>>> -to the most recent LTS release branch.
>>>>> +to the oldest maintained LTS release branch or the last 4 release branches if
>>>>> +the oldest LTS is newer.
>>>>>  
>>>>>  If the fix only affects a particular branch and not `master`, contributors
>>>>>  should submit the change with the target branch listed in the subject line of
>>>>> diff --git a/Documentation/internals/release-process.rst b/Documentation/internals/release-process.rst
>>>>> index 63080caab..c5475c49b 100644
>>>>> --- a/Documentation/internals/release-process.rst
>>>>> +++ b/Documentation/internals/release-process.rst
>>>>> @@ -78,13 +78,20 @@ Scheduling`_ for the timing of each stage:
>>>>>  At most two release branches are formally maintained at any given time: the
>>>>>  latest release and the latest release designed as LTS.  An LTS release is one
>>>>>  that the OVS project has designated as being maintained for a longer period of
>>>>> -time.  Currently, an LTS release is maintained until the next LTS is chosen.
>>>>> -There is not currently a strict guideline on how often a new LTS release is
>>>>> -chosen, but so far it has been about every 2 years.  That could change based on
>>>>> -the current state of OVS development.  For example, we do not want to designate
>>>>> -a new release as LTS that includes disruptive internal changes, as that may
>>>>> -make it harder to support for a longer period of time.  Discussion about
>>>>> -choosing the next LTS release occurs on the OVS development mailing list.
>>>>> +time.  Currently, an LTS release is maintained until the next major release
>>>>> +after the new LTS is chosen.  There is not currently a strict guideline on how
>>>>> +often a new LTS release is chosen, but so far it has been about every 2 years.
>>>>> +That could change based on the current state of OVS development.  For example,
>>>>> +we do not want to designate a new release as LTS that includes disruptive
>>>>> +internal changes, as that may make it harder to support for a longer period of
>>>>> +time.  Discussion about choosing the next LTS release occurs on the OVS
>>>>> +development mailing list.
>>>>> +
>>>>> +While branches other than LTS and the latest release are not formally
>>>>> +maintained, the OVS project usually provides stable releases for these branches
>>>>> +for at least 2 years, i.e. stable releases are provided for the last 4
>>>>> +release branches.  However, these branches includes only bug fixes that are
>>>>> +easy to backport, i.e. might not include all the fixes that LTS has.
>>>>
>>>> Thanks for working on this.  I think the last paragraph is not much
>>>> clear, because one can think that branches in between LTS and latest
>>>> might not receive all bug fixes and then there would be regressions
>>>> updating from LTS to one of them.
>>>
>>> I think that is exactly what current documentation says.
>>>
>>> FAQ says [1]: "Releases that are not LTS may not be fixed and may just be
>>> supplanted by the next major release."
>>>
>>> [1] https://docs.openvswitch.org/en/latest/faq/releases/
>>>
>>> Before this change (now) we had 2 formally maintained branches (LTS and latest).
>>> And there is no any guarantee that branches in-between receives any bug fixes
>>> and we're formally not obligated to provide any releases on these branches.
>>>
>>> After this change we're taking responsibility to provide releases for last 4
>>> releases, but we're still not obligated to backport every bug fix.
>>>
>>> I understand that this is a bit confusing and not very convenient for users of
>>> these branches, but it is, at least, something.
>>>
>>> IMHO, upgrades from LTS to non-LTS doesn't make much sense from the possible
>>> regressions perspective, unless it's an upgrade to latest stable.
>>>
>>> With new strategy users will have 1 release time frame to upgrade from the
>>> previous LTS to new LTS.  In case we will nominate LTS to move to current
>>> stable at the end of its support cycle, users will have 1 year (0.5 for stable
>>> + 0.5 for LTS transition) in order to upgrade, e.g.
>>>
>>>   1. 2.17 released on Feb 2022 --> new stable
>>>   2. 2.18 released on Aug 2022 --> new stable
>>>      2.17 nominated to be an LTS at the same time
>>>      Transition period started for 2.13 (old LTS).
>>>   3. 2.19 released on Feb 2023 --> new stable
>>>      Dropped support for 2.13 due to end of the transition period.
>>>
>>>   In this scenario 2.17 will be continuously supported for 1 year at the
>>>   moment 2.13 becomes unmaintained.  This should be enough time frame for
>>>   users to upgrade.
>>>
>>> We might want to formalize this process, though.
>>>
>>> Maybe something like this:
>>>
>>> """
>>> At most three release branches are formally maintained at any given time: the
>>> latest release, the latest release designed as LTS and a previous LTS release
>>> during the transition period.  An LTS release is one that the OVS project has
>>> designated as being maintained for a longer period of time.
>>> Currently, an LTS release is maintained until the next major release after the
>>> new LTS is chosen. This one release time frame is a transition period which is
>>> intended for users to upgrade from old LTS to new one.
>>> New LTS release is chosen every 2 years.  The process is that current latest
>>> stable release becomes an LTS release at the same time with the next major
>>> release.  That could change based on the current state of OVS development.  For
>>> example, we do not want to designate a new release as LTS that includes
>>> disruptive internal changes, as that may make it harder to support for a longer
>>> period of time.  Discussion about skipping designation of the next LTS release
>>> occurs on the OVS development mailing list.
>>>
>>> LTS designation schedule example (depends on current state of development):
>>>   2.14 released on Aug 2020 - new latest stable, 2.13 stable --> new LTS
>>>   2.15 released on Feb 2021 - new latest stable, 2.5     LTS --> EOL
>>>   2.16 released on Aug 2021 - new latest stable
>>>   2.17 released on Feb 2022 - new latest stable
>>>   2.18 released on Aug 2022 - new latest stable, 2.17 stable --> new LTS
>>>   2.19 released on Feb 2023 - new latest stable, 2.13    LTS --> EOL
>>>
>>> While branches other than LTS and the latest release are not formally
>>> maintained, the OVS project usually provides stable releases for these branches
>>> for at least 2 years, i.e. stable releases are provided for the last 4
>>> release branches.  However, these branches includes only bug fixes that are
>>> easy to backport, i.e. might not include all the fixes that LTS has.
>>> """
>>>
>>> The main difference here is that we're specifying that LTS nominated every 2
>>> years by default and "skipping" should be discussed on a list instead of
>>> "choosing".  And also specified the process that new LTS is a previous stable,
>>> so the branch is already maintained for 6 months before becoming an LTS and
>>> there is no unmaintained time for this branch until its EOL.
>>>
>>> What do you think?  Simon, Ian, anyone else?
>>>
>>
>> The only issue I can think of is, if someone is considering moving to
>> newer non-LTS branch for whatever reason, how would they know that there
>> is backports missing and which ones.
>>
>> A per (newer than LTS) branch list of skipped backports would solve it.
>> It could be handy for debug too - if an issue arises on the branch, you
>> could quickly check if there are known fixes missing. Obviously there
>> would be extra overhead for maintainers to keep the list though.
> 
> I hope that we will not need to have such a list since the default
> policy for maintainers is 'backport as far as it goes' and anyway
> backporting affects all intermediate branches while backporting to
> LTS.  The condition where users will want to upgrade from old non-LTS
> to the branch that is older than current LTS should not be likely
> and anyway we're maintaining hierarchy where newer branches always
> has greater or equal number of fixes, i.e. there might be buggy new
> feature (new relativety to branch from where upgrade is done), but
> there should be no regressions in old features.
> 

Ok, that seems clear, thanks. It was possible regressions in old
features moving fwd I was thinking about, but I must have misinterpreted
earlier in the thread.

In any case, I guess it would be very unlikely that a fix that has a
patch for both LTS and current branch would not be fairly easy to merge
on intermediate branches.

>>
>> Overall looks good and even though DPDK is not the only consideration,
>> it's nice that there is a 2 year block of time when an OVS/DPDK
>> combination are both maintained.
>>
>> thanks,
>> Kevin.
>>
>>> Best regards, Ilya Maximets.
>>>
>>>>
>>>> Perhaps:
>>>> However, stable branches older than LTS include only bug fixes that
>>>> are easy to backport, i.e. might not include all the fixes that LTS
>>>> has.
>>>>
>>>
>>> _______________________________________________
>>> dev mailing list
>>> dev at openvswitch.org
>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>>>
>>
> 