[ovs-dev] The way SDN controllers deal with MTU

[Ihar Hrachyshka]

> On 28 Jun 2016, at 17:41, Jesse Gross <jesse at kernel.org> wrote:
> 
> On Tue, Jun 28, 2016 at 1:37 AM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:
>> 
>>> On 27 Jun 2016, at 23:57, Jesse Gross <jesse at kernel.org> wrote:
>>> 
>>> On Tue, Jun 21, 2016 at 10:16 PM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:
>>>>> On 21 Jun 2016, at 23:46, Jesse Gross <jesse at kernel.org> wrote:
>>>>> On Mon, Jun 20, 2016 at 5:16 AM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:
>>>>>> So back to MTU. When I boot a VM using a VXLAN backed network, the tap device of MTU=1450 is plugged into the br-int bridge, which lowers the bridge MTU to 1450. Then when I plug a device that belongs to a GRE network (MTU = 1458) into that same bridge, the GRE network backed device also gets its MTU reduced to 1450, and no ‘ip link’ commands allow to raise it to the intended MTU=1458.
>>>>> 
>>>>> I'm not sure I fully understand the scenario. The part about having
>>>>> multiple, effectively independent networks on the same bridge I
>>>>> understand. I also understand how the bridge device's MTU is lowered
>>>>> to the minimum of the ports attached to the bridge.
>>>>> 
>>>>> The part about the tunnel device MTU sounds weird to me. Are these
>>>>> tunnel ports that have been created through OVSDB? Or are they created
>>>>> separately through something like ip link and then attached to OVS
>>>>> just as if they were regular Ethernet devices?
>>>> 
>>>> For the sake of the integration bridge, there is no difference between tunnelled networks and e.g. vlan networks. They are the same tap devices connected to the bridge. The only difference between them is that they should have different MTUs set.
>>>> 
>>>> (In Neutron, traffic from tunnelled devices that is meant for routing is redirected to another bridge (br-tun) connected via a patch port where routing actually happens.) The tunnelling bridge is also implemented as an Open vSwitch bridge.
>>>> 
>>>>> 
>>>>> In the former case, the ports essentially have infinite MTU so this
>>>>> shouldn't come up at all. And in the latter case, the ports don't even
>>>>> know that they are attached to OVS. (If this is happening, what occurs
>>>>> when you try to change the MTU with ip link? Do you get an error or it
>>>>> just doesn't work?)
>>>>> 
>>>> 
>>>> As I said above, tunneling is implemented by a separate bridge.
>>>> 
>>>> The correct MTU of devices is especially relevant for our routers (network namespaces) where the MTU value on an internal port connecting the router to the integration bridge influences how the router fragment packets coming from its other legs (internal or external).
>>>> 
>>>> When I try to set MTU with ip link on ports plugged into the integration bridge, it just doesn’t work if the desired MTU is higher than the bridge MTU (which is already the lowest of all other devices plugged into the bridge). No error produced. But when I move the device into a network namespace, it suddenly start to work, allowing any MTU to be set.
>>> 
>>> I guess I'm having a hard time understanding exactly what is happening
>>> based on your description. It's not clear to me whether there is a
>>> problem with changing the MTUs of the attached ports or just the
>>> effect that it has on the MTU of the internal device.
>> 
>> Yes, I can’t raise MTU on attached ports, neither on the bridge itself. I can lower it though (and revert back to the previous value).
>> 
>>> 
>>> I'm guessing that this is just the internal device - that's the
>>> intention of the code and I also tested to make sure that's what
>>> happens in practice. Presumably any other changes to MTUs are done by
>>> Neutron since OVS should not influence them.
>>> 
>>> In terms of the internal device, I agree that this MTU adjustment
>>> doesn't necessarily make sense in an OVS context. It was carried over
>>> from the Linux bridge but OVS allows the network to be divided up in a
>>> much more arbitrary manner.
>>> 
>>> I think the cleanest way to deal with this is to retain the MTU if the
>>> user sets it manually and not continue to make changes automatically.
>>> This would avoid adding a fairly obscure configuration option. The
>>> main problem that I see is ensuring that the MTU doesn't get reset in
>>> the event of an OVS restart. Perhaps you can find a clean way to do
>>> that.
>> 
>> Sadly I am not a kernel developer (well, I was once, but that was a quite long time ago). I can only suggest to relax the requirement, maybe pull some resources into fixing it. But first thing, I would like to get to an agreement on the direction we can take here. If that’s to fulfil all MTU set requests, I am fine with it, and may start looking into getting a patch up for review for just that.
> 
> The code to control the MTU of the bridge device is actually in OVS
> userspace, not the kernel. Here is the main hunk:
> https://github.com/openvswitch/ovs/blob/master/ofproto/ofproto.c#L2771

Doesn’t this loop update MTU for all devices plugged into a bridge to the lowest of all?
https://github.com/openvswitch/ovs/blob/master/ofproto/ofproto.c#L2810

> 
> I still don't understand how the MTU of other ports can be affected by
> this (and they didn't seem to be in my testing). Is it possible that
> there is something in Neutron that is doing this? In any case, the
> above code is the only logic that affects MTU in OVS that I am aware
> of, so maybe you can experiment with changing it and see if it fixes
> the problem.

I understand your concern that we still have Neutron in the picture.

I will try to capture specific ip link/ovs commands that Neutron triggers to isolate the issue from our SDN. I will get back to the thread when I complete that.

Thanks
Ihar