[ovs-discuss] Fw: TSO not working for VM iperf data packets.

Piyush R Srivastava1 pirsriva at in.ibm.com
Sat Dec 19 16:22:13 UTC 2015


Hi Jesse,

I'm sorry but i guess i have some differences here.

>>GRO is simply best effort aggregation and does not wait to get a full 64k
frame so when the packet is transmitted on the physical NIC, TSO will be
limited what was aggregated.
>> You're only guaranteed to get big TSO packets all the time when you
generate them on the local machine like you did in your later tests.

Yes I agree, its a software GRO ( best-effort) and that is why i do not see
full 64k sized offloaded packets

I did 10 sec iperf and tcpdump for full duration and recorded tcpdump data
at both egress port at router namespace and external network interface on
Network Node
( server IP- 1.1.1.34 and source SNAT IP- 1.1.4.92 )

[root at rhel7-23 ~]# ip netns exec
qrouter-223d91ae-3025-4070-83d0-63faa58d819b tcpdump -ennqi qg-ab10ec1d-2b
dst 1.1.1.34 > router_egress.log
[root at rhel7-23 ~]# tcpdump -ennqi eth1 dst 1.1.1.34 > external_nic.log


[root at rhel7-23 ~]# cat router_egress.log | head -n 30
21:35:07.628732 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 74:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 0
21:35:07.629839 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 66:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 0
21:35:07.629854 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 90:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 24
21:35:07.629859 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582
21:35:07.630609 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 7056:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 6990
21:35:07.630635 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 9852:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 9786
21:35:07.631272 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 8454:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 8388
21:35:07.631290 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 8454:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 8388
21:35:07.631974 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 8454:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 8388
21:35:07.631993 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582
21:35:07.632680 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194
21:35:07.632694 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.633215 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.633240 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194
21:35:07.633784 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582
21:35:07.633805 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 8454:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 8388
21:35:07.634372 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 21036:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 20970
21:35:07.634401 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194
21:35:07.634971 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582
21:35:07.634992 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582
21:35:07.635546 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194
21:35:07.635560 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.635577 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194
21:35:07.636120 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582
21:35:07.636142 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582
21:35:07.636707 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194
21:35:07.636722 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194
21:35:07.636731 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582
21:35:07.636749 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194
21:35:07.637250 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 9852:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 9786

As you said, its best effort and so i do not see all 64k sized offloaded
packets everytime.
However, majority of packets are offloaded at egress port of Router
Namespace.

[root at rhel7-23 ~]# cat router_egress.log | grep "length 1464" | wc -l
1448
[root at rhel7-23 ~]# cat router_egress.log | wc -l
119360

So out of 119360 packets captured only 1448 packets were actually not at
all offloaded ( Ethernet Frame of size 1464 )


However, when i take the same stats for external network interface

[root at rhel7-23 ~]# cat external_nic.log | head -n 200
21:35:07.641752 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641753 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641754 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641755 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641755 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641756 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641757 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641758 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641758 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641759 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641761 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641761 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641762 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641763 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641764 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641765 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641766 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641767 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641769 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641769 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641770 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641771 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641772 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641773 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641774 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641775 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641776 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.641776 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.642116 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.642223 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.642452 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.642743 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.642764 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.643112 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776
21:35:07.643301 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.643304 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.643305 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.643306 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.643308 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.643308 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.643309 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398
21:35:07.643310 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464:
1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398

There are only a handful of offloaded packets

[root at rhel7-23 ~]# cat external_nic.log | wc -l
4519467
[root at rhel7-23 ~]# cat external_nic.log | grep "length 1464" | wc -l
4504287

So out of 4519467 packets captured during full iperf duration, 4519467
packets actually had no offloading at all !!

Is this the way we would expect it to work?
The way i understood was that I should have seen a lot more offloaded
packets ( not necessarily 64k sized but more than 1464 sized Ethernet
frames ) on
External Network Interface ( similar to the packets we see on egress port
of router namespace )


Please rectify me if there is there something i misunderstood?

Thanks and regards,
Piyush Raman





From:	Jesse Gross <jesse at kernel.org>
To:	Piyush R Srivastava1/India/IBM at IBMIN
Cc:	discuss at openvswitch.org, Mala Anand <manand at us.ibm.com>
Date:	12/19/2015 09:26 PM
Subject:	Re: Fw: [ovs-discuss] TSO not working for VM iperf data
            packets.



On Sat, Dec 19, 2015 at 7:26 AM, Piyush R Srivastava1 <pirsriva at in.ibm.com>
wrote:
  Hi,

  One other thing i noticed is that... For starting I DO SEE offloaded
  packets... However, after a few initial packets, Offloading seems to be
  reduced drastically and i see purely 1464 sized Ethernet Frames.


[...]
  This seem to suggests that the issue is for the packets that are getting
  encapsulated at compute node and decapsulated at Network Node only
  We are using software GRO patch mentioned here at Network Nodes-
  http://openvswitch.org/pipermail/dev/2015-August/059335.html

  Does this info help in debugging the issue?



There is no issue then. GRO is simply best effort aggregation and does not
wait to get a full 64k frame so when the packet is transmitted on the
physical NIC, TSO will be limited what was aggregated. You're only
guaranteed to get big TSO packets all the time when you generate them on
the local machine like you did in your later tests.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20151219/3f3f10b5/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20151219/3f3f10b5/attachment-0002.gif>


More information about the discuss mailing list