[ovs-discuss] TC flow offload using OVS - failed to offload flow: Invalid argument: enp0s25
satish dhote
sdhote926 at gmail.com
Thu Mar 5 06:31:36 UTC 2020
Gentle reminder for above conversation :)
Also I noticed one thing that ingress qdisc for a port is getting deleted
after few seconds. Ingres qdisc created by openvswitch when we add any
netdevice to the openvswitch using ovs-vsctl utility.
Any reason why this discs are deleted automatically, I could not find
solution on web.
If I create qdisc manually using tc command and try add the flow using tc
commands then flows are getting added into hardware.
If qdisc is not there and if I try to add the flow using tc command, then
command throws "Invalid argument" error.
During offloading of flow through OVS, I still see the same logs as part
of ovs-vswitchd.log i.e. "failed to offload flow: Invalid argument: ens11"
even if qdisc exist.
Thanks
Satish
On Thu, Feb 27, 2020 at 4:26 PM satish dhote <sdhote926 at gmail.com> wrote:
> Hi Roni,
>
> Answer inline.
>
> *Any idea why all ipv4 rules has broadcast mac address? and flooded to all
> ports?*
>
> I believe these rules are derived by ovs-controller(user space daemon) as
> part of "NORMAL" actions given below. I didn't add them explicitly.
> cookie=0x0, duration=974.646s, table=0, n_packets=0, n_bytes=0, priority=0
> actions=NORMAL
>
> *Anyway, flooding is not supported currently it requires that we will
> clone the packet, and It will be supported on next release (high
> probability), but ctx5 and above as far as I know.*
>
> Does it mean, we can not offload actions like forward packet to the all
> ports (i.e. actions:1,3,4....n) ? OR we can not configure keys with
> broadcast addresses(i..e dst=ff:ff:ff:ff:ff:ff) ?
>
> how can I check flows offloaded in the mellanox card(firmware)? TC command
> "tc filter show dev ens11 ingress" doesn't show anything, does it mean no
> rule is configured in hardware?
>
> Even when deleted all flows and configured only the flows which says
> in_port=1, action=output:2 and vice versa, then also ping is working fine
> but traffic is always going through ovs kernel datapath since no flow is
> getting offloaded into mellanox-4 card. Still seeing the same error.
> 2020-02-27T10:32:54.046Z|00026|dpif_netlink(handler61)|ERR|failed to
> offload flow: Invalid argument: eth0
> 2020-02-27T10:33:08.344Z|00015|dpif_netlink(handler58)|ERR|failed to
> offload flow: Invalid argument: ens11
> 2020-02-27T10:33:09.344Z|00025|dpif_netlink(handler57)|ERR|failed to
> offload flow: Invalid argument: ens11
> 2020-02-27T10:33:25.380Z|00029|dpif_netlink(handler61)|ERR|failed to
> offload flow: Invalid argument: ens11
> 2020-02-27T10:33:40.067Z|00017|dpif_netlink(handler56)|ERR|failed to
> offload flow: Invalid argument: eth0
> 2020-02-27T10:34:21.743Z|00028|dpif_netlink(handler57)|ERR|failed to
> offload flow: Invalid argument: eth0
> 2020-02-27T10:34:26.012Z|00020|dpif_netlink(handler58)|ERR|failed to
> offload flow: Invalid argument: ens11
>
> *Apart from above issue I have one more query: *does OVS try to offload
> the TC flow while adding flow into ovs-kernel datapath OR TC flow is always
> offloaded while adding openflow rule using ovs-ofctl utility?
> As per my observation addition of tc flow is triggered when flow is added
> into kernel datapath as part of flow miss handling, but just wanted to know
> whether there is any specific flow for which tc flow is always added
> without triggering flow addition in ovs-kernel datapath?
>
> *Can you provide any ovs-flow command example which is supported for
> mellanox tc offload? I will try to configure the same in my setup. This
> will help me to narrow down the issue whether it is really a setup problem
> OR flow command problem OR issue with ovs daemon.*
>
> Thanks
> Satish
>
> On Thu, Feb 27, 2020 at 2:49 AM Roni Bar Yanai <roniba at mellanox.com>
> wrote:
>
>> Hi satish,
>>
>>
>>
>> I think the problem is in the rules of the form
>>
>> recirc_id(0),in_port(2),eth(src=e4:11:22:33:44:50,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
>> packets:195, bytes:66690, used:3.722s, actions:1,3,4
>>
>> Any idea why all ipv4 rules has broadcast mac address? and flooded to all
>> ports?
>>
>> Anyway, flooding is not supported currently it requires that we will
>> clone the packet, and
>>
>> It will be supported on next release (high probability), but ctx5 and
>> above as far as I know.
>>
>>
>>
>> BR,
>>
>> Roni
>>
>>
>>
>> *From:* satish dhote <sdhote926 at gmail.com>
>> *Sent:* Wednesday, February 26, 2020 3:27 PM
>> *To:* Roni Bar Yanai <roniba at mellanox.com>
>> *Cc:* Moshe Levi <moshele at mellanox.com>; ovs-discuss at openvswitch.org
>> *Subject:* Re: [ovs-discuss] TC flow offload using OVS - failed to
>> offload flow: Invalid argument: enp0s25
>>
>>
>>
>> Answer inline and provided other config related information below. Let me
>> know in case you need any other information.
>>
>>
>>
>> *eth1 is it the port representor of VF1?*
>>
>> eth0 and eth1 are port representor for VF1 and VF2 respectively.
>>
>>
>>
>> *eth0 is the up-link representor?*
>>
>> ens11 is pf(uplink) interface.
>>
>> *Can you share dpif/dump-flows so we can see what are the flows you are
>> trying to offload?*
>>
>> *[root at localhost ~]# ovs-appctl dpif/dump-flows ovs-sriov*
>> recirc_id(0),in_port(2),eth(src=50:6b:4b:0c:81:70,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
>> packets:63175587, bytes:21606050754, used:0.000s, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=e4:11:22:33:44:50,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(frag=no),
>> packets:8597565, bytes:773780850, used:0.000s, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=50:6b:4b:0c:81:70,dst=33:33:ff:db:1e:09),eth_type(0x86dd),ipv6(frag=no),
>> packets:209474316, bytes:18014791176, used:0.000s, actions:1,3,4
>> recirc_id(0),in_port(3),eth(src=e4:11:22:33:44:50,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
>> packets:3, bytes:1026, used:5.357s, actions:1,2,4
>> recirc_id(0),in_port(2),eth(src=9a:d7:9b:84:3a:5e,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(frag=no),
>> packets:355100402, bytes:31959036180, used:0.000s, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=e4:11:22:33:44:50,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
>> packets:0, bytes:0, used:never, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=50:6b:4b:0c:81:70,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(frag=no),
>> packets:152331284, bytes:13709822800, used:0.001s, actions:1,3,4
>>
>>
>>
>> *[root at localhost ~]# ovs-ofctl dump-flows ovs-sriov*
>> cookie=0x0, duration=974.646s, table=0, n_packets=0, n_bytes=0,
>> priority=0 actions=NORMAL
>>
>>
>>
>> *[root at localhost ~]# ovs-appctl dpif/dump-flows type=offloaded
>> ovs-sriov (Note sure whether this type parameter works here for
>> hardware offloaded flows:) )*
>> recirc_id(0),in_port(4),eth(src=e4:11:22:33:44:51,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
>> packets:3, bytes:1026, used:7.092s, actions:1,2,3
>> recirc_id(0),in_port(2),eth(src=50:6b:4b:0c:81:70,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
>> packets:209960759, bytes:71806579578, used:0.000s, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=e4:11:22:33:44:50,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(frag=no),
>> packets:151469827, bytes:13632285570, used:0.000s, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=50:6b:4b:0c:81:70,dst=33:33:ff:db:1e:09),eth_type(0x86dd),ipv6(frag=no),
>> packets:60983904, bytes:5244615744, used:0.001s, actions:1,3,4
>> recirc_id(0),in_port(3),eth(src=e4:11:22:33:44:50,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
>> packets:4, bytes:1368, used:3.748s, actions:1,2,4
>> recirc_id(0),in_port(2),eth(src=a2:b9:ce:03:0f:ba,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(frag=no),
>> packets:18142152, bytes:1632793680, used:0.001s, actions:1,3,4
>> recirc_id(0),in_port(4),eth(src=e4:11:22:33:44:51,dst=33:33:00:00:00:02),eth_type(0x86dd),ipv6(frag=no),
>> packets:2, bytes:124, used:4.037s, actions:1,2,3
>> recirc_id(0),in_port(2),eth(src=e4:11:22:33:44:51,dst=33:33:00:00:00:02),eth_type(0x86dd),ipv6(frag=no),
>> packets:890, bytes:55180, used:7.732s, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=9a:d7:9b:84:3a:5e,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(frag=no),
>> packets:497909550, bytes:44811859500, used:0.001s, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=e4:11:22:33:44:50,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
>> packets:195, bytes:66690, used:3.722s, actions:1,3,4
>> recirc_id(0),in_port(2),eth(src=50:6b:4b:0c:81:70,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(frag=no),
>> packets:309799865, bytes:27881997410, used:0.000s, actions:1,3,4
>>
>>
>>
>> TC commands shows empty response.
>>
>>
>>
>>
>> *[root at localhost ~]# tc filter show dev ovs-sriov ingress [root at localhost
>> ~]# tc filter show dev eth0 ingress [root at localhost ~]# tc filter show dev
>> eth1 ingress [root at localhost ~]# tc filter show dev ens11 ingress*
>>
>>
>>
>> *[root at localhost ~]# ovs-dpctl show*
>> 2020-02-26T13:10:28Z|00001|dpif_netlink|INFO|The kernel module does not
>> support meters.
>> system at ovs-system:
>> lookups: hit:2826089018 missed:439 lost:0
>> flows: 8
>> masks: hit:5034569510 total:2 hit/pkt:1.78
>> port 0: ovs-system (internal)
>> port 1: ovs-sriov (internal)
>> port 2: ens11
>> port 3: eth0
>> port 4: eth1
>>
>>
>>
>> *Is traffic running? Ping replay and ssh?*
>>
>> Yes. below are the response.
>> * VM1 ip: *192.168.122.115
>> *VM2 ip: *192.168.122.116
>>
>>
>> * [root at localhost ~]# ping 192.168.122.116*
>> PING 192.168.122.116 (192.168.122.116) 56(84) bytes of data.
>> 64 bytes from 192.168.122.116
>> <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.122.116%2F&data=02%7C01%7Croniba%40mellanox.com%7Cb4addd6f0606436fa3c608d7babf9f72%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637183204488838419&sdata=WnrK7QlqOM7YrMF80vkPo6Kvjmu8gSXKJaSuU6Numvk%3D&reserved=0>:
>> icmp_seq=1 ttl=64 time=3.08 ms
>> 64 bytes from 192.168.122.116
>> <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.122.116%2F&data=02%7C01%7Croniba%40mellanox.com%7Cb4addd6f0606436fa3c608d7babf9f72%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637183204488838419&sdata=WnrK7QlqOM7YrMF80vkPo6Kvjmu8gSXKJaSuU6Numvk%3D&reserved=0>:
>> icmp_seq=2 ttl=64 time=2.54 ms
>>
>> *[root at localhost ~]# ssh root at 192.168.122.116 <root at 192.168.122.116>*
>> The authenticity of host '192.168.122.116 (192.168.122.116)' can't be
>> established.
>> ECDSA key fingerprint is 0b:e0:20:e6:97:16:35:86:ca:3c:1b:86:5a:36:9e:3a.
>> Are you sure you want to continue connecting (yes/no)? yes
>> Warning: Permanently added '192.168.122.116' (ECDSA) to the list of known
>> hosts.
>> root at 192.168.122.116's password:
>>
>> *Error logs from ovs-vswitchd.log:*
>>
>> 2020-02-26T13:13:02.750Z|00038|dpif_netlink(handler11)|ERR|failed to
>> offload flow: Invalid argument: eth0
>> 2020-02-26T13:13:02.786Z|00100|dpif_netlink(handler8)|ERR|failed to
>> offload flow: Invalid argument: eth0
>> 2020-02-26T13:13:02.789Z|00101|dpif_netlink(handler8)|ERR|failed to
>> offload flow: Invalid argument: ens11
>> 2020-02-26T13:13:03.638Z|00102|dpif_netlink(handler8)|ERR|failed to
>> offload flow: Invalid argument: eth0
>> 2020-02-26T13:13:04.643Z|00078|dpif_netlink(handler9)|ERR|failed to
>> offload flow: Invalid argument: eth0
>> 2020-02-26T13:13:08.645Z|00055|dpif_netlink(handler10)|ERR|failed to
>> offload flow: Invalid argument: ens11
>> 2020-02-26T13:13:11.769Z|00103|dpif_netlink(handler8)|ERR|failed to
>> offload flow: Invalid argument: eth1
>> 2020-02-26T13:13:22.691Z|00079|dpif_netlink(handler9)|ERR|Dropped 10 log
>> messages in last 11 seconds (most recently, 8 seconds ago) due to excessive
>> rate
>> 2020-02-26T13:13:22.691Z|00080|dpif_netlink(handler9)|ERR|failed to
>> offload flow: Invalid argument: ens11
>> 2020-02-26T13:13:29.479Z|00104|dpif_netlink(handler8)|ERR|failed to
>> offload flow: Invalid argument: eth0
>> 2020-02-26T13:13:43.924Z|00105|dpif_netlink(handler8)|ERR|Dropped 3 log
>> messages in last 15 seconds (most recently, 11 seconds ago) due to
>> excessive rate
>> 2020-02-26T13:13:43.924Z|00106|dpif_netlink(handler8)|ERR|failed to
>> offload flow: Invalid argument: eth1
>> 2020-02-26T13:13:56.780Z|00056|dpif_netlink(handler10)|ERR|Dropped 7 log
>> messages in last 13 seconds (most recently, 7 seconds ago) due to excessive
>> rate
>> 2020-02-26T13:13:56.781Z|00057|dpif_netlink(handler10)|ERR|failed to
>> offload flow: Invalid argument: eth1
>> 2020-02-26T13:14:15.495Z|00107|dpif_netlink(handler8)|ERR|Dropped 6 log
>> messages in last 18 seconds (most recently, 12 seconds ago) due to
>> excessive rate
>> 2020-02-26T13:14:15.495Z|00108|dpif_netlink(handler8)|ERR|failed to
>> offload flow: Invalid argument: eth0
>> 2020-02-26T13:14:29.165Z|00081|dpif_netlink(handler9)|ERR|failed to
>> offload flow: Invalid argument: eth1
>> 2020-02-26T13:14:29.169Z|00082|dpif_netlink(handler9)|ERR|failed to
>> offload flow: Invalid argument: ens11
>> 2020-02-26T13:14:41.783Z|00058|dpif_netlink(handler10)|ERR|Dropped 6 log
>> messages in last 9 seconds (most recently, 4 seconds ago) due to excessive
>> rate
>>
>> *[root at localhost ~]# ovs-vsctl get Open_vSwitch . other_config:hw-offload*
>> "true"
>>
>>
>>
>> *[root at localhost ~]# ethtool -k ens11 | grep hw-tc-offload*
>> hw-tc-offload: on [requested off] (Not sure
>> what is "requested off" here)
>>
>> *[root at localhost ~]# ethtool -k eth0 | grep hw-tc-offload*
>> hw-tc-offload: on
>>
>> *[root at localhost ~]# ethtool -k eth1 | grep hw-tc-offload*
>> hw-tc-offload: on
>>
>>
>>
>> Thanks
>>
>> Satish
>>
>>
>>
>> On Wed, Feb 26, 2020 at 1:10 PM Roni Bar Yanai <roniba at mellanox.com>
>> wrote:
>>
>> Hi,
>>
>>
>>
>> eth1 is it the port representor of VF1?
>>
>> eth0 is the up-link representor?
>>
>>
>>
>> Can you share dpif/dump-flows so we can see what are the flows
>>
>> you are trying to offload? Not all flows are supported.
>>
>> Is traffic running? Ping replay and ssh?
>>
>>
>>
>> Thanks.
>>
>>
>>
>> *From:* satish dhote <sdhote926 at gmail.com>
>> *Sent:* Wednesday, February 26, 2020 8:39 AM
>> *To:* Moshe Levi <moshele at mellanox.com>
>> *Cc:* Roni Bar Yanai <roniba at mellanox.com>; ovs-discuss at openvswitch.org
>> *Subject:* Re: [ovs-discuss] TC flow offload using OVS - failed to
>> offload flow: Invalid argument: enp0s25
>>
>>
>>
>> Hi Moshe,
>>
>>
>>
>> Sorry for the confusion. My recent response has new setup details, so
>> please ignore the instructions and setup details which I sent as part of
>> very first email.
>>
>>
>>
>> Here is the fresh setup details and logs, and yes, ens11 is a physical
>> link.
>>
>>
>>
>> *Openvswitch version - *"2.12.0"
>>
>>
>>
>> *Host machine - *
>>
>> OS - Centos7.5
>>
>> mellanox card - ConnectX-4 Lx
>>
>> Physical interface ens11
>>
>>
>>
>> *Virtual machines - *
>>
>> 2 VM's
>>
>> OS - Minimal Centos7.5
>>
>> Attached each VF to each virtual machine
>>
>>
>>
>> *Steps followed to enable sriov and configuring ovs offload*
>>
>> *# lspci | grep -i mell*
>>
>> 01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx]
>>
>>
>>
>> *# ethtool -i ens11 | head -5*
>>
>> driver: mlx5_core
>>
>> version: 4.7-3.2.9
>>
>> firmware-version: 14.26.4012 (MT_2410110034)
>>
>> expansion-rom-version:
>>
>> bus-info: 0000:01:00.0
>>
>>
>>
>> *# cat /sys/class/net/ens11/device/sriov_totalvfs*
>>
>> 4
>>
>>
>>
>> *# echo 2 > /sys/class/net/ens11/device/sriov_numvfs*
>>
>>
>>
>> *# lspci | grep -i mell*
>>
>> 01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx]
>>
>> 01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx Virtual Function]
>>
>> 01:00.2 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx Virtual Function]
>>
>>
>>
>> *#cat /sys/class/net/ens11/device/sriov_numvfs*
>>
>> 2
>>
>>
>>
>> *# echo nic_netdev > /sys/class/net/ens11/compat/devlink/uplink_rep_mode*
>>
>> *# echo 0000:01:00.1 > /sys/bus/pci/drivers/mlx5_core/unbind*
>>
>> *# echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind*
>>
>>
>>
>> *# echo switchdev > /sys/class/net/ens11/compat/devlink/mode*
>>
>>
>>
>> *# systemctl start openvswitch*
>>
>> *# ovs-vsctl add-br ovs-sriov*
>>
>> *# ovs-vsctl set Open_vSwitch . other_config:hw-offload=true*
>>
>> *# systemctl restart openvswitch*
>>
>> *# ovs-vsctl add-port ovs-sriov ens11*
>>
>> *# ovs-vsctl add-port ovs-sriov eth0*
>>
>> *# ovs-vsctl add-port ovs-sriov eth1*
>>
>>
>>
>> # ovs-vsctl show
>> 8bcbcf4b-e1df-4e79-b218-075d9d77568b
>> Bridge ovs-sriov
>> Port "ens11"
>> Interface "ens11"
>> Port "eth1"
>> Interface "eth1"
>> Port ovs-sriov
>> Interface ovs-sriov
>> type: internal
>> Port "eth0"
>> Interface "eth0"
>>
>> *Traffic type from one from one VM to another VM:*
>>
>> ICMP(command: ping), TCP(command: ssh)
>>
>>
>>
>> *Logs:*
>>
>> dpif_netlink(handler6)|ERR|Dropped 5 log messages in last 7 seconds (most
>> recently, 3 seconds ago) due to excessive rate
>>
>> dpif_netlink(handler6)|ERR|failed to offload flow: Invalid argument: ens11
>>
>> dpif_netlink(handler6)|ERR|Dropped 3 log messages in last 8 seconds (most
>> recently, 1 seconds ago) due to excessive rate
>>
>> dpif_netlink(handler6)|ERR|failed to offload flow: Invalid argument: eth1
>>
>>
>>
>>
>>
>> Thanks
>>
>> Satish
>>
>>
>>
>>
>>
>> On Wed, Feb 26, 2020 at 11:38 AM Moshe Levi <moshele at mellanox.com> wrote:
>>
>> You uplink (or physical link) is ens11 so why did you add enp0s25 to the
>> bridge?
>>
>>
>>
>> *From:* discuss <ovs-discuss-bounces at openvswitch.org> *On Behalf Of *satish
>> dhote
>> *Sent:* Wednesday, February 26, 2020 8:01 AM
>> *To:* Roni Bar Yanai <roniba at mellanox.com>
>> *Cc:* ovs-discuss at openvswitch.org
>> *Subject:* Re: [ovs-discuss] TC flow offload using OVS - failed to
>> offload flow: Invalid argument: enp0s25
>>
>>
>>
>> Hi Roni,
>>
>>
>>
>> Thanks for your response.
>>
>> Below are the setup details.
>>
>>
>>
>> *Openvswitch version - *"2.12.0"
>>
>>
>>
>> *Host machine - *
>>
>> OS - Centos7.5
>>
>> mellanox card - ConnectX-4 Lx
>>
>> Physical interface ens11
>>
>>
>>
>> *Virtual machines - *
>>
>> 2 VM's
>>
>> OS - Minimal Centos7.5
>>
>> Attached each VF to each virtual machine
>>
>>
>>
>> *Steps followed to enable sriov and configuring ovs offload*
>>
>> *# lspci | grep -i mell*
>>
>> 01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx]
>>
>>
>>
>> *# ethtool -i ens11 | head -5*
>>
>> driver: mlx5_core
>>
>> version: 4.7-3.2.9
>>
>> firmware-version: 14.26.4012 (MT_2410110034)
>>
>> expansion-rom-version:
>>
>> bus-info: 0000:01:00.0
>>
>>
>>
>> *# cat /sys/class/net/ens11/device/sriov_totalvfs*
>>
>> 4
>>
>>
>>
>> *# echo 2 > /sys/class/net/ens11/device/sriov_numvfs*
>>
>>
>>
>> *# lspci | grep -i mell*
>>
>> 01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx]
>>
>> 01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx Virtual Function]
>>
>> 01:00.2 Ethernet controller: Mellanox Technologies MT27710 Family
>> [ConnectX-4 Lx Virtual Function]
>>
>>
>>
>> *#cat /sys/class/net/ens11/device/sriov_numvfs*
>>
>> 2
>>
>>
>>
>> *# echo nic_netdev > /sys/class/net/ens11/compat/devlink/uplink_rep_mode*
>>
>> *# echo 0000:01:00.1 > /sys/bus/pci/drivers/mlx5_core/unbind*
>>
>> *# echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind*
>>
>>
>>
>> *# echo switchdev > /sys/class/net/ens11/compat/devlink/mode*
>>
>>
>>
>> *# systemctl start openvswitch*
>>
>> *# ovs-vsctl add-br ovs-sriov*
>>
>> *# ovs-vsctl set Open_vSwitch . other_config:hw-offload=true*
>>
>> *# systemctl restart openvswitch*
>>
>> *# ovs-vsctl add-port ovs-sriov ens11*
>>
>> *# ovs-vsctl add-port ovs-sriov eth0*
>>
>> *# ovs-vsctl add-port ovs-sriov eth1*
>>
>>
>>
>> *Traffic type from one from one VM to another VM:* ICMP(command: ping),
>> TCP(command: ssh)
>>
>>
>>
>> let me know if I'm missing anything.
>>
>>
>>
>> Thanks
>>
>> Satish
>>
>>
>>
>> On Mon, Feb 24, 2020 at 2:50 PM Roni Bar Yanai <roniba at mellanox.com>
>> wrote:
>>
>> Hi Satish,
>>
>> Can you share more details of your setup. How it is connected and what is
>> the traffic you run?
>>
>> Are you in switchdev mode?
>>
>> See
>> https://www.mellanox.com/related-docs/prod_software/ASAP2_Hardware_Offloading_for_vSwitches_User_Manual_v4.4.pdf
>>
>> BR,
>>
>> Roni
>>
>> Hi Team,
>>
>>
>>
>> I'm using OVS to understand its feature and workflow and came across the
>> topic of "offloading flow in hardware" through TC.
>>
>> I installed OVS following official guide
>> http://docs.openvswitch.org/en/latest/intro/install/general/#bootstrapping
>> <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.openvswitch.org%2Fen%2Flatest%2Fintro%2Finstall%2Fgeneral%2F%23bootstrapping&data=02%7C01%7Croniba%40mellanox.com%7Cb4addd6f0606436fa3c608d7babf9f72%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637183204488848410&sdata=4CMhcIB9rvTbfGOw6XJTSzSHXrng%2BaWAI1YR5nbptXY%3D&reserved=0>
>>
>>
>>
>> https://docs.mellanox.com/display/MLNXOFEDv473290/OVS+Offload+Using+ASAP2+Direct#OVSOffloadUsingASAP2Direct-Overview
>>
>>
>>
>>
>> I tried to add flows after enabling hw_offload and then added few basic
>> flow using ovs-ofctl utilities. Flows are getting added into ovs-datapath
>> but not getting offloaded into hardware.
>>
>>
>>
>> I'm using mellanox card connectx4, centos 7.5, ovs 2.1.12.
>>
>>
>>
>> *Please let me know whether am I missing something? OR this could be a
>> bug? Below are the various command output for the references, let me know
>> for anything else. *
>>
>>
>>
>> *ovs-vsctl set Open_vSwitch . other_config:hw-offload=true*
>>
>>
>>
>> *root at satish:~# ovs-vsctl show*
>> 966880f9-7702-44b9-be38-3dfc2d3cfb44
>> Bridge test-br
>> Port "enp0s25"
>> Interface "enp0s25"
>> Port test-br
>> Interface test-br
>> type: internal
>> ovs_version: "2.12.0"
>>
>>
>>
>> *root at satish:~# ovs-ofctl dump-flows test-br*
>> cookie=0x0, duration=26540.500s, table=0, n_packets=0, n_bytes=0,
>> in_port=1 actions=output:enp0s25
>> cookie=0x0, duration=31399.514s, table=0, n_packets=18537,
>> n_bytes=2420553, priority=0 actions=NORMAL
>>
>>
>>
>> *root at satish:~# tc filter show dev test-br ingress*
>> root at satish:~#
>> *root at satish:~# tc filter show dev enp0s25 ingress*
>>
>>
>>
>> *satish at satish:~$ sudo tail -f
>> /usr/local/var/log/openvswitch/ovs-vswitchd.log*
>>
>> *..................*
>>
>> 2020-02-20T05:31:20.155Z|00255|dpif_netlink(handler6)|ERR|Dropped 5 log
>> messages in last 7 seconds (most recently, 3 seconds ago) due to excessive
>> rate
>> 2020-02-20T05:31:20.155Z|00256|dpif_netlink(handler6)|ERR|failed to
>> offload flow: Invalid argument: enp0s25
>> 2020-02-20T05:31:30.516Z|00257|dpif_netlink(handler6)|ERR|Dropped 3 log
>> messages in last 8 seconds (most recently, 1 seconds ago) due to excessive
>> rate
>> *2020-02-20T05:31:30.516Z|00258|dpif_netlink(handler6)|ERR|failed to
>> offload flow: Invalid argument: enp0s25*
>>
>>
>>
>>
>>
>> Thanks
>>
>> Satish
>>
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fmailman%2Flistinfo%2Fovs-discuss&data=02%7C01%7Croniba%40mellanox.com%7Cb4addd6f0606436fa3c608d7babf9f72%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637183204488858402&sdata=U7vCpuaynnqCvda3Ru6bb5iE6lAv4hxJgcsceAvJCjs%3D&reserved=0>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200305/976eb70b/attachment-0001.html>
More information about the discuss
mailing list