[ovs-discuss] [ovs-dev] Memory pool issue on delete and re-add of DPDK ports with jumbo MTU >1894

[Stokes, Ian]

> Hi,
> 
> WIth OVS + DPDK, after i add a port with MTU size greater than >1894, if i
> do port delete and re-add the port with same MTU, it leads to following
> memory pool error and MTU is set to 1500.
> While re-adding the port it the MTU is configured less than 1894, issue is
> not seen. To recover, i need to kill the OVS process and start it again.

What I suspect is happening here is that you are allocating too little hugepage memory to OVS with DPDK. From the details below it looks like you are assigning 1024 MB, can you confirm this?

I see the same error message you describe when requesting an MTU >= 1895 and with only 1024 hugepages memory set (although I see the error the first time the MTU is requested).

I'm surprised you only see the issue after deleting the port and re-adding it, can you confirm in your logs that you don't see the error when the port is added the first time with the 1895 MTU request? 

Also can you increase the hugepage memory being assigned to OVS with DPDK, it should allow you to request/set a larger MTU.

Below is a link to a very good article that explains the memory requirement's when using Jumbo frames with OVS with DPDK if you are interested 

https://software.intel.com/en-us/articles/jumbo-frames-in-open-vswitch-with-dpdk?language=ru

Ian

> 
> ovs-vswitchd[14264]: ovs|00002|dpdk|ERR|Insufficient memory to create
> memory pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0
> 
> Issue is not seen when ports use MTU size <=1894. Let me know if
> additional logs are needed.
> 
> OVS version: 2.6.1 and DPDK: 16.07
> ===========================
> 
> root at hp-dl20:/# ovs-vsctl --version
> ovs-vsctl (Open vSwitch) 2.6.1
> DB Schema 7.14.0
> 
> root at hp-dl20:/# ovs-vsctl show
> 176cae66-f14b-436e-9cb2-a7caa054c481
> 
> Issue case with MTU 1895:  (when port with mtu 1895 is readded error is
> seen, able to configure MTU <=1894) ==================== ovs-vsctl add-br
> trail -- set bridge trail datapath_type=netdev ovs-vsctl add-port trail
> dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser -- set
> Interface dpdkvhostuser0 mtu_request=1895 root at hp-dl20:/# ovs-vsctl get
> interface dpdkvhostuser0 mtu
> 1895
> 
> root at hp-dl20:/# ovs-vsctl show
> d5b44a7b-62c1-4096-82a4-722833d3d154
>     Bridge trail
>         Port "dpdkvhostuser0"
>             Interface "dpdkvhostuser0"
>                 type: dpdkvhostuser
>         Port trail
>             Interface trail
>                 type: internal
> 
> ovs-vsctl del-port trail dpdkvhostuser0
> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895
> 
> 2017-02-15T15:39:23.830Z|00002|dpdk|ERR|Insufficient memory to create
> memory pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0
> 
> ovs-vsctl get interface dpdkvhostuser0 mtu
> 1500
> 
> root at hp-dl20:/# ovs-vsctl set Interface dpdkvhostuser0 mtu_request=1894
> root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
> 1894
> 
> Issue case with MTU 1895:
> ====================
> pkill ovs and restart OVS
> 
> root     15206  0.0  0.0  23180  4788 ?        Ss   15:42   0:00
> /usr/sbin/ovsdb-server --remote=punix:/var/run/openvswitch/db.sock
> --remote=db:Open_vSwitch,Open_vSwitch,manager_options
> --private-key=db:Open_vSwitch,SSL,private_key
> --certificate=db:Open_vSwitch,SSL,certificate
> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --
> verbose=err
> root     15213 10.1  0.0 1836764 5960 ?        Ssl  15:42   0:01
> /usr/sbin/ovs-vswitchd --pidfile --log-file --verbose=err --detach
> 
> ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev ovs-vsctl
> add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894
> root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
> 1894
> 
> ovs-vsctl del-port trail dpdkvhostuser0
> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894 ovs-
> vsctl get interface dpdkvhostuser0 mtu
> 1894
> 
> When moved from 1894 to 1895
> ========================
> pkill ovs and restart OVS
> 
> ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev ovs-vsctl
> add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894 ovs-
> vsctl get interface dpdkvhostuser0 mtu
> 1894
> 
> ovs-vsctl del-port trail dpdkvhostuser0
> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895 ovs-
> vsctl get interface dpdkvhostuser0 mtu
> 1895
> ovs-vsctl del-port trail dpdkvhostuser0
> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895 2017-
> 02-15T15:43:28.066Z|00002|dpdk|ERR|Insufficient memory to create memory
> pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0 ovs-vsctl get
> interface dpdkvhostuser0 mtu
> 1500
> 
> root at hp-dl20:/# uname -r
> 4.1.26-yocto-standard
> 
> root at hp-dl20:/# cat /proc/meminfo
> HugePages_Total:    1536
> HugePages_Free:     1024
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB
> 
> Regards
> Kapil.
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev