[ovs-discuss] OVS+DPDK: pci_map_resource(): cannot mmap error

Kapil Adhikesavalu kapil20084 at gmail.com
Sun Jul 31 12:29:45 UTC 2016


Hello,

i am getting "EAL: pci_map_resource(): cannot mmap(18, 0x7f5040000000,
0x80000, 0x0): Invalid argument (0xffffffffffffffff)" when i start
ovs-vswitchd.

Setup:DL360gen8 CPU:E5-2967 NIC:82599ES 10-Gigabit SFI/SFP+ (2 Port) (PCI:
slot0: 04:00.0 04:00.1)
Kernel: 4.6.4-201.fc23.x86_64 ixgbe driver: 4.2.1-k

i have installed : DPDK-16.04  & OVS 2.5.90 - enabled with DPDK

steps after installation:
=====
1.  default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M
hugepages=2048 intel_iommu=off
2.  mount -t hugetlbfs nodev /mnt/huge -o pagesize=1GB
     mount -t hugetlbfs nodev /mnt/huge_2mb -o pagesize=2MB
2. sudo modprobe uio   [note, i am not using VFIO/IOMMU - i tried with that
as well after enable iommu in grub - didnt help - got a different error]
3. tools/dpdk_nic_bind.py -b igb_uio 04:00.0
    tools/dpdk_nic_bind.py -b igb_uio 04:00.1
4. mkdir -p $ovsdir/etc/openvswitch
ovsdb-tool create $ovsdir/etc/openvswitch/conf.db
$ovsdir/usr/share/openvswitch/vswitch.ovsschema

# Bring up ovsdb-server daemon
mkdir -p $ovsdir/var/run/openvswitch
export OVS_DB_SOCK=${ovsdir}/var/run/openvswitch/db.sock
$ovsdir/sbin/ovsdb-server --remote=punix:${OVS_DB_SOCK} \
               --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
               --private-key=db:Open_vSwitch,SSL,private_key \
               --certificate=db:Open_vSwitch,SSL,certificate \
               --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
               --pidfile --detach --verbose=dbg

# Intialize the ovs database
$ovsdir/bin/ovs-vsctl --no-wait init
5. $ovsdir/bin/ovs-vsctl --no-wait set Open_vSwitch .
other_config:dpdk-init=true
   $ovsdir/bin/ovs-vsctl --no-wait set Open_vSwitch .
other_config:dpdk-socket-mem="1024,1024"

   # Use Core 1 for user mode ovsvswitch
   $ovsdir/bin/ovs-vsctl --no-wait set Open_vSwitch .
other_config:dpdk-lcore-mask=0x2

   # Number of memory channels on targeted platform
   $ovsdir/bin/ovs-vsctl --no-wait set Open_vSwitch .
other_config:dpdk-extra="-n 4"

6. $ovsdir/sbin/ovs-vswitchd unix:${OVS_DB_SOCK} --pidfile --verbose=dbg
--detach

Note: i also have another setup with DL360gen9, with the same
configuration(UIO) and it is working without any issues. i am not able to
isolate to what is causing this issue. Appreciate any help.

Error logs:
========
2016-07-31T13:15:57Z|00001|reconnect|DBG|unix:/var/run/openvswitch/db.sock:
entering BACKOFF
2016-07-31T13:15:57Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA
node 0
2016-07-31T13:15:57Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA
node 1
2016-07-31T13:15:57Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU
cores
2016-07-31T13:15:57Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connecting...
2016-07-31T13:15:57Z|00006|reconnect|DBG|unix:/var/run/openvswitch/db.sock:
entering CONNECTING
2016-07-31T13:15:57Z|00007|poll_loop|DBG|wakeup due to [POLLOUT] on fd 10
(<->/var/run/openvswitch/db.sock) at lib/stream-fd.c:151
2016-07-31T13:15:57Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connected
2016-07-31T13:15:57Z|00009|reconnect|DBG|unix:/var/run/openvswitch/db.sock:
entering ACTIVE
2016-07-31T13:15:57Z|00010|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock:
send request, method="get_schema", params=["Open_vSwitch"], id=0
2016-07-31T13:15:57Z|00015|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock:
received reply, result={"locked":true}, id=1
2016-07-31T13:15:57Z|00016|poll_loop|DBG|wakeup due to [POLLIN] on fd 10
(<->/var/run/openvswitch/db.sock) at lib/stream-fd.c:155
2016-07-31T13:15:57Z|00017|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock:
received reply,
result={"Open_vSwitch":{"214ea68a-3cf4-4e8e-a319-4f0ba6205f17":{"initial":{"other_config":["map",[["dpdk-extra","-n
4"],["dpdk-init","true"],["dpdk-lcore-mask","0x2"],["dpdk-socket-mem","1024,1024"]]]}}}},
id=2
2016-07-31T13:15:57Z|00018|dpdk|INFO|DPDK Enabled, initializing
2016-07-31T13:15:57Z|00019|dpdk|INFO|No vhost-sock-dir provided -
defaulting to /var/run/openvswitch

2016-07-31T13:15:57Z|00020|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x2
--socket-mem 1024,1024 -n 4

EAL: Detected lcore 47 as core 13 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 48 lcore(s)
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000000 bytes
EAL: Virtual area found at 0x7f5000000000 (size = 0x200000000)
EAL: Ask a virtual area of 0x200000000 bytes
EAL: Virtual area found at 0x7f4dc0000000 (size = 0x200000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f52bca00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f52bc600000 (size = 0x200000)
EAL: Ask a virtual area of 0x8800000 bytes
EAL: Virtual area found at 0x7f52b3c00000 (size = 0x8800000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f52b3800000 (size = 0x200000)
EAL: Ask a virtual area of 0x3fc00000 bytes
EAL: Virtual area found at 0x7f5273a00000 (size = 0x3fc00000)
EAL: Ask a virtual area of 0x37400000 bytes
EAL: Virtual area found at 0x7f523c400000 (size = 0x37400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f523c000000 (size = 0x200000)
EAL: Ask a virtual area of 0x48000000 bytes
EAL: Virtual area found at 0x7f4d77e00000 (size = 0x48000000)
EAL: Ask a virtual area of 0x38000000 bytes
EAL: Virtual area found at 0x7f5203e00000 (size = 0x38000000)
EAL: Requesting 1 pages of size 1024MB from socket 0
EAL: Requesting 1 pages of size 1024MB from socket 1
EAL: TSC frequency is ~2693520 KHz
EAL: Master lcore 1 is ready (tid=c11bdbc0;cpuset=[1])
*EAL: PCI device 0000:04:00.0 on NUMA socket 0*
*EAL:   probe driver: 8086:10fb rte_ixgbe_pmd*
*EAL: pci_map_resource(): cannot mmap(18, 0x7f5040000000, 0x80000, 0x0):
Invalid argument (0xffffffffffffffff)*
*EAL: Error - exiting with code: 1*
*  Cause: Requested device 0000:04:00.0 cannot be used*

[root at localhost ~]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.6.4-201.fc23.x86_64 root=/dev/mapper/fedora-root ro
rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet
default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M
hugepages=2048 intel_iommu=off

[root at localhost ~]# cat /proc/meminfo | grep uge
AnonHugePages:         0 kB
HugePages_Total:      16
HugePages_Free:       14
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

[root at localhost ~]# lsmod | grep uio
igb_uio                16384  0
uio                    20480  1 igb_uio

[root at localhost ~]#  /localdisk/dpdk/dpdk-16.04/tools/dpdk_nic_bind.py
--status

Network devices using DPDK-compatible driver
============================================
0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio
unused=ixgbe
0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio
unused=ixgbe

Network devices using kernel driver
===================================
0000:03:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3
unused=igb_uio *Active*
0000:03:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3
unused=igb_uio
0000:03:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3
unused=igb_uio
0000:03:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3
unused=igb_uio

[root at localhost ~]# dmesg | grep 04:00
[    3.301717] pci 0000:04:00.0: [8086:10fb] type 00 class 0x020000
[    3.301728] pci 0000:04:00.0: reg 0x10: [mem 0xf7f80000-0xf7ffffff 64bit]
[    3.301733] pci 0000:04:00.0: reg 0x18: [io  0x6000-0x601f]
[    3.301744] pci 0000:04:00.0: reg 0x20: [mem 0xf7f70000-0xf7f73fff 64bit]

[    3.301750] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[    3.301781] pci 0000:04:00.0: PME# supported from D0 D3hot
[    3.301799] pci 0000:04:00.0: reg 0x184: [mem 0xf7e70000-0xf7e73fff
64bit]
[    3.301800] pci 0000:04:00.0: VF(n) BAR0 space: [mem
0xf7e70000-0xf7f6ffff 64bit] (contains BAR0 for 64 VFs)
[    3.301810] pci 0000:04:00.0: reg 0x190: [mem 0xf7d70000-0xf7d73fff
64bit]
[    3.301811] pci 0000:04:00.0: VF(n) BAR3 space: [mem
0xf7d70000-0xf7e6ffff 64bit] (contains BAR3 for 64 VFs)
[    3.301988] pci 0000:04:00.1: [8086:10fb] type 00 class 0x020000
[    3.301999] pci 0000:04:00.1: reg 0x10: [mem 0xf7c80000-0xf7cfffff 64bit]
[    3.302004] pci 0000:04:00.1: reg 0x18: [io  0x6020-0x603f]
[    3.302015] pci 0000:04:00.1: reg 0x20: [mem 0xf7c70000-0xf7c73fff 64bit]
[    3.302021] pci 0000:04:00.1: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[    3.302052] pci 0000:04:00.1: PME# supported from D0 D3hot
[    3.302066] pci 0000:04:00.1: reg 0x184: [mem 0xf7b70000-0xf7b73fff
64bit]
[    3.302067] pci 0000:04:00.1: VF(n) BAR0 space: [mem
0xf7b70000-0xf7c6ffff 64bit] (contains BAR0 for 64 VFs)
[    3.302076] pci 0000:04:00.1: reg 0x190: [mem 0xf7a70000-0xf7a73fff
64bit]
[    3.302078] pci 0000:04:00.1: VF(n) BAR3 space: [mem
0xf7a70000-0xf7b6ffff 64bit] (contains BAR3 for 64 VFs)
[    3.349170] pci 0000:04:00.0: BAR 6: no space for [mem size 0x00080000
pref]
[    3.349172] pci 0000:04:00.0: BAR 6: failed to assign [mem size
0x00080000 pref]
[    3.349173] pci 0000:04:00.1: BAR 6: no space for [mem size 0x00080000
pref]
[    3.349174] pci 0000:04:00.1: BAR 6: failed to assign [mem size
0x00080000 pref]
[    7.293595] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 48,
Tx Queue count = 48
[    7.293735] ixgbe 0000:04:00.0: PCI Express bandwidth of 32GT/s available
[    7.293736] ixgbe 0000:04:00.0: (Speed:5.0GT/s, Width: x8, Encoding
Loss:20%)
[    7.293826] ixgbe 0000:04:00.0: MAC: 2, PHY: 1, PBA No: E66560-005
[    7.293827] ixgbe 0000:04:00.0: 90:e2:ba:1d:18:50
[    7.296841] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[    8.397128] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 48,
Tx Queue count = 48
[    8.397254] ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s available
[    8.397255] ixgbe 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding
Loss:20%)
[    8.397336] ixgbe 0000:04:00.1: MAC: 2, PHY: 1, PBA No: E66560-005
[    8.397337] ixgbe 0000:04:00.1: 90:e2:ba:1d:18:51
[    8.398676] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[    8.399637] ixgbe 0000:04:00.1 ens1f1: renamed from eth1
[    8.409980] ixgbe 0000:04:00.0 ens1f0: renamed from eth0
[   18.835386] ixgbe 0000:04:00.1: registered PHC device on ens1f1
[   19.379900] ixgbe 0000:04:00.0: registered PHC device on ens1f0
[  522.645300] ixgbe 0000:04:00.0: removed PHC on ens1f0
[  522.645300] ixgbe 0000:04:00.0: removed PHC on ens1f0
[  523.099810] ixgbe 0000:04:00.0: complete
[  523.100283] igb_uio 0000:04:00.0: uio device registered with irq 1b
[  523.291072] ixgbe 0000:04:00.1: removed PHC on ens1f1
[  523.738970] ixgbe 0000:04:00.1: complete
[  523.739500] igb_uio 0000:04:00.1: uio device registered with irq 52
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20160731/cd9ef20d/attachment-0002.html>


More information about the discuss mailing list