[ovs-discuss] Error attaching device to DPDK

Darrell Ball dball at vmware.com
Thu Mar 30 19:51:22 UTC 2017



On 3/29/17, 7:40 AM, "O Mahony, Billy" <billy.o.mahony at intel.com> wrote:

    Hi All,
    
    
    
    Just to add something to this conversation that has not been explicitly mentioned below.
    
    
    
    Brad outlines how to set other_config:dpdk-socket-mem to reserve hugepages from both NUMA nodes in order to be sure to avoid pci topology issues.
    
    
    
    Strictly this doesn’t have to be done on both nodes – just on the correct node :) – but sometimes it’s easier to just cover all the bases.
    
    
    
    In addition you will need to ensure that:
    
     * That linux has hugepages available on both (or the right) nodes.
    
     * There is a pmd thread created on each node by setting other_config:pmd-cpu-mask correctly – or again at least on the right node.
    
    
    
    These config items are described in 
    
    * ./Documentation/intro/install/dpdk.rst
    
    * ./Documentation/howto/dpdk.rst
    
    
    
    Currently if there is not a pmd thread on the NUMA node belonging to the PCI device the device will not be polled (a warning message is issued to this effect).


[Darrell]
From Linux POV, the “default” memory policy is to allocate the hugepage pool across numa nodes, if possible;
this assumes an allocation was done, which we have no reports here otherwise.
iirc based on the logs in this case, which I cannot view anymore, there was no pmd mask configuration specified.
The advertised behavior should be, in this case, that a pmd thread is allocated on each numa node with at
least one local dpdk interface bound.
iirc, the dpdk socket memory was left at default, which only pre-allocated from the hugepage pool on node 0
So, does it make sense to also try to pre-allocate hugepages on a numa node, where a PMD thread is allocated ?
Pls correct me in any respects.

 
    
    
    
    There is currently a patch under consideration so that if a NUMA-local pmd is not available for a device then device will be assigned to some other pmd thread. While this is not as efficient due to the need for the data to travel between NUMA nodes it is less frustrating to configure. 
    
    
    
    https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_pipermail_ovs-2Ddev_2017-2DFebruary_329175.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=QmpnHwqCC8ZBxRGZgfCZuF-9SielPs89Yd-k7crooC8&e= 
    
    
    
    If you install the hwloc package you can use “hwloc-calc pci=0000:01:02.0 --intersect Socket” to show the NUMA node that is local to a pci device. However I have seen some systems which report that pci device is local to both nodes. The lstopo command can also be used for this.
    
    
    
    Hope some of that helps,
    
    
    
    Billy.
    
    
    
    
    
    From: ovs-discuss-bounces at openvswitch.org [mailto:ovs-discuss-bounces at openvswitch.org] On Behalf Of Darrell Ball
    
    Sent: Wednesday, March 29, 2017 2:36 AM
    
    To: Brad Cowie <brad at cowie.nz>; Shivaram Mysore <shivaram.mysore at gmail.com>
    
    Cc: ovs-discuss at openvswitch.org
    
    Subject: Re: [ovs-discuss] Error attaching device to DPDK
    
    
    
    
    
    
    
    From: <ovs-discuss-bounces at openvswitch.org> on behalf of Brad Cowie <brad at cowie.nz>
    
    Date: Tuesday, March 28, 2017 at 5:35 PM
    
    To: Shivaram Mysore <shivaram.mysore at gmail.com>
    
    Cc: "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
    
    Subject: Re: [ovs-discuss] Error attaching device to DPDK
    
    
    
    Hi folks,
    
    I have been helping Shivaram off-list with this problem and just wanted to update this thread with the solution.
    
    Basically the issue was with how hugepages were being preallocated by OVS for use with DPDK. I believe by default OVS sets other_config:dpdk-socket-mem to "1024,0". Which will give one 1GB hugepage to CPU0 and none to CPU1.
    
    The ports Shivaram was trying to assign to DPDK were connected physically via a PCI-E bus to CPU1 and since there are no hugepages preallocated there for DPDK then we can't add it.
    
    The fix is simple, to assign a hugepage to CPU1:
    
    
    
    $ sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024,1024"
    
    In OVS 2.7.0 unfortunately due to an issue (that is now fixed by commit ef6cca6fdc67f3cee21c6bb1c13c4ca7f8241314) you get a segfault rather than an error message telling you the problem.
    
    In OVS 2.7.90 however the issue becomes quite clear as there is a nice log message like such:
    
    
    
    2017-03-29T00:19:57Z|00067|netdev_dpdk|ERR|Insufficient memory to create memory pool for netdev dpdk-p0, with MTU 1500 on socket 1
    
    There is documentation regarding these fields, but for case, the documentation could simply state something like
    
    “you need to allocate hugepage memory on each numa node where dpdk ports are bound and or those ports
    
    will not work with dpdk”
    
    The error log could also be a little more succinct like “no memory pool allocated for netdev dpdk-p0,.., usage on socket 1”
    
    
    
    Possibly, going even further, when a PMD thread gets auto-allocated to run on a given numa node,
    
    memory allocation is defaulted there as well. If it cannot be allocated, an error log is generated.
    
    
    
    Unfortunately to those unfamiliar with DPDK and NUMA architectures this won't be very obvious. Potentially we could add some additional help to the DPDK documentation pages for OVS that explains the other_config options you may need to reconfigure on multi-CPU NUMA machines?
    
    Brad
    
    
    
    On 29 March 2017 at 10:02, Shivaram Mysore <shivaram.mysore at gmail.com> wrote:
    
    Ok thanks.  Will remember the same.  Right now, I will not specify anything as I have to get this to work first!
    
    
    
    On Tue, Mar 28, 2017 at 2:00 PM, Bodireddy, Bhanuprakash <bhanuprakash.bodireddy at intel.com> wrote:
    
    >Question:
    
    >
    
    >I don't have any specific cpu cores associated with DPDK.  Will this make any
    
    >difference?  I also did not see this requirement as a MUST have in the
    
    >documentation.
    
    
    
    I assume your question is on PMD threads. If you don’t specify any thing explicitly
    
    PMD pmd thread shall be created and pinned to core 0.  You can explicitly select the no. of pmd threads
    
    and the cores by explicitly setting the affinity mask using 'other_config:pmd-cpu-mask'.
    
    
    
    Based on the affinity mask, the PMD threads shall be spawned and the rx queues will be uniformly
    
    distributed among the PMD threads.
    
    
    
    Refer https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.openvswitch.org_en_latest_howto_dpdk_&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=JNQjCdBcYJG8H1n4bT6k_zFhtkk6wcRE27AhEuQoqjg&e= .
    
    
    
    Also note that 'dpdk-lcore-mask' should be used to set the dpdk lcore threads and vswitchd.
    
    
    
    - Bhanuprakash.
    
    
    
    >
    
    ># cat /proc/cmdline
    
    >BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic root=/dev/mapper/intelsdn-
    
    >-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G
    
    >hugepagesz=1G hugepages=4
    
    >
    
    >
    
    >Thanks
    
    >
    
    >/Shivaram
    
    >
    
    >On Tue, Mar 28, 2017 at 1:32 PM, Shivaram Mysore
    
    ><shivaram.mysore at gmail.com> wrote:
    
    >I was advised on this mailing list that VFIO drivers are better than IGB_UIO +
    
    >more forgiving [1].  Hence switched to the same.
    
    >
    
    >[1] https://urldefense.proofpoint.com/v2/url?u=http-3A__dpdk-2Dguide.gitlab.io_dpdk-2Dguide_setup_binding.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=KgBSWUsNwqMWB_Pm2CHmGAVAO5enRJxRlvbS9MR-7Ww&e= 
    
    >
    
    >/Shivaram
    
    >
    
    >On Tue, Mar 28, 2017 at 1:12 PM, Bodireddy, Bhanuprakash
    
    ><bhanuprakash.bodireddy at intel.com> wrote:
    
    >
    
    >>any hints for me based on the info provided?
    
    >
    
    >Your BIOS options and bootcmd line options are appropriate and no issues
    
    >there. I wanted to check if you tried igb_uio drivers
    
    >but saw your post here https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_pipermail_ovs-2D&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=qoh0Qu6HkVOjRIO_oNqdzBGTGjRdo5tppjkG2R3KpEw&e= 
    
    >discuss/2017-March/043993.html, reporting a crash with UIO drivers.
    
    >
    
    >Is that the reason why you moved to VFIO? I cross checked your logs but
    
    >found nothing suspicious there.
    
    >
    
    >-Bhanuprakash.
    
    >
    
    >>
    
    >>Thanks
    
    >>
    
    >>On Tue, Mar 28, 2017 at 8:41 AM, Shivaram Mysore
    
    >><shivaram.mysore at gmail.com> wrote:
    
    >>Hi,
    
    >>
    
    >># dmesg | grep -e DMAR -e IOMMU
    
    >>[    0.000000] ACPI: DMAR 0x00000000798AF340 000158 (v01 SUPERM SMCI--
    
    >>MB 00000001 INTL 20091013)
    
    >>[    0.000000] DMAR: IOMMU enabled
    
    >>[    0.109987] DMAR: Host address width 46
    
    >>[    0.109988] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
    
    >>[    0.109997] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
    
    >>d2078c106f0466 ecap f020de
    
    >>[    0.109998] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
    
    >>[    0.110002] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
    
    >>d2078c106f0466 ecap f020de
    
    >>[    0.110003] DMAR: RMRR base: 0x0000007ba75000 end: 0x0000007ba84fff
    
    >>[    0.110004] DMAR: ATSR flags: 0x0
    
    >>[    0.110005] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
    
    >>[    0.110006] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
    
    >>[    0.110008] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
    
    >>[    0.110008] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 1
    
    >>[    0.110009] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 1
    
    >>[    0.110010] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
    
    >>[    0.110011] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out
    
    >bit.
    
    >>[    0.110012] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the
    
    >>BIOS setting.
    
    >>[    0.110902] DMAR-IR: Enabled IRQ remapping in xapic mode
    
    >>[    1.687595] DMAR: dmar0: Using Queued invalidation
    
    >>[    1.687608] DMAR: dmar1: Using Queued invalidation
    
    >>[    1.688473] DMAR: Hardware identity mapping for device 0000:ff:08.0
    
    >>[    1.688475] DMAR: Hardware identity mapping for device 0000:ff:08.2
    
    >>[    1.688477] DMAR: Hardware identity mapping for device 0000:ff:08.3
    
    >>[    1.688478] DMAR: Hardware identity mapping for device 0000:ff:09.0
    
    >>[    1.688480] DMAR: Hardware identity mapping for device 0000:ff:09.2
    
    >>[    1.688482] DMAR: Hardware identity mapping for device 0000:ff:09.3
    
    >>[    1.688483] DMAR: Hardware identity mapping for device 0000:ff:0b.0
    
    >>[    1.688485] DMAR: Hardware identity mapping for device 0000:ff:0b.1
    
    >>[    1.688487] DMAR: Hardware identity mapping for device 0000:ff:0b.2
    
    >>[    1.688488] DMAR: Hardware identity mapping for device 0000:ff:0c.0
    
    >>[    1.688490] DMAR: Hardware identity mapping for device 0000:ff:0c.1
    
    >>[    1.688491] DMAR: Hardware identity mapping for device 0000:ff:0c.2
    
    >>[    1.688493] DMAR: Hardware identity mapping for device 0000:ff:0c.3
    
    >>[    1.688494] DMAR: Hardware identity mapping for device 0000:ff:0c.4
    
    >>[    1.688496] DMAR: Hardware identity mapping for device 0000:ff:0c.5
    
    >>[    1.688498] DMAR: Hardware identity mapping for device 0000:ff:0c.6
    
    >>[    1.688499] DMAR: Hardware identity mapping for device 0000:ff:0c.7
    
    >>[    1.688500] DMAR: Hardware identity mapping for device 0000:ff:0d.0
    
    >>[    1.688502] DMAR: Hardware identity mapping for device 0000:ff:0d.1
    
    >>[    1.688504] DMAR: Hardware identity mapping for device 0000:ff:0d.2
    
    >>[    1.688505] DMAR: Hardware identity mapping for device 0000:ff:0d.3
    
    >>[    1.688506] DMAR: Hardware identity mapping for device 0000:ff:0d.4
    
    >>[    1.688508] DMAR: Hardware identity mapping for device 0000:ff:0d.5
    
    >>[    1.688509] DMAR: Hardware identity mapping for device 0000:ff:0d.6
    
    >>[    1.688510] DMAR: Hardware identity mapping for device 0000:ff:0d.7
    
    >>[    1.688511] DMAR: Hardware identity mapping for device 0000:ff:0e.0
    
    >>[    1.688513] DMAR: Hardware identity mapping for device 0000:ff:0e.1
    
    >>[    1.688514] DMAR: Hardware identity mapping for device 0000:ff:0f.0
    
    >>[    1.688516] DMAR: Hardware identity mapping for device 0000:ff:0f.1
    
    >>[    1.688517] DMAR: Hardware identity mapping for device 0000:ff:0f.2
    
    >>[    1.688518] DMAR: Hardware identity mapping for device 0000:ff:0f.3
    
    >>[    1.688520] DMAR: Hardware identity mapping for device 0000:ff:0f.4
    
    >>[    1.688521] DMAR: Hardware identity mapping for device 0000:ff:0f.5
    
    >>[    1.688523] DMAR: Hardware identity mapping for device 0000:ff:0f.6
    
    >>[    1.688524] DMAR: Hardware identity mapping for device 0000:ff:10.0
    
    >>[    1.688526] DMAR: Hardware identity mapping for device 0000:ff:10.1
    
    >>[    1.688527] DMAR: Hardware identity mapping for device 0000:ff:10.5
    
    >>[    1.688529] DMAR: Hardware identity mapping for device 0000:ff:10.6
    
    >>[    1.688530] DMAR: Hardware identity mapping for device 0000:ff:10.7
    
    >>[    1.688532] DMAR: Hardware identity mapping for device 0000:ff:12.0
    
    >>[    1.688533] DMAR: Hardware identity mapping for device 0000:ff:12.1
    
    >>[    1.688534] DMAR: Hardware identity mapping for device 0000:ff:12.4
    
    >>[    1.688536] DMAR: Hardware identity mapping for device 0000:ff:12.5
    
    >>[    1.688537] DMAR: Hardware identity mapping for device 0000:ff:13.0
    
    >>[    1.688539] DMAR: Hardware identity mapping for device 0000:ff:13.1
    
    >>[    1.688541] DMAR: Hardware identity mapping for device 0000:ff:13.2
    
    >>[    1.688542] DMAR: Hardware identity mapping for device 0000:ff:13.3
    
    >>[    1.688543] DMAR: Hardware identity mapping for device 0000:ff:13.6
    
    >>[    1.688545] DMAR: Hardware identity mapping for device 0000:ff:13.7
    
    >>[    1.688546] DMAR: Hardware identity mapping for device 0000:ff:14.0
    
    >>[    1.688547] DMAR: Hardware identity mapping for device 0000:ff:14.1
    
    >>[    1.688549] DMAR: Hardware identity mapping for device 0000:ff:14.2
    
    >>[    1.688550] DMAR: Hardware identity mapping for device 0000:ff:14.3
    
    >>[    1.688552] DMAR: Hardware identity mapping for device 0000:ff:14.4
    
    >>[    1.688553] DMAR: Hardware identity mapping for device 0000:ff:14.5
    
    >>[    1.688559] DMAR: Hardware identity mapping for device 0000:ff:14.6
    
    >>[    1.688561] DMAR: Hardware identity mapping for device 0000:ff:14.7
    
    >>[    1.688562] DMAR: Hardware identity mapping for device 0000:ff:16.0
    
    >>[    1.688564] DMAR: Hardware identity mapping for device 0000:ff:16.1
    
    >>[    1.688565] DMAR: Hardware identity mapping for device 0000:ff:16.2
    
    >>[    1.688566] DMAR: Hardware identity mapping for device 0000:ff:16.3
    
    >>[    1.688568] DMAR: Hardware identity mapping for device 0000:ff:16.6
    
    >>[    1.688569] DMAR: Hardware identity mapping for device 0000:ff:16.7
    
    >>[    1.688571] DMAR: Hardware identity mapping for device 0000:ff:17.0
    
    >>[    1.688573] DMAR: Hardware identity mapping for device 0000:ff:17.1
    
    >>[    1.688574] DMAR: Hardware identity mapping for device 0000:ff:17.2
    
    >>[    1.688576] DMAR: Hardware identity mapping for device 0000:ff:17.3
    
    >>[    1.688577] DMAR: Hardware identity mapping for device 0000:ff:17.4
    
    >>[    1.688579] DMAR: Hardware identity mapping for device 0000:ff:17.5
    
    >>[    1.688581] DMAR: Hardware identity mapping for device 0000:ff:17.6
    
    >>[    1.688582] DMAR: Hardware identity mapping for device 0000:ff:17.7
    
    >>[    1.688584] DMAR: Hardware identity mapping for device 0000:ff:1e.0
    
    >>[    1.688586] DMAR: Hardware identity mapping for device 0000:ff:1e.1
    
    >>[    1.688588] DMAR: Hardware identity mapping for device 0000:ff:1e.2
    
    >>[    1.688590] DMAR: Hardware identity mapping for device 0000:ff:1e.3
    
    >>[    1.688592] DMAR: Hardware identity mapping for device 0000:ff:1e.4
    
    >>[    1.688594] DMAR: Hardware identity mapping for device 0000:ff:1f.0
    
    >>[    1.688596] DMAR: Hardware identity mapping for device 0000:ff:1f.2
    
    >>[    1.688610] DMAR: Hardware identity mapping for device 0000:7f:08.0
    
    >>[    1.688612] DMAR: Hardware identity mapping for device 0000:7f:08.2
    
    >>[    1.688614] DMAR: Hardware identity mapping for device 0000:7f:08.3
    
    >>[    1.688616] DMAR: Hardware identity mapping for device 0000:7f:09.0
    
    >>[    1.688617] DMAR: Hardware identity mapping for device 0000:7f:09.2
    
    >>[    1.688619] DMAR: Hardware identity mapping for device 0000:7f:09.3
    
    >>[    1.688621] DMAR: Hardware identity mapping for device 0000:7f:0b.0
    
    >>[    1.688623] DMAR: Hardware identity mapping for device 0000:7f:0b.1
    
    >>[    1.688625] DMAR: Hardware identity mapping for device 0000:7f:0b.2
    
    >>[    1.688627] DMAR: Hardware identity mapping for device 0000:7f:0c.0
    
    >>[    1.688629] DMAR: Hardware identity mapping for device 0000:7f:0c.1
    
    >>[    1.688631] DMAR: Hardware identity mapping for device 0000:7f:0c.2
    
    >>[    1.688632] DMAR: Hardware identity mapping for device 0000:7f:0c.3
    
    >>[    1.688634] DMAR: Hardware identity mapping for device 0000:7f:0c.4
    
    >>[    1.688636] DMAR: Hardware identity mapping for device 0000:7f:0c.5
    
    >>[    1.688637] DMAR: Hardware identity mapping for device 0000:7f:0c.6
    
    >>[    1.688639] DMAR: Hardware identity mapping for device 0000:7f:0c.7
    
    >>[    1.688641] DMAR: Hardware identity mapping for device 0000:7f:0d.0
    
    >>[    1.688642] DMAR: Hardware identity mapping for device 0000:7f:0d.1
    
    >>[    1.688644] DMAR: Hardware identity mapping for device 0000:7f:0d.2
    
    >>[    1.688646] DMAR: Hardware identity mapping for device 0000:7f:0d.3
    
    >>[    1.688647] DMAR: Hardware identity mapping for device 0000:7f:0d.4
    
    >>[    1.688649] DMAR: Hardware identity mapping for device 0000:7f:0d.5
    
    >>[    1.688650] DMAR: Hardware identity mapping for device 0000:7f:0d.6
    
    >>[    1.688652] DMAR: Hardware identity mapping for device 0000:7f:0d.7
    
    >>[    1.688653] DMAR: Hardware identity mapping for device 0000:7f:0e.0
    
    >>[    1.688655] DMAR: Hardware identity mapping for device 0000:7f:0e.1
    
    >>[    1.688657] DMAR: Hardware identity mapping for device 0000:7f:0f.0
    
    >>[    1.688658] DMAR: Hardware identity mapping for device 0000:7f:0f.1
    
    >>[    1.688660] DMAR: Hardware identity mapping for device 0000:7f:0f.2
    
    >>[    1.688661] DMAR: Hardware identity mapping for device 0000:7f:0f.3
    
    >>[    1.688663] DMAR: Hardware identity mapping for device 0000:7f:0f.4
    
    >>[    1.688664] DMAR: Hardware identity mapping for device 0000:7f:0f.5
    
    >>[    1.688666] DMAR: Hardware identity mapping for device 0000:7f:0f.6
    
    >>[    1.688668] DMAR: Hardware identity mapping for device 0000:7f:10.0
    
    >>[    1.688669] DMAR: Hardware identity mapping for device 0000:7f:10.1
    
    >>[    1.688671] DMAR: Hardware identity mapping for device 0000:7f:10.5
    
    >>[    1.688673] DMAR: Hardware identity mapping for device 0000:7f:10.6
    
    >>[    1.688674] DMAR: Hardware identity mapping for device 0000:7f:10.7
    
    >>[    1.688676] DMAR: Hardware identity mapping for device 0000:7f:12.0
    
    >>[    1.688678] DMAR: Hardware identity mapping for device 0000:7f:12.1
    
    >>[    1.688684] DMAR: Hardware identity mapping for device 0000:7f:12.4
    
    >>[    1.688686] DMAR: Hardware identity mapping for device 0000:7f:12.5
    
    >>[    1.688688] DMAR: Hardware identity mapping for device 0000:7f:13.0
    
    >>[    1.688689] DMAR: Hardware identity mapping for device 0000:7f:13.1
    
    >>[    1.688691] DMAR: Hardware identity mapping for device 0000:7f:13.2
    
    >>[    1.688692] DMAR: Hardware identity mapping for device 0000:7f:13.3
    
    >>[    1.688694] DMAR: Hardware identity mapping for device 0000:7f:13.6
    
    >>[    1.688696] DMAR: Hardware identity mapping for device 0000:7f:13.7
    
    >>[    1.688697] DMAR: Hardware identity mapping for device 0000:7f:14.0
    
    >>[    1.688699] DMAR: Hardware identity mapping for device 0000:7f:14.1
    
    >>[    1.688701] DMAR: Hardware identity mapping for device 0000:7f:14.2
    
    >>[    1.688703] DMAR: Hardware identity mapping for device 0000:7f:14.3
    
    >>[    1.688704] DMAR: Hardware identity mapping for device 0000:7f:14.4
    
    >>[    1.688706] DMAR: Hardware identity mapping for device 0000:7f:14.5
    
    >>[    1.688708] DMAR: Hardware identity mapping for device 0000:7f:14.6
    
    >>[    1.688710] DMAR: Hardware identity mapping for device 0000:7f:14.7
    
    >>[    1.688712] DMAR: Hardware identity mapping for device 0000:7f:16.0
    
    >>[    1.688714] DMAR: Hardware identity mapping for device 0000:7f:16.1
    
    >>[    1.688716] DMAR: Hardware identity mapping for device 0000:7f:16.2
    
    >>[    1.688718] DMAR: Hardware identity mapping for device 0000:7f:16.3
    
    >>[    1.688720] DMAR: Hardware identity mapping for device 0000:7f:16.6
    
    >>[    1.688722] DMAR: Hardware identity mapping for device 0000:7f:16.7
    
    >>[    1.688724] DMAR: Hardware identity mapping for device 0000:7f:17.0
    
    >>[    1.688726] DMAR: Hardware identity mapping for device 0000:7f:17.1
    
    >>[    1.688729] DMAR: Hardware identity mapping for device 0000:7f:17.2
    
    >>[    1.688731] DMAR: Hardware identity mapping for device 0000:7f:17.3
    
    >>[    1.688733] DMAR: Hardware identity mapping for device 0000:7f:17.4
    
    >>[    1.688735] DMAR: Hardware identity mapping for device 0000:7f:17.5
    
    >>[    1.688737] DMAR: Hardware identity mapping for device 0000:7f:17.6
    
    >>[    1.688739] DMAR: Hardware identity mapping for device 0000:7f:17.7
    
    >>[    1.688741] DMAR: Hardware identity mapping for device 0000:7f:1e.0
    
    >>[    1.688742] DMAR: Hardware identity mapping for device 0000:7f:1e.1
    
    >>[    1.688744] DMAR: Hardware identity mapping for device 0000:7f:1e.2
    
    >>[    1.688746] DMAR: Hardware identity mapping for device 0000:7f:1e.3
    
    >>[    1.688748] DMAR: Hardware identity mapping for device 0000:7f:1e.4
    
    >>[    1.688750] DMAR: Hardware identity mapping for device 0000:7f:1f.0
    
    >>[    1.688752] DMAR: Hardware identity mapping for device 0000:7f:1f.2
    
    >>[    1.688769] DMAR: Hardware identity mapping for device 0000:00:00.0
    
    >>[    1.688774] DMAR: Hardware identity mapping for device 0000:00:01.0
    
    >>[    1.688778] DMAR: Hardware identity mapping for device 0000:00:01.1
    
    >>[    1.688783] DMAR: Hardware identity mapping for device 0000:00:03.0
    
    >>[    1.688785] DMAR: Hardware identity mapping for device 0000:00:04.0
    
    >>[    1.688787] DMAR: Hardware identity mapping for device 0000:00:04.1
    
    >>[    1.688790] DMAR: Hardware identity mapping for device 0000:00:04.2
    
    >>[    1.688792] DMAR: Hardware identity mapping for device 0000:00:04.3
    
    >>[    1.688794] DMAR: Hardware identity mapping for device 0000:00:04.4
    
    >>[    1.688797] DMAR: Hardware identity mapping for device 0000:00:04.5
    
    >>[    1.688799] DMAR: Hardware identity mapping for device 0000:00:04.6
    
    >>[    1.688801] DMAR: Hardware identity mapping for device 0000:00:04.7
    
    >>[    1.688803] DMAR: Hardware identity mapping for device 0000:00:05.0
    
    >>[    1.688807] DMAR: Hardware identity mapping for device 0000:00:05.1
    
    >>[    1.688809] DMAR: Hardware identity mapping for device 0000:00:05.2
    
    >>[    1.688811] DMAR: Hardware identity mapping for device 0000:00:05.4
    
    >>[    1.688814] DMAR: Hardware identity mapping for device 0000:00:11.0
    
    >>[    1.688816] DMAR: Hardware identity mapping for device 0000:00:11.4
    
    >>[    1.688818] DMAR: Hardware identity mapping for device 0000:00:14.0
    
    >>[    1.688821] DMAR: Hardware identity mapping for device 0000:00:16.0
    
    >>[    1.688823] DMAR: Hardware identity mapping for device 0000:00:16.1
    
    >>[    1.688825] DMAR: Hardware identity mapping for device 0000:00:1a.0
    
    >>[    1.688828] DMAR: Hardware identity mapping for device 0000:00:1c.0
    
    >>[    1.688831] DMAR: Hardware identity mapping for device 0000:00:1c.2
    
    >>[    1.688833] DMAR: Hardware identity mapping for device 0000:00:1d.0
    
    >>[    1.688835] DMAR: Hardware identity mapping for device 0000:00:1f.0
    
    >>[    1.688837] DMAR: Hardware identity mapping for device 0000:00:1f.2
    
    >>[    1.688844] DMAR: Hardware identity mapping for device 0000:00:1f.3
    
    >>[    1.688846] DMAR: Hardware identity mapping for device 0000:00:1f.6
    
    >>[    1.688864] DMAR: Hardware identity mapping for device 0000:01:00.0
    
    >>[    1.688870] DMAR: Hardware identity mapping for device 0000:01:00.1
    
    >>[    1.688876] DMAR: Hardware identity mapping for device 0000:01:00.2
    
    >>[    1.688883] DMAR: Hardware identity mapping for device 0000:01:00.3
    
    >>[    1.688895] DMAR: Hardware identity mapping for device 0000:80:01.0
    
    >>[    1.688899] DMAR: Hardware identity mapping for device 0000:80:02.0
    
    >>[    1.688903] DMAR: Hardware identity mapping for device 0000:80:03.0
    
    >>[    1.688906] DMAR: Hardware identity mapping for device 0000:80:04.0
    
    >>[    1.688908] DMAR: Hardware identity mapping for device 0000:80:04.1
    
    >>[    1.688911] DMAR: Hardware identity mapping for device 0000:80:04.2
    
    >>[    1.688913] DMAR: Hardware identity mapping for device 0000:80:04.3
    
    >>[    1.688915] DMAR: Hardware identity mapping for device 0000:80:04.4
    
    >>[    1.688918] DMAR: Hardware identity mapping for device 0000:80:04.5
    
    >>[    1.688920] DMAR: Hardware identity mapping for device 0000:80:04.6
    
    >>[    1.688923] DMAR: Hardware identity mapping for device 0000:80:04.7
    
    >>[    1.688936] DMAR: Hardware identity mapping for device 0000:80:05.0
    
    >>[    1.688940] DMAR: Hardware identity mapping for device 0000:80:05.1
    
    >>[    1.688943] DMAR: Hardware identity mapping for device 0000:80:05.2
    
    >>[    1.688945] DMAR: Hardware identity mapping for device 0000:80:05.4
    
    >>[    1.688956] DMAR: Hardware identity mapping for device 0000:81:00.0
    
    >>[    1.688961] DMAR: Hardware identity mapping for device 0000:81:00.1
    
    >>[    1.688974] DMAR: Hardware identity mapping for device 0000:82:00.0
    
    >>[    1.688980] DMAR: Hardware identity mapping for device 0000:82:00.1
    
    >>[    1.688986] DMAR: Hardware identity mapping for device 0000:82:00.2
    
    >>[    1.688992] DMAR: Hardware identity mapping for device 0000:82:00.3
    
    >>[    1.689006] DMAR: Hardware identity mapping for device 0000:83:00.0
    
    >>[    1.689012] DMAR: Hardware identity mapping for device 0000:83:00.1
    
    >>[    1.689018] DMAR: Hardware identity mapping for device 0000:83:00.2
    
    >>[    1.689024] DMAR: Hardware identity mapping for device 0000:83:00.3
    
    >>[    1.689025] DMAR: Setting RMRR:
    
    >>[    1.689026] DMAR: Ignoring identity map for HW passthrough device
    
    >>0000:00:14.0 [0x7ba75000 - 0x7ba84fff]
    
    >>[    1.689027] DMAR: Ignoring identity map for HW passthrough device
    
    >>0000:00:1a.0 [0x7ba75000 - 0x7ba84fff]
    
    >>[    1.689028] DMAR: Ignoring identity map for HW passthrough device
    
    >>0000:00:1d.0 [0x7ba75000 - 0x7ba84fff]
    
    >>[    1.689036] DMAR: Prepare 0-16MiB unity mapping for LPC
    
    >>[    1.689036] DMAR: Ignoring identity map for HW passthrough device
    
    >>0000:00:1f.0 [0x0 - 0xffffff]
    
    >>[    1.689040] DMAR: Intel(R) Virtualization Technology for Directed I/O
    
    >>[    2.299766] DMAR: 32bit 0000:00:11.4 uses non-identity mapping
    
    >>[    2.301941] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping
    
    >>[    2.302679] DMAR: Setting identity map for device 0000:00:1a.0 [0x7ba75000
    
    >-
    
    >>0x7ba84fff]
    
    >>[    2.320910] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
    
    >>[    2.321685] DMAR: Setting identity map for device 0000:00:1d.0
    
    >[0x7ba75000
    
    >>- 0x7ba84fff]
    
    >>#
    
    >>
    
    >>
    
    >>On Tue, Mar 28, 2017 at 1:46 AM, Bodireddy, Bhanuprakash
    
    >><bhanuprakash.bodireddy at intel.com> wrote:
    
    >>Hello shivaram,
    
    >>
    
    >>Darrell has already asked if VT-d is enabled?
    
    >>Your cmdline options are appropriate as you have iommu
    
    >>enabled(intel_iommu=on iommu=pt).
    
    >>can you also check the output of below mentioned command to see if VT-d
    
    >is
    
    >>enabled in BIOS?
    
    >>
    
    >>$ dmesg | grep -e DMAR -e IOMMU
    
    >>
    
    >>Regards,
    
    >>Bhanuprakash.
    
    >>
    
    >>From: ovs-discuss-bounces at openvswitch.org [mailto:ovs-discuss-
    
    >>bounces at openvswitch.org] On Behalf Of Shivaram Mysore
    
    >>Sent: Tuesday, March 28, 2017 6:25 AM
    
    >>To: Darrell Ball <dball at vmware.com>
    
    >>Cc: ovs-discuss at openvswitch.org
    
    >>Subject: Re: [ovs-discuss] Error attaching device to DPDK
    
    >>
    
    >># uname -r
    
    >>4.10.0-14-generic
    
    >>
    
    >># cat /proc/cmdline
    
    >>BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic
    
    >root=/dev/mapper/intelsdn-
    
    >>-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G
    
    >>hugepagesz=1G hugepages=4
    
    >>
    
    >># uname -a
    
    >>Linux intelsdn 4.10.0-14-generic #16-Ubuntu SMP Fri Mar 17 15:19:26 UTC
    
    >2017
    
    >>x86_64 x86_64 x86_64 GNU/Linux
    
    >>
    
    >># cat /etc/*release
    
    >>DISTRIB_ID=Ubuntu
    
    >>DISTRIB_RELEASE=17.04
    
    >>DISTRIB_CODENAME=zesty
    
    >>DISTRIB_DESCRIPTION="Ubuntu Zesty Zapus (development branch)"
    
    >>NAME="Ubuntu"
    
    >>VERSION="17.04 (Zesty Zapus)"
    
    >>ID=ubuntu
    
    >>ID_LIKE=debian
    
    >>PRETTY_NAME="Ubuntu Zesty Zapus (development branch)"
    
    >>VERSION_ID="17.04"
    
    >>
    
    >>
    
    >>Nothing in the syslog related
    
    >>
    
    >>On Mon, Mar 27, 2017 at 10:02 PM, Darrell Ball <dball at vmware.com> wrote:
    
    >>I see the pastebins now.
    
    >>
    
    >>What kernel version are you using ?
    
    >>VT-d enabled ?
    
    >>Anything related in syslog ?
    
    >>
    
    >>From: Shivaram Mysore <shivaram.mysore at gmail.com>
    
    >>Date: Monday, March 27, 2017 at 9:34 PM
    
    >>To: Darrell Ball <dball at vmware.com>
    
    >>Cc: "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
    
    >>Subject: Re: [ovs-discuss] Error attaching device to DPDK
    
    >>
    
    >>One more datapoint, please check the paste bin log in URL sent
    
    >earlier.  Dpdk-
    
    >>devbind works and you can see results in the paste bin log
    
    >>
    
    >>/Shivaram
    
    >>::Sent from my mobile device::
    
    >>
    
    >>On Mar 27, 2017, at 8:34 PM, Darrell Ball <dball at vmware.com> wrote:
    
    >>see inline below
    
    >>
    
    >>But a few questions:
    
    >>1)      Are you using the recommended version of dpdk with OVS 2.7 – 16.11 ?
    
    >>2)      Are you getting a crash still or is this post crash ?
    
    >>3)      Was the previous crash still happening with 16.11 or only 17.02 ?
    
    >>
    
    >>Thanks
    
    >>Darrell
    
    >>
    
    >>From: <ovs-discuss-bounces at openvswitch.org> on behalf of Shivaram
    
    >>Mysore <shivaram.mysore at gmail.com>
    
    >>Date: Monday, March 27, 2017 at 7:45 PM
    
    >>To: "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
    
    >>Subject: [ovs-discuss] Error attaching device to DPDK
    
    >>
    
    >>Hello,
    
    >>
    
    >>I have installed OVS 2.7 running.  When I do:
    
    >>1.      # ovs-vsctl add-port ovs-ip64-br0 dpdk-p0 -- set Interface dpdk-p0
    
    >>type=dpdk options:dpdk-devargs=0000:82:00.0
    
    >>2.      ovs-vsctl: Error detected while setting up 'dpdk-p0': Error attaching
    
    >>device '0000:82:00.0' to DPDK.  See ovs-vswitchd log for details.
    
    >>
    
    >>Possibly, the port is not bound.
    
    >>Check using
    
    >>$DPDK_DIR/tools/dpdk-devbind.py –status
    
    >>and see the documentation for more detail
    
    >>
    
    >>and the log file shows:
    
    >>1.      2017-03-28T02:04:40.004Z|00064|netdev_dpdk|WARN|Error attaching
    
    >>device '0000:82:00.0' to DPDK
    
    >>2.      2017-03-28T02:04:40.004Z|00065|netdev|WARN|dpdk-p0: could
    
    >>not set configuration (Invalid argument)
    
    >>
    
    >>What would cause such a situation?
    
    >>
    
    >>Commands run: https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_0P754x0t&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=XEhP0isSM6FAfGw-HXbOfMLfomYnhSo1G2UxvC1qjp8&e= 
    
    >>Detailed OVS Log: https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_ctBhLuRG&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=90tudSRNE_HlxrCclX83wTmqlUL5xMPrRha7_-fMAKs&e= 
    
    >>
    
    >>
    
    >
    
    
    
    
    
    _______________________________________________
    
    discuss mailing list
    
    discuss at openvswitch.org
    
    https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-2Ddiscuss&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=9-oI5HFUKs-b_kDAFcnn2vFwTcuE2xw2SFvd-zE6vWY&e= 
    
    
    
    





More information about the discuss mailing list