[ovs-discuss] Error attaching device to DPDK
O Mahony, Billy
billy.o.mahony at intel.com
Tue Apr 4 16:14:17 UTC 2017
Hi Shivram,
Apologies if I missed this info in the email thread.
For the bug report:
* What version of DPDK were you using?
* Also what is your hardware - NICs (out from dpdk_devbind.py --status) and processor architecture (last few lines from cpuid -1 )
Thanks,
Billy.
> -----Original Message-----
> From: O Mahony, Billy
> Sent: Tuesday, April 4, 2017 12:00 PM
> To: 'Darrell Ball' <dball at vmware.com>; Brad Cowie <brad at cowie.nz>;
> Shivaram Mysore <shivaram.mysore at gmail.com>
> Cc: ovs-discuss at openvswitch.org
> Subject: RE: [ovs-discuss] Error attaching device to DPDK
>
> Hi Darrell,
>
> /Billy
>
> > -----Original Message-----
> > From: Darrell Ball [mailto:dball at vmware.com]
> > Sent: Thursday, March 30, 2017 8:51 PM
> > To: O Mahony, Billy <billy.o.mahony at intel.com>; Brad Cowie
> > <brad at cowie.nz>; Shivaram Mysore <shivaram.mysore at gmail.com>
> > Cc: ovs-discuss at openvswitch.org
> > Subject: Re: [ovs-discuss] Error attaching device to DPDK
> >
> >
> >
> > On 3/29/17, 7:40 AM, "O Mahony, Billy" <billy.o.mahony at intel.com>
> wrote:
> >
> > Hi All,
> >
> >
> >
> > Just to add something to this conversation that has not been
> > explicitly mentioned below.
> >
> >
> >
> > Brad outlines how to set other_config:dpdk-socket-mem to reserve
> > hugepages from both NUMA nodes in order to be sure to avoid pci
> > topology issues.
> >
> >
> >
> > Strictly this doesn’t have to be done on both nodes – just on the
> > correct node :) – but sometimes it’s easier to just cover all the bases.
> >
> >
> >
> > In addition you will need to ensure that:
> >
> > * That linux has hugepages available on both (or the right) nodes.
> >
> > * There is a pmd thread created on each node by setting
> > other_config:pmd-cpu-mask correctly – or again at least on the right node.
> >
> >
> >
> > These config items are described in
> >
> > * ./Documentation/intro/install/dpdk.rst
> >
> > * ./Documentation/howto/dpdk.rst
> >
> >
> >
> > Currently if there is not a pmd thread on the NUMA node belonging
> > to the PCI device the device will not be polled (a warning message is
> > issued to this effect).
> >
> >
> > [Darrell]
> > From Linux POV, the “default” memory policy is to allocate the
> > hugepage pool across numa nodes, if possible; this assumes an
> > allocation was done, which we have no reports here otherwise.
> > iirc based on the logs in this case, which I cannot view anymore,
> > there was no pmd mask configuration specified.
> > The advertised behavior should be, in this case, that a pmd thread is
> > allocated on each numa node with at least one local dpdk interface bound.
> > iirc, the dpdk socket memory was left at default, which only
> > pre-allocated from the hugepage pool on node 0 So, does it make sense
> > to also try to pre- allocate hugepages on a numa node, where a PMD
> thread is allocated ?
> > Pls correct me in any respects.
> >
> [[BO'M]]
>
> What you're looking for here is something that will work (even if not
> optimally). And leave the optimization of exact configurations of
> pmds/hugepage-allocations and so on until later?
>
> So even if pmd-cpu-mask and dpdk-socket-mem other_config items are
> omitted OVS will 'do the right thing' ? ie. place a pmd on each numa node and
> take some hugepages on each NUMA node.
>
> It's hard to argue with that and I can put that on our internal backlog of tasks.
>
> How about these rules?
> * If pmd-cpu-mask is not set OVS creates one pmd thread for each NUMA
> node. (current case)
> * if dpdk-socket-mem is not set then for each NUMA node dpdk will reserve:
> ** one quarter of the available hugepage memory
> ** 512MB of 2MB hugepages
> If hugepage granularity requires more memory may be reserved.
>
>
>
>
> >
> >
> >
> >
> > There is currently a patch under consideration so that if a
> > NUMA-local pmd is not available for a device then device will be
> > assigned to some other pmd thread. While this is not as efficient due
> > to the need for the data to travel between NUMA nodes it is less
> frustrating to configure.
> >
> >
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__mail.openvswitch.org_pipermail_ovs-2Ddev_2017-
> >
> 2DFebruary_329175.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVh
> > FA09CGX7JQ5Ih-
> >
> uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=QmpnH
> > wqCC8ZBxRGZgfCZuF-9SielPs89Yd-k7crooC8&e=
> >
> >
> >
> > If you install the hwloc package you can use “hwloc-calc
> > pci=0000:01:02.0 -- intersect Socket” to show the NUMA node that is local
> to a pci device.
> > However I have seen some systems which report that pci device is local
> > to both nodes. The lstopo command can also be used for this.
> >
> >
> >
> > Hope some of that helps,
> >
> >
> >
> > Billy.
> >
> >
> >
> >
> >
> > From: ovs-discuss-bounces at openvswitch.org [mailto:ovs-discuss-
> > bounces at openvswitch.org] On Behalf Of Darrell Ball
> >
> > Sent: Wednesday, March 29, 2017 2:36 AM
> >
> > To: Brad Cowie <brad at cowie.nz>; Shivaram Mysore
> > <shivaram.mysore at gmail.com>
> >
> > Cc: ovs-discuss at openvswitch.org
> >
> > Subject: Re: [ovs-discuss] Error attaching device to DPDK
> >
> >
> >
> >
> >
> >
> >
> > From: <ovs-discuss-bounces at openvswitch.org> on behalf of Brad
> > Cowie <brad at cowie.nz>
> >
> > Date: Tuesday, March 28, 2017 at 5:35 PM
> >
> > To: Shivaram Mysore <shivaram.mysore at gmail.com>
> >
> > Cc: "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
> >
> > Subject: Re: [ovs-discuss] Error attaching device to DPDK
> >
> >
> >
> > Hi folks,
> >
> > I have been helping Shivaram off-list with this problem and just
> > wanted to update this thread with the solution.
> >
> > Basically the issue was with how hugepages were being preallocated
> > by OVS for use with DPDK. I believe by default OVS sets
> > other_config:dpdk- socket-mem to "1024,0". Which will give one 1GB
> > hugepage to CPU0 and none to CPU1.
> >
> > The ports Shivaram was trying to assign to DPDK were connected
> > physically via a PCI-E bus to CPU1 and since there are no hugepages
> > preallocated there for DPDK then we can't add it.
> >
> > The fix is simple, to assign a hugepage to CPU1:
> >
> >
> >
> > $ sudo ovs-vsctl --no-wait set Open_vSwitch .
> > other_config:dpdk-socket- mem="1024,1024"
> >
> > In OVS 2.7.0 unfortunately due to an issue (that is now fixed by
> > commit
> > ef6cca6fdc67f3cee21c6bb1c13c4ca7f8241314) you get a segfault rather
> > than an error message telling you the problem.
> >
> > In OVS 2.7.90 however the issue becomes quite clear as there is a
> > nice log message like such:
> >
> >
> >
> > 2017-03-29T00:19:57Z|00067|netdev_dpdk|ERR|Insufficient memory to
> > create memory pool for netdev dpdk-p0, with MTU 1500 on socket 1
> >
> > There is documentation regarding these fields, but for case, the
> > documentation could simply state something like
> >
> > “you need to allocate hugepage memory on each numa node where
> dpdk
> > ports are bound and or those ports
> >
> > will not work with dpdk”
> >
> > The error log could also be a little more succinct like “no memory
> > pool allocated for netdev dpdk-p0,.., usage on socket 1”
> >
> >
> >
> > Possibly, going even further, when a PMD thread gets
> > auto-allocated to run on a given numa node,
> >
> > memory allocation is defaulted there as well. If it cannot be
> > allocated, an error log is generated.
> >
> >
> >
> > Unfortunately to those unfamiliar with DPDK and NUMA architectures
> > this won't be very obvious. Potentially we could add some additional
> > help to the DPDK documentation pages for OVS that explains the
> > other_config options you may need to reconfigure on multi-CPU NUMA
> machines?
> >
> > Brad
> >
> >
> >
> > On 29 March 2017 at 10:02, Shivaram Mysore
> > <shivaram.mysore at gmail.com> wrote:
> >
> > Ok thanks. Will remember the same. Right now, I will not specify
> > anything as I have to get this to work first!
> >
> >
> >
> > On Tue, Mar 28, 2017 at 2:00 PM, Bodireddy, Bhanuprakash
> > <bhanuprakash.bodireddy at intel.com> wrote:
> >
> > >Question:
> >
> > >
> >
> > >I don't have any specific cpu cores associated with DPDK. Will
> > this make any
> >
> > >difference? I also did not see this requirement as a MUST have
> > in the
> >
> > >documentation.
> >
> >
> >
> > I assume your question is on PMD threads. If you don’t specify any
> > thing explicitly
> >
> > PMD pmd thread shall be created and pinned to core 0. You can
> > explicitly select the no. of pmd threads
> >
> > and the cores by explicitly setting the affinity mask using
> > 'other_config:pmd-cpu-mask'.
> >
> >
> >
> > Based on the affinity mask, the PMD threads shall be spawned and
> > the rx queues will be uniformly
> >
> > distributed among the PMD threads.
> >
> >
> >
> > Refer https://urldefense.proofpoint.com/v2/url?u=http-
> >
> 3A__docs.openvswitch.org_en_latest_howto_dpdk_&d=DwIGaQ&c=uilaK9
> > 0D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
> >
> uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=JNQjCdB
> > cYJG8H1n4bT6k_zFhtkk6wcRE27AhEuQoqjg&e= .
> >
> >
> >
> > Also note that 'dpdk-lcore-mask' should be used to set the dpdk
> > lcore threads and vswitchd.
> >
> >
> >
> > - Bhanuprakash.
> >
> >
> >
> > >
> >
> > ># cat /proc/cmdline
> >
> > >BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic
> > root=/dev/mapper/intelsdn-
> >
> > >-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G
> >
> > >hugepagesz=1G hugepages=4
> >
> > >
> >
> > >
> >
> > >Thanks
> >
> > >
> >
> > >/Shivaram
> >
> > >
> >
> > >On Tue, Mar 28, 2017 at 1:32 PM, Shivaram Mysore
> >
> > ><shivaram.mysore at gmail.com> wrote:
> >
> > >I was advised on this mailing list that VFIO drivers are better
> > than IGB_UIO
> > +
> >
> > >more forgiving [1]. Hence switched to the same.
> >
> > >
> >
> > >[1] https://urldefense.proofpoint.com/v2/url?u=http-3A__dpdk-
> > 2Dguide.gitlab.io_dpdk-
> >
> 2Dguide_setup_binding.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=
> > BVhFA09CGX7JQ5Ih-
> >
> uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=KgBSWU
> > sNwqMWB_Pm2CHmGAVAO5enRJxRlvbS9MR-7Ww&e=
> >
> > >
> >
> > >/Shivaram
> >
> > >
> >
> > >On Tue, Mar 28, 2017 at 1:12 PM, Bodireddy, Bhanuprakash
> >
> > ><bhanuprakash.bodireddy at intel.com> wrote:
> >
> > >
> >
> > >>any hints for me based on the info provided?
> >
> > >
> >
> > >Your BIOS options and bootcmd line options are appropriate and no
> > issues
> >
> > >there. I wanted to check if you tried igb_uio drivers
> >
> > >but saw your post here
> > https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__mail.openvswitch.org_pipermail_ovs-
> > 2D&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
> >
> uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=qoh0Qu6
> > HkVOjRIO_oNqdzBGTGjRdo5tppjkG2R3KpEw&e=
> >
> > >discuss/2017-March/043993.html, reporting a crash with UIO drivers.
> >
> > >
> >
> > >Is that the reason why you moved to VFIO? I cross checked your
> > logs but
> >
> > >found nothing suspicious there.
> >
> > >
> >
> > >-Bhanuprakash.
> >
> > >
> >
> > >>
> >
> > >>Thanks
> >
> > >>
> >
> > >>On Tue, Mar 28, 2017 at 8:41 AM, Shivaram Mysore
> >
> > >><shivaram.mysore at gmail.com> wrote:
> >
> > >>Hi,
> >
> > >>
> >
> > >># dmesg | grep -e DMAR -e IOMMU
> >
> > >>[ 0.000000] ACPI: DMAR 0x00000000798AF340 000158 (v01 SUPERM
> > SMCI--
> >
> > >>MB 00000001 INTL 20091013)
> >
> > >>[ 0.000000] DMAR: IOMMU enabled
> >
> > >>[ 0.109987] DMAR: Host address width 46
> >
> > >>[ 0.109988] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
> >
> > >>[ 0.109997] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
> >
> > >>d2078c106f0466 ecap f020de
> >
> > >>[ 0.109998] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
> >
> > >>[ 0.110002] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
> >
> > >>d2078c106f0466 ecap f020de
> >
> > >>[ 0.110003] DMAR: RMRR base: 0x0000007ba75000 end:
> > 0x0000007ba84fff
> >
> > >>[ 0.110004] DMAR: ATSR flags: 0x0
> >
> > >>[ 0.110005] DMAR: RHSA base: 0x000000c7ffc000 proximity domain:
> 0x0
> >
> > >>[ 0.110006] DMAR: RHSA base: 0x000000fbffc000 proximity domain:
> 0x1
> >
> > >>[ 0.110008] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000
> IOMMU
> > 0
> >
> > >>[ 0.110008] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000
> IOMMU
> > 1
> >
> > >>[ 0.110009] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000
> IOMMU
> > 1
> >
> > >>[ 0.110010] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
> >
> > >>[ 0.110011] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt
> > out
> >
> > >bit.
> >
> > >>[ 0.110012] DMAR-IR: Use 'intremap=no_x2apic_optout' to override
> > the
> >
> > >>BIOS setting.
> >
> > >>[ 0.110902] DMAR-IR: Enabled IRQ remapping in xapic mode
> >
> > >>[ 1.687595] DMAR: dmar0: Using Queued invalidation
> >
> > >>[ 1.687608] DMAR: dmar1: Using Queued invalidation
> >
> > >>[ 1.688473] DMAR: Hardware identity mapping for device 0000:ff:08.0
> >
> > >>[ 1.688475] DMAR: Hardware identity mapping for device 0000:ff:08.2
> >
> > >>[ 1.688477] DMAR: Hardware identity mapping for device 0000:ff:08.3
> >
> > >>[ 1.688478] DMAR: Hardware identity mapping for device 0000:ff:09.0
> >
> > >>[ 1.688480] DMAR: Hardware identity mapping for device 0000:ff:09.2
> >
> > >>[ 1.688482] DMAR: Hardware identity mapping for device 0000:ff:09.3
> >
> > >>[ 1.688483] DMAR: Hardware identity mapping for device 0000:ff:0b.0
> >
> > >>[ 1.688485] DMAR: Hardware identity mapping for device 0000:ff:0b.1
> >
> > >>[ 1.688487] DMAR: Hardware identity mapping for device 0000:ff:0b.2
> >
> > >>[ 1.688488] DMAR: Hardware identity mapping for device 0000:ff:0c.0
> >
> > >>[ 1.688490] DMAR: Hardware identity mapping for device 0000:ff:0c.1
> >
> > >>[ 1.688491] DMAR: Hardware identity mapping for device 0000:ff:0c.2
> >
> > >>[ 1.688493] DMAR: Hardware identity mapping for device 0000:ff:0c.3
> >
> > >>[ 1.688494] DMAR: Hardware identity mapping for device 0000:ff:0c.4
> >
> > >>[ 1.688496] DMAR: Hardware identity mapping for device 0000:ff:0c.5
> >
> > >>[ 1.688498] DMAR: Hardware identity mapping for device 0000:ff:0c.6
> >
> > >>[ 1.688499] DMAR: Hardware identity mapping for device 0000:ff:0c.7
> >
> > >>[ 1.688500] DMAR: Hardware identity mapping for device 0000:ff:0d.0
> >
> > >>[ 1.688502] DMAR: Hardware identity mapping for device 0000:ff:0d.1
> >
> > >>[ 1.688504] DMAR: Hardware identity mapping for device 0000:ff:0d.2
> >
> > >>[ 1.688505] DMAR: Hardware identity mapping for device 0000:ff:0d.3
> >
> > >>[ 1.688506] DMAR: Hardware identity mapping for device 0000:ff:0d.4
> >
> > >>[ 1.688508] DMAR: Hardware identity mapping for device 0000:ff:0d.5
> >
> > >>[ 1.688509] DMAR: Hardware identity mapping for device 0000:ff:0d.6
> >
> > >>[ 1.688510] DMAR: Hardware identity mapping for device 0000:ff:0d.7
> >
> > >>[ 1.688511] DMAR: Hardware identity mapping for device 0000:ff:0e.0
> >
> > >>[ 1.688513] DMAR: Hardware identity mapping for device 0000:ff:0e.1
> >
> > >>[ 1.688514] DMAR: Hardware identity mapping for device 0000:ff:0f.0
> >
> > >>[ 1.688516] DMAR: Hardware identity mapping for device 0000:ff:0f.1
> >
> > >>[ 1.688517] DMAR: Hardware identity mapping for device 0000:ff:0f.2
> >
> > >>[ 1.688518] DMAR: Hardware identity mapping for device 0000:ff:0f.3
> >
> > >>[ 1.688520] DMAR: Hardware identity mapping for device 0000:ff:0f.4
> >
> > >>[ 1.688521] DMAR: Hardware identity mapping for device 0000:ff:0f.5
> >
> > >>[ 1.688523] DMAR: Hardware identity mapping for device 0000:ff:0f.6
> >
> > >>[ 1.688524] DMAR: Hardware identity mapping for device 0000:ff:10.0
> >
> > >>[ 1.688526] DMAR: Hardware identity mapping for device 0000:ff:10.1
> >
> > >>[ 1.688527] DMAR: Hardware identity mapping for device 0000:ff:10.5
> >
> > >>[ 1.688529] DMAR: Hardware identity mapping for device 0000:ff:10.6
> >
> > >>[ 1.688530] DMAR: Hardware identity mapping for device 0000:ff:10.7
> >
> > >>[ 1.688532] DMAR: Hardware identity mapping for device 0000:ff:12.0
> >
> > >>[ 1.688533] DMAR: Hardware identity mapping for device 0000:ff:12.1
> >
> > >>[ 1.688534] DMAR: Hardware identity mapping for device 0000:ff:12.4
> >
> > >>[ 1.688536] DMAR: Hardware identity mapping for device 0000:ff:12.5
> >
> > >>[ 1.688537] DMAR: Hardware identity mapping for device 0000:ff:13.0
> >
> > >>[ 1.688539] DMAR: Hardware identity mapping for device 0000:ff:13.1
> >
> > >>[ 1.688541] DMAR: Hardware identity mapping for device 0000:ff:13.2
> >
> > >>[ 1.688542] DMAR: Hardware identity mapping for device 0000:ff:13.3
> >
> > >>[ 1.688543] DMAR: Hardware identity mapping for device 0000:ff:13.6
> >
> > >>[ 1.688545] DMAR: Hardware identity mapping for device 0000:ff:13.7
> >
> > >>[ 1.688546] DMAR: Hardware identity mapping for device 0000:ff:14.0
> >
> > >>[ 1.688547] DMAR: Hardware identity mapping for device 0000:ff:14.1
> >
> > >>[ 1.688549] DMAR: Hardware identity mapping for device 0000:ff:14.2
> >
> > >>[ 1.688550] DMAR: Hardware identity mapping for device 0000:ff:14.3
> >
> > >>[ 1.688552] DMAR: Hardware identity mapping for device 0000:ff:14.4
> >
> > >>[ 1.688553] DMAR: Hardware identity mapping for device 0000:ff:14.5
> >
> > >>[ 1.688559] DMAR: Hardware identity mapping for device 0000:ff:14.6
> >
> > >>[ 1.688561] DMAR: Hardware identity mapping for device 0000:ff:14.7
> >
> > >>[ 1.688562] DMAR: Hardware identity mapping for device 0000:ff:16.0
> >
> > >>[ 1.688564] DMAR: Hardware identity mapping for device 0000:ff:16.1
> >
> > >>[ 1.688565] DMAR: Hardware identity mapping for device 0000:ff:16.2
> >
> > >>[ 1.688566] DMAR: Hardware identity mapping for device 0000:ff:16.3
> >
> > >>[ 1.688568] DMAR: Hardware identity mapping for device 0000:ff:16.6
> >
> > >>[ 1.688569] DMAR: Hardware identity mapping for device 0000:ff:16.7
> >
> > >>[ 1.688571] DMAR: Hardware identity mapping for device 0000:ff:17.0
> >
> > >>[ 1.688573] DMAR: Hardware identity mapping for device 0000:ff:17.1
> >
> > >>[ 1.688574] DMAR: Hardware identity mapping for device 0000:ff:17.2
> >
> > >>[ 1.688576] DMAR: Hardware identity mapping for device 0000:ff:17.3
> >
> > >>[ 1.688577] DMAR: Hardware identity mapping for device 0000:ff:17.4
> >
> > >>[ 1.688579] DMAR: Hardware identity mapping for device 0000:ff:17.5
> >
> > >>[ 1.688581] DMAR: Hardware identity mapping for device 0000:ff:17.6
> >
> > >>[ 1.688582] DMAR: Hardware identity mapping for device 0000:ff:17.7
> >
> > >>[ 1.688584] DMAR: Hardware identity mapping for device 0000:ff:1e.0
> >
> > >>[ 1.688586] DMAR: Hardware identity mapping for device 0000:ff:1e.1
> >
> > >>[ 1.688588] DMAR: Hardware identity mapping for device 0000:ff:1e.2
> >
> > >>[ 1.688590] DMAR: Hardware identity mapping for device 0000:ff:1e.3
> >
> > >>[ 1.688592] DMAR: Hardware identity mapping for device 0000:ff:1e.4
> >
> > >>[ 1.688594] DMAR: Hardware identity mapping for device 0000:ff:1f.0
> >
> > >>[ 1.688596] DMAR: Hardware identity mapping for device 0000:ff:1f.2
> >
> > >>[ 1.688610] DMAR: Hardware identity mapping for device 0000:7f:08.0
> >
> > >>[ 1.688612] DMAR: Hardware identity mapping for device 0000:7f:08.2
> >
> > >>[ 1.688614] DMAR: Hardware identity mapping for device 0000:7f:08.3
> >
> > >>[ 1.688616] DMAR: Hardware identity mapping for device 0000:7f:09.0
> >
> > >>[ 1.688617] DMAR: Hardware identity mapping for device 0000:7f:09.2
> >
> > >>[ 1.688619] DMAR: Hardware identity mapping for device 0000:7f:09.3
> >
> > >>[ 1.688621] DMAR: Hardware identity mapping for device 0000:7f:0b.0
> >
> > >>[ 1.688623] DMAR: Hardware identity mapping for device 0000:7f:0b.1
> >
> > >>[ 1.688625] DMAR: Hardware identity mapping for device 0000:7f:0b.2
> >
> > >>[ 1.688627] DMAR: Hardware identity mapping for device 0000:7f:0c.0
> >
> > >>[ 1.688629] DMAR: Hardware identity mapping for device 0000:7f:0c.1
> >
> > >>[ 1.688631] DMAR: Hardware identity mapping for device 0000:7f:0c.2
> >
> > >>[ 1.688632] DMAR: Hardware identity mapping for device 0000:7f:0c.3
> >
> > >>[ 1.688634] DMAR: Hardware identity mapping for device 0000:7f:0c.4
> >
> > >>[ 1.688636] DMAR: Hardware identity mapping for device 0000:7f:0c.5
> >
> > >>[ 1.688637] DMAR: Hardware identity mapping for device 0000:7f:0c.6
> >
> > >>[ 1.688639] DMAR: Hardware identity mapping for device 0000:7f:0c.7
> >
> > >>[ 1.688641] DMAR: Hardware identity mapping for device 0000:7f:0d.0
> >
> > >>[ 1.688642] DMAR: Hardware identity mapping for device 0000:7f:0d.1
> >
> > >>[ 1.688644] DMAR: Hardware identity mapping for device 0000:7f:0d.2
> >
> > >>[ 1.688646] DMAR: Hardware identity mapping for device 0000:7f:0d.3
> >
> > >>[ 1.688647] DMAR: Hardware identity mapping for device 0000:7f:0d.4
> >
> > >>[ 1.688649] DMAR: Hardware identity mapping for device 0000:7f:0d.5
> >
> > >>[ 1.688650] DMAR: Hardware identity mapping for device 0000:7f:0d.6
> >
> > >>[ 1.688652] DMAR: Hardware identity mapping for device 0000:7f:0d.7
> >
> > >>[ 1.688653] DMAR: Hardware identity mapping for device 0000:7f:0e.0
> >
> > >>[ 1.688655] DMAR: Hardware identity mapping for device 0000:7f:0e.1
> >
> > >>[ 1.688657] DMAR: Hardware identity mapping for device 0000:7f:0f.0
> >
> > >>[ 1.688658] DMAR: Hardware identity mapping for device 0000:7f:0f.1
> >
> > >>[ 1.688660] DMAR: Hardware identity mapping for device 0000:7f:0f.2
> >
> > >>[ 1.688661] DMAR: Hardware identity mapping for device 0000:7f:0f.3
> >
> > >>[ 1.688663] DMAR: Hardware identity mapping for device 0000:7f:0f.4
> >
> > >>[ 1.688664] DMAR: Hardware identity mapping for device 0000:7f:0f.5
> >
> > >>[ 1.688666] DMAR: Hardware identity mapping for device 0000:7f:0f.6
> >
> > >>[ 1.688668] DMAR: Hardware identity mapping for device 0000:7f:10.0
> >
> > >>[ 1.688669] DMAR: Hardware identity mapping for device 0000:7f:10.1
> >
> > >>[ 1.688671] DMAR: Hardware identity mapping for device 0000:7f:10.5
> >
> > >>[ 1.688673] DMAR: Hardware identity mapping for device 0000:7f:10.6
> >
> > >>[ 1.688674] DMAR: Hardware identity mapping for device 0000:7f:10.7
> >
> > >>[ 1.688676] DMAR: Hardware identity mapping for device 0000:7f:12.0
> >
> > >>[ 1.688678] DMAR: Hardware identity mapping for device 0000:7f:12.1
> >
> > >>[ 1.688684] DMAR: Hardware identity mapping for device 0000:7f:12.4
> >
> > >>[ 1.688686] DMAR: Hardware identity mapping for device 0000:7f:12.5
> >
> > >>[ 1.688688] DMAR: Hardware identity mapping for device 0000:7f:13.0
> >
> > >>[ 1.688689] DMAR: Hardware identity mapping for device 0000:7f:13.1
> >
> > >>[ 1.688691] DMAR: Hardware identity mapping for device 0000:7f:13.2
> >
> > >>[ 1.688692] DMAR: Hardware identity mapping for device 0000:7f:13.3
> >
> > >>[ 1.688694] DMAR: Hardware identity mapping for device 0000:7f:13.6
> >
> > >>[ 1.688696] DMAR: Hardware identity mapping for device 0000:7f:13.7
> >
> > >>[ 1.688697] DMAR: Hardware identity mapping for device 0000:7f:14.0
> >
> > >>[ 1.688699] DMAR: Hardware identity mapping for device 0000:7f:14.1
> >
> > >>[ 1.688701] DMAR: Hardware identity mapping for device 0000:7f:14.2
> >
> > >>[ 1.688703] DMAR: Hardware identity mapping for device 0000:7f:14.3
> >
> > >>[ 1.688704] DMAR: Hardware identity mapping for device 0000:7f:14.4
> >
> > >>[ 1.688706] DMAR: Hardware identity mapping for device 0000:7f:14.5
> >
> > >>[ 1.688708] DMAR: Hardware identity mapping for device 0000:7f:14.6
> >
> > >>[ 1.688710] DMAR: Hardware identity mapping for device 0000:7f:14.7
> >
> > >>[ 1.688712] DMAR: Hardware identity mapping for device 0000:7f:16.0
> >
> > >>[ 1.688714] DMAR: Hardware identity mapping for device 0000:7f:16.1
> >
> > >>[ 1.688716] DMAR: Hardware identity mapping for device 0000:7f:16.2
> >
> > >>[ 1.688718] DMAR: Hardware identity mapping for device 0000:7f:16.3
> >
> > >>[ 1.688720] DMAR: Hardware identity mapping for device 0000:7f:16.6
> >
> > >>[ 1.688722] DMAR: Hardware identity mapping for device 0000:7f:16.7
> >
> > >>[ 1.688724] DMAR: Hardware identity mapping for device 0000:7f:17.0
> >
> > >>[ 1.688726] DMAR: Hardware identity mapping for device 0000:7f:17.1
> >
> > >>[ 1.688729] DMAR: Hardware identity mapping for device 0000:7f:17.2
> >
> > >>[ 1.688731] DMAR: Hardware identity mapping for device 0000:7f:17.3
> >
> > >>[ 1.688733] DMAR: Hardware identity mapping for device 0000:7f:17.4
> >
> > >>[ 1.688735] DMAR: Hardware identity mapping for device 0000:7f:17.5
> >
> > >>[ 1.688737] DMAR: Hardware identity mapping for device 0000:7f:17.6
> >
> > >>[ 1.688739] DMAR: Hardware identity mapping for device 0000:7f:17.7
> >
> > >>[ 1.688741] DMAR: Hardware identity mapping for device 0000:7f:1e.0
> >
> > >>[ 1.688742] DMAR: Hardware identity mapping for device 0000:7f:1e.1
> >
> > >>[ 1.688744] DMAR: Hardware identity mapping for device 0000:7f:1e.2
> >
> > >>[ 1.688746] DMAR: Hardware identity mapping for device 0000:7f:1e.3
> >
> > >>[ 1.688748] DMAR: Hardware identity mapping for device 0000:7f:1e.4
> >
> > >>[ 1.688750] DMAR: Hardware identity mapping for device 0000:7f:1f.0
> >
> > >>[ 1.688752] DMAR: Hardware identity mapping for device 0000:7f:1f.2
> >
> > >>[ 1.688769] DMAR: Hardware identity mapping for device
> 0000:00:00.0
> >
> > >>[ 1.688774] DMAR: Hardware identity mapping for device
> 0000:00:01.0
> >
> > >>[ 1.688778] DMAR: Hardware identity mapping for device
> 0000:00:01.1
> >
> > >>[ 1.688783] DMAR: Hardware identity mapping for device
> 0000:00:03.0
> >
> > >>[ 1.688785] DMAR: Hardware identity mapping for device
> 0000:00:04.0
> >
> > >>[ 1.688787] DMAR: Hardware identity mapping for device
> 0000:00:04.1
> >
> > >>[ 1.688790] DMAR: Hardware identity mapping for device
> 0000:00:04.2
> >
> > >>[ 1.688792] DMAR: Hardware identity mapping for device
> 0000:00:04.3
> >
> > >>[ 1.688794] DMAR: Hardware identity mapping for device
> 0000:00:04.4
> >
> > >>[ 1.688797] DMAR: Hardware identity mapping for device
> 0000:00:04.5
> >
> > >>[ 1.688799] DMAR: Hardware identity mapping for device
> 0000:00:04.6
> >
> > >>[ 1.688801] DMAR: Hardware identity mapping for device
> 0000:00:04.7
> >
> > >>[ 1.688803] DMAR: Hardware identity mapping for device
> 0000:00:05.0
> >
> > >>[ 1.688807] DMAR: Hardware identity mapping for device
> 0000:00:05.1
> >
> > >>[ 1.688809] DMAR: Hardware identity mapping for device
> 0000:00:05.2
> >
> > >>[ 1.688811] DMAR: Hardware identity mapping for device
> 0000:00:05.4
> >
> > >>[ 1.688814] DMAR: Hardware identity mapping for device
> 0000:00:11.0
> >
> > >>[ 1.688816] DMAR: Hardware identity mapping for device
> 0000:00:11.4
> >
> > >>[ 1.688818] DMAR: Hardware identity mapping for device
> 0000:00:14.0
> >
> > >>[ 1.688821] DMAR: Hardware identity mapping for device
> 0000:00:16.0
> >
> > >>[ 1.688823] DMAR: Hardware identity mapping for device
> 0000:00:16.1
> >
> > >>[ 1.688825] DMAR: Hardware identity mapping for device
> 0000:00:1a.0
> >
> > >>[ 1.688828] DMAR: Hardware identity mapping for device 0000:00:1c.0
> >
> > >>[ 1.688831] DMAR: Hardware identity mapping for device 0000:00:1c.2
> >
> > >>[ 1.688833] DMAR: Hardware identity mapping for device
> 0000:00:1d.0
> >
> > >>[ 1.688835] DMAR: Hardware identity mapping for device 0000:00:1f.0
> >
> > >>[ 1.688837] DMAR: Hardware identity mapping for device 0000:00:1f.2
> >
> > >>[ 1.688844] DMAR: Hardware identity mapping for device 0000:00:1f.3
> >
> > >>[ 1.688846] DMAR: Hardware identity mapping for device 0000:00:1f.6
> >
> > >>[ 1.688864] DMAR: Hardware identity mapping for device
> 0000:01:00.0
> >
> > >>[ 1.688870] DMAR: Hardware identity mapping for device
> 0000:01:00.1
> >
> > >>[ 1.688876] DMAR: Hardware identity mapping for device
> 0000:01:00.2
> >
> > >>[ 1.688883] DMAR: Hardware identity mapping for device
> 0000:01:00.3
> >
> > >>[ 1.688895] DMAR: Hardware identity mapping for device
> 0000:80:01.0
> >
> > >>[ 1.688899] DMAR: Hardware identity mapping for device
> 0000:80:02.0
> >
> > >>[ 1.688903] DMAR: Hardware identity mapping for device
> 0000:80:03.0
> >
> > >>[ 1.688906] DMAR: Hardware identity mapping for device
> 0000:80:04.0
> >
> > >>[ 1.688908] DMAR: Hardware identity mapping for device
> 0000:80:04.1
> >
> > >>[ 1.688911] DMAR: Hardware identity mapping for device
> 0000:80:04.2
> >
> > >>[ 1.688913] DMAR: Hardware identity mapping for device
> 0000:80:04.3
> >
> > >>[ 1.688915] DMAR: Hardware identity mapping for device
> 0000:80:04.4
> >
> > >>[ 1.688918] DMAR: Hardware identity mapping for device
> 0000:80:04.5
> >
> > >>[ 1.688920] DMAR: Hardware identity mapping for device
> 0000:80:04.6
> >
> > >>[ 1.688923] DMAR: Hardware identity mapping for device
> 0000:80:04.7
> >
> > >>[ 1.688936] DMAR: Hardware identity mapping for device
> 0000:80:05.0
> >
> > >>[ 1.688940] DMAR: Hardware identity mapping for device
> 0000:80:05.1
> >
> > >>[ 1.688943] DMAR: Hardware identity mapping for device
> 0000:80:05.2
> >
> > >>[ 1.688945] DMAR: Hardware identity mapping for device
> 0000:80:05.4
> >
> > >>[ 1.688956] DMAR: Hardware identity mapping for device
> 0000:81:00.0
> >
> > >>[ 1.688961] DMAR: Hardware identity mapping for device
> 0000:81:00.1
> >
> > >>[ 1.688974] DMAR: Hardware identity mapping for device
> 0000:82:00.0
> >
> > >>[ 1.688980] DMAR: Hardware identity mapping for device
> 0000:82:00.1
> >
> > >>[ 1.688986] DMAR: Hardware identity mapping for device
> 0000:82:00.2
> >
> > >>[ 1.688992] DMAR: Hardware identity mapping for device
> 0000:82:00.3
> >
> > >>[ 1.689006] DMAR: Hardware identity mapping for device
> 0000:83:00.0
> >
> > >>[ 1.689012] DMAR: Hardware identity mapping for device
> 0000:83:00.1
> >
> > >>[ 1.689018] DMAR: Hardware identity mapping for device
> 0000:83:00.2
> >
> > >>[ 1.689024] DMAR: Hardware identity mapping for device
> 0000:83:00.3
> >
> > >>[ 1.689025] DMAR: Setting RMRR:
> >
> > >>[ 1.689026] DMAR: Ignoring identity map for HW passthrough device
> >
> > >>0000:00:14.0 [0x7ba75000 - 0x7ba84fff]
> >
> > >>[ 1.689027] DMAR: Ignoring identity map for HW passthrough device
> >
> > >>0000:00:1a.0 [0x7ba75000 - 0x7ba84fff]
> >
> > >>[ 1.689028] DMAR: Ignoring identity map for HW passthrough device
> >
> > >>0000:00:1d.0 [0x7ba75000 - 0x7ba84fff]
> >
> > >>[ 1.689036] DMAR: Prepare 0-16MiB unity mapping for LPC
> >
> > >>[ 1.689036] DMAR: Ignoring identity map for HW passthrough device
> >
> > >>0000:00:1f.0 [0x0 - 0xffffff]
> >
> > >>[ 1.689040] DMAR: Intel(R) Virtualization Technology for Directed I/O
> >
> > >>[ 2.299766] DMAR: 32bit 0000:00:11.4 uses non-identity mapping
> >
> > >>[ 2.301941] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping
> >
> > >>[ 2.302679] DMAR: Setting identity map for device 0000:00:1a.0
> > [0x7ba75000
> >
> > >-
> >
> > >>0x7ba84fff]
> >
> > >>[ 2.320910] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
> >
> > >>[ 2.321685] DMAR: Setting identity map for device 0000:00:1d.0
> >
> > >[0x7ba75000
> >
> > >>- 0x7ba84fff]
> >
> > >>#
> >
> > >>
> >
> > >>
> >
> > >>On Tue, Mar 28, 2017 at 1:46 AM, Bodireddy, Bhanuprakash
> >
> > >><bhanuprakash.bodireddy at intel.com> wrote:
> >
> > >>Hello shivaram,
> >
> > >>
> >
> > >>Darrell has already asked if VT-d is enabled?
> >
> > >>Your cmdline options are appropriate as you have iommu
> >
> > >>enabled(intel_iommu=on iommu=pt).
> >
> > >>can you also check the output of below mentioned command to see
> > if VT-d
> >
> > >is
> >
> > >>enabled in BIOS?
> >
> > >>
> >
> > >>$ dmesg | grep -e DMAR -e IOMMU
> >
> > >>
> >
> > >>Regards,
> >
> > >>Bhanuprakash.
> >
> > >>
> >
> > >>From: ovs-discuss-bounces at openvswitch.org [mailto:ovs-discuss-
> >
> > >>bounces at openvswitch.org] On Behalf Of Shivaram Mysore
> >
> > >>Sent: Tuesday, March 28, 2017 6:25 AM
> >
> > >>To: Darrell Ball <dball at vmware.com>
> >
> > >>Cc: ovs-discuss at openvswitch.org
> >
> > >>Subject: Re: [ovs-discuss] Error attaching device to DPDK
> >
> > >>
> >
> > >># uname -r
> >
> > >>4.10.0-14-generic
> >
> > >>
> >
> > >># cat /proc/cmdline
> >
> > >>BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic
> >
> > >root=/dev/mapper/intelsdn-
> >
> > >>-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G
> >
> > >>hugepagesz=1G hugepages=4
> >
> > >>
> >
> > >># uname -a
> >
> > >>Linux intelsdn 4.10.0-14-generic #16-Ubuntu SMP Fri Mar 17
> > 15:19:26 UTC
> >
> > >2017
> >
> > >>x86_64 x86_64 x86_64 GNU/Linux
> >
> > >>
> >
> > >># cat /etc/*release
> >
> > >>DISTRIB_ID=Ubuntu
> >
> > >>DISTRIB_RELEASE=17.04
> >
> > >>DISTRIB_CODENAME=zesty
> >
> > >>DISTRIB_DESCRIPTION="Ubuntu Zesty Zapus (development branch)"
> >
> > >>NAME="Ubuntu"
> >
> > >>VERSION="17.04 (Zesty Zapus)"
> >
> > >>ID=ubuntu
> >
> > >>ID_LIKE=debian
> >
> > >>PRETTY_NAME="Ubuntu Zesty Zapus (development branch)"
> >
> > >>VERSION_ID="17.04"
> >
> > >>
> >
> > >>
> >
> > >>Nothing in the syslog related
> >
> > >>
> >
> > >>On Mon, Mar 27, 2017 at 10:02 PM, Darrell Ball
> > <dball at vmware.com>
> > wrote:
> >
> > >>I see the pastebins now.
> >
> > >>
> >
> > >>What kernel version are you using ?
> >
> > >>VT-d enabled ?
> >
> > >>Anything related in syslog ?
> >
> > >>
> >
> > >>From: Shivaram Mysore <shivaram.mysore at gmail.com>
> >
> > >>Date: Monday, March 27, 2017 at 9:34 PM
> >
> > >>To: Darrell Ball <dball at vmware.com>
> >
> > >>Cc: "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
> >
> > >>Subject: Re: [ovs-discuss] Error attaching device to DPDK
> >
> > >>
> >
> > >>One more datapoint, please check the paste bin log in URL sent
> >
> > >earlier. Dpdk-
> >
> > >>devbind works and you can see results in the paste bin log
> >
> > >>
> >
> > >>/Shivaram
> >
> > >>::Sent from my mobile device::
> >
> > >>
> >
> > >>On Mar 27, 2017, at 8:34 PM, Darrell Ball <dball at vmware.com> wrote:
> >
> > >>see inline below
> >
> > >>
> >
> > >>But a few questions:
> >
> > >>1) Are you using the recommended version of dpdk with OVS 2.7 –
> > 16.11 ?
> >
> > >>2) Are you getting a crash still or is this post crash ?
> >
> > >>3) Was the previous crash still happening with 16.11 or only 17.02 ?
> >
> > >>
> >
> > >>Thanks
> >
> > >>Darrell
> >
> > >>
> >
> > >>From: <ovs-discuss-bounces at openvswitch.org> on behalf of
> > Shivaram
> >
> > >>Mysore <shivaram.mysore at gmail.com>
> >
> > >>Date: Monday, March 27, 2017 at 7:45 PM
> >
> > >>To: "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
> >
> > >>Subject: [ovs-discuss] Error attaching device to DPDK
> >
> > >>
> >
> > >>Hello,
> >
> > >>
> >
> > >>I have installed OVS 2.7 running. When I do:
> >
> > >>1. # ovs-vsctl add-port ovs-ip64-br0 dpdk-p0 -- set Interface dpdk-p0
> >
> > >>type=dpdk options:dpdk-devargs=0000:82:00.0
> >
> > >>2. ovs-vsctl: Error detected while setting up 'dpdk-p0': Error
> attaching
> >
> > >>device '0000:82:00.0' to DPDK. See ovs-vswitchd log for details.
> >
> > >>
> >
> > >>Possibly, the port is not bound.
> >
> > >>Check using
> >
> > >>$DPDK_DIR/tools/dpdk-devbind.py –status
> >
> > >>and see the documentation for more detail
> >
> > >>
> >
> > >>and the log file shows:
> >
> > >>1. 2017-03-28T02:04:40.004Z|00064|netdev_dpdk|WARN|Error
> > attaching
> >
> > >>device '0000:82:00.0' to DPDK
> >
> > >>2. 2017-03-28T02:04:40.004Z|00065|netdev|WARN|dpdk-p0: could
> >
> > >>not set configuration (Invalid argument)
> >
> > >>
> >
> > >>What would cause such a situation?
> >
> > >>
> >
> > >>Commands run: https://urldefense.proofpoint.com/v2/url?u=https-
> >
> 3A__pastebin.com_0P754x0t&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=
> > BVhFA09CGX7JQ5Ih-
> >
> uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=XEhP0isS
> > M6FAfGw-HXbOfMLfomYnhSo1G2UxvC1qjp8&e=
> >
> > >>Detailed OVS Log:
> > https://urldefense.proofpoint.com/v2/url?u=https-
> >
> 3A__pastebin.com_ctBhLuRG&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=
> > BVhFA09CGX7JQ5Ih-
> >
> uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=90tudSR
> > NE_HlxrCclX83wTmqlUL5xMPrRha7_-fMAKs&e=
> >
> > >>
> >
> > >>
> >
> > >
> >
> >
> >
> >
> >
> > _______________________________________________
> >
> > discuss mailing list
> >
> > discuss at openvswitch.org
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__mail.openvswitch.org_mailman_listinfo_ovs-
> >
> 2Ddiscuss&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih
> > -uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=9-
> > oI5HFUKs-b_kDAFcnn2vFwTcuE2xw2SFvd-zE6vWY&e=
> >
> >
> >
> >
> >
> >
More information about the discuss
mailing list