<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:SimSun;
mso-fareast-language:ZH-CN;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">How was ovs installed? I don’t recall ever seeing ovs-vswitchd running as a non-root user.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
</span><span lang="ZH-CN" style="font-size:11.0pt">王嵘</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> [mailto:rong.wang@easystack.cn]
<br>
<b>Sent:</b> 23 November 2017 13:09<br>
<b>To:</b> O'Reilly, Darragh <darragh.oreilly@hpe.com><br>
<b>Cc:</b> ovs-discuss@openvswitch.org<br>
<b>Subject:</b> Re:RE: [ovs-discuss] ovs-2.8 Cannot open /dev/uio0<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">O, my fault.<br>
And now find the log like :<br>
<span style="background:silver">2017-11-23T13:02:43.869Z|00071|dpdk|ERR|EAL: Cannot open /dev/uio0: Permission denied<br>
2017-11-23T13:02:43.869Z|00072|dpdk|ERR|EAL: Current userid: </span></span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:red;background:silver">988</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">The uid 988 belongs to the lbvirt? why the uid is 988?<br>
And even this uid don't have the right to open the uio0</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">。</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">[root@localhost rpm-2.8]# cat /etc/group | grep openvswitch<br>
openvswitch:x:983:<br>
hugetlbfs:x:1001:openvswitch<br>
[root@localhost rpm-2.8]# cat /etc/group | grep 988<br>
libvirt:x:988:<br>
[root@localhost rpm-2.8]# </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">在</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> 2017-11-23 20:24:04</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">，</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">"O'Reilly,
Darragh" <<a href="mailto:darragh.oreilly@hpe.com">darragh.oreilly@hpe.com</a>> </span>
<span lang="ZH-CN" style="font-size:10.5pt;color:black">写道：</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
<br>
<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I think you got the address of the function getuid, but you want to call
it by adding ():</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"> RTE_LOG(ERR, EAL, "Current userid: %d\n", getuid() );</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"> </span><span style="color:black"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">
<a href="mailto:ovs-discuss-bounces@openvswitch.org">ovs-discuss-bounces@openvswitch.org</a> [mailto:<a href="mailto:ovs-discuss-bounces@openvswitch.org">ovs-discuss-bounces@openvswitch.org</a>]
<b>On Behalf Of </b>??<br>
<b>Sent:</b> 23 November 2017 08:20<br>
<b>To:</b> </span><span lang="ZH-CN" style="font-size:11.0pt;color:black">王嵘</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> <<a href="mailto:rong.wang@easystack.cn">rong.wang@easystack.cn</a>><br>
<b>Cc:</b> <a href="mailto:ovs-discuss@openvswitch.org">ovs-discuss@openvswitch.org</a><br>
<b>Subject:</b> Re: [ovs-discuss] ovs-2.8 Cannot open /dev/uio0</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">And I add a log in the code, found thath the current uid is not
</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:red">0</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">(root),
<br>
if there is some other config I missed ?<br>
<br>
<span style="background:silver"> /* save fd if in primary process */<br>
dev->intr_handle.fd = open(devname, O_RDWR);<br>
if (dev->intr_handle.fd < 0) {<br>
RTE_LOG(ERR, EAL, "Cannot open %s: %s\n",<br>
devname, strerror(errno));<br>
</span></span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:red;background:silver"> RTE_LOG(ERR, EAL, "Current userid: %d\n", getuid);</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver"><br>
goto error;<br>
}</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
<br>
#systemctl restart openvswitch<br>
<span style="background:silver">2017-11-23T07:57:13.476Z|00071|dpdk|ERR|EAL: Cannot open /dev/uio0: Permission denied<br>
2017-11-23T07:57:13.476Z|00072|dpdk|ERR|EAL: Current userid: </span></span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:red;background:silver">-534781824</span><span style="color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">在</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> 2017-11-22 21:05:05</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">，</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">"</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">王嵘</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">"
<<a href="mailto:rong.wang@easystack.cn">rong.wang@easystack.cn</a>> </span><span lang="ZH-CN" style="font-size:10.5pt;color:black">写道：</span><span style="color:black"><o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">hi,<br>
That's because I'm using the Broadcom Nic for performing-test and found the uio problem("Cannot open /dev/uio0: Permission denied").<br>
and I now RECHECKED the problem in my own VM environment</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">。</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
Pity that, the uio Permission denied problem still alway exists.(the ovs2.7 is ok).</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">and the ovs is</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#880000"> 2.8.1</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">,
dpdk-1705.2</span><span style="color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">[root@localhost test]# rpm -qa | grep openvswitch<br>
openvswitch-2.8.1-1.el7.centos.x86_64<br>
openvswitch-debuginfo-2.8.1-1.el7.centos.x86_64<br>
[root@localhost test]# rpm -qa | grep dpdk<br>
dpdk-17.05.2-1.x86_64<br>
dpdk-devel-17.05.2-1.x86_64<br>
dpdk-debuginfo-17.05.2-1.x86_64</span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
<br>
<span style="background:silver">2017-11-22T11:37:24.909Z|00061|dpdk|WARN|EAL: rte_pci_probe_one execing.<br>
2017-11-22T11:37:24.909Z|00062|dpdk|INFO|EAL: PCI device 0000:00:09.0 on NUMA socket -1<br>
2017-11-22T11:37:24.909Z|00063|dpdk|INFO|EAL: probe driver: 8086:100e net_e1000_em<br>
2017-11-22T11:37:24.909Z|00064|dpdk|ERR|EAL: Cannot open /dev/uio0: Permission denied<br>
2017-11-22T11:37:24.909Z|00065|dpdk|WARN|EAL: Requested device 0000:00:09.0 cannot be used<br>
2017-11-22T11:37:24.909Z|00066|dpdk|ERR|EAL: Driver cannot attach the device (0000:00:09.0)</span><br>
<br>
<br>
<span style="background:silver">[root@localhost test]# dpdk-devbind --status<br>
<br>
Network devices using DPDK-compatible driver<br>
============================================<br>
0000:00:09.0 '82540EM Gigabit Ethernet Controller 100e' drv=igb_uio unused=<br>
0000:00:0a.0 '82540EM Gigabit Ethernet Controller 100e' drv=igb_uio unused=<br>
<br>
Network devices using kernel driver<br>
===================================<br>
0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=igb_uio *Active*</span><br>
<br>
<br>
<span style="background:silver">[root@localhost test]# lspci | egrep -i --color 'network|ethernet'<br>
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device<br>
00:09.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)<br>
00:0a.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)</span><br>
<br>
<span style="background:silver">[root@localhost test]# lshw -c network -businfo<br>
Bus info Device Class Description<br>
========================================================<br>
<a href="mailto:pci@0000:00:03.0">pci@0000:00:03.0</a> network Virtio network device<br>
virtio@0 eth0 network Ethernet interface<br>
<a href="mailto:pci@0000:00:09.0">pci@0000:00:09.0</a> network 82540EM Gigabit Ethernet Controller<br>
<a href="mailto:pci@0000:00:0a.0">pci@0000:00:0a.0</a> network 82540EM Gigabit Ethernet Controller<br>
br0 network Ethernet interface<br>
ovs-system network Ethernet interface<br>
virbr0-nic network Ethernet interface<br>
virbr0 network Ethernet interface</span><br>
<br>
<br>
<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">在</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> 2017-11-16 02:21:54</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">，</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">"Shivaram
Mysore" <<a href="mailto:shivaram.mysore@gmail.com">shivaram.mysore@gmail.com</a>>
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">写道：</span><span style="color:black"><o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Glad to hear that you solved the problem.</span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">What you have are Intel NIC - 82540em. The driver igb_uio should work for this - which is what you have.
I am confused as to why you said that you were using Broadcom Nextreme card </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Anyways, it is good to hear that you were able to rectify the issue.</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">On Wed, Nov 15, 2017 at 4:40 AM,
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">王嵘</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> <<a href="mailto:rong.wang@easystack.cn" target="_blank">rong.wang@easystack.cn</a>> wrote:</span><span style="color:black"><o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">hi, I solved the NIC bind problem.<br>
I checked the log found that<br>
<span style="background:silver"> netdev_dpdk|ERR|Interface dpdk-eth2(rxq:1 txq:3) configure error: Invalid argument</span><br>
so, I changed the rxq of the NIC to 3 by "options:n_rxq=3", and it works. ^^ <br>
<br>
I tested the ovs2.8-dpdk in another VM of Centos7, but the "</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:red;background:silver">Permission denied</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">"
problem still exists.<br>
Here is the results of the commands you asked to execute, </span><span style="color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
[root@localhost ~]# lshw -c network -businfo<br>
Bus info Device Class Description<br>
========================================================<br>
<a href="mailto:pci@0000:00:03.0">pci@0000:00:03.0</a> network Virtio network device<br>
virtio@0 eth0 network Ethernet interface<br>
<a href="mailto:pci@0000:00:09.0">pci@0000:00:09.0</a> network 82540EM Gigabit Ethernet Controller<br>
<a href="mailto:pci@0000:00:0a.0">pci@0000:00:0a.0</a> ens10 network 82540EM Gigabit Ethernet Controller<br>
br0 network Ethernet interface<br>
ovs-netdev network Ethernet interface<br>
virbr0-nic network Ethernet interface<br>
virbr0 network Ethernet interface<br>
[root@localhost ~]# lspci | egrep -i --color 'network|ethernet'<br>
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device<br>
00:09.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)<br>
00:0a.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)<br>
[root@localhost ~]# lsmod | grep uio<br>
igb_uio 13224 1 <br>
uio 19259 3 igb_uio<br>
<br>
[root@localhost test]# lsmod | grep uio<br>
igb_uio 13224 0 <br>
uio 19259 1 igb_uio<br>
<br>
<br>
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">在</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> 2017-11-15 04:29:34</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">，</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">"Shivaram
Mysore" <<a href="mailto:shivaram.mysore@gmail.com" target="_blank">shivaram.mysore@gmail.com</a>>
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">写道：</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt" id="m_-6536986810327580421isReplyContent">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">What is the output of the following commands:</span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E">lshw -c network -businfo</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:Consolas;color:black">lspci | egrep -i --color 'network|ethernet'</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:Consolas;color:black">lsmod | grep uio</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E">/sbin/dpdk-devbind --status</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">On Mon, Nov 13, 2017 at 9:29 PM,
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">王嵘</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> <<a href="mailto:rong.wang@easystack.cn" target="_blank">rong.wang@easystack.cn</a>> wrote:</span><span style="color:black"><o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">I just want to use ovs-2.8 for userspace connection-track, and I choose the latest LST DPDK 1705.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">But, you mean theres is something like broadcom-uio.ko when using broadcom NIC? </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">I just have changed the config/common_base:</span><span style="color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">CONFIG_RTE_LIBRTE_BNX2X_PMD=y</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">CONFIG_RTE_LIBRTE_BNX2X_DEBUG=y</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">CONFIG_RTE_LIBRTE_BNX2X_DEBUG_INIT=y</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">CONFIG_RTE_LIBRTE_BNX2X_DEBUG_RX=y</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">CONFIG_RTE_LIBRTE_BNX2X_DEBUG_TX=y</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">CONFIG_RTE_LIBRTE_BNX2X_MF_SUPPORT=y</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC=y</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">在</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> 2017-11-14 12:49:44</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">，</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">"Shivaram
Mysore" <<a href="mailto:shivaram.mysore@gmail.com" target="_blank">shivaram.mysore@gmail.com</a>>
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">写道：</span><span style="color:black"><o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Is there a specific reason as to why you would want to use this version of DPDK? I have never had to check
permission.</span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Also, igb is mainly for Intel NICs. You are using a Broadcom NIC - <a href="http://dpdk.org/doc/guides/nics/bnxt.html" target="_blank">http://dpdk.org/doc/guides/nics/bnxt.html</a>
for driver info. I think what you have is a misconfiguration.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">You may want to check out the scripts here <a href="https://github.com/shivarammysore/faucetsdn-intel/tree/master/src/ubuntu/zesty/ovs_281" target="_blank">https://github.com/shivarammysore/faucetsdn-intel/tree/master/src/ubuntu/zesty/ovs_281</a>
- to setup and configure DPDK. I have only used Intel NICs.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Cheers!</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">On Mon, Nov 13, 2017 at 6:50 PM,
</span><span lang="ZH-CN" style="font-size:10.5pt;color:black">王嵘</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> <<a href="mailto:rong.wang@easystack.cn" target="_blank">rong.wang@easystack.cn</a>> wrote:</span><span style="color:black"><o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">hi, I'm using ovs2.8 with dpdk-17052, and the '</span><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:red">NetXtreme
II BCM57810 10 Gigabit Ethernet 168e</span></b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">'.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">There comes the error, while I try to bind the NIC using igb_uio.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:25:18.400Z|00362|dpdk|INFO|EAL: probe driver: 14e4:168e net_bnx2x</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:25:18.400Z|00363|dpdk|ERR|EAL:
</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:red;background:silver">Cannot open /dev/uio0: Permission denied</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Then I chmod the file, like "chmod 777 /dev/uio0",(but I dont think this is right way.)</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">but there comes other files with the same error, and I also chmod them.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">And I do the actions:</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">chmod 777 /sys/bus/pci/devices/0000:20:00.1/*</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">chmod 777 /sys/class/uio/uio0/device/config</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Then , restart the ovs-dpdk,</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">But, there comes the following error:</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:40:05.436Z|8384580|dpdk|ERR|PMD: pci_read(): Can't read from PCI config
space</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:40:05.436Z|8384581|dpdk|ERR|PMD: pci_read(): Can't read from PCI config
space</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:40:05.436Z|8384582|dpdk|ERR|PMD: pci_read(): Can't read from PCI config
space</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:40:05.436Z|8384583|dpdk|ERR|PMD:
</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:red;background:silver">pci_read(): Can't read from PCI config space</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:40:05.436Z|8384584|dpdk|INFO|PMD: bnx2x_pci_get_caps(): Failed to allocate
memory</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:40:05.436Z|8384585|dpdk|INFO|PMD: bnx2x_attach(): PCIe caps reading was
failed</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:40:05.436Z|8384586|dpdk|ERR|PMD: bnx2x_common_dev_init(): bnx2x_attach
failed (-12)</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:silver">2017-11-14T02:40:05.436Z|8384587|dpdk|ERR|EAL: Requested device 0000:20:00.1 cannot be
used</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">So, why there comes the Permission prolem? And what should I do for the PCIe error? Or it is the NIC drivers'
problem?</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Thanks a lot.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><br>
_______________________________________________<br>
discuss mailing list<br>
<a href="mailto:discuss@openvswitch.org" target="_blank">discuss@openvswitch.org</a><br>
<a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" target="_blank">https://mail.openvswitch.org/mailman/listinfo/ovs-discuss</a></span><span style="color:black"><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>