[ovs-discuss] [bug]ovs-dpdk can't open dpdk port and dpdkvhostuser port added before

Christian Ehrhardt christian.ehrhardt at canonical.com
Thu Feb 25 08:13:50 UTC 2016


I still don't see the EAL messages of DPDK in these logs.
And with the "dpdk0: Cannot allocate memory" issue you reported I really
think in those messages you will find what is going on.
So IMHO take a look around where those end up on your system.

Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

On Thu, Feb 25, 2016 at 8:49 AM, 王华夏 <wanghuaxia at jd.com> wrote:

> Thanks for your sharing , maybe this
>
>
>
>
>
> Syslog :
>
>
>
> Feb 24 18:44:43 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00513|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap57b09341-c9
>
> Feb 24 18:44:43 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00515|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap6e8f58fa-c1
>
> Feb 24 18:44:43 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00517|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap1522aef1-14
>
> Feb 24 18:44:43 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00519|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap78ec2293-ae
>
> Feb 24 18:44:43 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00521|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tapb6496d1c-4c
>
> Feb 24 18:44:43 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00523|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap511dcfea-e1
>
> Feb 24 18:44:43 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00525|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tapb94777b3-5d
>
> Feb 24 18:44:44 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00528|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap57b09341-c9
>
> Feb 24 18:44:44 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00530|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap6e8f58fa-c1
>
> Feb 24 18:44:44 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00532|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap1522aef1-14
>
> Feb 24 18:44:44 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00534|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tap78ec2293-ae
>
> Feb 24 18:44:44 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00536|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tapb6496d1c-4c
>
> Feb 24 18:44:44 A06-R12-302F0111-I32-13 ovs-vswitchd[123203]:
> ovs|00538|dpdk|ERR|vhost-user socket device setup failure for socket
> /var/run/openvswitch/tapb94777b3-5d
>
>
>
> Ovs-vswitch log:
>
>
>
> 2016-02-24T10:24:09.555Z|00152|bridge|WARN|could not open network device
> tap9f4bd15d-51 (No such device)
>
> 2016-02-24T10:24:09.603Z|00153|dpdk|INFO|Socket
> /var/run/openvswitch/tap9f4bd15d-51 created for vhost-user port
> tap9f4bd15d-51
>
> 2016-02-24T10:24:09.604Z|00002|dpif_netdev(pmd71)|INFO|Core 10 processing
> port 'tap6e8f58fa-c1'
>
> 2016-02-24T10:24:09.604Z|00003|dpif_netdev(pmd71)|INFO|Core 10 processing
> port 'tap9f4bd15d-51'
>
> 2016-02-24T10:24:09.604Z|00154|bridge|INFO|bridge br-int: added interface
> tap9f4bd15d-51 on port 8
>
> 2016-02-24T10:24:10.471Z|00155|connmgr|INFO|br-tun<->unix: 1 flow_mods in
> the last 0 s (1 modifications)
>
> 2016-02-24T10:24:10.475Z|00156|connmgr|INFO|br-tun<->unix: 1 flow_mods in
> the last 0 s (1 adds)
>
> 2016-02-24T10:24:10.687Z|00157|connmgr|INFO|br-int<->unix: 1 flow_mods in
> the last 0 s (1 deletes)
>
> 2016-02-24T10:24:10.692Z|00158|connmgr|INFO|br-int<->unix: 1 flow_mods in
> the last 0 s (1 deletes)
>
> 2016-02-24T10:24:10.697Z|00159|ofp_util|INFO|Dropped 1 log messages in
> last 4012 seconds (most recently, 4012 seconds ago) due to excessive rate
>
> 2016-02-24T10:24:10.697Z|00160|ofp_util|INFO|normalization changed
> ofp_match, details:
>
> 2016-02-24T10:24:10.697Z|00161|ofp_util|INFO| pre:
> in_port=8,nw_proto=58,tp_src=136
>
> 2016-02-24T10:24:10.697Z|00162|ofp_util|INFO|post: in_port=8
>
> 2016-02-24T10:24:10.697Z|00163|connmgr|INFO|br-int<->unix: 1 flow_mods in
> the last 0 s (1 deletes)
>
> 2016-02-24T10:24:10.702Z|00164|connmgr|INFO|br-int<->unix: 1 flow_mods in
> the last 0 s (1 deletes)
>
> 2016-02-24T10:24:10.707Z|00165|connmgr|INFO|br-int<->unix: 1 flow_mods in
> the last 0 s (1 adds)
>
> 2016-02-24T10:24:10.712Z|00166|connmgr|INFO|br-int<->unix: 1 flow_mods in
> the last 0 s (1 adds)
>
> 2016-02-24T10:24:10.717Z|00167|connmgr|INFO|br-int<->unix: 1 flow_mods in
> the last 0 s (1 adds)
>
> 2016-02-24T10:24:10.722Z|00168|connmgr|INFO|br-int<->unix: 1 flow_mods in
> the last 0 s (1 adds)
>
> 2016-02-24T10:24:14.718Z|00003|daemon_unix(monitor)|ERR|1 crashes: pid
> 86760 died, killed (Aborted), core dumped, restarting
>
> 2016-02-24T10:24:14.743Z|00004|ovs_numa|INFO|Discovered 16 CPU cores on
> NUMA node 0
>
> 2016-02-24T10:24:14.743Z|00005|ovs_numa|INFO|Discovered 16 CPU cores on
> NUMA node 1
>
> 2016-02-24T10:24:14.743Z|00006|ovs_numa|INFO|Discovered 2 NUMA nodes and
> 32 CPU cores
>
> 2016-02-24T10:24:14.743Z|00007|memory|INFO|55836 kB peak resident set size
> after 4137.6 seconds
>
> 2016-02-24T10:24:14.743Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
> connecting...
>
> 2016-02-24T10:24:14.747Z|00009|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
> connected
>
> 2016-02-24T10:24:14.781Z|00010|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath supports recirculation
>
> 2016-02-24T10:24:14.781Z|00011|ofproto_dpif|INFO|netdev at ovs-netdev: MPLS
> label stack length probed as 3
>
> 2016-02-24T10:24:14.781Z|00012|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath supports unique flow ids
>
> 2016-02-24T10:24:14.781Z|00013|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath does not support ct_state
>
> 2016-02-24T10:24:14.781Z|00014|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath does not support ct_zone
>
> 2016-02-24T10:24:14.781Z|00015|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath does not support ct_mark
>
> 2016-02-24T10:24:14.781Z|00016|ofproto_dpif|INFO|netdev at ovs-netdev:
> Datapath does not support ct_label
>
> 2016-02-24T10:24:14.815Z|00017|dpdk|ERR|vhost-user socket device setup
> failure for socket /var/run/openvswitch/tap9f4bd15d-51
>
> 2016-02-24T10:24:14.815Z|00018|bridge|WARN|could not open network device
> tap9f4bd15d-51 (Unknown error -1)
>
> 2016-02-24T10:24:14.815Z|00019|dpdk|ERR|vhost-user socket device setup
> failure for socket /var/run/openvswitch/tap57b09341-c9
>
> 2016-02-24T10:24:14.815Z|00020|bridge|WARN|could not open network device
> tap57b09341-c9 (Unknown error -1)
>
> 2016-02-24T10:24:14.815Z|00021|dpdk|ERR|vhost-user socket device setup
> failure for socket /var/run/openvswitch/tap1522aef1-14
>
> 2016-02-24T10:24:14.816Z|00022|bridge|WARN|could not open network device
> tap1522aef1-14 (Unknown error -1)
>
> 2016-02-24T10:24:14.816Z|00023|dpdk|ERR|vhost-user socket device setup
> failure for socket /var/run/openvswitch/tap6e8f58fa-c1
>
> 2016-02-24T10:24:14.816Z|00024|bridge|WARN|could not open network device
> tap6e8f58fa-c1 (Unknown error -1)
>
> 2016-02-24T10:24:14.816Z|00025|dpdk|ERR|vhost-user socket device setup
> failure for socket /var/run/openvswitch/tap78ec2293-ae
>
> 2016-02-24T10:24:14.816Z|00026|bridge|WARN|could not open network device
> tap78ec2293-ae (Unknown error -1)
>
> 2016-02-24T10:24:14.816Z|00027|bridge|INFO|bridge br-int: added interface
> patch-tun on port 1
>
> 2016-02-24T10:24:14.816Z|00028|dpdk|ERR|vhost-user socket device setup
> failure for socket /var/run/openvswitch/tapb6496d1c-4c
>
> 2016-02-24T10:24:14.816Z|00029|bridge|WARN|could not open network device
> tapb6496d1c-4c (Unknown error -1)
>
> 2016-02-24T10:24:14.817Z|00030|bridge|INFO|bridge br-int: added interface
> br-int on port 65534
>
> 2016-02-24T10:24:14.817Z|00031|dpdk|ERR|vhost-user socket device setup
> failure for socket /var/run/openvswitch/tapb94777b3-5d
>
> 2016-02-24T10:24:14.817Z|00032|bridge|WARN|could not open network device
> tapb94777b3-5d (Unknown error -1)
>
> 2016-02-24T10:24:14.817Z|00033|bridge|INFO|bridge br-tun: added interface
> patch-int on port 1
>
> 2016-02-24T10:24:14.817Z|00034|bridge|INFO|bridge br-tun: added interface
> vxlan-0a001f8d on port 3
>
> 2016-02-24T10:24:14.818Z|00035|bridge|INFO|bridge br-tun: added interface
> br-tun on port 65534
>
> 2016-02-24T10:24:14.818Z|00036|bridge|INFO|bridge br-tun: added interface
> vxlan-0a002016 on port 4
>
> 2016-02-24T10:24:14.818Z|00037|bridge|INFO|bridge br-tun: added interface
> vxlan-0a001f7e on port 2
>
> 2016-02-24T10:24:14.818Z|00038|bridge|WARN|could not open network device
> dpdk0 (Cannot allocate memory)
>
> 2016-02-24T10:24:14.818Z|00039|bridge|INFO|bridge br-dpdk: added interface
> br-dpdk on port 65534
>
> 2016-02-24T10:24:14.819Z|00040|bridge|INFO|bridge br-int: using datapath
> ID 00000ac7455f064c
>
> 2016-02-24T10:24:14.819Z|00041|connmgr|INFO|br-int: added service
> controller "punix:/var/run/openvswitch/br-int.mgmt"
>
> 2016-02-24T10:24:14.819Z|00042|bridge|INFO|bridge br-tun: using datapath
> ID 00008affbc1d314f
>
> 2016-02-24T10:24:14.819Z|00043|connmgr|INFO|br-tun: added service
> controller "punix:/var/run/openvswitch/br-tun.mgmt"
>
> 2016-02-24T10:24:14.819Z|00044|bridge|INFO|bridge br-dpdk: using datapath
> ID 00001e3c7b157245
>
> 2016-02-24T10:24:14.819Z|00045|connmgr|INFO|br-dpdk: added service
> controller "punix:/var/run/openvswitch/br-dpdk.mgmt"
>
> 2016-02-24T10:24:14.850Z|00046|dpdk|ERR|vhost-user socket device setup
> failure for socket /var/run/openvswitch/tap9f4bd15d-51
>
>
>
>
>
> Indeed I have set as --socket-mem 400  and tested for long time and did
> not have the same problem ,so it would not be lack of memory
>
>
>
>
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Christian Ehrhardt <christian.ehrhardt at canonical.com>
> *Sent: *2016年2月25日 15:07
> *To: *王华夏 <wanghuaxia at jd.com>
> *Cc: *discuss at openvswitch.org
> *Subject: *Re: [ovs-discuss] [bug]ovs-dpdk can't open dpdk port and
> dpdkvhostuser port added before
>
>
> Hi Eric,
> when debugging similar issues I realized that while the openvswitch log
> (like /var/log/openvswitch/ovs-vswitchd.log depending on your Distro) is
> great.
> The DPDK messages can bypass that and at least in my case ended up in
> syslog.
>
> Therein you will find messages like
> ovs-ctl[3560]: EAL: ...
> ovs-vswitchd[3592]: EAL: ...
>
> And DPDK there is rather verbose on its initialization (which is good).
> Since you can trigger the issue by different --socket-mem options I'd
> recommend you to check these logs.
> Most of the time, once I looked there it was eventually rather obvious.
>
> Regarding the memory allocation you will find for example (in my case non
> socket based -m 4096)
> ovs-ctl[3560]: EAL: Requesting 4 pages of size 1024MB from socket 0
>
> If it is not obvious please share the logs you find there and people on
> the list might be able to help you more.
>
>
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
>
> On Thu, Feb 25, 2016 at 7:22 AM, 王华夏 <wanghuaxia at jd.com> wrote:
>
>>
>>
>> HI
>>
>>
>>
>> I am trying to restart ovs ,find  error as fellows, all thess bridges and
>> ports I have added before
>>
>>
>>
>> Bridge br-dpdk
>>         Port "dpdk0"
>>             Interface "dpdk0"
>>                 type: dpdk
>>
>> *   error: "could not open network device dpdk0 (Cannot allocate memory)*
>> "
>>         Port br-dpdk
>>             Interface br-dpdk
>>                 type: internal
>>     Bridge br-tun
>>         fail_mode: secure
>>         Port "vxlan-0a001f8d"
>>             Interface "vxlan-0a001f8d"
>>                 type: vxlan
>>
>>                 options: {df_default="true", in_key=flow, local_ip="10.0.32.13", out_key=flow, remote_ip="10.0.31.141"}
>>         Port "vxlan-0a001f7e"
>>             Interface "vxlan-0a001f7e"
>>                 type: vxlan
>>
>>                 options: {df_default="true", in_key=flow, local_ip="10.0.32.13", out_key=flow, remote_ip="10.0.31.126"}
>>         Port "vxlan-0a002016"
>>             Interface "vxlan-0a002016"
>>                 type: vxlan
>>
>>                 options: {df_default="true", in_key=flow, local_ip="10.0.32.13", out_key=flow, remote_ip="10.0.32.22"}
>>         Port patch-int
>>             Interface
>>  patch-int
>>                 type: patch
>>                 options: {peer=patch-tun}
>>         Port br-tun
>>             Interface br-tun
>>                 type: internal
>>     Bridge br-int
>>         fail_mode: secure
>>         Port br-int
>>             Interface br-int
>>                 type: internal
>>         Port "tap78ec2293-ae"
>>             tag: 1
>>             Interface "tap78ec2293-ae"
>>                 type: dpdkvhostuser
>>
>> *         error: "could not open network device tap78ec2293-ae (Unknown error -1)*
>> "
>>         Port "tapb94777b3-5d"
>>             tag: 1
>>
>>
>>
>>
>>
>> When start the ovs like this ,this problem will occurs:
>>
>>
>>
>> ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem=2048,2048 --
>> unix:/var/run/openvswitch/db.sock --pidfile --detach -vconsole:emer
>> -vsyslog:err -vfile:info --mlockall --no-chdir
>> --log-file=/var/log/openvswitch/ovs-vswitchd.log –monitor
>>
>>
>>
>> if change the --socket-mem=2048,2048   to   –socket-mem= 2048 ,  it will
>> have no problem
>>
>>
>>
>> if it  means the numa node1 can not use ?
>>
>>
>>
>> my ovs version is 2.5 and dpdk version is 2.2 with kernel Linux version
>> 3.10.0-229.20.1.el7.x86_64 (builder at kbuilder.dev.centos.org) (gcc
>> version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Nov 3 19:10:07
>> UTC 2015
>>
>>
>>
>>
>>
>> Thanks & regards
>>
>>
>>
>> Eric wang
>>
>>
>>
>> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>>
>>
>>
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> http://openvswitch.org/mailman/listinfo/discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20160225/e2eb684b/attachment-0002.html>


More information about the discuss mailing list