[ovs-discuss] IPFIX crash

Neelakantam Gaddam neelugaddam at gmail.com
Thu Apr 6 05:24:45 UTC 2017


While testing the IPFIX feature with openvswitch-2.6.1, observed the below
crash when traffic is on.

[  347.143417] BUG: unable to handle kernel NULL pointer dereference
at           (null)
[  347.143452] IP: [<ffffffffa04e8b4c>] dst_cache_get_ip4+0xc/0x50 [openvswitch]
[  347.143484] PGD 0
[  347.143493] Oops: 0000 [#1] SMP
[  347.143506] Modules linked in: vhost_net vhost macvtap macvlan
vport_geneve(OE) vport_gre(OE) vport_vxlan(OE) openvswitch(OE)
nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 geneve gre libcrc32c
xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter
ebtables ip6table_filter ip6_tables iptable_filter fuse dm_mirror
dm_region_hash dm_log dm_mod intel_powerclamp coretemp intel_rapl
kvm_intel ixgbe(OE) kvm iTCO_wdt crc32_pclmul ipmi_ssif ipmi_devintf
iTCO_vendor_support ses ghash_clmulni_intel enclosure ipmi_si vxlan
ip6_udp_tunnel udp_tunnel aesni_intel mei_me sg sb_edac dcdbas
edac_core lpc_ich mei shpchp wmi mfd_core acpi_power_meter lrw
gf128mul glue_helper
[  347.143809]  ablk_helper cryptd pcspkr ipmi_msghandler nfsd
auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2
sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea
sysfillrect sysimgblt drm_kms_helper ttm crct10dif_pclmul drm
crct10dif_common ahci crc32c_intel libahci igb dca libata i2c_algo_bit
ptp i2c_core megaraid_sas pps_core [last unloaded: liquidio_ovs]
[  347.143960] CPU: 4 PID: 3818 Comm: vhost-3811 Tainted: G
OE  ------------   3.10.0-327.el7.x86_64 #1
[  347.143993] Hardware name: Dell Inc. PowerEdge T630/0W9WXC, BIOS
1.1.4 11/04/2014
[  347.144019] task: ffff880847ec1700 ti: ffff8808516f8000 task.ti:
ffff8808516f8000
[  347.144043] RIP: 0010:[<ffffffffa04e8b4c>]  [<ffffffffa04e8b4c>]
dst_cache_get_ip4+0xc/0x50 [openvswitch]
[  347.144079] RSP: 0018:ffff8808516fb5d8  EFLAGS: 00010246
[  347.144097] RAX: 0000000000000000 RBX: ffff880849023210 RCX: ffff880849023208
[  347.144120] RDX: 0000000000000000 RSI: ffff880849023210 RDI: 0000000000000000
[  347.144144] RBP: ffff8808516fb5e8 R08: 000000000a04a8c0 R09: ffff880849023210
[  347.144167] R10: ffff880849023e00 R11: ffff8808516fb890 R12: 0000000000000000
[  347.144190] R13: ffff8800784d8c00 R14: 0000000000000000 R15: ffff8808514568f8
[  347.144214] FS:  0000000000000000(0000) GS:ffff88085e480000(0000)
knlGS:0000000000000000
[  347.144240] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  347.144259] CR2: 0000000000000000 CR3: 000000081c865000 CR4: 00000000001427e0
[  347.144283] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  347.144306] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  347.144329] Stack:
[  347.144337]  ffff880849023210 0000000000000000 ffff8808516fb658
ffffffffa04f6b0a
[  347.144365]  0a04a8c04d5e1de0 ffff880000000000 ffff8808516fb658
ffffffff81524289
[  347.144393]  020e0e0e64000000 00000000030e0e0e 0000000080da67a9
ffff880849023208
[  347.144421] Call Trace:
[  347.144436]  [<ffffffffa04f6b0a>]
vxlan_get_route.isra.41+0xea/0x130 [openvswitch]
[  347.144464]  [<ffffffff81524289>] ? __skb_get_hash+0x39/0x160
[  347.144487]  [<ffffffffa04f7ec0>]
ovs_vxlan_fill_metadata_dst+0x170/0x1e0 [openvswitch]
[  347.144517]  [<ffffffffa04e88ee>]
ovs_dev_fill_metadata_dst+0x9e/0xd0 [openvswitch]
[  347.144544]  [<ffffffffa04d717e>] output_userspace+0xfe/0x180 [openvswitch]
[  347.144569]  [<ffffffffa04d86dd>] do_execute_actions+0x63d/0x8f0
[openvswitch]
[  347.144594]  [<ffffffffa04d89d1>] ovs_execute_actions+0x41/0x130
[openvswitch]
[  347.145578]  [<ffffffffa04de904>] ovs_dp_process_packet+0x94/0x140
[openvswitch]
[  347.146549]  [<ffffffffa04e73a3>] ovs_vport_receive+0x73/0xd0 [openvswitch]
[  347.147510]  [<ffffffff810c3f12>] ? enqueue_task_fair+0x402/0x6c0
[  347.148470]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[  347.149432]  [<ffffffff810b5d85>] ? check_preempt_curr+0x75/0xa0
[  347.150378]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[  347.151316]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[  347.152249]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[  347.153168]  [<ffffffff810b0d84>] ? __wake_up+0x44/0x50
[  347.154090]  [<ffffffffa04e7d84>] internal_dev_xmit+0x24/0x60 [openvswitch]
[  347.155012]  [<ffffffff8152c671>] dev_hard_start_xmit+0x171/0x3b0
[  347.155919]  [<ffffffff8154cd74>] sch_direct_xmit+0x104/0x200
[  347.156800]  [<ffffffff8152cae6>] dev_queue_xmit+0x236/0x570
[  347.157662]  [<ffffffffa03897bc>] macvlan_start_xmit+0x3c/0xc0 [macvlan]
[  347.158503]  [<ffffffff8152c671>] dev_hard_start_xmit+0x171/0x3b0
[  347.159327]  [<ffffffff8154cd74>] sch_direct_xmit+0x104/0x200
[  347.160127]  [<ffffffff8152cae6>] dev_queue_xmit+0x236/0x570
[  347.160901]  [<ffffffffa0517a74>] macvtap_get_user+0x414/0x720 [macvtap]
[  347.161658]  [<ffffffffa0517dab>] macvtap_sendmsg+0x2b/0x30 [macvtap]
[  347.162391]  [<ffffffffa052e10c>] handle_tx+0x2fc/0x550 [vhost_net]
[  347.163111]  [<ffffffffa052e395>] handle_tx_kick+0x15/0x20 [vhost_net]
[  347.163807]  [<ffffffffa051e60b>] vhost_worker+0xfb/0x1e0 [vhost]
[  347.164489]  [<ffffffffa051e510>] ? vhost_dev_reset_owner+0x50/0x50 [vhost]
[  347.165165]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[  347.165826]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[  347.166482]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[  347.167131]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[  347.167784] Code: 65 48 03 34 25 e8 ce 00 00 48 83 c7 08 e8 4d ff
ff ff 5d c3 0f 1f 00 31 c0 c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89
e5 41 54 53 <48> 8b 1f 48 85 db 74 2c 49 89 f4 65 48 03 1c 25 e8 ce 00
00 48
[  347.169207] RIP  [<ffffffffa04e8b4c>] dst_cache_get_ip4+0xc/0x50
[openvswitch]
[  347.169904]  RSP <ffff8808516fb5d8>
[  347.170584] CR2: 0000000000000000


Setup details:

Host-1                                         Host-2
                                                 VM in macvatp mode via OVS
bridge
Intel 10G NIC(p1)  -----VXLAN-----  (p1)Intel 10G NIC

     p2 (collector) ------------------------- p2 (IPFIX exporter)

Further digging into the code, revealed the below information.
The pointer dst_cache pointer is NULL in the function rpl_dst_cache_get_ip4().
There is no NULL check for pointer dst_cache.

Is this issue fixed after the release 2.6.1 version? Please let me know if
anybody has the proper fix for this issue.

-- 
Thanks & Regards
Neelakantam Gaddam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20170406/fab04793/attachment.html>


More information about the discuss mailing list