[ovs-dev] kernel panic

Gregory Rose gvrose8192 at gmail.com
Fri Apr 6 14:51:39 UTC 2018


On 4/5/2018 11:47 AM, Nicolas Bouliane via dev wrote:
> Hey folks,
>
> We have experienced a kernel panic and we are trying to nail down the
> culprit: NIC, network stack itself, OvS datapath, etc.
>
> I can provide more info in a subsequent email, but at first I just wanted
> to have a couple of eye balls to see if it could potentially
> be caused by OvS, or if from the trace we just can't tell.
>
> Please let me know what you think ! thanks.

"Potentially" it could be a lot of things - there's just no way to tell 
from looking at a kernel panic stack dump.  We need more to go on.

Thanks,

- Greg

>
> # ovs-appctl version
> ovs-vswitchd (Open vSwitch) 2.7.3
>
> [ 1810.969828] ------------[ cut here ]------------
> [ 1810.975024] kernel BUG at /build/linux-4.9.77-1.4/net/core/skbuff.c:3236!
> [ 1810.982644] invalid opcode: 0000 [#1] SMP
> [ 1810.992505] Modules linked in: act_police cls_basic sch_ingress
> ebtable_filter ebtables ip6table_filter iptable_filter nbd ip6tablem
> [ 1811.092848] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 4.9.77-1-generic
> #4+1516907002do~691dc34
> [ 1811.103887] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.5.5
> 08/16/2017
> [ 1811.112305] task: ffff98eee64b8000 task.stack: ffffb8ffb1390000
> [ 1811.118949] RIP: 0010:[<ffffffff9074106e>] [<ffffffff9074106e>]
> skb_segment+0xabe/0xc80
> [ 1811.128064] RSP: 0018:ffff98eeff9036f0 EFLAGS: 00010206
> [ 1811.134030] RAX: 0000000000000000 RBX: ffff98eeda3f0b00 RCX:
> 00000000000060e2
> [ 1811.142037] RDX: 0000000000000011 RSI: 00000000000060e2 RDI:
> 0000000000000078
> [ 1811.150045] RBP: ffff98eeff9037c0 R08: ffff98eed965f200 R09:
> 0000000000000011
> [ 1811.158053] R10: 0000000000006142 R11: 000000000000606a R12:
> ffff98e871305f00
> [ 1811.166060] R13: 00000000000060e2 R14: ffff98eecbe7fb00 R15:
> ffff98eed965fd00
> [ 1811.174069] FS: 0000000000000000(0000) GS:ffff98eeff900000(0000)
> knlGS:0000000000000000
> [ 1811.183166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 1811.189619] CR2: 00007fd4152c809d CR3: 0000007ea5402000 CR4:
> 0000000000362670
> [ 1811.197624] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 1811.205631] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [ 1811.213636] Stack:
> [ 1811.215908] ffff98eed965f200 ffffffffffffffbe 00000000b48ee580
> ffff98eed965f200
> [ 1811.224275] 00005b2efffffff4 0000000000000000 0000000000000001
> 0000000000000088
> [ 1811.232640] 0000000000000042 000000100000000c ffff98e800000000
> 00000400000058c1
> [ 1811.241009] Call Trace:
> [ 1811.243768] <IRQ> [ 1811.245928] [<ffffffff9073a0a0>] ?
> skb_panic+0x70/0x70
> [ 1811.251809] [<ffffffff90739740>] ? reqsk_fastopen_remove+0x170/0x170
> [ 1811.259044] [<ffffffff907bdf08>] tcp_gso_segment+0xf8/0x4e0
> [ 1811.265403] [<ffffffffc065a638>] ? find_bucket.isra.2+0x58/0x60
> [openvswitch]
> [ 1811.273531] [<ffffffff907be32e>] tcp4_gso_segment+0x3e/0xa0
> [ 1811.279885] [<ffffffff907cf32d>] inet_gso_segment+0x16d/0x3c0
> [ 1811.286437] [<ffffffff9075293a>] skb_mac_gso_segment+0xaa/0x110
> [ 1811.293179] [<ffffffff90752a66>] __skb_gso_segment+0xc6/0x190
> [ 1811.299727] [<ffffffff90752c43>] ? netif_skb_features+0x113/0x240
> [ 1811.306664] [<ffffffff90752ec4>] validate_xmit_skb+0x154/0x280
> [ 1811.313309] [<ffffffff90753033>] validate_xmit_skb_list+0x43/0x70
> [ 1811.320249] [<ffffffff9077bc9b>] sch_direct_xmit+0x10b/0x190
> [ 1811.326700] [<ffffffff907537fa>] __dev_queue_xmit+0x4ca/0x670
> [ 1811.333247] [<ffffffff907539b0>] dev_queue_xmit+0x10/0x20
> [ 1811.339412] [<ffffffffc065c61c>] ovs_vport_send+0x4c/0xc0 [openvswitch]
> [ 1811.346923] [<ffffffffc064b8c0>] do_output.isra.32+0x50/0x230
> [openvswitch]
> [ 1811.354832] [<ffffffffc064cc49>] do_execute_actions+0x11a9/0x1510
> [openvswitch]
> [ 1811.363153] [<ffffffffc065aabc>] ? flow_lookup.isra.6+0x7c/0xb0
> [openvswitch]
> [ 1811.371277] [<ffffffffc064cff3>] ovs_execute_actions+0x43/0x110
> [openvswitch]
> [ 1811.379393] [<ffffffffc0652e76>] ovs_dp_process_packet+0x86/0x120
> [openvswitch]
> [ 1811.387716] [<ffffffffc0653bd4>] ? key_extract+0x914/0xc80 [openvswitch]
> [ 1811.395337] [<ffffffffc065d040>] ? netdev_port_receive+0x100/0x100
> [openvswitch]
> [ 1811.403753] [<ffffffffc065c576>] ovs_vport_receive+0x76/0xd0
> [openvswitch]
> [ 1811.411569] [<ffffffff9074fc3c>] ? netif_rx+0x1c/0x70
> [ 1811.417341] [<ffffffffc065d3ec>] ? ovs_ip_tunnel_rcv+0x8c/0xe0
> [openvswitch]
> [ 1811.425347] [<ffffffffc065f888>] ? nf_ip_hook+0x738/0xde0 [openvswitch]
> [ 1811.432871] [<ffffffff9078ae2b>] ? nf_iterate+0x5b/0x70
> [ 1811.438835] [<ffffffff9078ae9b>] ? nf_hook_slow+0x5b/0xa0
> [ 1811.444994] [<ffffffff907925de>] ? ip_rcv+0xce/0x390
> [ 1811.450670] [<ffffffff90739d59>] ? kfree_skbmem+0x59/0x60
> [ 1811.456830] [<ffffffff9073c39a>] ? kfree_skb+0x3a/0x90
> [ 1811.462701] [<ffffffffc065cff0>] netdev_port_receive+0xb0/0x100
> [openvswitch]
> [ 1811.470825] [<ffffffffc065d040>] ? netdev_port_receive+0x100/0x100
> [openvswitch]
> [ 1811.479239] [<ffffffffc065d078>] netdev_frame_hook+0x38/0x60
> [openvswitch]
> [ 1811.487051] [<ffffffff907501b0>] __netif_receive_skb_core+0x220/0xac0
> [ 1811.494378] [<ffffffff90750ad3>] ? netif_receive_skb_internal+0x23/0x80
> [ 1811.501899] [<ffffffff907518b5>] ? napi_gro_receive+0xc5/0xf0
> [ 1811.508447] [<ffffffff90750a68>] __netif_receive_skb+0x18/0x60
> [ 1811.515093] [<ffffffff90751b99>] process_backlog+0x89/0x140
> [ 1811.521447] [<ffffffff907511ac>] net_rx_action+0x10c/0x360
> [ 1811.527706] [<ffffffffc0305bac>] ? i40e_msix_clean_rings+0x2c/0x50 [i40e]
> [ 1811.535421] [<ffffffff9086eb0f>] __do_softirq+0xdf/0x2bb
> [ 1811.541484] [<ffffffff90087816>] irq_exit+0xb6/0xc0
> [ 1811.547051] [<ffffffff9086dc3b>] do_IRQ+0x8b/0xd0
> [ 1811.552435] [<ffffffff9086b956>] common_interrupt+0x96/0x96
> [ 1811.558787] <EOI> [ 1811.560942] [<ffffffff906f38e2>] ?
> cpuidle_enter_state+0xa2/0x2e0
> [ 1811.567909] [<ffffffff906f38d2>] ? cpuidle_enter_state+0x92/0x2e0
> [ 1811.574844] [<ffffffff906f3b57>] cpuidle_enter+0x17/0x20
> [ 1811.580907] [<ffffffff900c6563>] call_cpuidle+0x23/0x40
> [ 1811.586871] [<ffffffff900c67d0>] cpu_startup_entry+0x150/0x230
> [ 1811.593520] [<ffffffff90050280>] start_secondary+0x160/0x190
> [ 1811.599969] Code: 75 08 3b 90 84 00 00 00 74 da c7 44 24 14 00 00 00 00
> e9 94 f6 ff ff 48 8b 93 d0 00 00 00 8b 83 d8 00 00 00 29 d0
> [ 1811.621952] RIP [<ffffffff9074106e>] skb_segment+0xabe/0xc80
> [ 1811.628419] RSP <ffff98eeff9036f0>
> [ 1811.632900] ---[ end trace 218aa8a38802887a ]---
> [ 1811.681195] Kernel panic - not syncing: Fatal exception in interrupt
> [ 1811.694518] Kernel Offset: 0xf000000 from 0xffffffff81000000 (relocation
> range: 0xffffffff80000000-0xffffffffbfffffff)
> [ 1811.749633] ---[ end Kernel panic - not syncing: Fatal exception in
> interrupt
> [ 1811.757725] ------------[ cut here ]------------
> [ 1811.762982] WARNING: CPU: 8 PID: 0 at
> /build/linux-4.9.77-1.4/arch/x86/kernel/smp.c:127
> native_smp_send_reschedule+0x40/0x50
> [ 1811.775637] Modules linked in: act_police cls_basic sch_ingress
> ebtable_filter ebtables ip6table_filter iptable_filter nbd ip6tablea
> [ 1811.882174] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G D OE
> 4.9.77-1-generic #4+1516907002do~691dc34
> [ 1811.893273] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.5.5
> 08/16/2017
> [ 1811.901751] ffff98eeff903018 ffffffff90405811 0000000000000000
> 0000000000000000
> [ 1811.910434] ffff98eeff903058 ffffffff900816d1 0000007f00000021
> 0000000000000000
> [ 1811.919117] ffff98eeff819b00 ffff98eee0128e80 0000000000000046
> ffff98eeff819b00
> [ 1811.927799] Call Trace:
> [ 1811.930619] <IRQ> [ 1811.932838] [<ffffffff90405811>]
> dump_stack+0x63/0x82
> [ 1811.938748] [<ffffffff900816d1>] __warn+0xe1/0x100
> [ 1811.944288] [<ffffffff900817bd>] warn_slowpath_null+0x1d/0x20
> [ 1811.950898] [<ffffffff9004f7c0>] native_smp_send_reschedule+0x40/0x50
> [ 1811.958288] [<ffffffff900ab6cd>] resched_curr+0x5d/0xc0
> [ 1811.964314] [<ffffffff900ac1da>] check_preempt_curr+0x7a/0x90
> [ 1811.970924] [<ffffffff900ac209>] ttwu_do_wakeup+0x19/0xf0
> [ 1811.977145] [<ffffffff900ac34f>] ttwu_do_activate+0x6f/0x80
> [ 1811.983559] [<ffffffff900acf48>] try_to_wake_up+0x188/0x3c0
> [ 1811.989975] [<ffffffff9040f0fa>] ? number+0x30a/0x330
> [ 1811.995808] [<ffffffff900ad232>] default_wake_function+0x12/0x20
> [ 1812.002710] [<ffffffff900c6012>] autoremove_wake_function+0x12/0x40
> [ 1812.009903] [<ffffffff900c5a63>] __wake_up_common+0x53/0x90
> [ 1812.016317] [<ffffffff900c5ad9>] __wake_up+0x39/0x50
> [ 1812.022055] [<ffffffff900dc840>] wake_up_klogd_work_func+0x40/0x60
> [ 1812.029154] [<ffffffff9017c3be>] irq_work_run_list+0x4e/0x80
> [ 1812.035668] [<ffffffff90103db0>] ? tick_sched_handle.isra.13+0x60/0x60
> [ 1812.043152] [<ffffffff9017c59b>] irq_work_tick+0x3b/0x50
> [ 1812.049276] [<ffffffff900f4572>] update_process_times+0x42/0x60
> [ 1812.056081] [<ffffffff90103d73>] tick_sched_handle.isra.13+0x23/0x60
> [ 1812.063370] [<ffffffff90103db0>] ? tick_sched_handle.isra.13+0x60/0x60
> [ 1812.070854] [<ffffffff90103ded>] tick_sched_timer+0x3d/0x70
> [ 1812.077268] [<ffffffff900f4ffe>] __hrtimer_run_queues+0xde/0x260
> [ 1812.084168] [<ffffffff900f5737>] hrtimer_interrupt+0x97/0x180
> [ 1812.090778] [<ffffffff900518be>] local_apic_timer_interrupt+0x3e/0x60
> [ 1812.098168] [<ffffffff9086e5d3>] smp_apic_timer_interrupt+0x43/0x60
> [ 1812.105360] [<ffffffff9086ce86>] apic_timer_interrupt+0x96/0xa0
> [ 1812.112168] [<ffffffff9019da0e>] ? panic+0x1ea/0x230
> [ 1812.117903] [<ffffffff9019da07>] ? panic+0x1e3/0x230
> [ 1812.123642] [<ffffffff90030a66>] oops_end+0xb6/0xd0
> [ 1812.129280] [<ffffffff90030eeb>] die+0x4b/0x70
> [ 1812.134431] [<ffffffff9002e249>] do_trap+0x139/0x150
> [ 1812.140157] [<ffffffff9002e557>] do_error_trap+0x77/0xe0
> [ 1812.146283] [<ffffffff9074106e>] ? skb_segment+0xabe/0xc80
> [ 1812.152601] [<ffffffff9086b956>] ? common_interrupt+0x96/0x96
> [ 1812.159212] [<ffffffff90739740>] ? reqsk_fastopen_remove+0x170/0x170
> [ 1812.166501] [<ffffffff9002ea80>] do_invalid_op+0x20/0x30
> [ 1812.172614] [<ffffffff9086bdce>] invalid_op+0x1e/0x30
> [ 1812.178448] [<ffffffff9074106e>] ? skb_segment+0xabe/0xc80
> [ 1812.184766] [<ffffffff9073a0a0>] ? skb_panic+0x70/0x70
> [ 1812.190694] [<ffffffff90739740>] ? reqsk_fastopen_remove+0x170/0x170
> [ 1812.197988] [<ffffffff907bdf08>] tcp_gso_segment+0xf8/0x4e0
> [ 1812.204405] [<ffffffffc065a638>] ? find_bucket.isra.2+0x58/0x60
> [openvswitch]
> [ 1812.217907] [<ffffffff907be32e>] tcp4_gso_segment+0x3e/0xa0
> [ 1812.224321] [<ffffffff907cf32d>] inet_gso_segment+0x16d/0x3c0
> [ 1812.230932] [<ffffffff9075293a>] skb_mac_gso_segment+0xaa/0x110
> [ 1812.237736] [<ffffffff90752a66>] __skb_gso_segment+0xc6/0x190
> [ 1812.244346] [<ffffffff90752c43>] ? netif_skb_features+0x113/0x240
> [ 1812.251344] [<ffffffff90752ec4>] validate_xmit_skb+0x154/0x280
> [ 1812.258052] [<ffffffff90753033>] validate_xmit_skb_list+0x43/0x70
> [ 1812.265051] [<ffffffff9077bc9b>] sch_direct_xmit+0x10b/0x190
> [ 1812.271563] [<ffffffff907537fa>] __dev_queue_xmit+0x4ca/0x670
> [ 1812.278173] [<ffffffff907539b0>] dev_queue_xmit+0x10/0x20
> [ 1812.284396] [<ffffffffc065c61c>] ovs_vport_send+0x4c/0xc0 [openvswitch]
> [ 1812.291980] [<ffffffffc064b8c0>] do_output.isra.32+0x50/0x230
> [openvswitch]
> [ 1812.299950] [<ffffffffc064cc49>] do_execute_actions+0x11a9/0x1510
> [openvswitch]
> [ 1812.308332] [<ffffffffc065aabc>] ? flow_lookup.isra.6+0x7c/0xb0
> [openvswitch]
> [ 1812.316517] [<ffffffffc064cff3>] ovs_execute_actions+0x43/0x110
> [openvswitch]
> [ 1812.324703] [<ffffffffc0652e76>] ovs_dp_process_packet+0x86/0x120
> [openvswitch]
> [ 1812.333084] [<ffffffffc0653bd4>] ? key_extract+0x914/0xc80 [openvswitch]
> [ 1812.340762] [<ffffffffc065d040>] ? netdev_port_receive+0x100/0x100
> [openvswitch]
> [ 1812.349241] [<ffffffffc065c576>] ovs_vport_receive+0x76/0xd0
> [openvswitch]
> [ 1812.357114] [<ffffffff9074fc3c>] ? netif_rx+0x1c/0x70
> [ 1812.362948] [<ffffffffc065d3ec>] ? ovs_ip_tunnel_rcv+0x8c/0xe0
> [openvswitch]
> [ 1812.371016] [<ffffffffc065f888>] ? nf_ip_hook+0x738/0xde0 [openvswitch]
> [ 1812.378598] [<ffffffff9078ae2b>] ? nf_iterate+0x5b/0x70
> [ 1812.384624] [<ffffffff9078ae9b>] ? nf_hook_slow+0x5b/0xa0
> [ 1812.390845] [<ffffffff907925de>] ? ip_rcv+0xce/0x390
> [ 1812.396580] [<ffffffff90739d59>] ? kfree_skbmem+0x59/0x60
> [ 1812.402801] [<ffffffff9073c39a>] ? kfree_skb+0x3a/0x90
> [ 1812.408730] [<ffffffffc065cff0>] netdev_port_receive+0xb0/0x100
> [openvswitch]
> [ 1812.416916] [<ffffffffc065d040>] ? netdev_port_receive+0x100/0x100
> [openvswitch]
> [ 1812.425392] [<ffffffffc065d078>] netdev_frame_hook+0x38/0x60
> [openvswitch]
> [ 1812.433265] [<ffffffff907501b0>] __netif_receive_skb_core+0x220/0xac0
> [ 1812.440652] [<ffffffff90750ad3>] ? netif_receive_skb_internal+0x23/0x80
> [ 1812.448234] [<ffffffff907518b5>] ? napi_gro_receive+0xc5/0xf0
> [ 1812.454844] [<ffffffff90750a68>] __netif_receive_skb+0x18/0x60
> [ 1812.461552] [<ffffffff90751b99>] process_backlog+0x89/0x140
> [ 1812.467966] [<ffffffff907511ac>] net_rx_action+0x10c/0x360
> [ 1812.474287] [<ffffffffc0305bac>] ? i40e_msix_clean_rings+0x2c/0x50 [i40e]
> [ 1812.482063] [<ffffffff9086eb0f>] __do_softirq+0xdf/0x2bb
> [ 1812.488187] [<ffffffff90087816>] irq_exit+0xb6/0xc0
> [ 1812.493815] [<ffffffff9086dc3b>] do_IRQ+0x8b/0xd0
> [ 1812.499250] [<ffffffff9086b956>] common_interrupt+0x96/0x96
> [ 1812.505662] <EOI> [ 1812.507879] [<ffffffff906f38e2>] ?
> cpuidle_enter_state+0xa2/0x2e0
> [ 1812.514971] [<ffffffff906f38d2>] ? cpuidle_enter_state+0x92/0x2e0
> [ 1812.521969] [<ffffffff906f3b57>] cpuidle_enter+0x17/0x20
> [ 1812.528092] [<ffffffff900c6563>] call_cpuidle+0x23/0x40
> [ 1812.534120] [<ffffffff900c67d0>] cpu_startup_entry+0x150/0x230
> [ 1812.540827] [<ffffffff90050280>] start_secondary+0x160/0x190
> [ 1812.547339] ---[ end trace 218aa8a38802887b ]---
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev



More information about the dev mailing list