[ovs-discuss] kernel BUG at /build/buildd/linux-3.13.0/net/core/skbuff.c:1043 (invalid opcode) (#81)

[Jesse Gross]

On Mon, Jun 1, 2015 at 8:35 AM, Ben Pfaff <blp at nicira.com> wrote:
> This was reported on github.  I'm forwarding it here to make sure it
> gets noticed.
>
> There is some additional information in a followup on github.
>
> ----- Forwarded message from George Shuklin <notifications at github.com> -----
>
> Date: Mon, 01 Jun 2015 00:12:51 -0700
> From: George Shuklin <notifications at github.com>
> To: openvswitch/ovs-issues <ovs-issues at noreply.github.com>
> Subject: [ovs-issues] kernel BUG at
>         /build/buildd/linux-3.13.0/net/core/skbuff.c:1043 (invalid opcode)
>         (#81)
>
> One of our nodes starts to trace aperiodically. We're gathing logs via remote syslog, here trace (it is a bit unordered because of UDP in syslog).
>
> Configuration:
> openvswitch-switch                  2.3.1
> Linux: 3.13.0-52-generic
>
> We using: neutron with vlans, sFlow (inside OVS) on 'external' bridge.

Here is the trace reordered by timestamp (I haven't done any further
analysis on it):

[12803.874856] ------------[ cut here ]------------
[12803.874870] kernel BUG at /build/buildd/linux-3.13.0/net/core/skbuff.c:1043!
[12803.874889] invalid opcode: 0000 [#1] SMP
[12803.874916] Modules linked in: ip6table_nat nf_conntrack_ipv6
nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw iptable_raw
8021q garp mrp vhost_net vhost macvtap macvlan dm_multipath scsi_dh
nbd kvm_intel ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT
xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc xfs
ip6table_filter ip6_tables iptable_filter ip_tables x_tables
openvswitch(OX) gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich
dcdbas x86_pkg_temp_thermal intel_powerclamp coretemp kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd lpc_ich ipmi_si
mei_me mei wmi shpchp acpi_power_meter mac_hid squashfs aufs igb
i2c_algo_bit ixgbe e1000e tg3 dca ptp ahci pps_core libahci
megaraid_sas mdio [last unloaded: kvm_intel]
[12803.875430] CPU: 0 PID: 17654 Comm: vhost-17644 Tainted: G
 OX 3.13.0-53-generic #89-Ubuntu
[12803.875451] Hardware name: Dell Inc. PowerEdge R530/03XKDV, BIOS
1.1.10 03/10/2015
[12803.875473] task: ffff88301f389800 ti: ffff88300fcce000 task.ti:
ffff88300fcce000
[12803.875495] RIP: 0010:[<ffffffff8161853e>]  [<ffffffff8161853e>]
pskb_expand_head+0x20e/0x250
[12803.875548] RSP: 0018:ffff881fff403a90  EFLAGS: 00010202
[12803.875587] RAX: 0000000000000002 RBX: ffff8800720fa100 RCX: 0000000000000020
[12803.875630] RDX: 00000000000006aa RSI: 0000000000000000 RDI: ffff8800720fa100
[12803.875673] RBP: ffff881fff403ac8 R08: 00000000e0400000 R09: ffff881fff403bd0
[12803.875715] R10: 0000000000000010 R11: ffff881f31d29d40 R12: 0000000000000000
[12803.875758] R13: ffff8800720fa100 R14: ffff881f5c0e6cfc R15: 00000000ffffffff
[12803.875802] FS:  0000000000000000(0000) GS:ffff881fff400000(0000)
knlGS:0000000000000000
[12803.875868] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12803.875907] CR2: 00007eff8c072400 CR3: 0000001f86331000 CR4: 00000000001427f0
[12803.875950] Stack:
[12803.875980]  ffff8800720fa100 ffff881fff403ac0 ffff8800720fa100
ffff881fb6e58200
[12803.876060]  00000000000005b4 ffff881f5c0e6cfc 00000000ffffffff
ffff881fff403b10
[12803.876140]  ffffffff81618ef0 ffffffffa01a41c0 000005b4cd428000
ffff8800720fa100
[12803.876219] Call Trace:
[12803.876251]  <IRQ>
[12803.876258]  [<ffffffff81618ef0>] __pskb_pull_tail+0x50/0x360
[12803.876331]  [<ffffffff816252d3>] skb_checksum_help+0x123/0x190
[12803.876374]  [<ffffffffa0156a2e>]
queue_userspace_packet+0x3ae/0x400 [openvswitch]
[12803.876441]  [<ffffffffa0157fe5>] ovs_dp_upcall+0x65/0x70 [openvswitch]
[12803.876484]  [<ffffffffa0154796>] do_execute_actions+0x376/0xbf0
[openvswitch]
[12803.876551]  [<ffffffffa015fe0b>] ? netdev_send+0x4b/0xc0 [openvswitch]
[12803.876594]  [<ffffffffa01547cc>] do_execute_actions+0x3ac/0xbf0
[openvswitch]
[12803.876661]  [<ffffffffa015d01a>] ? flow_lookup.isra.4+0x4a/0xc0
[openvswitch]
[12803.876727]  [<ffffffffa0155077>] ovs_execute_actions+0x67/0x100
[openvswitch]
[12803.876793]  [<ffffffffa015806f>]
ovs_dp_process_packet_with_key+0x7f/0xf0 [openvswitch]
[12803.876861]  [<ffffffffa0158c98>] ? key_extract+0x6f8/0xa20 [openvswitch]
[12803.876904]  [<ffffffffa015812b>]
ovs_dp_process_received_packet+0x4b/0x80 [openvswitch]
[12803.876973]  [<ffffffffa015e4da>] ovs_vport_receive+0x2a/0x30 [openvswitch]
[12803.877017]  [<ffffffffa01600e1>] netdev_frame_hook+0xc1/0x120 [openvswitch]
[12803.877061]  [<ffffffff81626892>] __netif_receive_skb_core+0x262/0x840
[12803.877103]  [<ffffffff81626e88>] __netif_receive_skb+0x18/0x60
[12803.877145]  [<ffffffff81627a1e>] process_backlog+0xae/0x1a0
[12803.877185]  [<ffffffff81627272>] net_rx_action+0x152/0x250
[12803.877226]  [<ffffffff8106cc6c>] __do_softirq+0xec/0x2c0
[12803.877268]  [<ffffffff8173531c>] do_softirq_own_stack+0x1c/0x30
[12803.877307]  <EOI>
[12803.877314]  [<ffffffff8106cee5>] do_softirq+0x55/0x60
[12803.877383]  [<ffffffff816263b8>] netif_rx_ni+0x28/0x30
[12803.877425]  [<ffffffff81527dc3>] tun_get_user+0x403/0x840
[12803.877464]  [<ffffffff8152825a>] tun_sendmsg+0x5a/0x80
[12803.877504]  [<ffffffffa02ecd99>] handle_tx+0x1b9/0x4d0 [vhost_net]
[12803.877546]  [<ffffffffa02ed0e5>] handle_tx_kick+0x15/0x20 [vhost_net]
[12803.877589]  [<ffffffffa02dedb1>] vhost_worker+0x101/0x1f0 [vhost]
[12803.877632]  [<ffffffffa02decb0>] ? vhost_dev_reset_owner+0x30/0x30 [vhost]
[12803.877676]  [<ffffffff8108b6b2>] kthread+0xd2/0xf0
[12803.877714]  [<ffffffff8108b5e0>] ? kthread_create_on_node+0x1c0/0x1c0
[12803.877757]  [<ffffffff81733868>] ret_from_fork+0x58/0x90
[12803.877797]  [<ffffffff8108b5e0>] ? kthread_create_on_node+0x1c0/0x1c0
[12803.877838] Code: ff ff ff 48 89 d7 48 89 55 c8 e8 ae 4f b4 ff 84
c0 48 8b 55 c8 75 a1 eb 9b 0f 1f 40 00 0f 0b 41 81 cf 00 20 00 00 e9
43 fe ff ff <0f> 0b 44 89 fe 4c 89 ef e8 e5 f4 ff ff 85 c0 74 12 48 89
df e8
[12803.878203] RIP  [<ffffffff8161853e>] pskb_expand_head+0x20e/0x250
[12803.878247]  RSP <ffff881fff403a90>
[12803.878792] ---[ end trace 4e8d5a14aaae59db ]---