[ovs-dev] 答复: [PATCH] pkt reassemble: fix kernel panic for ovs reassemble

Tue Jun 27 04:05:48 UTC 2017

Hi Joe, Greg,

I tried to create a pull request, please check whether it works. Thanks.

https://github.com/openvswitch/ovs/pull/187

Br,
Wang Zhike
-----Original Message-----
From: Joe Stringer [mailto:joe at ovn.org] 
Sent: Saturday, June 24, 2017 5:15 AM
To: 王志克
Cc: dev at openvswitch.org
Subject: Re: 答复: [ovs-dev] [PATCH] pkt reassemble: fix kernel panic for ovs reassemble

Hi Wang Zhike,

I'd like if others like Greg could take a look as well, since this
code is delicate. The more review it gets, the better. It seems like
maybe the version of your email that goes to the list does not get the
attachment. Perhaps you could try sending the patch using git
send-email or putting the patch on GitHub instead, and linking to it
here.

For what it's worth, I did run your patch for a while and it seemed
OK, but when I tried again today on an Ubuntu Trusty (Linux
3.13.0-119-generic) box, running make check-kmod, I saw an issue with
get_next_timer_interrupt():

[181250.892557] BUG: unable to handle kernel paging request at ffffffffa03317e0
[181250.892557] IP: [<ffffffff81079606>] get_next_timer_interrupt+0x86/0x250
[181250.892557] PGD 1c11067 PUD 1c12063 PMD 1381a2067 PTE 0
[181250.892557] Oops: 0000 [#1] SMP
[181250.892557] Modules linked in: nf_nat_ipv6 nf_nat_ipv4 nf_nat
gre(-) nf_conntrack_ipv6 nf_conntrack_ipv4 nf_defrag_ipv6
nf_defrag_ipv4 nf_conntrack_netlink nfnetlink nf_conntrack bonding
8021q garp stp mrp llc veth nfsd auth_rpcgss nfs_acl nfs lockd sunrpc
fscache dm_crypt kvm_intel kvm serio_raw netconsole configfs
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy
ahci libahci [last unloaded: libcrc32c]
[181250.892557] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OX
3.13.0-119-generic #166-Ubuntu
[181250.892557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[181250.892557] task: ffffffff81c15480 ti: ffffffff81c00000 task.ti:
ffffffff81c00000
[181250.892557] RIP: 0010:[<ffffffff81079606>]  [<ffffffff81079606>]
get_next_timer_interrupt+0x86/0x250
[181250.892557] RSP: 0018:ffffffff81c01e00  EFLAGS: 00010002
[181250.892557] RAX: ffffffffa03317c8 RBX: 0000000102b245da RCX:
00000000000000db
[181250.892557] RDX: ffffffff81ebac58 RSI: 00000000000000db RDI:
0000000102b245db
[181250.892557] RBP: ffffffff81c01e48 R08: 0000000000c88c1c R09:
0000000000000000
[181250.892557] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000142b245d9
[181250.892557] R13: ffffffff81eb9e80 R14: 0000000102b245da R15:
0000000000cd63e8
[181250.892557] FS:  0000000000000000(0000) GS:ffff88013fc00000(0000)
knlGS:0000000000000000
[181250.892557] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[181250.892557] CR2: ffffffffa03317e0 CR3: 000000003707f000 CR4:
00000000000006f0
[181250.892557] Stack:
[181250.892557]  0000000000000000 ffffffff81c01e30 ffffffff810a3af5
ffff88013fc13bc0
[181250.892557]  ffff88013fc0dce0 0000000102b245da 0000000000000000
00000063ae154000
[181250.892557]  0000000000cd63e8 ffffffff81c01ea8 ffffffff810da655
0000a4d8c2cb6200
[181250.892557] Call Trace:
[181250.892557]  [<ffffffff810a3af5>] ? set_next_entity+0x95/0xb0
[181250.892557]  [<ffffffff810da655>] tick_nohz_stop_sched_tick+0x1e5/0x340
[181250.892557]  [<ffffffff810da851>] __tick_nohz_idle_enter+0xa1/0x160
[181250.892557]  [<ffffffff810dab4d>] tick_nohz_idle_enter+0x3d/0x70
[181250.892557]  [<ffffffff810c2af7>] cpu_startup_entry+0x87/0x2b0
[181250.892557]  [<ffffffff8171b387>] rest_init+0x77/0x80
[181250.892557]  [<ffffffff81d34f6a>] start_kernel+0x432/0x43d
[181250.892557]  [<ffffffff81d34941>] ? repair_env_string+0x5c/0x5c
[181250.892557]  [<ffffffff81d34120>] ? early_idt_handler_array+0x120/0x120
[181250.892557]  [<ffffffff81d345ee>] x86_64_start_reservations+0x2a/0x2c
[181250.892557]  [<ffffffff81d34733>] x86_64_start_kernel+0x143/0x152
[181250.892557] Code: 8b 7d 10 4d 8b 75 18 4c 39 f7 78 5c 40 0f b6 cf
89 ce 48 63 c6 48 c1 e0 04 49 8d 54 05 00 48 8b 42 28 48 83 c2 28 48
39 d0 74 0e <f6> 40 18 01 74 24 48 8b 00 48 39 d0 75 f2 83 c6 01 40 0f
b6 f6
[181250.892557] RIP  [<ffffffff81079606>] get_next_timer_interrupt+0x86/0x250
[181250.892557]  RSP <ffffffff81c01e00>
[181250.892557] CR2: ffffffffa03317e0

It seems like perhaps a fragment timer signed up by OVS is still
remaining when the OVS module is unloaded, so it may attempt to clean
up an entry using OVS code but the OVS code has been unloaded at that
point. This might be related to IPv6 cvlan test - that seems to be
where my VM froze and went to 100% CPU, but I would think that the
IPv6 fragmentation cleanup test is a more likely to cause this, since
it leaves fragments behind in the cache after the test finishes. I've
only hit this when running all of the tests in make check-kmod.

Cheers,
Joe

On 22 June 2017 at 17:53, 王志克 <wangzhike at jd.com> wrote:
> Hi Joe,
>
> Please check the attachment. Thanks.
>
> Br,
> Wang Zhike
>
> -----邮件原件-----
> 发件人: Joe Stringer [mailto:joe at ovn.org]
> 发送时间: 2017年6月23日 8:20
> 收件人: 王志克
> 抄送: dev at openvswitch.org
> 主题: Re: [ovs-dev] [PATCH] pkt reassemble: fix kernel panic for ovs reassemble
>
> On 21 June 2017 at 18:54, 王志克 <wangzhike at jd.com> wrote:
>> Ovs and kernel stack would add frag_queue to same netns_frags list.
>> As result, ovs and kernel may access the fraq_queue without correct
>> lock. Also the struct ipq may be different on kernel(older than 4.3),
>> which leads to invalid pointer access.
>>
>> The fix creates specific netns_frags for ovs.
>>
>> Signed-off-by: wangzhike <wangzhike at jd.com>
>> ---
>
> Hi,
>
> It looks like the whitespace has been corrupted in this version of the patch that you sent, I cannot apply it. Probably your email client mistreats it when sending the email out. A reliable method to send patches correctly via email is to use the commandline client 'git send-email'. This is the preferred method. If you are unable to set that up, consider attaching the patch to the email (or send a pull request on GitHub).
>
> Cheers,
> Joe