[ovs-dev] [PATCH] datapath: Fix deadlock on STT device destroy.

[Pravin Shelar]

On Mon, Jan 11, 2016 at 6:53 PM, Joe Stringer <joe at ovn.org> wrote:
> On 11 January 2016 at 16:23, Pravin B Shelar <pshelar at nicira.com> wrote:
>> STT unregisters nf-hook when there are no other STT devices
>> left in the namespace. On some kernel versions the nf-unreg API
>> take RTNL lock, but it is already taken in the tunnel device
>> destroy code path which results in deadlock. To fix the issue
>> I moved the unreg call into net-exit.
>>
>> Bug: #1582410
>> Reported-by: Joe Stringer <joe at ovn.org>
>> Signed-off-by: Pravin B Shelar <pshelar at nicira.com>
>
> Running the kernel module testsuite, I'm getting some complaints and
> crashes from 25 - conntrack - IPv6 FTP test. On one ubuntu kernel-3.13
> host the test failed, I attempted to remove the kernel module and
> found this in dmesg:
>
> [2796049.160967] BUG: unable to handle kernel NULL pointer dereference
> at 0000000000000008
> [2796049.162370] IP: [<ffffffff81656ff7>] nf_unregister_hook+0x27/0x80
> [2796049.163450] PGD 0
> [2796049.163872] Oops: 0002 [#1] SMP
> [2796049.164512] Modules linked in: openvswitch(OX-) nf_defrag_ipv6
> vxlan gre ip_tunnel nf_defrag_ipv4 nf_conntrack_netlink nfnetlink
> nf_conntrack 8021q garp stp mrp llc veth btrfs raid6_pq xor ufs msdos
> xfs libcrc32c netconsole configfs dm_crypt ppdev vmw_balloon vmw_vmci
> parport_pc parport vmxnet3 vmw_pvscsi floppy [last unloaded:
> nf_conntrack_ipv4]
> [2796049.170608] CPU: 0 PID: 22897 Comm: rmmod Tainted: G           OX
> 3.13.0-68-generic #111-Ubuntu
> [2796049.172063] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
> [2796049.173813] task: ffff880003204800 ti: ffff880005df0000 task.ti:
> ffff880005df0000
> [2796049.175052] RIP: 0010:[<ffffffff81656ff7>]  [<ffffffff81656ff7>]
> nf_unregister_hook+0x27/0x80
> [2796049.176495] RSP: 0018:ffff880005df1eb8  EFLAGS: 00010246
> [2796049.177391] RAX: 0000000000000000 RBX: ffffffffa031c3c0 RCX:
> 00000000c0000100
> [2796049.178632] RDX: 0000000000000000 RSI: ffff880003204800 RDI:
> ffffffff81cde560
> [2796049.179963] RBP: ffff880005df1ec0 R08: ffff880005df0000 R09:
> 0000000000000000
> [2796049.181258] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffffffffa031c520
> [2796049.182565] R13: 0000000000000800 R14: 0000000000000000 R15:
> 00007f5a25e141a0
> [2796049.183872] FS:  00007f5a24d7b740(0000) GS:ffff88003f600000(0000)
> knlGS:0000000000000000
> [2796049.185321] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [2796049.186349] CR2: 0000000000000008 CR3: 000000003b971000 CR4:
> 00000000000007f0
> [2796049.187707] Stack:
> [2796049.188150]  0000000000000000 ffff880005df1ed0 ffffffffa030f852
> ffff880005df1ee0
> [2796049.189865]  ffffffffa0302b5e ffff880005df1ef0 ffffffffa02fa520
> ffff880005df1f78
> [2796049.191346]  ffffffff810e08d2 00000000c87dacc0 ffffffffa031c520
> 0000000000000800
> [2796049.192829] Call Trace:
> [2796049.193356]  [<ffffffffa030f852>]
> ovs_stt_cleanup_module+0x22/0x40 [openvswitch]
> [2796049.194697]  [<ffffffffa0302b5e>] ovs_vport_exit+0xe/0x40 [openvswitch]
> [2796049.195900]  [<ffffffffa02fa520>] dp_cleanup+0x60/0x90 [openvswitch]
> [2796049.197035]  [<ffffffff810e08d2>] SyS_delete_module+0x162/0x200
> [2796049.198098]  [<ffffffff81013ed7>] ? do_notify_resume+0x97/0xb0
> [2796049.199145]  [<ffffffff81734cdd>] system_call_fastpath+0x1a/0x1f
> [2796049.200213] Code: 1f 44 00 00 66 66 66 66 90 55 48 89 e5 53 48 89
> fb 48 c7 c7 60 e5 cd 81 e8 07 39 0d 00 48 8b 43 08 48 8b 13 48 c7 c7
> 60 e5 cd 81 <48> 89 42 08 48 89 10 48 b8 00 02 20 00 00 00 ad de 48 89
> 43 08
> [2796049.205628] RIP  [<ffffffff81656ff7>] nf_unregister_hook+0x27/0x80
> [2796049.206755]  RSP <ffff880005df1eb8>
> [2796049.207421] CR2: 0000000000000008
> [2796049.208366] ---[ end trace 62f941e5b9590b3e ]---
>
> The module wasn't removed; subsequent attempts to remove it ended with
> errors like this:
>

Thanks for the bug report. I posted patch to fix this issue.