[ovs-git] [openvswitch/ovs] b26788: tun_metadata: Fix coredump caused by use-after-fre...

[yifsun]

  Branch: refs/heads/branch-2.13
  Home:   https://github.com/openvswitch/ovs
  Commit: b267889057c47503f4c4f41fff7c268a4bec3ed7
      https://github.com/openvswitch/ovs/commit/b267889057c47503f4c4f41fff7c268a4bec3ed7
  Author: Yifeng Sun <pkusunyifeng at gmail.com>
  Date:   2020-04-10 (Fri, 10 Apr 2020)

  Changed paths:
    M ofproto/ofproto-dpif-rid.h
    M ofproto/ofproto-dpif-upcall.c
    M ofproto/ofproto-dpif-xlate.c

  Log Message:
  -----------
  tun_metadata: Fix coredump caused by use-after-free bug

Tun_metadata can be referened by flow and frozen_state at the same
time. When ovs-vswitchd handles TLV table mod message, the involved
tun_metadata gets freed. The call trace to free tun_metadata is
shown as below:

ofproto_run
- handle_openflow
  - handle_single_part_openflow
    - handle_tlv_table_mod
      - tun_metadata_table_mod
        - tun_metadata_postpone_free

Unfortunately, this tun_metadata can be still used by some frozen_state,
and later on when frozen_state tries to access its tun_metadata table,
ovs-vswitchd crashes. The call trace to access tun_metadata from
frozen_state is shown as below:

udpif_upcall_handler
- recv_upcalls
  - process_upcall
    - frozen_metadata_to_flow

It is unsafe for frozen_state to reference tun_table because tun_table
is protected by RCU while the lifecycle of frozen_state can span several
RCU quiesce states. Current code violates OVS's RCU protection mechanism.

This patch fixes it by simply stopping frozen_state from referencing
tun_table. If frozen_state needs tun_table, the latest valid tun_table
can be found through ofproto_get_tun_tab() efficiently.

A previous commit seems fixing the samiliar issue:
254878c18874f6 (ofproto-dpif-xlate: Fix segmentation fault caused by tun_table)

VMware-BZ: #2526222
Signed-off-by: Yifeng Sun <pkusunyifeng at gmail.com>
Signed-off-by: William Tu <u9012063 at gmail.com>


  Commit: a09e22dbdd01797138bf8bc2bb0a0874dd752980
      https://github.com/openvswitch/ovs/commit/a09e22dbdd01797138bf8bc2bb0a0874dd752980
  Author: Yifeng Sun <pkusunyifeng at gmail.com>
  Date:   2020-04-10 (Fri, 10 Apr 2020)

  Changed paths:
    M tests/system-traffic.at

  Log Message:
  -----------
  system-traffic: Check frozen state handling with TLV map change

This patch enhances a system traffic test to prevent regression on
the tunnel metadata table (tun_table) handling with frozen state.
Without a proper fix this test can crash ovs-vswitchd due to a
use-after-free bug on tun_table.

These are the timed sequence of how this bug is triggered:

- Adds an OpenFlow rule in OVS that matches Geneve tunnel metadata that
contains a controller action.
- When the first packet matches the aforementioned OpenFlow rule,
during the miss upcall, OVS stores a pointer to the tun_table (that
decodes the Geneve tunnel metadata) in a frozen state and pushes down
a datapath flow into kernel datapath.
- Issues a add-tlv-map command to reprogram the tun_table on OVS.
OVS frees the old tun_table and create a new tun_table.
- A subsequent packet hits the kernel datapath flow again. Since
there is a controller action associated with that flow, it triggers
slow path controller upcall.
- In the slow path controller upcall, OVS derives the tun_table
from the frozen state, which points to the old tun_table that is
already being freed at this time point.
- In order to access the tunnel metadata, OVS uses the invalid
pointer that points to the old tun_table and triggers the core dump.

Signed-off-by: Yi-Hung Wei <yihung.wei at gmail.com>
Signed-off-by: Yifeng Sun <pkusunyifeng at gmail.com>
Co-authored-by: Yi-Hung Wei <yihung.wei at gmail.com>
Signed-off-by: William Tu <u9012063 at gmail.com>


Compare: https://github.com/openvswitch/ovs/compare/d323c5398fad...a09e22dbdd01