[ovs-discuss] the raft_is_connected state of a raft server stays as false and cannot recover

Yun Zhou yunz at nvidia.com
Thu Aug 20 23:34:20 UTC 2020


Han,

I just find out that we are using the ovs directly build from upstream branch-2.13. It seems this branch does not have the following commit:

# git log -p -1 cdae6100f8
commit cdae6100f89d04c5c29dc86a490b936a204622b7
Author: Han Zhou <hzhou at ovn.org<mailto:hzhou at ovn.org>>
Date:   Thu Mar 5 23:48:46 2020 -0800

  raft: Unset leader when starting election.

From my read of the code, lack of this commit could cause a missing raft_set_leader() call therefore candidate_retrying could stay as true

Please let me know if my understand is correct. If so, the problem should be fixed in upstream/master branch.

Sorry for the confusion about the version of the ovs.

Thanks
Yun




From: Han Zhou <zhouhan at gmail.com>
Sent: Sunday, August 16, 2020 10:14 PM
To: Yun Zhou <yunz at nvidia.com>
Cc: ovs-discuss at openvswitch.org; ovn-kubernetes at googlegroups.com; Girish Moodalbail <gmoodalbail at nvidia.com>
Subject: Re: the raft_is_connected state of a raft server stays as false and cannot recover

External email: Use caution opening links or attachments



On Thu, Aug 13, 2020 at 5:26 PM Yun Zhou <yunz at nvidia.com<mailto:yunz at nvidia.com>> wrote:
Hi,

Need expert's view to address a problem we are seeing now and then:  A ovsdb-server node in a 3-nodes raft cluster keeps printing out the "raft_is_connected: false" message, and its "connected" state in its _Server DB stays as false.

According to the ovsdb-server(5) manpage, it means this server is not contacting with a majority of its cluster.

Except its "connected" state, from what we can see, this server is in the follower state and works fine, and connection between it and the other two servers appear healthy as well.

Below is its raft structure snapshot at the time of the problem. Note that its candidate_retrying field stays as true.

Hopefully the provide information can help to figure out what goes wrong here. Unfortunately we don't have a solid case to reproduce it:

Thanks for reporting the issue. This looks really strange. In the below state, leader_sid is non-zero, but candidate_retrying is true.
According to the latest code, whenever leader_sid is set to non-zero (in raft_set_leader()), candidate_retrying will be set to false; whenever candidate_retrying is set to true (in raft_start_election()), leader_sid will be set to UUID_ZERO. And the data struct is initialized with xzalloc, making sure candidate_retrying is false in the beginning. So, sorry that I can't explain how it ends up with this conflict situation. It would be helpful if there is a way to reproduce. How often does it happen?

Thanks,
Han


(gdb) print *(struct raft *)0xa872c0
$19 = {
  hmap_node = {
    hash = 2911123117,
    next = 0x0
  },
  log = 0xa83690,
  cid = {
    parts = {2699238234, 2258650653, 3035282424, 813064186}
  },
  sid = {
    parts = {1071328836, 400573240, 2626104521, 1746414343}
  },
  local_address = 0xa874e0 "tcp:10.8.51.55:6643<http://10.8.51.55:6643>",
  local_nickname = 0xa876d0 "3fdb",
  name = 0xa876b0 "OVN_Northbound",
  servers = {
    buckets = 0xad4bc0,
    one = 0x0,
    mask = 3,
    n = 3
  },
  election_timer = 1000,
  election_timer_new = 0,
  term = 3,
  vote = {
    parts = {1071328836, 400573240, 2626104521, 1746414343}
  },
  synced_term = 3,
  synced_vote = {
    parts = {1071328836, 400573240, 2626104521, 1746414343}
  },
  entries = 0xbf0fe0,
  log_start = 2,
  log_end = 312,
  log_synced = 311,
  allocated_log = 512,
  snap = {
    term = 1,
    data = 0xaafb10,
    eid = {
      parts = {1838862864, 1569866528, 2969429118, 3021055395}
    },
    servers = 0xaafa70,
    election_timer = 1000
  },
  role = RAFT_FOLLOWER,
  commit_index = 311,
  last_applied = 311,
  leader_sid = {
    parts = {642765114, 43797788, 2533161504, 3088745929}
  },
  election_base = 6043283367,
  election_timeout = 6043284593,
  joining = false,
  remote_addresses = {
    map = {
      buckets = 0xa87410,
      one = 0xa879c0,
      mask = 0,
      n = 1
    }
  },
  join_timeout = 6037634820,
  leaving = false,
  left = false,
  leave_timeout = 0,
  failed = false,
  waiters = {
    prev = 0xa87448,
    next = 0xa87448
  },
  listener = 0xaafad0,
  listen_backoff = -9223372036854775808,
  conns = {
    prev = 0xbcd660,
    next = 0xaafc20
  },
  add_servers = {
    buckets = 0xa87480,
    one = 0x0,
    mask = 0,
    n = 0
  },
  remove_server = 0x0,
  commands = {
    buckets = 0xa874a8,
    one = 0x0,
    mask = 0,
    n = 0
  },
  ping_timeout = 6043283700,
  n_votes = 1,
  candidate_retrying = true,
  had_leader = false,
  ever_had_leader = true
}

Thanks
- Yun

--
You received this message because you are subscribed to the Google Groups "ovn-kubernetes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ovn-kubernetes+unsubscribe at googlegroups.com<mailto:ovn-kubernetes%2Bunsubscribe at googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/ovn-kubernetes/BY5PR12MB4132F190E4BFE9F381BC5A82B0400%40BY5PR12MB4132.namprd12.prod.outlook.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200820/7a3a9efc/attachment-0001.html>


More information about the discuss mailing list