[ovs-discuss] open vSwitch port input buffer capacity

[Ali Volkan Atli]

Hi 

@Ben, the proposal is from (https://mailman.stanford.edu/pipermail/openflow-discuss/2010-June/thread.html#1024). Also I don't think it violets the Openflow specs. Could you please give us more information?

@Aaron, actually your design completely works for me and also better than my pre-design for performance in kernel mode OvS because there will not any context-switching between kernel and userspace unnecessarily but I thought that if I change pktbuf_save before packet_in message leave I could handle the buffering issue for kernel and userspace (DPDK) together.

- Volkan
________________________________________
From: Aaron Conole [aconole at redhat.com]
Sent: Friday, April 15, 2016 9:32 PM
To: discuss at openvswitch.org; Ben Pfaff; Ali Volkan Atli
Subject: Re: [ovs-discuss] open vSwitch port input buffer capacity

Hi Ben,

Ben Pfaff <blp at ovn.org> writes:
> On Fri, Apr 15, 2016 at 08:08:02AM +0000, Ali Volkan Atli wrote:
> > I would like to take this opportunity to ask a question about
> > buffering. I've been trying to figure the buffering issue out for a
> > couple of days. I tested it using ryu-controller and OvS produced
> > ~1700 packet-in messages (and also ~1700 packet-out and ~1700
> > flow-mod) for one udp-flow on 1G (packet size is 1024). I think I
> > should change pktbuf_save functions with a hash (like microflow but
> > not in cache) for 5-tuple key (src_ip, dst_ip, src_port, dst_port and
> > protocol) and also add a linked-list for each flow. It will be
> > suitable for your proposal below which discussed before in a different
> > thread.
>
> What proposal did I make?  Where?
>
> > Also I should add a parameter into vswitch.xml to enable/disable
> > buffering. Does my pre-design make sense?
> >
> > ******STARTS *****
> >
> > Here's a new copy of the change that I proposed to Open vSwitch:
> >
> > - When a packet misses in the flow table, check for a buffered packet whose
> > flow is identical.
> >
> > * If there is one, just append the new packet to a linked list attached to
> > that buffered packet. Don't send a packet-in.
> >
> > * If there is none, make a new packet buffer and send a packet-in to the
> > controller.
> >
> > - If the controller sends a flow-add for a buffered packet, apply it to the
> > buffered packet itself and to every packet on the linked list.
> >
> > - If the controller sends a packet-out for a buffered packet, send it. Then
> > if there's a list attached to the buffered packet, send the first packet
> > in the list to the controller as a packet-in.
> >
> > - If a buffered packet times out, send the first packet in the list to the
> > controller as a packet-in.
>
> It's a nice idea but it violates the OpenFlow spec.

Just want to understand the difference between the design that is
discussed here, and the work I've started doing to try and address
an implementation detail with OvS kernel -> userspace communication.

In my case, I have the following:

1. A per-flow 'pending-upcall' list in datapath which queues SKBs
2. A new netlink message which tells kernel to flush a specific list
   by flow key
3. An option to that netlink message that indicates the flush should
   invoke the dp_execute for the skbs

So, this case, A) skb arrives in the kernel datapath, B) doesn't find
an entry in the pending upcall table, and misses in the flow table,
so, it C) adds a new pending-upcall entry in the pending upcall table
and delivers the skb to the vswitchd (which does any of the openflow
controller communication, etc.). D) Meanwhile, additional skbs which
arrive in the kernel datapth and hash to the same flow key will be
queued to the pending upcall table. E) Vswitchd finally installs the
flow in the flow table, and F) sends the netlink message to flush and
remove the pending upcall table.

Basically the (simplified) receive call-flow moves from:

[skb] --> ovs_vport_receive()
             --> ovs_dp_process_packet()
                    --> if (!ovs_flow_tbl_lookup_stats())
                           --> ovs_dp_upcall()
                                  --> tx to userspace
                    --> else
                           --> ovs_execute_actions()


to:

[skb] --> ovs_vport_receive()
             --> if (!ovs_dp_pending_upcall())
                    --> ovs_dp_process_packet()
                           --> if (!ovs_flow_tbl_lookup_stats())
                                  --> ovs_dp_upcall()
                                         --> find or create pending upcall
                                         --> tx to userspace
                           --> else
                                  --> ovs_execute_actions()
             --> else
                    --> enqueue to pending upcall

Then when OVS choses to either install / deny the flow, it sends the
netlink message which clears the queue.

The actual code handles a lot of corner cases that exist, but I want to
make sure that the general outline I have above works to solve the
_implentation_ detail of kernel/userspace upcalls in ovs in an
acceptable manner and doesn't violate openflow.

-Aaron