[ovs-discuss] OVS not forwarding packets to port

[Chris Picton]

Hi Gurusharan

Thanks for the extremely helpful tip

I did as you suggested, and ran a strace on the kvm pid:
  # strace  -e trace=select -p 3740
Process 3740 attached - interrupt to quit
select(44, [7 11 14 15 17 19 20 40 41 42 43], [], [], {0, 584192}) = 0 
(Timeout)
select(44, [7 11 14 15 17 19 20 40 41 42 43], [], [], {0, 999000}) = 1 
(in [19], left {0, 665094})
select(44, [7 11 14 15 17 19 20 40 41 42 43], [], [], {0, 665000}) = 2 
(in [7 17], left {0, 664997})
...

The FD of the vnet0 interface is not in the readfds fd_set (it should 
include 39)  This indicates why the network device is not pulling any 
traffic.

Also, even though the network interface is not registering any drops or 
errors, the tc queue attached to it does show drops (not sure the 
reasons for that)
  # tc -s qdisc show dev vnet0
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 
1 1 1 1 1
  Sent 359863 bytes 1983 pkt (dropped 2804865, overlimits 0 requeues 2)
  rate 0bit 0pps backlog 82148b 500p requeues 2

Commits like this:
http://git.qemu.org/?p=qemu.git;a=commitdiff;h=68e5ec64009812dbaa03ed9cfded9344986f5304 

possibly fix the problem at the qemu side.

So even though the fault is not in openvswitch, in this process, I have 
learned a lot more about how ovs and flows work - thanks :)

Regards
Chris

On 2015/03/18 05:27 PM, Gurucharan Shetty wrote:
> Fwiw, I had seen a similar issue a couple of years ago. It turned out
> to be a system bug. I debugged (with help from Ben) like this:
> ps -ef | grep kvm : Choose the process id of the kvm
> ls -ltr /proc/$pid/fd
>
> This will show you the fd of /dev/net/tun
> So the KVM process will be listening on that fd for all the packets
> that comes from host and is headed to VM.
>
> "strace -o logfile -p $Pid" for a few seconds.
>
> If you open the logfile, you will see a select system call on that fd
> and then read system calls on that fd.
>
> In my case, there was nothing being read. I don't remember very well
> what exactly was the problem. The fix was to use a different network
> device .
>
> On Wed, Mar 18, 2015 at 8:14 AM, Chris Picton <chris at picton.nom.za> wrote:
>> On 03/18/15 16:52, Ben Pfaff wrote:
>>> On Wed, Mar 18, 2015 at 12:44:08PM +0200, Chris Picton wrote:
>>>> On 03/18/15 08:47, Chris Picton wrote:
>>>>> The relevant flow is:
>>>>>
>>>>> recirc_id(0),skb_priority(0),in_port(2),eth(src=00:15:17:91:05:4c,dst=52:54:00:14:54:c6),eth_type(0x8100),vlan(vid=3,pcp=0),encap(eth_type(0x0806)),
>>>>> packets:2, bytes:120, used:0.014s, actions:pop_vlan,7
>>>>>
>>>>> I can't find any definitive documentation for an action element
>>>>> consisting
>>>>> purely of a number - in this case "7".
>>>> I have just re-read the man page, and found the following:
>>>> "The  target  may  be  a  decimal  port  number  designating the physical
>>>> port on which to  output the packet."
>>>>
>>>> so the packet should, according to the above, be forwarded to port 7,
>>>> which
>>>> is vnet0.
>>>>
>>>> However, the packet is not arriving on vnet0.  Any ideas where I can look
>>>> to
>>>> find out why not?
>>> I'd start by checking the kernel log (with "dmesg").
>>
>> Hi Ben
>>
>> I had already done system level debugging to try find out where issues may
>> lie.  There are no kernel logs indicating dropped packets or transmit
>> failures.
>>
>> A diff between the contents of /sys/class/net/vnet0 (which is the port that
>> gets no traffic) and /sys/class/net/veth-carbon1-3/ (which is the port that
>> gets traffic) shows very few differences at the kernel level (rx/tx
>> counts/address/etc differ, but type/features/carrier/flags/etc are the same)
>>
>> What I know so far is that while openvswitch should be forwarding packets
>> onto vnet0 (which is what the flow indicates), vnet0 does not receive those
>> packets (as indicated with tcpdump).  Also the tx_packets counter for vnet0
>> does not increment (as it should when packets get sent to the interface and
>> transmitted on)
>>
>> Is this a kernel/openvswitch bug?  I can probably get the interface working
>> by migrating the VM to a different host and migrating it back, but I would
>> rather leave it in a non working state to try and pinpoint the issue.
>>
>> Regards
>> Chris
>>
>>
>>
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> http://openvswitch.org/mailman/listinfo/discuss