[ovs-discuss] Is it possible to limit bandwidth for particular OVS port?

Gurucharan Shetty shettyg at nicira.com
Fri Mar 27 14:46:29 UTC 2015


On Thu, Mar 26, 2015 at 6:35 PM, Tim Bagr <timbgr at gmail.com> wrote:
> Thanks for your reply! It helps a lot. I didn't use tc before. I just
> watched at large ping packets and if they are delayed progressively - then I
> concluded QoS is actually working.
>
> I tried again with patch-port and you're right.
> # tc class show dev b2p
> had shown nothing.
>
> So I tried to do the trick with LOCAL interface again, which is the same
> name as name of bridge, in my case "br3".
>
> # ovs-vsctl set port br3 qos=@nqos -- --id=@nqos create qos type=linux-htb
> other-config:max-rate=100000 queues=111=@nq -- --id=@nq create queue
> other-config:max-rate=13000
>
> And then, tried to alter max-rate for QoS object:
> # ovs-vsctl set qos cbba20c1-9afb-4092-ad37-628d70ce2139
> other-config:max-rate=100000
> # ovs-vsctl set qos cbba20c1-9afb-4092-ad37-628d70ce2139
> other-config:max-rate=10000
> # ovs-vsctl set qos cbba20c1-9afb-4092-ad37-628d70ce2139
> other-config:max-rate=1000000
>
> And after each command I've sent pings from VM to outside network, and see
> if they are limited. The values actually worked - when I set rate 10000, for
> example, then pings from VM looked like:
> [root at VM ~]# ping -i 0.1 outside.host -s 500
> PING outside.host (10.1.1.168) 500(528) bytes of data.
> 508 bytes from outside.host (10.1.1.168): icmp_seq=1 ttl=63 time=0.242 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=2 ttl=63 time=0.270 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=3 ttl=63 time=0.224 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=4 ttl=63 time=67.7 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=5 ttl=63 time=85.7 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=6 ttl=63 time=75.3 ms
> ^C508 bytes from 10.1.1.168: icmp_seq=7 ttl=63 time=84.8 ms
>
> And when I set rate about max-rate=100000000 then pings went flawlessly:
>
> [root at localhost ~]# ping -i 0.1 outside.host -s 500
> PING outside.host (10.1.1.168) 500(528) bytes of data.
> 508 bytes from outside.host (10.1.1.168): icmp_seq=1 ttl=63 time=0.270 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=2 ttl=63 time=0.201 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=3 ttl=63 time=0.256 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=4 ttl=63 time=0.249 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=5 ttl=63 time=0.203 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=6 ttl=63 time=0.224 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=7 ttl=63 time=0.216 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=8 ttl=63 time=0.217 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=9 ttl=63 time=0.226 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=10 ttl=63 time=0.204 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=11 ttl=63 time=0.227 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=12 ttl=63 time=0.220 ms
> 508 bytes from outside.host (10.1.1.168): icmp_seq=13 ttl=63 time=0.260 ms
>
>
>
> Then I set openflow rules to assign queue number to specific traffic (with
> src and dst address equal to that otside.host, which I send ping requests
> to):
>
> The rules are matched and n_packets grown when I sent pings:
> # ovs-ofctl dump-flows br3 --sort=priority
>  cookie=0x0, duration=155295.034s, table=0, n_packets=12692,
> n_bytes=6188205, priority=0 actions=NORMAL
>  cookie=0x0, duration=482.039s, table=0, n_packets=112, n_bytes=116704,
> priority=15,ip,nw_src=10.1.1.168 actions=set_queue:111,NORMAL
>  cookie=0x0, duration=531.879s, table=0, n_packets=259, n_bytes=269878,
> priority=15,ip,nw_dst=10.1.1.168 actions=set_queue:111,NORMAL
>  cookie=0x0, duration=420.512s, table=0, n_packets=640, n_bytes=751480,
> priority=20,ip,nw_src=10.1.1.168 actions=enqueue:6:111
>  cookie=0x0, duration=347.242s, table=0, n_packets=573, n_bytes=681666,
> priority=20,ip,nw_dst=10.1.1.168 actions=enqueue:LOCAL:111
>
> But tc command shown that queue (as I suppose it's 1:70, is not active
> against that traffic):
> # tc -s class show dev br3
> class htb 1:1 parent 1:fffe prio 0 rate 12000bit ceil 1000Kbit burst 1563b
> cburst 1564b
>  Sent 92820 bytes 128 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
>  lended: 8 borrowed: 120 giants: 0
>  tokens: 15854166 ctokens: 190250
When you do the above, ping, you can look at the actual kernel
datapath flows with a 'ovs-dpctl dump-flows'. You will see a flow of
the following form (my ip addresses are different):

recirc_id(0),skb_priority(0),in_port(1),eth(src=00:0c:29:d6:c0:2e,dst=00:0c:29:23:ee:7f),eth_type(0x0800),ipv4(src=192.168.1.1/255.255.255.255,dst=192.168.1.2/0.0.0.0,proto=1/0,tos=0/0,ttl=64/0,frag=no/0xff),
packets:5, bytes:490, used:0.212s,
actions:set(skb_priority(0x10070)),2

The set(skb_priority(0x10070)),2 is the actual marking of packets to
go to queue 70 (decimal 111, I think).
So my take is that the packet has been marked and sent to output port
2 (in my case eth1). But since the queue is only in br3, I think it
never gets hit.

This is unchartered territory for me. My reading is that QoS can only
be applied on the egress port.





>
> class htb 1:fffe root rate 1000Kbit ceil 1000Kbit burst 1500b cburst 1500b
>  Sent 92820 bytes 128 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
>  lended: 120 borrowed: 0 giants: 0
>  tokens: 182250 ctokens: 182250
>
> class htb 1:70 parent 1:fffe prio 0 rate 12000bit ceil 13000bit burst 1563b
> cburst 1563b
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
>  lended: 0 borrowed: 0 giants: 0
>  tokens: 16291666 ctokens: 15038461
>
>
> Maybe it's some undiscovered bug? Or it works so just by design?
>
>
>
>
>
>
>
> On Thu, Mar 26, 2015 at 7:44 PM, Gurucharan Shetty <shettyg at nicira.com>
> wrote:
>>
>> On Wed, Mar 25, 2015 at 7:03 PM, Tim Bagr <timbgr at gmail.com> wrote:
>> > Hello Gurucharan,
>> >
>> > Thanks for the reply.
>> >
>> > I have no physical ports attached to that OVS bridge. My intention was
>> > exactly to limit bandwidth for br3 port (i.e. openflow keyword LOCAL),
>> > and
>> > another try was to limit bandwidth for internal patch-port, which
>> > connects
>> > br3 bridge with br2 bridge. In both bridges there are VMs attached and
>> > they
>> > may communicate with each other through the patch-port.
>> >
>> > # ovs-vsctl show
>> > b11e83cb-d741-4a59-90f7-ea9693d508cf
>> >     Bridge "br2"
>> >         Port "b3p"
>> >             Interface "b3p"
>> >                 type: patch
>> >                 options: {peer="b2p"}
>> >         Port "vnet1"
>> >             Interface "vnet1"
>> >         Port "br2"
>> >             Interface "br2"
>> >                 type: internal
>> >     Bridge "br3"
>> >         Port "br3"
>> >             Interface "br3"
>> >                 type: internal
>> >         Port "vnet0"
>> >             Interface "vnet0"
>> >         Port "b2p"
>> >             Interface "b2p"
>> >                 type: patch
>> >                 options: {peer="b3p"}
>> >
>> > Here vnet1 is port of 1st VM and vnet0 is port of 2nd VM. Now I just
>> > want to
>> > limit bandwidth from VM2 to VM1.
>> > So I run:
>> > # ovs-vsctl set Port b2p qos=@newq -- --id=@newq create qos
>> > type=linux-htb
>> > other-config:max-rate=100000000 queues=111=@q1 -- --id=@q1 create queue
>> > other-config:min-rate=0 other-config:max-rate=10
>> > 65c5488d-2066-4d97-b21f-ba369a8b2920
>> > 1e3726b4-12e2-4184-b8e4-13f9c692095f
>> I _think_(not 100% sure) that you cannot apply QoS on a patch port.
>> Patch port has no netdevice backing it. OVS uses Linux tc to apply
>> QoS. Since tc cannot see the OVS patch port, I don't think anything
>> will come out of it.
>>
>> The way to verify that QoS is really working is to look at tc stats
>> for that queue. e.g: If you had applied QoS on eth1 with 3 queues: 1,
>> 2, 3, you will get:
>>
>> sudo tc class show dev eth1
>> class htb 1:fffe root rate 900000Kbit ceil 900000Kbit burst 1462b cburst
>> 1462b
>> class htb 1:1 parent 1:fffe prio 0 rate 720000Kbit ceil 900000Kbit
>> burst 1530b cburst 1462b
>> class htb 1:2 parent 1:fffe prio 0 rate 12000bit ceil 90000Kbit burst
>> 1563b cburst 1563b
>> class htb 1:3 parent 1:fffe prio 0 rate 12000bit ceil 90000Kbit burst
>> 1563b cburst 1563b
>>
>> For statistics: (to see how many packets went through etc)
>> tc -s class show dev eth1
>>
>>
>>
>> >
>> > # ovs-ofctl show br3
>> > OFPT_FEATURES_REPLY (xid=0x2): dpid:00007a42d1ca7e05
>> > n_tables:254, n_buffers:256
>> > capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
>> > actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC
>> > SET_DL_DST
>> > SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
>> >  3(b2p): addr:ae:a4:9d:8b:9f:a9
>> >      config:     0
>> >      state:      0
>> >      speed: 0 Mbps now, 0 Mbps max
>> >  6(vnet0): addr:fe:54:00:ac:e1:a6
>> >      config:     0
>> >      state:      0
>> >      current:    10MB-FD COPPER
>> >      speed: 10 Mbps now, 0 Mbps max
>> >  LOCAL(br3): addr:7a:42:d1:ca:7e:05
>> >      config:     0
>> >      state:      0
>> >      speed: 0 Mbps now, 0 Mbps max
>> > OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
>> >
>> > # ovs-ofctl add-flow br3 priority=5,in_port=6,actions=enqueue:3:111
>> > # ovs-ofctl dump-flows br3
>> > NXST_FLOW reply (xid=0x4):
>> >  cookie=0x0, duration=3.285s, table=0, n_packets=0, n_bytes=0,
>> > idle_age=3,
>> > priority=5,in_port=6 actions=enqueue:3:111
>> >  cookie=0x0, duration=79191.532s, table=0, n_packets=744,
>> > n_bytes=431875,
>> > idle_age=7, hard_age=65534, priority=0 actions=NORMAL
>> >
>> > Then I start to ping6 from VM2 to VM1 and pings will go through that
>> > patch-ports b3p and b2p. As I expected, the QoS queue will be in effect.
>> > And
>> > limit the bandwidth to very small amount of max-rate=10
>> How do you know that QOS is in effect above? I guess based on the
>> n_packets of openflow flow? That only tells that OVS marked the packet
>> to go to a QoS queue. But if QoS queue has not been created in Linux
>> kernel, there is really nothing that will happen.
>>
>> I imagine that OVS should show some error in the logs, but I did not
>> see any (during my small test). That is the reason I am not 100% sure
>> what will actually  happen.
>
>



More information about the discuss mailing list