[ovs-discuss] OVS DPDK|DPDK|ERR|interface dpdk0 start error:operation not permitted.

sothy shan sothy.e98 at gmail.com
Thu Jul 28 07:43:39 UTC 2016


On Thu, Jul 28, 2016 at 2:29 AM, Yong Wang <yongwang at vmware.com> wrote:
>
>> On Jul 27, 2016, at 12:25 AM, sothy shan <sothy.e98 at gmail.com> wrote:
>>
>> On Tue, Jul 26, 2016 at 6:32 PM, Mauricio Vasquez
>> <mauricio.vasquez at polito.it> wrote:
>>> Hello Sothy,
>>>
>>> Sorry for the delay in answering, some busy days here.
>>>
>>>
>>> On 07/25/2016 12:03 PM, sothy shan wrote:
>>>>
>>>> Hello Mauricio,
>>>>
>>>> On Sun, Jul 24, 2016 at 12:17 PM, Mauricio Vasquez
>>>> <mauricio.vasquez at polito.it> wrote:
>>>>>
>>>>> Hi Sothy,
>>>>>
>>>>>
>>>>> On 07/21/2016 11:55 AM, sothy shan wrote:
>>>>>>
>>>>>> On Thu, Jul 21, 2016 at 11:43 AM, Mauricio Vasquez
>>>>>> <mauricio.vasquez at polito.it> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 07/21/2016 10:51 AM, sothy shan wrote:
>>>>>>>>
>>>>>>>> On Thu, Jul 21, 2016 at 10:41 AM, sothy shan <sothy.e98 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On Wed, Jul 20, 2016 at 9:02 PM, Mauricio Vasquez
>>>>>>>>> <mauricio.vasquez at polito.it> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Sothy,
>>>>>>>>>>
>>>>>>>>>> On 07/20/2016 07:40 PM, Aaron Conole wrote:
>>>>>>>>>>>
>>>>>>>>>>> sothy shan <sothy.e98 at gmail.com> writes:
>>>>>>>>>>>
>>>>>>>>>>>> Hello all,
>>>>>>>>>>>
>>>>>>>>>>> Hi Sothy,
>>>>>>>>>>>
>>>>>>>>>>>> I followed the installion guide provided in the web site
>>>>>>>>>>>>
>>>>>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openvswitch_ovs_blob_master_INSTALL.DPDK.md-23ovstc&d=CwIBaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=44mSO5N5yEs4CeCdtQE0xt0F7J0p67_mApYVAzyYms0&m=vpaHcL64W3vOTY9P8yRZOdCJzbL_oWzZH6J2u4iT614&s=ojiXK7Tj1ANsHE_AlchVROP4R7LrgLQqVqTx7BjJQas&e=
>>>>>>>>>>>> to install dpdk OVS.
>>>>>>>>>>>>
>>>>>>>>>>>> I am rurnning Ubuntu 14.04 (kernel 3.13). It is a VM running
>>>>>>>>>>>> VMware.
>>>>>>>>>>>> One intereface is enabled by VMXNET3.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It is not clear for me, are you trying to install OvS within the VM?
>>>>>>>>>>
>>>>>>>>>>>> I used dpdk 16.04
>>>>>>>>>>>> Today I made clone of OVS (committ
>>>>>>>>>>>> :bf32e3e2c029da18b5d6fd9210cb0ea12a1d0383).
>>>>>>>>>>>>
>>>>>>>>>>>> The following are different configuration compared to above link.
>>>>>>>>>>>>
>>>>>>>>>>>> sysctl -w vm.nr_hugepages=2048
>>>>>>>>>>>> Binded interface via igb_uio.
>>>>>>>>>>>>
>>>>>>>>>>>> Then followed same configuration to start ovs-vsswitch with dpdk.
>>>>>>>>>>>
>>>>>>>>>>> Can you please post the steps exactly that you followed?  Which
>>>>>>>>>>> user
>>>>>>>>>>> you
>>>>>>>>>>> were running as, etc?  Can you also post how you expect packets to
>>>>>>>>>>> arrive into the vm?
>>>>>>>>>>>
>>>>>>>>>>>> I am able to creatre bridge. When I create a port. It gives error
>>>>>>>>>>>> message.
>>>>>>>>>>>
>>>>>>>>>>> Please show the steps for this, too.  The exact commands you used.
>>>>>>>>>>>
>>>>>>>>>>>> When I check ovs-vswitchd.log file,
>>>>>>>>>>>>
>>>>>>>>>>>> There is warning that dpif_netlink|WARN|genric netlink family,
>>>>>>>>>>>> ovs_datapath does not exist. The openvswitch kernel module may be
>>>>>>>>>>>> not
>>>>>>>>>>>> loaded.
>>>>>>>>>>>
>>>>>>>>>>> This is definitely a problem.  How did you start the ovs-vswitchd
>>>>>>>>>>> daemon?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is not a problem Aaron, the ovs kernel module is not needed
>>>>>>>>>> when
>>>>>>>>>> ovs is
>>>>>>>>>> compiled with DPDK.
>>>>>>>>>>>>
>>>>>>>>>>>> Then ERR: ERR|interface dpdk0 start error:operation not permitted.
>>>>>>>>>>>>
>>>>>>>>>>>> The corresponding port is loaded by dpdk driver as shown in
>>>>>>>>>>>> dpdk_nic_bind --status.
>>>>>>>>>>>>
>>>>>>>>>>>> After that I did sudo ovs-vsctl show. it show br and port details
>>>>>>>>>>>> as
>>>>>>>>>>>> well.
>>>>>>>>>>>> I am able to add one flow rule as well. But ovs bridge didnt get
>>>>>>>>>>>> packets from the interface.
>>>>>>>>>>>>
>>>>>>>>>>>> Do you know why this error is popup? anyway to solve the problem?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you very much
>>>>>>>>>>>>
>>>>>>>>>>>> Best regars
>>>>>>>>>>>> Sothy
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> discuss mailing list
>>>>>>>>>>> discuss at openvswitch.org
>>>>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__openvswitch.org_mailman_listinfo_discuss&d=CwIBaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=44mSO5N5yEs4CeCdtQE0xt0F7J0p67_mApYVAzyYms0&m=vpaHcL64W3vOTY9P8yRZOdCJzbL_oWzZH6J2u4iT614&s=DZpyuDSP-5BYH4bvfNHk68XRFB9-M-M43FxZCCpZKho&e=
>>>>>>>>>>
>>>>>>>>>> As Aaron wrote, having the exact commands is important to help you.
>>>>>>>>>
>>>>>>>>> My configuration and log info:
>>>>>>>>>
>>>>>>>>> sudo sysctl vm.nr_hugepages=2048
>>>>>>>>> sudo mkdir -p /dev/hugepages
>>>>>>>>> sudo  mount -t hugetlbfs hugetlbfs /dev/hugepages
>>>>>>>>> sudo modprobe uio
>>>>>>>>> sudo insmod $DPDK_BUILD/kmod/igb_uio.ko
>>>>>>>>> sudo $DPDK_DIR/tools/dpdk_nic_bind.py --status
>>>>>>>>> sudo $DPDK_DIR/tools/dpdk_nic_bind.py -b igb_uio 0000:0b:00.0
>>>>>>>>> sudo ./dpdk-16.04/tools/dpdk_nic_bind.py --status
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Network devices using DPDK-compatible driver
>>>>>>>>> ============================================
>>>>>>>>> 0000:0b:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=
>>>>>>>>> 0000:13:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=
>>>>>>>>>
>>>>>>>>> Network devices using kernel driver
>>>>>>>>> ===================================
>>>>>>>>> 0000:03:00.0 'VMXNET3 Ethernet Controller' if=eth0 drv=vmxnet3
>>>>>>>>> unused=igb_uio *Active*
>>>>>>>>>
>>>>>>>>> Other network devices
>>>>>>>>> =====================
>>>>>>>>> <none>
>>>>>>>>>
>>>>>>>>> .........Then......................
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> sudo ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
>>>>>>>>> sudo ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
>>>>>>>
>>>>>>> These two commands are inverted.
>>>>>>> The bridge has to be created first and then ports have to be added to
>>>>>>> it.
>>>>>>>
>>>>>> It is my mistake when coping.
>>>>>>>>>
>>>>>>>>> Log file/
>>>>>>>>> 2016-07-21T07:53:35.527Z|00001|vlog|INFO|opened log file
>>>>>>>>> /var/log/openvswitch/ovs-vswitchd.log
>>>>>>>>> 2016-07-21T07:53:35.529Z|00002|ovs_numa|INFO|Discovered 2 CPU cores
>>>>>>>>> on
>>>>>>>>> NUMA node 0
>>>>>>>>> 2016-07-21T07:53:35.529Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes
>>>>>>>>> and 2 CPU cores
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2016-07-21T07:53:35.529Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>>>>>>> connecting...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2016-07-21T07:53:35.529Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>>>>>>> connected
>>>>>>>>> 2016-07-21T07:53:35.533Z|00006|dpdk|INFO|DPDK Enabled, initializing
>>>>>>>>> 2016-07-21T07:53:35.533Z|00007|dpdk|INFO|No vhost-sock-dir provided -
>>>>>>>>> defaulting to /usr/local/var/run/openvswitch
>>>>>>>>> 2016-07-21T07:53:35.533Z|00008|dpdk|INFO|EAL ARGS: ovs-vswitchd
>>>>>>>>> --socket-mem 1024,0 -c 0x00000001
>>>>>>>>> 2016-07-21T07:53:36.606Z|00009|timeval|WARN|Unreasonably long 1076ms
>>>>>>>>> poll interval (80ms user, 875ms system)
>>>>>>>>> 2016-07-21T07:53:36.606Z|00010|timeval|WARN|faults: 3306 minor, 0
>>>>>>>>> major
>>>>>>>>> 2016-07-21T07:53:36.606Z|00011|timeval|WARN|context switches: 3
>>>>>>>>> voluntary, 101 involuntary
>>>>>>>>> 2016-07-21T07:53:36.606Z|00012|coverage|INFO|Event coverage, avg rate
>>>>>>>>> over last: 5 seconds, last minute, last hour,  hash=398361de:
>>>>>>>>> 2016-07-21T07:53:36.606Z|00013|coverage|INFO|bridge_reconfigure
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>>>>> 2016-07-21T07:53:36.606Z|00014|coverage|INFO|cmap_expand
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 9
>>>>>>>>> 2016-07-21T07:53:36.606Z|00015|coverage|INFO|miniflow_malloc
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 9
>>>>>>>>> 2016-07-21T07:53:36.606Z|00016|coverage|INFO|hmap_pathological
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>>>>> 2016-07-21T07:53:36.606Z|00017|coverage|INFO|hmap_expand
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 640
>>>>>>>>> 2016-07-21T07:53:36.606Z|00018|coverage|INFO|txn_unchanged
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 2
>>>>>>>>> 2016-07-21T07:53:36.606Z|00019|coverage|INFO|txn_incomplete
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00020|coverage|INFO|poll_create_node
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 35
>>>>>>>>> 2016-07-21T07:53:36.607Z|00021|coverage|INFO|seq_change
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 38
>>>>>>>>> 2016-07-21T07:53:36.607Z|00022|coverage|INFO|pstream_open
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00023|coverage|INFO|stream_open
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00024|coverage|INFO|util_xalloc
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 11049
>>>>>>>>> 2016-07-21T07:53:36.607Z|00025|coverage|INFO|netdev_get_hwaddr
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00026|coverage|INFO|netlink_received
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 3
>>>>>>>>> 2016-07-21T07:53:36.607Z|00027|coverage|INFO|netlink_sent
>>>>>>>>>    0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00028|coverage|INFO|86 events never hit
>>>>>>>>> 2016-07-21T07:53:36.607Z|00029|bridge|INFO|ovs-vswitchd (Open
>>>>>>>>> vSwitch)
>>>>>>>>> 2.5.90
>>>>>>>>> 2016-07-21T07:54:50.861Z|00030|memory|INFO|18980 kB peak resident set
>>>>>>>>> size after 75.3 seconds
>>>>>>>>> 2016-07-21T07:54:50.866Z|00031|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> Datapath supports recirculation
>>>>>>>>> 2016-07-21T07:54:50.866Z|00032|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> MPLS label stack length probed as 3
>>>>>>>>> 2016-07-21T07:54:50.866Z|00033|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> Datapath supports truncate action
>>>>>>>>> 2016-07-21T07:54:50.866Z|00034|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> Datapath supports unique flow ids
>>>>>>>>> 2016-07-21T07:54:50.866Z|00035|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> Datapath does not support ct_state
>>>>>>>>> 2016-07-21T07:54:50.866Z|00036|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> Datapath does not support ct_zone
>>>>>>>>> 2016-07-21T07:54:50.866Z|00037|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> Datapath does not support ct_mark
>>>>>>>>> 2016-07-21T07:54:50.866Z|00038|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> Datapath does not support ct_label
>>>>>>>>> 2016-07-21T07:54:50.866Z|00039|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>>>>> Datapath does not support ct_state_nat
>>>>>>>>> 2016-07-21T07:54:50.875Z|00040|bridge|INFO|bridge br0: added
>>>>>>>>> interface
>>>>>>>>> br0 on port 65534
>>>>>>>>> 2016-07-21T07:54:50.887Z|00041|dpif_netlink|WARN|Generic Netlink
>>>>>>>>> family 'ovs_datapath' does not exist. The Open vSwitch kernel module
>>>>>>>>> is probably not loaded.
>>>>>>>>> 2016-07-21T07:54:50.887Z|00042|bridge|INFO|bridge br0: using datapath
>>>>>>>>> ID 0000c664fbd86348
>>>>>>>>> 2016-07-21T07:54:50.887Z|00043|connmgr|INFO|br0: added service
>>>>>>>>> controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
>>>>>>>>> 2016-07-21T07:58:18.796Z|00044|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>>>>>>> 2016-07-21T07:58:18.810Z|00045|dpdk|ERR|Interface dpdk0 start error:
>>>>>>>>> Operation not permitted
>>>>>>>>> 2016-07-21T07:58:18.810Z|00046|dpif|WARN|netdev at ovs-netdev: failed to
>>>>>>>>> add dpdk0 as port: Operation not permitted
>>>>>>>>> 2016-07-21T08:10:35.440Z|00047|connmgr|INFO|br0<->unix: 1 flow_mods
>>>>>>>>> in
>>>>>>>>> the last 0 s (1 adds)
>>>>>>>
>>>>>>> Could you also provide the full terminal output when you run
>>>>>>> ovs-vswitchd?,
>>>>>>> I would like to see the DPDK output
>>>>>>
>>>>>> vrouter at vrouter-virtual-machine:~/ovs$ sudo ovs-vswitchd unix:$DB_SOCK
>>>>>> --pidfile --detach --log-file=/var/log/openvswitch/ovs-vswitchd.log
>>>>>> 2016-07-21T09:33:35Z|00001|vlog|INFO|opened log file
>>>>>> /var/log/openvswitch/ovs-vswitchd.log
>>>>>> 2016-07-21T09:33:35Z|00002|ovs_numa|INFO|Discovered 2 CPU cores on NUMA
>>>>>> node 0
>>>>>> 2016-07-21T09:33:35Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 2
>>>>>> CPU
>>>>>> cores
>>>>>>
>>>>>>
>>>>>> 2016-07-21T09:33:35Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>>>> connecting...
>>>>>>
>>>>>>
>>>>>> 2016-07-21T09:33:35Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>>>> connected
>>>>>> 2016-07-21T09:33:35Z|00006|dpdk|INFO|DPDK Enabled, initializing
>>>>>> 2016-07-21T09:33:35Z|00007|dpdk|INFO|No vhost-sock-dir provided -
>>>>>> defaulting to /usr/local/var/run/openvswitch
>>>>>> 2016-07-21T09:33:35Z|00008|dpdk|INFO|EAL ARGS: ovs-vswitchd
>>>>>> --socket-mem 1024,0 -c 0x00000001
>>>>>> EAL: Detected lcore 0 as core 0 on socket 0
>>>>>> EAL: Detected lcore 1 as core 0 on socket 0
>>>>>> EAL: Support maximum 128 logical core(s) by configuration.
>>>>>> EAL: Detected 2 lcore(s)
>>>>>> EAL: Probing VFIO support...
>>>>>> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
>>>>>> directory)
>>>>>> EAL: VFIO modules not loaded, skipping VFIO support...
>>>>>> EAL: Setting up physically contiguous memory...
>>>>>> EAL: Ask a virtual area of 0x5e400000 bytes
>>>>>> EAL: Virtual area found at 0x7fa618400000 (size = 0x5e400000)
>>>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>>>> EAL: Virtual area found at 0x7fa618000000 (size = 0x200000)
>>>>>> EAL: Ask a virtual area of 0xa1800000 bytes
>>>>>> EAL: Virtual area found at 0x7fa576600000 (size = 0xa1800000)
>>>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>>>> EAL: Virtual area found at 0x7fa576200000 (size = 0x200000)
>>>>>> EAL: Requesting 512 pages of size 2MB from socket 0
>>>>>> EAL: TSC frequency is ~2799213 KHz
>>>>>> EAL: Master lcore 0 is ready (tid=78ad7b80;cpuset=[0])
>>>>>> EAL: PCI device 0000:03:00.0 on NUMA socket -1
>>>>>> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>>>> EAL:   Not managed by a supported kernel driver, skipped
>>>>>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>>>>>> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>>>> EAL:   PCI memory mapped at 0x7fa658400000
>>>>>> EAL:   PCI memory mapped at 0x7fa658401000
>>>>>> EAL:   PCI memory mapped at 0x7fa658402000
>>>>>> EAL: PCI device 0000:13:00.0 on NUMA socket -1
>>>>>> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>>>> EAL:   PCI memory mapped at 0x7fa658404000
>>>>>> EAL:   PCI memory mapped at 0x7fa658405000
>>>>>> EAL:   PCI memory mapped at 0x7fa658406000
>>>>>> Zone 0: name:<RG_MP_log_history>, phys:0x82bfdec0, len:0x2080,
>>>>>> virt:0x7fa6583fdec0, socket_id:0, flags:0
>>>>>> Zone 1: name:<MP_log_history>, phys:0x82973d40, len:0x28a0c0,
>>>>>> virt:0x7fa658173d40, socket_id:0, flags:0
>>>>>> Zone 2: name:<rte_eth_dev_data>, phys:0x82943200, len:0x2f800,
>>>>>> virt:0x7fa658143200, socket_id:0, flags:0
>>>>>> 2016-07-21T09:33:36Z|00009|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>> Datapath supports recirculation
>>>>>> 2016-07-21T09:33:36Z|00010|ofproto_dpif|INFO|netdev at ovs-netdev: MPLS
>>>>>> label stack length probed as 3
>>>>>> 2016-07-21T09:33:36Z|00011|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>> Datapath supports truncate action
>>>>>> 2016-07-21T09:33:36Z|00012|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>> Datapath supports unique flow ids
>>>>>> 2016-07-21T09:33:36Z|00013|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>> Datapath does not support ct_state
>>>>>> 2016-07-21T09:33:36Z|00014|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>> Datapath does not support ct_zone
>>>>>> 2016-07-21T09:33:36Z|00015|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>> Datapath does not support ct_mark
>>>>>> 2016-07-21T09:33:36Z|00016|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>> Datapath does not support ct_label
>>>>>> 2016-07-21T09:33:36Z|00017|ofproto_dpif|INFO|netdev at ovs-netdev:
>>>>>> Datapath does not support ct_state_nat
>>>>>> 2016-07-21T09:33:36Z|00018|bridge|INFO|bridge br0: added interface br0
>>>>>> on port 65534
>>>>>> 2016-07-21T09:33:36Z|00019|dpif_netlink|WARN|Generic Netlink family
>>>>>> 'ovs_datapath' does not exist. The Open vSwitch kernel module is
>>>>>> probably not loaded.
>>>>>> 2016-07-21T09:33:36Z|00020|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>>>> 2016-07-21T09:33:36Z|00021|dpdk|ERR|Interface dpdk0 start error:
>>>>>> Operation not permitted
>>>>>> 2016-07-21T09:33:36Z|00022|bridge|INFO|bridge br0: using datapath ID
>>>>>> 0000c664fbd86348
>>>>>> 2016-07-21T09:33:36Z|00023|connmgr|INFO|br0: added service controller
>>>>>> "punix:/usr/local/var/run/openvswitch/br0.mgmt"
>>>>>> 2016-07-21T09:33:36Z|00024|timeval|WARN|Unreasonably long 1244ms poll
>>>>>> interval (233ms user, 880ms system)
>>>>>> 2016-07-21T09:33:36Z|00025|timeval|WARN|faults: 3902 minor, 0 major
>>>>>> 2016-07-21T09:33:36Z|00026|timeval|WARN|context switches: 10
>>>>>> voluntary, 214 involuntary
>>>>>> 2016-07-21T09:33:36Z|00027|coverage|INFO|Event coverage, avg rate over
>>>>>> last: 5 seconds, last minute, last hour,  hash=75cfd2fc:
>>>>>> 2016-07-21T09:33:36Z|00028|coverage|INFO|bridge_reconfigure
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00029|coverage|INFO|ofproto_flush
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00030|coverage|INFO|ofproto_update_port
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00031|coverage|INFO|rev_flow_table
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00032|coverage|INFO|cmap_expand
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 22
>>>>>> 2016-07-21T09:33:36Z|00033|coverage|INFO|cmap_shrink
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 7
>>>>>> 2016-07-21T09:33:36Z|00034|coverage|INFO|dpif_port_add
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 2
>>>>>> 2016-07-21T09:33:36Z|00035|coverage|INFO|dpif_flow_flush
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 2
>>>>>> 2016-07-21T09:33:36Z|00036|coverage|INFO|dpif_flow_get
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 5
>>>>>> 2016-07-21T09:33:36Z|00037|coverage|INFO|dpif_flow_put
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 10
>>>>>> 2016-07-21T09:33:36Z|00038|coverage|INFO|dpif_flow_del
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 5
>>>>>> 2016-07-21T09:33:36Z|00039|coverage|INFO|dpif_execute
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 3
>>>>>> 2016-07-21T09:33:36Z|00040|coverage|INFO|flow_extract
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 4
>>>>>> 2016-07-21T09:33:36Z|00041|coverage|INFO|miniflow_malloc
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 16
>>>>>> 2016-07-21T09:33:36Z|00042|coverage|INFO|hmap_pathological
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00043|coverage|INFO|hmap_expand
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 653
>>>>>> 2016-07-21T09:33:36Z|00044|coverage|INFO|netdev_get_stats
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00045|coverage|INFO|poll_create_node
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 15
>>>>>> 2016-07-21T09:33:36Z|00046|coverage|INFO|poll_zero_timeout
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00047|coverage|INFO|seq_change
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 575
>>>>>> 2016-07-21T09:33:36Z|00048|coverage|INFO|pstream_open
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 3
>>>>>> 2016-07-21T09:33:36Z|00049|coverage|INFO|stream_open
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00050|coverage|INFO|util_xalloc
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 11820
>>>>>> 2016-07-21T09:33:36Z|00051|coverage|INFO|netdev_set_policing
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00052|coverage|INFO|netdev_get_ifindex
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00053|coverage|INFO|netdev_get_hwaddr
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 2
>>>>>> 2016-07-21T09:33:36Z|00054|coverage|INFO|netdev_set_hwaddr
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00055|coverage|INFO|netdev_get_ethtool
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 2
>>>>>> 2016-07-21T09:33:36Z|00056|coverage|INFO|netlink_received
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 6
>>>>>> 2016-07-21T09:33:36Z|00057|coverage|INFO|netlink_recv_jumbo
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 1
>>>>>> 2016-07-21T09:33:36Z|00058|coverage|INFO|netlink_sent
>>>>>> 0.0/sec     0.000/sec        0.0000/sec   total: 5
>>>>>> 2016-07-21T09:33:36Z|00059|coverage|INFO|70 events never hit
>>>>>> 2016-07-21T09:33:36Z|00060|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>>>> 2016-07-21T09:33:36Z|00061|dpdk|ERR|Interface dpdk0 start error:
>>>>>> Operation not permitted
>>>>>
>>>>> The PMD start function is failing, unfortunately that error does not tell
>>>>> that much information.
>>>>>
>>>>> I would suggest you to enable the debug of the vmxnet3 PMD in DPDK, set
>>>>> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y in the config/common_base file of
>>>>> DPDK, then recompile DPDK and OvS.
>>>>>
>>>>> Please try once again and send the terminal output, I hope we can get a
>>>>> more
>>>>> information of what is going on with that log.
>>>>
>>>> Based on your request, I reproduce the log with
>>>> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
>>>> sudo ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock
>>>> --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile --detach
>>>> 2016-07-25T09:24:17Z|00001|vlog|INFO|opened log file
>>>> /var/log/openvswitch/ovs-vswitchd.log
>>>> 2016-07-25T09:24:17Z|00002|ovs_numa|INFO|Discovered 2 CPU cores on NUMA
>>>> node 0
>>>> 2016-07-25T09:24:17Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 2 CPU
>>>> cores
>>>>
>>>> 2016-07-25T09:24:17Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>> connecting...
>>>>
>>>> 2016-07-25T09:24:17Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>> connected
>>>> 2016-07-25T09:24:17Z|00006|dpdk|INFO|DPDK Enabled, initializing
>>>> 2016-07-25T09:24:17Z|00007|dpdk|INFO|No vhost-sock-dir provided -
>>>> defaulting to /usr/local/var/run/openvswitch
>>>> 2016-07-25T09:24:17Z|00008|dpdk|INFO|EAL ARGS: ovs-vswitchd
>>>> --socket-mem 1024,0 -c 0x00000001
>>>> EAL: Detected lcore 0 as core 0 on socket 0
>>>> EAL: Detected lcore 1 as core 0 on socket 0
>>>> EAL: Support maximum 128 logical core(s) by configuration.
>>>> EAL: Detected 2 lcore(s)
>>>> EAL: Probing VFIO support...
>>>> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
>>>> directory)
>>>> EAL: VFIO modules not loaded, skipping VFIO support...
>>>> EAL: Setting up physically contiguous memory...
>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>> EAL: Virtual area found at 0x7fdea6000000 (size = 0x200000)
>>>> EAL: Ask a virtual area of 0x5e000000 bytes
>>>> EAL: Virtual area found at 0x7fde47e00000 (size = 0x5e000000)
>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>> EAL: Virtual area found at 0x7fde47a00000 (size = 0x200000)
>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>> EAL: Virtual area found at 0x7fde47600000 (size = 0x200000)
>>>> EAL: Ask a virtual area of 0xa1800000 bytes
>>>> EAL: Virtual area found at 0x7fdda5c00000 (size = 0xa1800000)
>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>> EAL: Virtual area found at 0x7fdda5800000 (size = 0x200000)
>>>> EAL: Requesting 512 pages of size 2MB from socket 0
>>>> EAL: TSC frequency is ~2799209 KHz
>>>> EAL: Master lcore 0 is ready (tid=a855eb80;cpuset=[0])
>>>> PMD: rte_vmxnet3_pmd_init():  >>
>>>> EAL: PCI device 0000:03:00.0 on NUMA socket -1
>>>> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>> EAL:   Not managed by a supported kernel driver, skipped
>>>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>>>> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>> EAL:   PCI memory mapped at 0x7fdea6200000
>>>> EAL:   PCI memory mapped at 0x7fdea6201000
>>>> EAL:   PCI memory mapped at 0x7fdea6202000
>>>> PMD: eth_vmxnet3_dev_init():  >>
>>>> PMD: eth_vmxnet3_dev_init(): Hardware version : 1
>>>> PMD: eth_vmxnet3_dev_init(): UPT hardware version : 1
>>>> PMD: eth_vmxnet3_dev_init(): MAC Address : 00:0c:29:b5:9f:5b
>>>> EAL: PCI device 0000:13:00.0 on NUMA socket -1
>>>> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>> EAL:   Not managed by a supported kernel driver, skipped
>>>> Zone 0: name:<RG_MP_log_history>, phys:0x555fdec0, len:0x2080,
>>>> virt:0x7fdea61fdec0, socket_id:0, flags:0
>>>> Zone 1: name:<MP_log_history>, phys:0x95375f00, len:0x28a0c0,
>>>> virt:0x7fde87975f00, socket_id:0, flags:0
>>>> Zone 2: name:<rte_eth_dev_data>, phys:0x555cd300, len:0x2f800,
>>>> virt:0x7fdea61cd300, socket_id:0, flags:0
>>>>
>>>> PMD: vmxnet3_dev_configure():  >>
>>>> PMD: vmxnet3_dev_tx_queue_setup():  >>
>>>> PMD: vmxnet3_dev_rx_queue_setup():  >>
>>>> PMD: vmxnet3_dev_start():  >>
>>>> PMD: vmxnet3_rss_configure():  >>
>>>> PMD: vmxnet3_write_mac(): Writing MAC Address : 00:0c:29:b5:9f:5b
>>>> PMD: vmxnet3_disable_intr():  >>
>>>> PMD: vmxnet3_dev_rxtx_init():  >>
>>>> 2016-07-25T09:40:23Z|00020|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>> PMD: vmxnet3_dev_stop():  >>
>>>> PMD: vmxnet3_disable_intr():  >>
>>>> PMD: vmxnet3_dev_stop(): Device reset.
>>>> PMD: vmxnet3_dev_clear_queues():  >>
>>>> PMD: vmxnet3_dev_configure():  >>
>>>> PMD: vmxnet3_dev_tx_queue_setup():  >>
>>>> PMD: vmxnet3_dev_tx_queue_setup():  >>
>>>> PMD: vmxnet3_dev_tx_queue_setup():  >>
>>>> PMD: vmxnet3_dev_rx_queue_setup():  >>
>>>> PMD: vmxnet3_dev_start():  >>
>>>> PMD: vmxnet3_rss_configure():  >>
>>>> PMD: vmxnet3_write_mac(): Writing MAC Address : 00:0c:29:b5:9f:5b
>>>> PMD: vmxnet3_dev_start(): Device activation: UNSUCCESSFUL
>>>
>>>
>>> Unfortunately it is the most I can help you, it is strange for me why
>>> test-pmd is working and why OvS  is not.
>>> I think it is a DPDK issue, then the next step is to post on the DPDK
>>> mailing (dev at dpdk.org, add also in CC Yong Wang <yongwang at vmware.com> that
>>> is the vmxnet3 maintainer), send a link to this thread in order to be clear
>>> what you have tested.
>>
>>
>> Hello Yong Wang,
>>
>> I am running fedora VM in ESXi 5.5.0 environment. I tried to install
>> OVS and DPDK. OVS couldnt start DPDK device.
>> You can see the error in the mail below. Do you know why? IF you need
>> more information. I can provide more information.
>>
>> Good news, when I tried in Fedora 23 VM in ESXi 5.5.0, it works. It is
>> possible to bind an dpdk device and get packets on port.
>>
>> Thanks Mauricio for your help and reply.
>>
>> Best regards
>> Sothy
>
> Sothy,
>

Hello,
> You mentioned that on Fedora 23, it works.  Then on some version Fedora that it failed?

Initially, I started working on Ubuntu 14.04. It produce error.
Therefore, I changed in Fedora. I worked directly on Fedora 23.
For fedora, I only tried on Fedora 23. It works. I didnt tried on
other version.  All Vms are in ESX 5.5. The problem is in Ubuntu vm
ONLY,

> From your previous post, you mentioned that it’s a Ubuntu 14.04 VM?  Can you clarify?  Other than the changes you >mentioned already, are there any other changes (such as ring buffer size, number of rx queues, etc) that have been >changed?
I didnt do any changes.  I just followed basic commands given in OVDK
install.md  and
 >Is there any updates installed on your ESX 5.5?

no
>If yes, what’s the exact update version?  Can you also send me a copy of your vmkernel.log offline?

I will send you offline.
>
> Also copying Daniele as he is trying to help reproduce the reported issue.
>
>>
>>>
>>>> I deleted exisiting bridge and recreated::::::
>>>>
>>>>
>>>> Then I added bridge. After I try to create bridge. It gives error. The
>>>> erros is same as shown in log file.
>>>>
>>>> 2016-07-25T09:29:43.248Z|00020|bridge|INFO|bridge br0: added interface
>>>> br0 on port 65534
>>>> 2016-07-25T09:29:43.253Z|00021|dpif_netlink|WARN|Generic Netlink
>>>> family 'ovs_datapath' does not exist. The Open vSwitch kernel module
>>>> is probably not loaded.
>>>> 2016-07-25T09:29:43.253Z|00022|bridge|INFO|bridge br0: using datapath
>>>> ID 0000e6aac740584b
>>>> 2016-07-25T09:29:43.253Z|00023|connmgr|INFO|br0: added service
>>>> controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
>>>> 2016-07-25T09:30:54.008Z|00024|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>> 2016-07-25T09:30:54.021Z|00025|dpdk|ERR|Interface dpdk0 start error:
>>>> Operation not permitted
>>>> 2016-07-25T09:30:54.021Z|00026|dpif|WARN|netdev at ovs-netdev: failed to
>>>> add dpdk0 as port: Operation not permitted
>>>>
>>>>
>>>>
>>>>> Another thing to do would be to bind the NICs to the uio_pci_generic and
>>>>> VFIO modules, just to be sure that the issue is also present there.
>>>>
>>>> I tried in uio and uio_pci_genric also. Both are giving same error.
>>>> I suspect I cannt activate VFIO in VM since it is BIOS suport also. AM I
>>>> right?
>>>>
>>>> Thanks
>>>>
>>>>>
>>>>>> in command line some information for your verification:
>>>>>>
>>>>>>   sudo ovs-vsctl show
>>>>>> d53612e0-5d82-4cc1-af7d-ee887a737ef2
>>>>>>      Bridge "br0"
>>>>>>          Port "dpdk0"
>>>>>>              Interface "dpdk0"
>>>>>>                  type: dpdk
>>>>>>          Port "br0"
>>>>>>              Interface "br0"
>>>>>>                  type: internal
>>>>>> vrouter at vrouter-virtual-machine:~/ovs$ sudo ovs-vsctl del-br br0
>>>>>> vrouter at vrouter-virtual-machine:~/ovs$ sudo ovs-vsctl add-br br0 --
>>>>>> set bridge br0 datapath_type=netdev
>>>>>> vrouter at vrouter-virtual-machine:~/ovs$ sudo ovs-vsctl add-port br0
>>>>>> dpdk0 -- set Interface dpdk0 type=dpdk
>>>>>> ovs-vsctl: Error detected while setting up 'dpdk0'.  See ovs-vswitchd
>>>>>> log for details.
>>>>>>
>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> sudo ovs-vsctl show
>>>>>>>>> d53612e0-5d82-4cc1-af7d-ee887a737ef2
>>>>>>>>>       Bridge "br0"
>>>>>>>>>           Port "dpdk0"
>>>>>>>>>               Interface "dpdk0"
>>>>>>>>>                   type: dpdk
>>>>>>>>>           Port "br0"
>>>>>>>>>               Interface "br0"
>>>>>>>>>                   type: internal
>>>>>>>>>
>>>>>>>>> sudo ovs-ofctl dump-flows br0
>>>>>>>>> NXST_FLOW reply (xid=0x4):
>>>>>>>>>    cookie=0x0, duration=487.638s, table=0, n_packets=0, n_bytes=0,
>>>>>>>>> idle_age=487, priority=0 actions=NORMAL
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> In addition I would recommend to use the test-pmd [1] application
>>>>>>>>>> from
>>>>>>>>>> DPDK
>>>>>>>>>> to check if the issue is also present in that case.
>>>>>>>>
>>>>>>>> I run testpmd command.The result is
>>>>>>>>
>>>>>>>> Configuring Port 0 (socket 0)
>>>>>>>> Port 0: 00:0C:29:B5:9F:5B
>>>>>>>> Configuring Port 1 (socket 0)
>>>>>>>> Port 1: 00:0C:29:B5:9F:65
>>>>>>>> Checking link statuses...
>>>>>>>> Port 0 Link Up - speed 10000 Mbps - full-duplex
>>>>>>>> Port 1 Link Up - speed 10000 Mbps - full-duplex
>>>>>>>> Done
>>>>>>>> No commandline core given, start packet forwarding
>>>>>>>>     io packet forwarding - CRC stripping disabled - packets/burst=32
>>>>>>>>     nb forwarding cores=1 - nb forwarding ports=2
>>>>>>>>     RX queues=1 - RX desc=128 - RX free threshold=0
>>>>>>>>     RX threshold registers: pthresh=0 hthresh=0 wthresh=0
>>>>>>>>     TX queues=1 - TX desc=512 - TX free threshold=0
>>>>>>>>     TX threshold registers: pthresh=0 hthresh=0 wthresh=0
>>>>>>>>     TX RS bit threshold=0 - TXQ flags=0x200
>>>>>>>> Press enter to exit
>>>>>>>>
>>>>>>>> Telling cores to stop...
>>>>>>>> Waiting for lcores to finish...
>>>>>>>>
>>>>>>>>     ---------------------- Forward statistics for port 0
>>>>>>>> ----------------------
>>>>>>>>     RX-packets: 88             RX-dropped: 0             RX-total: 88
>>>>>>>>     TX-packets: 2              TX-dropped: 0             TX-total: 2
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>     ---------------------- Forward statistics for port 1
>>>>>>>> ----------------------
>>>>>>>>     RX-packets: 2              RX-dropped: 0             RX-total: 2
>>>>>>>>     TX-packets: 88             TX-dropped: 0             TX-total: 88
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>     +++++++++++++++ Accumulated forward statistics for all
>>>>>>>> ports+++++++++++++++
>>>>>>>>     RX-packets: 90             RX-dropped: 0             RX-total: 90
>>>>>>>>     TX-packets: 90             TX-dropped: 0             TX-total: 90
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>
>>>>>>>> make sense?
>>>>>>>
>>>>>>>
>>>>>>> Yes, it makes sense. Ports are being recognized by DPDK and some
>>>>>>> packets
>>>>>>> were exchanged.
>>>>>>>
>>>>>>>
>>>>>>>>>> [1] https://urldefense.proofpoint.com/v2/url?u=http-3A__dpdk.org_doc_guides-2D16.04_testpmd-5Fapp-5Fug_index.html&d=CwIBaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=44mSO5N5yEs4CeCdtQE0xt0F7J0p67_mApYVAzyYms0&m=vpaHcL64W3vOTY9P8yRZOdCJzbL_oWzZH6J2u4iT614&s=JnLeHoyKuiecYI95-fcfFiuev5g2CDweT2MnyUiKWzM&e=
>>>>>>>>>>
>>>>>>>>>> Mauricio Vasquez,
>>>>>>>
>>>>>>>
>>>>> Mauricio V,
>>>
>>>
>



More information about the discuss mailing list