[ovs-discuss] OvS build (w DPDK) failed - Undefined Mellanox references

Avi Cohen (A) avi.cohen at huawei.com
Wed Feb 21 09:56:53 UTC 2018


Olga

./boot.sh  
./configure --with-dpdk=$DPDK_BUILD
Make install
     
Best Regards
Avi

> -----Original Message-----
> From: Olga Shern [mailto:olgas at mellanox.com]
> Sent: Tuesday, 20 February, 2018 7:20 PM
> To: Avi Cohen (A); ovs-discuss at openvswitch.org
> Cc: Erez Ferber
> Subject: RE: [ovs-discuss] OvS build (w DPDK) failed - Undefined Mellanox
> references
> 
> How do you compile OVS?
> 
> -----Original Message-----
> From: Avi Cohen (A) [mailto:avi.cohen at huawei.com]
> Sent: Tuesday, February 20, 2018 1:39 PM
> To: Olga Shern <olgas at mellanox.com>; ovs-discuss at openvswitch.org
> Cc: Erez Ferber <erezf at mellanox.com>
> Subject: RE: [ovs-discuss] OvS build (w DPDK) failed - Undefined Mellanox
> references
> 
> Thank you Olga
> 
> Now dpdk build is ok but ovs build fail because of ibv* undefined reference
> 
> 
> 
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5.o): In function `mlx5_dev_close':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:195: undefined
> reference to `ibv_dealloc_pd'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:196: undefined
> reference to `ibv_close_device'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5.o): In function `mlx5_pci_probe':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:453: undefined
> reference to `ibv_get_device_list'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:513: undefined
> reference to `ibv_open_device'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:533: undefined
> reference to `ibv_query_device'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:829: undefined
> reference to `ibv_close_device'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:831: undefined
> reference to `ibv_free_device_list'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:518: undefined
> reference to `ibv_free_device_list'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:568: undefined
> reference to `ibv_open_device'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5.o): In function `___ibv_query_port':
> /usr/include/infiniband/verbs.h:1227: undefined reference to `ibv_query_port'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5.o): In function `mlx5_pci_probe':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:591: undefined
> reference to `ibv_alloc_pd'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:808: undefined
> reference to `ibv_dealloc_pd'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:810: undefined
> reference to `ibv_close_device'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:749: undefined
> reference to `ibv_get_device_name'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:518: undefined
> reference to `ibv_free_device_list'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5.o): In function `rte_mlx5_pmd_init':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:897: undefined
> reference to `ibv_fork_init'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_rxq.o): In function `priv_create_hash_rxqs':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxq.c:499: undefined
> reference to `ibv_destroy_qp'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_rxq.o): In function `priv_destroy_hash_rxqs':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxq.c:549: undefined
> reference to `ibv_destroy_qp'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_rxq.o): In function `rxq_cleanup':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxq.c:754: undefined
> reference to `ibv_destroy_cq'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxq.c:756: undefined
> reference to `ibv_destroy_comp_channel'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxq.c:758: undefined
> reference to `ibv_dereg_mr'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_rxq.o): In function `rxq_ctrl_setup':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxq.c:979: undefined
> reference to `ibv_create_comp_channel'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_rxq.o): In function `mlx5_rx_intr_disable':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxq.c:1456: undefined
> reference to `ibv_get_cq_event'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxq.c:1464: undefined
> reference to `ibv_ack_cq_events'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_txq.o): In function `txq_cleanup':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_txq.c:148: undefined
> reference to `ibv_destroy_qp'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_txq.c:150: undefined
> reference to `ibv_destroy_cq'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_txq.c:155: undefined
> reference to `ibv_dereg_mr'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_txq.o): In function `ibv_exp_create_qp':
> /usr/include/infiniband/verbs_exp.h:3009: undefined reference to
> `ibv_create_qp'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_ethdev.o): In function
> `priv_dev_link_status_handler':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_ethdev.c:1240:
> undefined reference to `ibv_ack_async_event'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_ethdev.c:1233:
> undefined reference to `ibv_get_async_event'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_ethdev.c:1240:
> undefined reference to `ibv_ack_async_event'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_ethdev.c:1233:
> undefined reference to `ibv_get_async_event'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_fdir.o): In function `priv_fdir_queue_create':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_fdir.c:540: undefined
> reference to `ibv_destroy_cq'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_fdir.o): In function `priv_fdir_queue_destroy':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_fdir.c:431: undefined
> reference to `ibv_destroy_qp'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_fdir.c:437: undefined
> reference to `ibv_destroy_cq'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_mr.o): In function `mlx5_mp2mr':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_mr.c:162: undefined
> reference to `ibv_reg_mr'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_mr.o): In function `txq_mp2mr_reg':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_mr.c:204: undefined
> reference to `ibv_dereg_mr'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_mr.c:204: undefined
> reference to `ibv_dereg_mr'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_flow.o): In function
> `priv_flow_create_drop_queue':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_flow.c:1440: undefined
> reference to `ibv_destroy_qp'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_flow.c:1448: undefined
> reference to `ibv_destroy_cq'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_flow.o): In function `priv_flow_destroy':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_flow.c:1258: undefined
> reference to `ibv_destroy_qp'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_flow.c:1264: undefined
> reference to `ibv_destroy_cq'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_flow.o): In function
> `priv_flow_create_action_queue':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_flow.c:1130: undefined
> reference to `ibv_destroy_qp'
> /home/avi/dpdk-stable-17.05.2/x86_64-native-linuxapp-
> gcc/lib/librte_pmd_mlx5.a(mlx5_flow.o): In function
> `priv_flow_delete_drop_queue':
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_flow.c:1470: undefined
> reference to `ibv_destroy_qp'
> /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_flow.c:1478: undefined
> reference to `ibv_destroy_cq'
> 
> Best Regards
> Avi
> 
> 
> > -----Original Message-----
> > From: Olga Shern [mailto:olgas at mellanox.com]
> > Sent: Tuesday, 20 February, 2018 1:17 PM
> > To: Avi Cohen (A); ovs-discuss at openvswitch.org
> > Cc: Erez Ferber
> > Subject: RE: [ovs-discuss] OvS build (w DPDK) failed - Undefined
> > Mellanox references
> >
> > Avi,
> >
> > Every DPDK version has specific MLNX_OFED that is supports:
> > If you will follow the instruction that I sent to you will see that:
> > Mellanox OFED version should be  4.0-2.0.0.0 and installation should
> > be ./mlnxofedinstall
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpdk
> > .org%2Fdoc%2Fguides-
> 17.05%2Fnics%2Fmlx5.html&data=02%7C01%7Colgas%40me
> >
> llanox.com%7Caf7aca2bf64f4dc1752508d578567bef%7Ca652971c7d2e4d9ba6
> a4d1
> >
> 49256f461b%7C0%7C0%7C636547235204887682&sdata=X6L6dwwsOaQXEY4yS
> 1BwgHAM
> > Dpumfcuni4pB6dxP6eU%3D&reserved=0
> >
> > Now, you can stay with MLNX_OFED 4.2, but you will need to reinstall
> > it without --upstream-libs flags.
> >
> > Starting DPDK 17.11 Mellanox PMDs supports upstream rdma-core libs,
> > therefore the installation is different for DPDK versions starting
> > 17.11 Since user shouldn't be aware about this info, user should
> > follow the exact instructions!
> >
> > Best Regards,
> > Olga
> >
> > -----Original Message-----
> > From: Avi Cohen (A) [mailto:avi.cohen at huawei.com]
> > Sent: Tuesday, February 20, 2018 1:03 PM
> > To: Olga Shern <olgas at mellanox.com>; ovs-discuss at openvswitch.org
> > Cc: Erez Ferber <erezf at mellanox.com>
> > Subject: RE: [ovs-discuss] OvS build (w DPDK) failed - Undefined
> > Mellanox references
> >
> > Olga
> >
> > I have OFED -  MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64
> > I install it like this :  ./mlnxofedinstall --force --dpdk
> > --upstream-libs NOTE  that I don't have this problem with dpdk.17.11/
> > dpdk.18.02
> >
> > Best Regards
> > Avi
> >
> >
> >
> > > -----Original Message-----
> > > From: Olga Shern [mailto:olgas at mellanox.com]
> > > Sent: Tuesday, 20 February, 2018 12:19 PM
> > > To: Avi Cohen (A); ovs-discuss at openvswitch.org
> > > Cc: Erez Ferber
> > > Subject: RE: [ovs-discuss] OvS build (w DPDK) failed - Undefined
> > > Mellanox references
> > >
> > > Looks like you didn't install MLNX_OFED ir something wrong with your
> > > installation, the following file is missing:
> > > /usr/include/infiniband/mlx5_hw.h
> > >
> > > What OFED did you install and how?
> > >
> > > -----Original Message-----
> > > From: Avi Cohen (A) [mailto:avi.cohen at huawei.com]
> > > Sent: Tuesday, February 20, 2018 10:21 AM
> > > To: Olga Shern <olgas at mellanox.com>; ovs-discuss at openvswitch.org
> > > Cc: Erez Ferber <erezf at mellanox.com>
> > > Subject: RE: [ovs-discuss] OvS build (w DPDK) failed - Undefined
> > > Mellanox references
> > >
> > > Thank you Olga
> > > What I'm trying to do is to run ovs-dpdk w connectx4 card, I don't
> > > care about the versions but only try to  find a  combination of
> > > versions (dpdk/ovs/kernel) that is working.
> > > As you've suggested,  now I'm working with dpdk.17.05 (ovs 2.8.1) .
> > > I'm following the
> > > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdp
> > > dk
> > > .org
> > > %2Fdoc%2Fguides-
> > >
> >
> 17.05%2Fnics%2Fmlx5.html&data=02%7C01%7Colgas%40mellanox.com%7C45
> > >
> >
> 2a99fe115941a1fd0508d5783add13%7Ca652971c7d2e4d9ba6a4d149256f461b
> > >
> >
> %7C0%7C0%7C636547116571396518&sdata=YyijRGomnVwvz8epD90ec1lx4YyB
> > > 2Fn2InZvgK5fqfY%3D&reserved=0
> > > And I have compilation error when building dpdk  - [ note that  dpdk
> > > 18.02 and
> > > dpdk.17.11 build pass ok, but then ovs failed with undefined mlx
> > > ref.]
> > >
> > > Dpdk.17.05 - compilation error:
> > > --------------
> > > In file included from /home/avi/dpdk-stable-
> > > 17.05.2/drivers/net/mlx5/mlx5.h:68:0,
> > > from /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5.c:67:
> > > /home/avi/dpdk-stable-17.05.2/drivers/net/mlx5/mlx5_rxtx.h:46:32:
> > > fatal
> > > error: infiniband/mlx5_hw.h: No such file or directory
> > > -------------------
> > >
> > > Best Regards
> > > Avi
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Olga Shern [mailto:olgas at mellanox.com]
> > > > Sent: Monday, 19 February, 2018 11:31 PM
> > > > To: Avi Cohen (A); ovs-discuss at openvswitch.org
> > > > Cc: Erez Ferber
> > > > Subject: RE: [ovs-discuss] OvS build (w DPDK) failed - Undefined
> > > > Mellanox references
> > > >
> > > > Hi Avi,
> > > >
> > > > Can you please explain what you are trying to do?
> > > > Are you trying to test latest  DPDK with OVS 2.8.1?
> > > > OVS 2.8.1 supports DPDK 17.05.1 , and as far as I know,  it also
> > > > working with DPDK 17.08 DPDK 17.11 is supported with OVS 2.9.
> > > >
> > > > If you want help regarding DPDK you should send emails to the
> > > > users at dpdk.org
> > > >
> > > > Now, regarding what I stated before, you need to follow procedure
> > > > from DPDK.org for mlx PMDs :
> > > > For 17.08:
> > > > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F
> > > > dp
> > > > dk
> > > > .org%2Fdoc%2Fguides-
> > > 17.08%2Fnics%2Fmlx5.html&data=02%7C01%7Colgas%40me
> > > >
> > >
> >
> llanox.com%7C452a99fe115941a1fd0508d5783add13%7Ca652971c7d2e4d9ba
> > > 6a4d1
> > > >
> > >
> >
> 49256f461b%7C0%7C0%7C636547116571396518&sdata=ITJ%2FVuP09A2911IU
> > > %2BN%2
> > > > BQqscF4aYBGl2vYbEK7qU9QfM%3D&reserved=0 , Please go to section
> > 20.7.
> > > > Quick Start Guide and check section  20.5. Prerequisites MLNX_OFED
> > > > 4.1 ./mlnxofedinstall For 17.11:
> > > > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F
> > > > dp
> > > > dk
> > > > .org%2Fdoc%2Fguides-
> > > &data=02%7C01%7Colgas%40mellanox.com%7C452a99fe115
> > > >
> > >
> >
> 941a1fd0508d5783add13%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0
> > > %7C636
> > > >
> > >
> >
> 547116571396518&sdata=gxX93MLNYo3vhR0p32hQn37I8yuSm5g4xqsWpBsKQ
> > > 3U%3D&r
> > > > eserved=0 17.11/nics/mlx5.html ,  please go to section 21.7 (Quick
> > > > Start Guide on OFED) and check section  21.5.1.2. Mellanox OFED
> > > > for the right MLNX_OFED to use MLNX_OFED 4.2 ./mlnxofedinstall
> > > > --upstream-libs --dpdk
> > > >
> > > > Etc ...
> > > >
> > > > Hope this helps
> > > >
> > > > Best Regards,
> > > > Olga
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: ovs-discuss-bounces at openvswitch.org [mailto:ovs-discuss-
> > > > bounces at openvswitch.org] On Behalf Of Avi Cohen (A)
> > > > Sent: Monday, February 19, 2018 6:21 PM
> > > > To: ovs-discuss at openvswitch.org
> > > > Subject: [ovs-discuss] OvS build (w DPDK) failed - Undefined
> > > > Mellanox references
> > > >
> > > > Hi,
> > > > I build dpdk-18.02 with MLX5_PMD - ok..
> > > > Then I failed to build ovs - 2.8.1]  with dpdk and get many
> > > > undefined reference for mlx I have mellanox OFED installed
> > > >
> > > > Some of the errors here :
> > > >
> > > >
> > > > /home/avi/dpdk-18.02/x86_64-native-linuxapp-
> > > > gcc/lib/librte_pmd_mlx5.a(mlx5_glue.o): In function
> > `mlx5_glue_dv_init_obj':
> > > > /home/avi/dpdk-18.02/drivers/net/mlx5/mlx5_glue.c:307: undefined
> > > > reference to `mlx5dv_init_obj'
> > > > /home/avi/dpdk-18.02/x86_64-native-linuxapp-
> > > > gcc/lib/librte_pmd_mlx5.a(mlx5_glue.o): In function
> > > > `mlx5_glue_dv_set_context_attr':
> > > > /home/avi/dpdk-18.02/drivers/net/mlx5/mlx5_glue.c:301: undefined
> > > > reference to `mlx5dv_set_context_attr'
> > > > /home/avi/dpdk-18.02/x86_64-native-linuxapp-
> > > > gcc/lib/librte_pmd_mlx5.a(mlx5_glue.o): In function
> > > > `mlx5_glue_dv_query_device':
> > > > /home/avi/dpdk-18.02/drivers/net/mlx5/mlx5_glue.c:294: undefined
> > > > reference to `mlx5dv_query_device'
> > > > /home/avi/dpdk-18.02/x86_64-native-linuxapp-
> > > > gcc/lib/librte_pmd_mlx5.a(mlx5_glue.o): In function
> > > > `mlx5_glue_dv_create_cq':
> > > > /home/avi/dpdk-18.02/drivers/net/mlx5/mlx5_glue.c:287: undefined
> > > > reference to `mlx5dv_create_cq'
> > > > /home/avi/dpdk-18.02/x86_64-native-linuxapp-
> > > > gcc/lib/librte_pmd_mlx5.a(mlx5_glue.o): In function
> > > > `mlx5_glue_port_state_str':
> > > > /home/avi/dpdk-18.02/drivers/net/mlx5/mlx5_glue.c:273: undefined
> > > > reference to `ibv_port_state_str'
> > > > /home/avi/dpdk-18.02/x86_64-native-linuxapp-
> > > > gcc/lib/librte_pmd_mlx5.a(mlx5_glue.o): In function
> > > > `mlx5_glue_get_async_event':
> > > > /home/avi/dpdk-18.02/drivers/net/mlx5/mlx5_glue.c:267: undefined
> > > > reference to `ibv_get_async_event'
> > > > /home/avi/dpdk-18.02/x86_64-native-linuxapp-
> > > > gcc/lib/librte_pmd_mlx5.a(mlx5_glue.o): In function
> > > > `mlx5_glue_ack_async_event':
> > > > /home/avi/dpdk-18.02/drivers/net/mlx5/mlx5_glue.c:260: undefined
> > > > reference to `ibv_ack_async_event'
> > > > /home/avi/dpdk-18.02/x86_64-native-linuxapp-
> > > > gcc/lib/librte_pmd_mlx5.a(mlx5_glue.o): In function
> > `mlx5_glue_dereg_mr':
> > > >
> > > > Who can help ?
> > > > Best Regards
> > > > Avi
> > > > _______________________________________________
> > > > discuss mailing list
> > > > discuss at openvswitch.org
> > > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2
> > > > Fm
> > > > ai
> > > > l.op
> > > > envswitch.org%2Fmailman%2Flistinfo%2Fovs-
> > > >
> > >
> >
> discuss&data=02%7C01%7Colgas%40mellanox.com%7Cbd7017779e624a247e3
> > > >
> > >
> >
> 508d577b4d322%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C6365
> > > >
> > >
> >
> 46540872320081&sdata=qPOVcZSoBvPQzpSKw9632jOC0iRhPwpnGdULr3u8KU
> > > > Q%3D&reserved=0


More information about the discuss mailing list