[ovs-dev] Cannot open /dev/vfio/noiommu-0: Permission denied

Leon Goldberg lgoldber at redhat.com
Tue May 8 14:41:09 UTC 2018


Hey Aaron, sorry for bugging you so often, but this is very odd...

I'm not sure what has changed, but using a new env, I am now getting
operation not permitted for /dev/vfio/noiommu-0.

Here's sestatus:

*[root at lago-network-suite-master-host-0 ~]# sestatus*
*SELinux status:                 enabled*
*SELinuxfs mount:                /sys/fs/selinux*
*SELinux root directory:         /etc/selinux*
*Loaded policy name:             targeted*
*Current mode:                   permissive*
*Mode from config file:          enforcing*
*Policy MLS status:              enabled*
*Policy deny_unknown status:     allowed*
*Max kernel policy version:      28*

Trying to add dpdk port:

*[root at lago-network-suite-master-host-0 ~]# ovs-vsctl add-port br0 dpdk-p0
-- set Interface dpdk-p0 type=dpdk options:dpdk-devargs=0000:00:04.0*
*ovs-vsctl: Error detected while setting up 'dpdk-p0': Error attaching
device '0000:00:04.0' to DPDK.  See ovs-vswitchd log for details.*
*ovs-vsctl: The default log directory is "/var/log/openvswitch".*
*[root at lago-network-suite-master-host-0 ~]# cat
/var/log/openvswitch/ovs-vswitchd.log | grep permitted*
*2018-05-08T14:29:02.755Z|00227|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:29:22.532Z|00239|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:29:22.549Z|00250|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:29:23.525Z|00023|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:29:23.573Z|00082|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:29:23.595Z|00105|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:29:23.712Z|00124|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:29:23.721Z|00135|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:29:23.725Z|00146|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*
*2018-05-08T14:32:32.713Z|00160|dpdk|ERR|EAL: Cannot open
/dev/vfio/noiommu-0: Operation not permitted*

audit.log snippet:

*[root at lago-network-suite-master-host-0 ~]# tail /var/log/audit/audit.log |
grep openvswitch_t*
*type=AVC msg=audit(1525789763.711:1210): avc:  denied  { open } for
pid=22757 comm="ovs-vswitchd" path="/dev/vfio/noiommu-0" dev="devtmpfs"
ino=33747 scontext=system_u:system_r:openvswitch_t:s0
tcontext=system_u:object_r:device_t:s0 tclass=chr_file*
*type=SYSCALL msg=audit(1525789763.711:1210): arch=c000003e syscall=2
success=no exit=-1 a0=7fff0045bfa0 a1=2 a2=7fff0045bfb3 a3=0 items=0 ppid=1
pid=22757 auid=4294967295 uid=994 gid=1000 euid=994 suid=994 fsuid=994
egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295
comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd"
subj=system_u:system_r:openvswitch_t:s0 key=(null)*

permissions:

*[root at lago-network-suite-master-host-0 ~]# ps aux | grep ovs-vswitchd*
*openvsw+ 22757  0.3  5.1 1122780 96568 ?       S<Lsl 10:29   0:01
ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err
-vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir
--log-file=/var/log/openvswitch/ovs-vswitchd.log
--pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach*
*root     22933  0.0  0.0 112660   972 pts/0    S+   10:34   0:00 grep
--color=auto ovs-vswitchd*
*[root at lago-network-suite-master-host-0 ~]# ls -lah /dev/vfio*
*total 0*
*drwxr-xr-x.  2 root root            80 May  8 06:34 .*
*drwxr-xr-x. 19 root root          3.2K May  8 06:34 ..*
*crw-rw----.  1 root hugetlbfs 244,   0 May  8 06:34 noiommu-0*
*crw-rw-rw-.  1 root root       10, 196 May  8 06:34 vfio*



*[root at lago-network-suite-master-host-0 ~]# *

On Tue, May 8, 2018 at 4:14 PM, Aaron Conole <aconole at redhat.com> wrote:

> Leon Goldberg <lgoldber at redhat.com> writes:
>
> > On Mon, May 7, 2018 at 9:00 PM, Aaron Conole <aconole at redhat.com> wrote:
> >
> >  Leon Goldberg <lgoldber at redhat.com> writes:
> >
> >  > I stand correct, I was not using permissive mode. With permissive
> mode, noiommu-0
> >  issue seems to
> >  > be resolved, however:
> >
> >  Cool.  I caught other issues turning on the -DB flag... although maybe
> >  that was using enforcing mode as well.
> >
> >  > type=AVC msg=audit(1525707587.009:447): avc:  denied  { remove_name }
> for
> >  pid=4497
> >  > comm="qemu-kvm" name="vhost-user-5" dev="vda3" ino=8742121
> >  > scontext=system_u:system_r:svirt_t:s0:c794,c950
> >  tcontext=unconfined_u:object_r:default_t:s0
> >  > tclass=dir
> >  > type=AVC msg=audit(1525707587.009:447): avc:  denied  { unlink } for
> pid=4497
> >  > comm="qemu-kvm" name="vhost-user-5" dev="vda3" ino=8742121
> >  > scontext=system_u:system_r:svirt_t:s0:c794,c950
> >  tcontext=system_u:object_r:default_t:s0
> >  > tclass=sock_file
> >  > type=AVC msg=audit(1525707587.009:448): avc:  denied  { add_name }
> for
> >  pid=4497
> >  > comm="qemu-kvm" name="vhost-user-5"
> >  scontext=system_u:system_r:svirt_t:s0:c794,c950
> >  > tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
> >  > type=AVC msg=audit(1525707587.009:448): avc:  denied  { create } for
> pid=4497
> >  > comm="qemu-kvm" name="vhost-user-5"
> >  scontext=system_u:system_r:svirt_t:s0:c794,c950
> >  > tcontext=system_u:object_r:default_t:s0 tclass=sock_file
> >  >
> >  > Still occurs.
> >  >
> >  > OVS log shows:
> >  >
> >  > 2018-05-07T15:27:15.059Z|00072|dpdk|INFO|VHOST_CONFIG: vhost-user
> client:
> >  socket created, fd:
> >  > 55
> >  > 2018-05-07T15:27:15.059Z|00073|netdev_dpdk|INFO|vHost User device
> >  'dpdkvhostclient1' created
> >  > in 'client' mode, using client socket '/vhostusers/vhost-user-5'
> >  > 2018-05-07T15:27:15.062Z|00074|dpdk|WARN|VHOST_CONFIG: failed to
> connect to
> >  > /vhostusers/vhost-user-5: Permission denied
> >
> >  Does that directory exist?  What are the permissions?  What are the
> >  permissions of the sock file that exist in that directory?
> >
> > [root at lago-network-suite-master-host-0 ~]# ll /vhostusers/
> > total 0
> > srwxrwxr-x. 1 qemu kvm 0 May  7 11:55 vhost-user-5
>
> That looks like a problem.  Have a look at /etc/libvirt/qemu.conf and
> change the group to hugetlbfs, restart libvirtd, and see if that allows
> ovs+libvirt to proceed.
>
> More information could be available at:
> https://developers.redhat.com/blog/2018/03/23/non-root-open-vswitch-rhel/
>
> -Aaron
>
> >
> >  > On Mon, May 7, 2018 at 6:21 PM, Leon Goldberg <lgoldber at redhat.com>
> wrote:
> >  >
> >  >  Aha, indeed, I see:
> >  >
> >  >  type=AVC msg=audit(1525649015.102:1305): avc:  denied  { open } for
> >  pid=12993
> >  >  comm="ovs-vswitchd" path="/dev/vfio/noiommu-0" dev="devtmpfs"
> ino=708920
> >  >  scontext=system_u:system_r:openvswitch_t:s0
> >  tcontext=system_u:object_r:device_t:s0
> >  >  tclass=chr_file
> >  >  type=AVC msg=audit(1525649177.311:1326): avc:  denied  { open } for
> >  pid=13241
> >  >  comm="ovs-vswitchd" path="/dev/vfio/noiommu-0" dev="devtmpfs"
> ino=708920
> >  >  scontext=system_u:system_r:openvswitch_t:s0
> >  tcontext=system_u:object_r:device_t:s0
> >  >  tclass=chr_file
> >  >
> >  >  and I'm using permissive mode.
> >  >
> >  >  I also see:
> >  >
> >  >  [root at lago-network-suite-master-host-0 ~]# cat
> /var/log/audit/audit.log | grep
> >  vhost-user-5
> >  >  type=AVC msg=audit(1525636067.061:757): avc:  denied  { create }
> for  pid=7533
> >  >  comm="qemu-kvm" name="vhost-user-5"
> >  scontext=system_u:system_r:svirt_t:s0:c423,c510
> >  >  tcontext=system_u:object_r:tmp_t:s0 tclass=sock_file
> >  >  type=AVC msg=audit(1525648910.361:1276): avc:  denied  { add_name }
> for
> >  pid=12734
> >  >  comm="qemu-kvm" name="vhost-user-5"
> >  scontext=system_u:system_r:svirt_t:s0:c245,c301
> >  >  tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
> >  >  type=AVC msg=audit(1525648910.361:1276): avc:  denied  { create }
> for
> >  pid=12734
> >  >  comm="qemu-kvm" name="vhost-user-5"
> >  scontext=system_u:system_r:svirt_t:s0:c245,c301
> >  >  tcontext=system_u:object_r:default_t:s0 tclass=sock_file
> >  >  type=AVC msg=audit(1525648979.442:1290): avc:  denied  {
> remove_name } for
> >  pid=12822
> >  >  comm="qemu-kvm" name="vhost-user-5" dev="vda3" ino=8742121
> >  >  scontext=system_u:system_r:svirt_t:s0:c515,c819
> >  tcontext=unconfined_u:object_r:default_t:s0
> >  >  tclass=dir
> >  >  type=AVC msg=audit(1525648979.442:1290): avc:  denied  { unlink }
> for
> >  pid=12822
> >  >  comm="qemu-kvm" name="vhost-user-5" dev="vda3" ino=8742121
> >  >  scontext=system_u:system_r:svirt_t:s0:c515,c819
> >  tcontext=system_u:object_r:default_t:s0
> >  >  tclass=sock_file
> >  >  type=AVC msg=audit(1525648979.442:1291): avc:  denied  { add_name }
> for
> >  pid=12822
> >  >  comm="qemu-kvm" name="vhost-user-5"
> >  scontext=system_u:system_r:svirt_t:s0:c515,c819
> >  >  tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
> >  >  type=AVC msg=audit(1525648979.442:1291): avc:  denied  { create }
> for
> >  pid=12822
> >  >  comm="qemu-kvm" name="vhost-user-5"
> >  scontext=system_u:system_r:svirt_t:s0:c515,c819
> >  >  tcontext=system_u:object_r:default_t:s0 tclass=sock_file
> >  >
> >  >  This is my vhostuser client.
> >  >
> >  >  On Mon, May 7, 2018 at 4:39 PM, Aaron Conole <aconole at redhat.com>
> wrote:
> >  >
> >  >  Leon Goldberg <lgoldber at redhat.com> writes:
> >  >
> >  >  > On Fri, May 4, 2018 at 10:19 PM, Aaron Conole <aconole at redhat.com>
> wrote:
> >  >  >
> >  >  >  Leon Goldberg <lgoldber at redhat.com> writes:
> >  >  >
> >  >  >  > Hi list,
> >  >  >  >
> >  >  >  > I'm trying to integrate ovs-dpdk into oVirt. For testing
> purposes, I'm
> >  >  >  > writing a test that looks to run a VM on top of a dpdk port.
> >  >  >  >
> >  >  >  > The testing environment consists of nested virtualization:
> >  >  >  >
> >  >  >  > Physical machine -> Jenkins CI VM -> Target VM
> >  >  >  >
> >  >  >  > The test merely looks to see that the various components are
> properly
> >  >  >  > configured for the real world. For that purpose, I'm using
> NOIOMMU mode of
> >  >  >  > VFIO.
> >  >  >  >
> >  >  >  > The select virtio device fails to to be attached to dpdk, and I
> suspect it
> >  >  >  > is due to $subject.
> >  >  >  >
> >  >  >  > Here are the CI logs[1]. I see some other red lights, but
> $subject seems
> >  >  >  > the brightest.
> >  >  >
> >  >  >  Can you provide:
> >  >  >
> >  >  >  $ ps aux | grep ovs-vswitchd
> >  >  >  $ ls -lah /dev/vfio
> >  >  >
> >  >  > Hey Aaron,
> >  >  >
> >  >  > Here it is:
> >  >  >
> >  >  > [root at lago-network-suite-master-host-0 ~]# ps aux | grep
> ovs-vswitchd
> >  >  > openvsw+   840  0.6  6.2 1273732 116716 ?      S<Lsl 07:28   0:10
> ovs-vswitchd
> >  >  > unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err
> -vfile:info --mlockall
> >  --user
> >  >  > openvswitch:hugetlbfs --no-chdir --log-file=/var/log/
> openvswitch/ovs-vswitchd.log
> >  >  > --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
> >  >  > root      4425  0.0  0.0 112660   976 pts/0    R+   07:55   0:00
> grep --color=auto
> >  >  ovs-vswitchd
> >  >  >
> >  >  > [root at lago-network-suite-master-host-0 ~]# ls -lah /dev/vfio
> >  >  > total 0
> >  >  > drwxr-xr-x.  2 root root            80 May  6 07:28 .
> >  >  > drwxr-xr-x. 19 root root          3.2K May  6 07:28 ..
> >  >  > crw-rw----.  1 root hugetlbfs 244,   0 May  6 07:28 noiommu-0
> >  >  > crw-rw-rw-.  1 root root       10, 196 May  6 07:28 vfio
> >  >
> >  >  Okay - that looks like it should be okay.
> >  >
> >  >  Can you check if there are any selinux violations in audit.log
> >  >  (specifically from the openvswitch_t domain)?  Maybe there is a
> missing
> >  >  selinux policy directive.
> >  >
> >  >  >  Just want to see if there's a disconnect between the userid for
> ovs
> >  >  >  and the permissions on the vfio file.  If that's the case, we may
> need
> >  >  >  to update the vfio rules.
> >  >  >
> >  >  >  > Any tips will be greatly appreciated!
> >  >  >  >
> >  >  >  > Thanks,
> >  >  >  > Leon
> >  >  >  >
> >  >  >  > [1]
> >  >  >  >
> >  >  >
> >  >
> >  http://jenkins.ovirt.org/job/ovirt-system-tests_standard-
> check-patch/642/artifact/exported-artifacts/check-
> patch.network_suite_master.el7.x86_64/tests.test_dpdk/
> lago-network-suite-master-host-0/_var_log/openvswitch/ovs-vswitchd.log
> >
> >  >
> >  >  >
> >  >  >  >
> >  >  >
> >  >
> >  <http://jenkins.ovirt.org/job/ovirt-system-tests_standard-
> check-patch/642/artifact/exported-artifacts/check-
> patch.network_suite_master.el7.x86_64/tests.test_dpdk/
> lago-network-suite-master-host-0/_var_log/openvswitch/ovs-vswitchd.log>
> >
> >  >
> >  >  >
> >  >  >  > _______________________________________________
> >  >  >  > dev mailing list
> >  >  >  > dev at openvswitch.org
> >  >  >  > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>


More information about the dev mailing list