[ovs-dev] Cannot open /dev/vfio/noiommu-0: Permission denied
Aaron Conole
aconole at redhat.com
Wed May 9 13:21:43 UTC 2018
Leon Goldberg <lgoldber at redhat.com> writes:
> Hey Aaron, sorry for bugging you so often, but this is very odd...
>
> I'm not sure what has changed, but using a new env, I am now getting operation not permitted for
> /dev/vfio/noiommu-0.
Strange.
With selinux mode in Permissive, and noiommu device with appropriate
permissions, I'm not sure what's happening.
Wild guess, can you look at /proc/cmdline and your BIOS and make sure
that the system is setup in a way that works for your particular case?
Sometimes it happens that a new system has incorrect PCI bus settings,
or invalid iommu / other settings that impede the use of devices with
DPDK.
> Here's sestatus:
>
> [root at lago-network-suite-master-host-0 ~]# sestatus
> SELinux status: enabled
> SELinuxfs mount: /sys/fs/selinux
> SELinux root directory: /etc/selinux
> Loaded policy name: targeted
> Current mode: permissive
> Mode from config file: enforcing
> Policy MLS status: enabled
> Policy deny_unknown status: allowed
> Max kernel policy version: 28
>
> Trying to add dpdk port:
>
> [root at lago-network-suite-master-host-0 ~]# ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0
> type=dpdk options:dpdk-devargs=0000:00:04.0
> ovs-vsctl: Error detected while setting up 'dpdk-p0': Error attaching device '0000:00:04.0' to DPDK.
> See ovs-vswitchd log for details.
> ovs-vsctl: The default log directory is "/var/log/openvswitch".
> [root at lago-network-suite-master-host-0 ~]# cat /var/log/openvswitch/ovs-vswitchd.log | grep
> permitted
> 2018-05-08T14:29:02.755Z|00227|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:29:22.532Z|00239|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:29:22.549Z|00250|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:29:23.525Z|00023|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:29:23.573Z|00082|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:29:23.595Z|00105|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:29:23.712Z|00124|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:29:23.721Z|00135|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:29:23.725Z|00146|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
> 2018-05-08T14:32:32.713Z|00160|dpdk|ERR|EAL: Cannot open /dev/vfio/noiommu-0: Operation not
> permitted
>
> audit.log snippet:
>
> [root at lago-network-suite-master-host-0 ~]# tail /var/log/audit/audit.log | grep openvswitch_t
> type=AVC msg=audit(1525789763.711:1210): avc: denied { open } for pid=22757
> comm="ovs-vswitchd" path="/dev/vfio/noiommu-0" dev="devtmpfs" ino=33747
> scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:device_t:s0
> tclass=chr_file
> type=SYSCALL msg=audit(1525789763.711:1210): arch=c000003e syscall=2 success=no exit=-1
> a0=7fff0045bfa0 a1=2 a2=7fff0045bfb3 a3=0 items=0 ppid=1 pid=22757 auid=4294967295
> uid=994 gid=1000 euid=994 suid=994 fsuid=994 egid=1000 sgid=1000 fsgid=1000 tty=(none)
> ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd"
> subj=system_u:system_r:openvswitch_t:s0 key=(null)
>
> permissions:
>
> [root at lago-network-suite-master-host-0 ~]# ps aux | grep ovs-vswitchd
> openvsw+ 22757 0.3 5.1 1122780 96568 ? S<Lsl 10:29 0:01 ovs-vswitchd
> unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user
> openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log
> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
> root 22933 0.0 0.0 112660 972 pts/0 S+ 10:34 0:00 grep --color=auto ovs-vswitchd
> [root at lago-network-suite-master-host-0 ~]# ls -lah /dev/vfio
> total 0
> drwxr-xr-x. 2 root root 80 May 8 06:34 .
> drwxr-xr-x. 19 root root 3.2K May 8 06:34 ..
> crw-rw----. 1 root hugetlbfs 244, 0 May 8 06:34 noiommu-0
> crw-rw-rw-. 1 root root 10, 196 May 8 06:34 vfio
> [root at lago-network-suite-master-host-0 ~]#
>
> On Tue, May 8, 2018 at 4:14 PM, Aaron Conole <aconole at redhat.com> wrote:
>
> Leon Goldberg <lgoldber at redhat.com> writes:
>
> > On Mon, May 7, 2018 at 9:00 PM, Aaron Conole <aconole at redhat.com> wrote:
> >
> > Leon Goldberg <lgoldber at redhat.com> writes:
> >
> > > I stand correct, I was not using permissive mode. With permissive mode, noiommu-0
> > issue seems to
> > > be resolved, however:
> >
> > Cool. I caught other issues turning on the -DB flag... although maybe
> > that was using enforcing mode as well.
> >
> > > type=AVC msg=audit(1525707587.009:447): avc: denied { remove_name } for
> > pid=4497
> > > comm="qemu-kvm" name="vhost-user-5" dev="vda3" ino=8742121
> > > scontext=system_u:system_r:svirt_t:s0:c794,c950
> > tcontext=unconfined_u:object_r:default_t:s0
> > > tclass=dir
> > > type=AVC msg=audit(1525707587.009:447): avc: denied { unlink } for pid=4497
> > > comm="qemu-kvm" name="vhost-user-5" dev="vda3" ino=8742121
> > > scontext=system_u:system_r:svirt_t:s0:c794,c950
> > tcontext=system_u:object_r:default_t:s0
> > > tclass=sock_file
> > > type=AVC msg=audit(1525707587.009:448): avc: denied { add_name } for
> > pid=4497
> > > comm="qemu-kvm" name="vhost-user-5"
> > scontext=system_u:system_r:svirt_t:s0:c794,c950
> > > tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
> > > type=AVC msg=audit(1525707587.009:448): avc: denied { create } for pid=4497
> > > comm="qemu-kvm" name="vhost-user-5"
> > scontext=system_u:system_r:svirt_t:s0:c794,c950
> > > tcontext=system_u:object_r:default_t:s0 tclass=sock_file
> > >
> > > Still occurs.
> > >
> > > OVS log shows:
> > >
> > > 2018-05-07T15:27:15.059Z|00072|dpdk|INFO|VHOST_CONFIG: vhost-user client:
> > socket created, fd:
> > > 55
> > > 2018-05-07T15:27:15.059Z|00073|netdev_dpdk|INFO|vHost User device
> > 'dpdkvhostclient1' created
> > > in 'client' mode, using client socket '/vhostusers/vhost-user-5'
> > > 2018-05-07T15:27:15.062Z|00074|dpdk|WARN|VHOST_CONFIG: failed to connect to
> > > /vhostusers/vhost-user-5: Permission denied
> >
> > Does that directory exist? What are the permissions? What are the
> > permissions of the sock file that exist in that directory?
> >
> > [root at lago-network-suite-master-host-0 ~]# ll /vhostusers/
> > total 0
> > srwxrwxr-x. 1 qemu kvm 0 May 7 11:55 vhost-user-5
>
> That looks like a problem. Have a look at /etc/libvirt/qemu.conf and
> change the group to hugetlbfs, restart libvirtd, and see if that allows
> ovs+libvirt to proceed.
>
> More information could be available at:
> https://developers.redhat.com/blog/2018/03/23/non-root-open-vswitch-rhel/
>
> -Aaron
>
> >
> > > On Mon, May 7, 2018 at 6:21 PM, Leon Goldberg <lgoldber at redhat.com> wrote:
> > >
> > > Aha, indeed, I see:
> > >
> > > type=AVC msg=audit(1525649015.102:1305): avc: denied { open } for
> > pid=12993
> > > comm="ovs-vswitchd" path="/dev/vfio/noiommu-0" dev="devtmpfs" ino=708920
> > > scontext=system_u:system_r:openvswitch_t:s0
> > tcontext=system_u:object_r:device_t:s0
> > > tclass=chr_file
> > > type=AVC msg=audit(1525649177.311:1326): avc: denied { open } for
> > pid=13241
> > > comm="ovs-vswitchd" path="/dev/vfio/noiommu-0" dev="devtmpfs" ino=708920
> > > scontext=system_u:system_r:openvswitch_t:s0
> > tcontext=system_u:object_r:device_t:s0
> > > tclass=chr_file
> > >
> > > and I'm using permissive mode.
> > >
> > > I also see:
> > >
> > > [root at lago-network-suite-master-host-0 ~]# cat /var/log/audit/audit.log | grep
> > vhost-user-5
> > > type=AVC msg=audit(1525636067.061:757): avc: denied { create } for pid=7533
> > > comm="qemu-kvm" name="vhost-user-5"
> > scontext=system_u:system_r:svirt_t:s0:c423,c510
> > > tcontext=system_u:object_r:tmp_t:s0 tclass=sock_file
> > > type=AVC msg=audit(1525648910.361:1276): avc: denied { add_name } for
> > pid=12734
> > > comm="qemu-kvm" name="vhost-user-5"
> > scontext=system_u:system_r:svirt_t:s0:c245,c301
> > > tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
> > > type=AVC msg=audit(1525648910.361:1276): avc: denied { create } for
> > pid=12734
> > > comm="qemu-kvm" name="vhost-user-5"
> > scontext=system_u:system_r:svirt_t:s0:c245,c301
> > > tcontext=system_u:object_r:default_t:s0 tclass=sock_file
> > > type=AVC msg=audit(1525648979.442:1290): avc: denied { remove_name } for
> > pid=12822
> > > comm="qemu-kvm" name="vhost-user-5" dev="vda3" ino=8742121
> > > scontext=system_u:system_r:svirt_t:s0:c515,c819
> > tcontext=unconfined_u:object_r:default_t:s0
> > > tclass=dir
> > > type=AVC msg=audit(1525648979.442:1290): avc: denied { unlink } for
> > pid=12822
> > > comm="qemu-kvm" name="vhost-user-5" dev="vda3" ino=8742121
> > > scontext=system_u:system_r:svirt_t:s0:c515,c819
> > tcontext=system_u:object_r:default_t:s0
> > > tclass=sock_file
> > > type=AVC msg=audit(1525648979.442:1291): avc: denied { add_name } for
> > pid=12822
> > > comm="qemu-kvm" name="vhost-user-5"
> > scontext=system_u:system_r:svirt_t:s0:c515,c819
> > > tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
> > > type=AVC msg=audit(1525648979.442:1291): avc: denied { create } for
> > pid=12822
> > > comm="qemu-kvm" name="vhost-user-5"
> > scontext=system_u:system_r:svirt_t:s0:c515,c819
> > > tcontext=system_u:object_r:default_t:s0 tclass=sock_file
> > >
> > > This is my vhostuser client.
> > >
> > > On Mon, May 7, 2018 at 4:39 PM, Aaron Conole <aconole at redhat.com> wrote:
> > >
> > > Leon Goldberg <lgoldber at redhat.com> writes:
> > >
> > > > On Fri, May 4, 2018 at 10:19 PM, Aaron Conole <aconole at redhat.com> wrote:
> > > >
> > > > Leon Goldberg <lgoldber at redhat.com> writes:
> > > >
> > > > > Hi list,
> > > > >
> > > > > I'm trying to integrate ovs-dpdk into oVirt. For testing purposes, I'm
> > > > > writing a test that looks to run a VM on top of a dpdk port.
> > > > >
> > > > > The testing environment consists of nested virtualization:
> > > > >
> > > > > Physical machine -> Jenkins CI VM -> Target VM
> > > > >
> > > > > The test merely looks to see that the various components are properly
> > > > > configured for the real world. For that purpose, I'm using NOIOMMU mode of
> > > > > VFIO.
> > > > >
> > > > > The select virtio device fails to to be attached to dpdk, and I suspect it
> > > > > is due to $subject.
> > > > >
> > > > > Here are the CI logs[1]. I see some other red lights, but $subject seems
> > > > > the brightest.
> > > >
> > > > Can you provide:
> > > >
> > > > $ ps aux | grep ovs-vswitchd
> > > > $ ls -lah /dev/vfio
> > > >
> > > > Hey Aaron,
> > > >
> > > > Here it is:
> > > >
> > > > [root at lago-network-suite-master-host-0 ~]# ps aux | grep ovs-vswitchd
> > > > openvsw+ 840 0.6 6.2 1273732 116716 ? S<Lsl 07:28 0:10 ovs-vswitchd
> > > > unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall
> > --user
> > > > openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log
> > > > --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
> > > > root 4425 0.0 0.0 112660 976 pts/0 R+ 07:55 0:00 grep --color=auto
> > > ovs-vswitchd
> > > >
> > > > [root at lago-network-suite-master-host-0 ~]# ls -lah /dev/vfio
> > > > total 0
> > > > drwxr-xr-x. 2 root root 80 May 6 07:28 .
> > > > drwxr-xr-x. 19 root root 3.2K May 6 07:28 ..
> > > > crw-rw----. 1 root hugetlbfs 244, 0 May 6 07:28 noiommu-0
> > > > crw-rw-rw-. 1 root root 10, 196 May 6 07:28 vfio
> > >
> > > Okay - that looks like it should be okay.
> > >
> > > Can you check if there are any selinux violations in audit.log
> > > (specifically from the openvswitch_t domain)? Maybe there is a missing
> > > selinux policy directive.
> > >
> > > > Just want to see if there's a disconnect between the userid for ovs
> > > > and the permissions on the vfio file. If that's the case, we may need
> > > > to update the vfio rules.
> > > >
> > > > > Any tips will be greatly appreciated!
> > > > >
> > > > > Thanks,
> > > > > Leon
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/642/artifact/exported-artifacts/check-patch.network_suite_master.el7.x86_64/tests.test_dpdk/lago-network-suite-master-host-0/_var_log/openvswitch/ovs-vswitchd.log
>
> >
> > >
> > > >
> > > > >
> > > >
> > >
> >
> <http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/642/artifact/exported-artifacts/check-patch.network_suite_master.el7.x86_64/tests.test_dpdk/lago-network-suite-master-host-0/_var_log/openvswitch/ovs-vswitchd.log>
>
> >
> > >
> > > >
> > > > > _______________________________________________
> > > > > dev mailing list
> > > > > dev at openvswitch.org
> > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
More information about the dev
mailing list