[ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

Thu Apr 25 11:44:33 UTC 2019

> As a "security concern" you mean something among the lines where one
> of ovs-* processes running under openvswitch user would go ahead and
> create a file with its owner that later one of ovn processes would
> blindly reuse without checking that it actually belongs to root:root?

One example is that the OVS user could create link as any OVN log file
to other file owned by root and then OVN would write to that file.

> Can you give a more concrete example? I believe logrotate is running
> under root and should be able to rotate everything?

The logrotate configuration for openvswitch logs has a 'su' directive
to run under the openvswitch user, precisely to prevent something
similar to the above. So it wont be able to rotate root owned logs in
the openvswitch directory. How it fails precisely depends on global
logrotate configuration. For example it will fail to create the new log
file after rotation if the 'create' directive is used. Or it may fail
to compress the rotated log file as it wont be able to read it.

BR
Jaime

-----Original Message-----
From: Ansis Atteka <ansisatteka at gmail.com>
To: jcaamano at suse.de
Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
changes
Date: Wed, 24 Apr 2019 19:07:24 -0700

On Tue, 23 Apr 2019 at 09:30, Jaime Caamaño Ruiz <jcaamano at suse.de>
wrote:
> 
> Hello
> 
> So the non root owned log directory (and run directory) is shared
> between non root OVS processes and root OVN processes. Doesn't this
> raise some security concerns?

As a "security concern" you mean something among the lines where one
of ovs-* processes running under openvswitch user would go ahead and
create a file with its owner that later one of ovn processes would
blindly reuse without checking that it actually belongs to root:root?

> 
> Also, since logrotate rotates the logs with the OVS user, it may fail
> to some extent to rotate the root owned OVN log files. Consequences
> vary but in it's current state some logging might be lost. This could
> be improved but I would guess that the approprioate thing to do is to
> etiher use a different log directory for OVN or make its processes
> run
> with the OVS user also. Has any of this been considered?

Can you give a more concrete example? I believe logrotate is running
under root and should be able to rotate everything?

> 
> BR
> Jaime.
> 
> 
> -----Original Message-----
> From: Jaime Caamaño Ruiz <jcaamano at suse.de>
> Reply-to: jcaamano at suse.com
> To: Ansis Atteka <ansisatteka at gmail.com>
> Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
> Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
> changes
> Date: Wed, 17 Apr 2019 12:52:32 +0200
> 
> > You also need to chown /var/log/openvswitch.*.log files.
> 
> OVS seems to be already handling this. I dont know the details but I
> guess that before dropping capabilities, OVS chowns these by itself.
> 
> > However,
> > what about other daemons, like ovn? Do they share run time dir?
> > Haven't looked into this in a while and don't have a setup to check
> > myself.
> 
> The permisions of the openvswitch run directory are already being
> handled by the service unit file. There is read access to files
> created
> there by OVS and AFAIK what it ultimately makes it a no problem for
> OVN
> is that it still runs as root with the provided unit files. If anyone
> changes this to a user different than the openvswitch user, then that
> is an already existing issue. Am I missing something here?
> 
> Agree with you other considerations.
> 
> BR
> Jaime.
> 
> -----Original Message-----
> From: Ansis Atteka <ansisatteka at gmail.com>
> To: jcaamano at suse.de
> Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
> Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
> changes
> Date: Tue, 16 Apr 2019 18:21:48 -0700
> 
> On Tue, 16 Apr 2019 at 17:20, Jaime Caamaño Ruiz <jcaamano at suse.de>
> wrote:
> > 
> > My intention was doing it at systemd unit (prestart) for file
> > conf.db
> > (and .conf.db.~lock~ that stays around) only.
> 
> You also need to chown /var/log/openvswitch.*.log files.
> > 
> > I was not thinking on absolutely fool proof mechanism, among other
> > things because the admin might have customized the location for the
> > database. Also, updating and restarting the service are things that
> > would usually be monitored. So best effort.
> 
> make sure that best effort solution informs admin in case of error so
> that he does not have to go file by file to find the offending one.
> 
> > 
> > At systemd unit prestart the service is stopped and OVS nor nobody
> > should really be messing with the database files owned by OVS. If
> > the
> > spec file is changing things back to root unappropriately then that
> > should be changed too.
> 
> Yes.
> 
> > 
> > The runtime /run/openvswitch directory is also controlled by
> > systemd
> > and at least on my system is removed if the service is not active,
> > either by graceful or ungraceful exit.
> 
> I somewhat agree with this, because back in those days we were using
> SystemV where the init.d managed lifecycle of run directory..
> However,
> what about other daemons, like ovn? Do they share run time dir?
> Haven't looked into this in a while and don't have a setup to check
> myself.
> 
> > 
> > I guess we will need to look for specific issues. Any hint to find
> > that
> > patch?
> 
> 1. test upgrade to your patched OVS and also from your patched OVS to
> another future version
> 2. if you change existing spec files that are shared with on Fedora
> or
> RHEL/CentOS make sure you don't break anything for them.
> 3. prefer consistency w.r.t. default behavior across deb and rpm
> packages (if you change user by default consider to do that for all
> platforms)
> 4. besides database you will also need to set right access bits to
> log
> files. On my Ubuntu it is "-rw-r----- 1 root adm " which means:
> 4.1. you have to chown it; OR
> 4.2. add openvswitch user to adm group and chmod that file
> 5. test logrotation. Maybe you can use a one forced logrotation
> invocation to change the ownrer.
> 6. check that other stuff like ovs-monitor-ipsec is not affected. It
> stil need to talk with StrongSwan or libreswan that best to my
> knowledge still runs as root.
> 7. if you use USER= then:
> 7.1. For Linux datapath make sure you retain CAP_NET_ADMIN Linux
> Capabilities
> 7.2. I don't know if there are special privileges you need to retain
> for DPDK. And in future possibly AF_XDP.
> 8. I haven't looked in a while but check if ovn (that has its own
> Systemd service file) shares run dir with OVS
> 
> > 
> > BR
> > Jaime.
> > 
> > -----Original Message-----
> > From: Ansis Atteka <ansisatteka at gmail.com>
> > To: jcaamano at suse.de
> > Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
> > Subject: Re: [ovs-discuss] Handling conf.db ownership on
> > OVS_USER_ID
> > changes
> > Date: Tue, 16 Apr 2019 16:11:44 -0700
> > 
> > On Tue, 16 Apr 2019 at 15:36, Jaime Caamaño Ruiz <jcaamano at suse.de>
> > wrote:
> > > 
> > > > On any given system, I would expect ovsdb-server to run as the
> > > > same
> > > > user
> > > > every time.  Is there a reason to sometimes use a different
> > > > user?
> > > 
> > > Well, changing from root which was the only supported option in
> > > the
> > > past and current default to a different user (like the suggested
> > > openvswitch), this one time, might be somewhat common. And the
> > > scenario
> > > that I am looking at is suporting a pianless upgrade through it.
> > 
> > Actually debian packages still by default use "root" user. Not
> > "openvswitch".
> > 
> > Long time ago there was an attempt to change the default user
> > across
> > all different flavor packages to openvswitch (you can lookup the
> > patch
> > on mailing list). However, as you noticed upgrades are tricky.
> > Hence
> > packages built from our debian/rules and rhel/openvswitch.spec.in
> > files still use "root" as default user. The packages built with
> > rhel/openvswitch-fedora.spec.in are an exception where the user
> > indeed
> > is "openvswitch". Unless you passed --without libcapng flag to
> > rpmbuild invocation. Then the user would still be "root".
> > 
> > Here are the difficulties with automating the change of file
> > ownership:
> > 1. from which context to change ownership? As chown must be invoked
> > from something that runs under root then there is not much choice -
> > either package installation time (%post scriptlet) or daemon
> > startup
> > time (assuming you are not using Systemd's USER= feature in spec
> > file)
> > or something that admin explicitly has to invoke from bash as
> > "root".
> > 1.1. if it is package installation time then you can't update user
> > gracefully at daemon init time. It will be possible only at
> > installation time.
> > 1.2. if it is installation time then you can run into weird race
> > conditions where some code that was executed under root (as package
> > %post scriptlets) or previous instance of OVS that was still
> > running
> > as root could change ownership back to root for some files and
> > cause
> > unexpected "permission denied" issues (the patch I mentioned was
> > affected with these rare race conditions)
> > 2. for which files to change ownership? What happens if Unix domain
> > socket file remained on fileystem with "root" permissions when OVS
> > was
> > abruptly killed? What if someone else put a file under ovs
> > directories
> > with non-root user - should you blindly chown that too? What if
> > %post
> > scriptlet that runs under root non-atomically created a file and in
> > next line chown() it back - is there a window for race? Should you
> > follow directories/symlinks recursively? Changing user for conf.db
> > file is not enough...
> > 
> > I am not saying it is impossible to change user gracefully on
> > upgrades. I am saying that code may have to be reorganized quite a
> > bit
> > to eliminate the issues I raised above.
> > 
> > Alternatively, for Centos, RHEL and Fedora one can use SElinux to
> > confine processes and leave them running as root (instead of
> > running
> > them under limited privilege user). I believe on SUSE (and Debian)
> > once could use AppArmor to achieve the same results. At least for
> > SElinux it was a little bit easier to make upgrade seamless than
> > with
> > Linux user confinement approach.
> > 
> > > 
> > > So if that makes sense to you, I will go ahead with the patch.
> > > 
> > > BR
> > > Jaime.
> > > 
> > > -----Original Message-----
> > > From: Ben Pfaff <blp at ovn.org>
> > > To: jcaamano at suse.de
> > > Cc: ovs-discuss at openvswitch.org
> > > Subject: Re: [ovs-discuss] Handling conf.db ownership on
> > > OVS_USER_ID
> > > changes
> > > Date: Tue, 16 Apr 2019 15:25:24 -0700
> > > 
> > > On Tue, Apr 16, 2019 at 11:32:21PM +0200, Jaime Caamaño Ruiz
> > > wrote:
> > > > When sysconfig OVS_USER_ID is changed to a different user, it
> > > > requires
> > > > a manual ownership change of the OVS conf.db database if
> > > > existing
> > > > or
> > > > otherwise ovsdb-server will fail to (re)start. I was wondering
> > > > if
> > > > I
> > > > am
> > > > missing any particular reason why this is change of ownership
> > > > is
> > > > not
> > > > automatically being handled through the service unit file as it
> > > > is
> > > > being done with other items.
> > > 
> > > On any given system, I would expect ovsdb-server to run as the
> > > same
> > > user
> > > every time.  Is there a reason to sometimes use a different user?
> > > 
> > > Or, perhaps you are saying that, just before invoking ovsdb-
> > > server,
> > > the
> > > service unit should chown (or whatever) the database file to the
> > > user
> > > that it is going to use to invoke ovsdb-server.  That might be
> > > reasonable, but no one has thought to do it yet.  Do you want to
> > > submit
> > > a patch?
> > > 
> > > Thanks,
> > > 
> > > Ben.
> > > 
> > > _______________________________________________
> > > discuss mailing list
> > > discuss at openvswitch.org
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > 
> > 
> 
> 