[ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

Mon May 6 10:15:35 UTC 2019

> Agree. I will try to address this issue. I think we can have a
> separate
> run time/log directory for OVN. ovn-controller needs to talk to the
> local ovsdb-server
> and br-int.mgmt and other related socket interfaces, so it needs
> access
> to
> the /var/run/openvswitch/ folder. I think we can solve this.

I have a (yet to be submitted patch) on my side that changes OVN to run
as the same user as OVS. Thought it as less disruptive than also
changing the log directory.

openvswitch-ipsec still logs as root though and I dont see a way around
that so it probably needs to log to a different directory.

Jaime.

-----Original Message-----
From: Numan Siddique <nusiddiq at redhat.com>
To: Aaron Conole <aconole at redhat.com>
Cc: Jaime Caamaño Ruiz <jcaamano at suse.de>, ovs-discuss <ovs-discuss at ope
nvswitch.org>
Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
changes
Date: Thu, 2 May 2019 19:20:28 +0530

On Mon, Apr 29, 2019, 9:36 PM Aaron Conole <aconole at redhat.com> wrote:
> Jaime Caamaño Ruiz <jcaamano at suse.de> writes:
> 
> >> As a "security concern" you mean something among the lines where
> one
> >> of ovs-* processes running under openvswitch user would go ahead
> and
> >> create a file with its owner that later one of ovn processes would
> >> blindly reuse without checking that it actually belongs to
> root:root?
> >
> > One example is that the OVS user could create link as any OVN log
> file
> > to other file owned by root and then OVN would write to that file.
> 
> There's a long-standing issue of OVN running as root.  I don't see
> any
> reason it should.

Agree. I will try to address this issue. I think we can have a separate
run time/log directory for OVN. ovn-controller needs to talk to the
local ovsdb-server
and br-int.mgmt and other related socket interfaces, so it needs access
to
the /var/run/openvswitch/ folder. I think we can solve this.

Thanks
Numan

> >> Can you give a more concrete example? I believe logrotate is
> running
> >> under root and should be able to rotate everything?
> >
> > The logrotate configuration for openvswitch logs has a 'su'
> directive
> > to run under the openvswitch user, precisely to prevent something
> > similar to the above. So it wont be able to rotate root owned logs
> in
> > the openvswitch directory. How it fails precisely depends on global
> > logrotate configuration. For example it will fail to create the new
> log
> > file after rotation if the 'create' directive is used. Or it may
> fail
> > to compress the rotated log file as it wont be able to read it.
> >
> > BR
> > Jaime
> >
> > -----Original Message-----
> > From: Ansis Atteka <ansisatteka at gmail.com>
> > To: jcaamano at suse.de
> > Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
> > Subject: Re: [ovs-discuss] Handling conf.db ownership on
> OVS_USER_ID
> > changes
> > Date: Wed, 24 Apr 2019 19:07:24 -0700
> >
> > On Tue, 23 Apr 2019 at 09:30, Jaime Caamaño Ruiz <jcaamano at suse.de>
> > wrote:
> >> 
> >> Hello
> >> 
> >> So the non root owned log directory (and run directory) is shared
> >> between non root OVS processes and root OVN processes. Doesn't
> this
> >> raise some security concerns?
> >
> > As a "security concern" you mean something among the lines where
> one
> > of ovs-* processes running under openvswitch user would go ahead
> and
> > create a file with its owner that later one of ovn processes would
> > blindly reuse without checking that it actually belongs to
> root:root?
> >
> >
> >> 
> >> Also, since logrotate rotates the logs with the OVS user, it may
> fail
> >> to some extent to rotate the root owned OVN log files.
> Consequences
> >> vary but in it's current state some logging might be lost. This
> could
> >> be improved but I would guess that the approprioate thing to do is
> to
> >> etiher use a different log directory for OVN or make its processes
> >> run
> >> with the OVS user also. Has any of this been considered?
> >
> > Can you give a more concrete example? I believe logrotate is
> running
> > under root and should be able to rotate everything?
> >
> >
> >> 
> >> BR
> >> Jaime.
> >> 
> >> 
> >> -----Original Message-----
> >> From: Jaime Caamaño Ruiz <jcaamano at suse.de>
> >> Reply-to: jcaamano at suse.com
> >> To: Ansis Atteka <ansisatteka at gmail.com>
> >> Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
> >> Subject: Re: [ovs-discuss] Handling conf.db ownership on
> OVS_USER_ID
> >> changes
> >> Date: Wed, 17 Apr 2019 12:52:32 +0200
> >> 
> >> > You also need to chown /var/log/openvswitch.*.log files.
> >> 
> >> OVS seems to be already handling this. I dont know the details but
> I
> >> guess that before dropping capabilities, OVS chowns these by
> itself.
> >> 
> >> > However,
> >> > what about other daemons, like ovn? Do they share run time dir?
> >> > Haven't looked into this in a while and don't have a setup to
> check
> >> > myself.
> >> 
> >> The permisions of the openvswitch run directory are already being
> >> handled by the service unit file. There is read access to files
> >> created
> >> there by OVS and AFAIK what it ultimately makes it a no problem
> for
> >> OVN
> >> is that it still runs as root with the provided unit files. If
> anyone
> >> changes this to a user different than the openvswitch user, then
> that
> >> is an already existing issue. Am I missing something here?
> >> 
> >> Agree with you other considerations.
> >> 
> >> BR
> >> Jaime.
> >> 
> >> -----Original Message-----
> >> From: Ansis Atteka <ansisatteka at gmail.com>
> >> To: jcaamano at suse.de
> >> Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
> >> Subject: Re: [ovs-discuss] Handling conf.db ownership on
> OVS_USER_ID
> >> changes
> >> Date: Tue, 16 Apr 2019 18:21:48 -0700
> >> 
> >> On Tue, 16 Apr 2019 at 17:20, Jaime Caamaño Ruiz <jcaamano at suse.de
> >
> >> wrote:
> >> > 
> >> > My intention was doing it at systemd unit (prestart) for file
> >> > conf.db
> >> > (and .conf.db.~lock~ that stays around) only.
> >> 
> >> You also need to chown /var/log/openvswitch.*.log files.
> >> > 
> >> > I was not thinking on absolutely fool proof mechanism, among
> other
> >> > things because the admin might have customized the location for
> the
> >> > database. Also, updating and restarting the service are things
> that
> >> > would usually be monitored. So best effort.
> >> 
> >> make sure that best effort solution informs admin in case of error
> so
> >> that he does not have to go file by file to find the offending
> one.
> >> 
> >> > 
> >> > At systemd unit prestart the service is stopped and OVS nor
> nobody
> >> > should really be messing with the database files owned by OVS.
> If
> >> > the
> >> > spec file is changing things back to root unappropriately then
> that
> >> > should be changed too.
> >> 
> >> Yes.
> >> 
> >> > 
> >> > The runtime /run/openvswitch directory is also controlled by
> >> > systemd
> >> > and at least on my system is removed if the service is not
> active,
> >> > either by graceful or ungraceful exit.
> >> 
> >> I somewhat agree with this, because back in those days we were
> using
> >> SystemV where the init.d managed lifecycle of run directory..
> >> However,
> >> what about other daemons, like ovn? Do they share run time dir?
> >> Haven't looked into this in a while and don't have a setup to
> check
> >> myself.
> >> 
> >> > 
> >> > I guess we will need to look for specific issues. Any hint to
> find
> >> > that
> >> > patch?
> >> 
> >> 1. test upgrade to your patched OVS and also from your patched OVS
> to
> >> another future version
> >> 2. if you change existing spec files that are shared with on
> Fedora
> >> or
> >> RHEL/CentOS make sure you don't break anything for them.
> >> 3. prefer consistency w.r.t. default behavior across deb and rpm
> >> packages (if you change user by default consider to do that for
> all
> >> platforms)
> >> 4. besides database you will also need to set right access bits to
> >> log
> >> files. On my Ubuntu it is "-rw-r----- 1 root adm " which means:
> >> 4.1. you have to chown it; OR
> >> 4.2. add openvswitch user to adm group and chmod that file
> >> 5. test logrotation. Maybe you can use a one forced logrotation
> >> invocation to change the ownrer.
> >> 6. check that other stuff like ovs-monitor-ipsec is not affected.
> It
> >> stil need to talk with StrongSwan or libreswan that best to my
> >> knowledge still runs as root.
> >> 7. if you use USER= then:
> >> 7.1. For Linux datapath make sure you retain CAP_NET_ADMIN Linux
> >> Capabilities
> >> 7.2. I don't know if there are special privileges you need to
> retain
> >> for DPDK. And in future possibly AF_XDP.
> >> 8. I haven't looked in a while but check if ovn (that has its own
> >> Systemd service file) shares run dir with OVS
> >> 
> >> > 
> >> > BR
> >> > Jaime.
> >> > 
> >> > -----Original Message-----
> >> > From: Ansis Atteka <ansisatteka at gmail.com>
> >> > To: jcaamano at suse.de
> >> > Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
> >> > Subject: Re: [ovs-discuss] Handling conf.db ownership on
> >> > OVS_USER_ID
> >> > changes
> >> > Date: Tue, 16 Apr 2019 16:11:44 -0700
> >> > 
> >> > On Tue, 16 Apr 2019 at 15:36, Jaime Caamaño Ruiz <jcaamano at suse.
> de>
> >> > wrote:
> >> > > 
> >> > > > On any given system, I would expect ovsdb-server to run as
> the
> >> > > > same
> >> > > > user
> >> > > > every time.  Is there a reason to sometimes use a different
> >> > > > user?
> >> > > 
> >> > > Well, changing from root which was the only supported option
> in
> >> > > the
> >> > > past and current default to a different user (like the
> suggested
> >> > > openvswitch), this one time, might be somewhat common. And the
> >> > > scenario
> >> > > that I am looking at is suporting a pianless upgrade through
> it.
> >> > 
> >> > Actually debian packages still by default use "root" user. Not
> >> > "openvswitch".
> >> > 
> >> > Long time ago there was an attempt to change the default user
> >> > across
> >> > all different flavor packages to openvswitch (you can lookup the
> >> > patch
> >> > on mailing list). However, as you noticed upgrades are tricky.
> >> > Hence
> >> > packages built from our debian/rules and
> rhel/openvswitch.spec.in
> >> > files still use "root" as default user. The packages built with
> >> > rhel/openvswitch-fedora.spec.in are an exception where the user
> >> > indeed
> >> > is "openvswitch". Unless you passed --without libcapng flag to
> >> > rpmbuild invocation. Then the user would still be "root".
> >> > 
> >> > Here are the difficulties with automating the change of file
> >> > ownership:
> >> > 1. from which context to change ownership? As chown must be
> invoked
> >> > from something that runs under root then there is not much
> choice -
> >> > either package installation time (%post scriptlet) or daemon
> >> > startup
> >> > time (assuming you are not using Systemd's USER= feature in spec
> >> > file)
> >> > or something that admin explicitly has to invoke from bash as
> >> > "root".
> >> > 1.1. if it is package installation time then you can't update
> user
> >> > gracefully at daemon init time. It will be possible only at
> >> > installation time.
> >> > 1.2. if it is installation time then you can run into weird race
> >> > conditions where some code that was executed under root (as
> package
> >> > %post scriptlets) or previous instance of OVS that was still
> >> > running
> >> > as root could change ownership back to root for some files and
> >> > cause
> >> > unexpected "permission denied" issues (the patch I mentioned was
> >> > affected with these rare race conditions)
> >> > 2. for which files to change ownership? What happens if Unix
> domain
> >> > socket file remained on fileystem with "root" permissions when
> OVS
> >> > was
> >> > abruptly killed? What if someone else put a file under ovs
> >> > directories
> >> > with non-root user - should you blindly chown that too? What if
> >> > %post
> >> > scriptlet that runs under root non-atomically created a file and
> in
> >> > next line chown() it back - is there a window for race? Should
> you
> >> > follow directories/symlinks recursively? Changing user for
> conf.db
> >> > file is not enough...
> >> > 
> >> > I am not saying it is impossible to change user gracefully on
> >> > upgrades. I am saying that code may have to be reorganized quite
> a
> >> > bit
> >> > to eliminate the issues I raised above.
> >> > 
> >> > Alternatively, for Centos, RHEL and Fedora one can use SElinux
> to
> >> > confine processes and leave them running as root (instead of
> >> > running
> >> > them under limited privilege user). I believe on SUSE (and
> Debian)
> >> > once could use AppArmor to achieve the same results. At least
> for
> >> > SElinux it was a little bit easier to make upgrade seamless than
> >> > with
> >> > Linux user confinement approach.
> >> > 
> >> > > 
> >> > > So if that makes sense to you, I will go ahead with the patch.
> >> > > 
> >> > > BR
> >> > > Jaime.
> >> > > 
> >> > > -----Original Message-----
> >> > > From: Ben Pfaff <blp at ovn.org>
> >> > > To: jcaamano at suse.de
> >> > > Cc: ovs-discuss at openvswitch.org
> >> > > Subject: Re: [ovs-discuss] Handling conf.db ownership on
> >> > > OVS_USER_ID
> >> > > changes
> >> > > Date: Tue, 16 Apr 2019 15:25:24 -0700
> >> > > 
> >> > > On Tue, Apr 16, 2019 at 11:32:21PM +0200, Jaime Caamaño Ruiz
> >> > > wrote:
> >> > > > When sysconfig OVS_USER_ID is changed to a different user,
> it
> >> > > > requires
> >> > > > a manual ownership change of the OVS conf.db database if
> >> > > > existing
> >> > > > or
> >> > > > otherwise ovsdb-server will fail to (re)start. I was
> wondering
> >> > > > if
> >> > > > I
> >> > > > am
> >> > > > missing any particular reason why this is change of
> ownership
> >> > > > is
> >> > > > not
> >> > > > automatically being handled through the service unit file as
> it
> >> > > > is
> >> > > > being done with other items.
> >> > > 
> >> > > On any given system, I would expect ovsdb-server to run as the
> >> > > same
> >> > > user
> >> > > every time.  Is there a reason to sometimes use a different
> user?
> >> > > 
> >> > > Or, perhaps you are saying that, just before invoking ovsdb-
> >> > > server,
> >> > > the
> >> > > service unit should chown (or whatever) the database file to
> the
> >> > > user
> >> > > that it is going to use to invoke ovsdb-server.  That might be
> >> > > reasonable, but no one has thought to do it yet.  Do you want
> to
> >> > > submit
> >> > > a patch?
> >> > > 
> >> > > Thanks,
> >> > > 
> >> > > Ben.
> >> > > 
> >> > > _______________________________________________
> >> > > discuss mailing list
> >> > > discuss at openvswitch.org
> >> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >> > 
> >> > 
> >> 
> >> 
> >
> >
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss