[ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

Aaron Conole aconole at redhat.com
Mon May 6 12:33:59 UTC 2019


Jaime Caamaño Ruiz <jcaamano at suse.de> writes:

>> Agree. I will try to address this issue. I think we can have a
>> separate
>> run time/log directory for OVN. ovn-controller needs to talk to the
>> local ovsdb-server
>> and br-int.mgmt and other related socket interfaces, so it needs
>> access
>> to
>> the /var/run/openvswitch/ folder. I think we can solve this.
>
> I have a (yet to be submitted patch) on my side that changes OVN to run
> as the same user as OVS. Thought it as less disruptive than also
> changing the log directory.

That sounds interesting.

> openvswitch-ipsec still logs as root though and I dont see a way around
> that so it probably needs to log to a different directory.
>
> Jaime.
>
> -----Original Message-----
> From: Numan Siddique <nusiddiq at redhat.com>
> To: Aaron Conole <aconole at redhat.com>
> Cc: Jaime Caamaño Ruiz <jcaamano at suse.de>, ovs-discuss <ovs-discuss at ope
> nvswitch.org>
> Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
> changes
> Date: Thu, 2 May 2019 19:20:28 +0530
>
>
>
>
> On Mon, Apr 29, 2019, 9:36 PM Aaron Conole <aconole at redhat.com> wrote:
>> Jaime Caamaño Ruiz <jcaamano at suse.de> writes:
>> 
>> >> As a "security concern" you mean something among the lines where
>> one
>> >> of ovs-* processes running under openvswitch user would go ahead
>> and
>> >> create a file with its owner that later one of ovn processes would
>> >> blindly reuse without checking that it actually belongs to
>> root:root?
>> >
>> > One example is that the OVS user could create link as any OVN log
>> file
>> > to other file owned by root and then OVN would write to that file.
>> 
>> There's a long-standing issue of OVN running as root.  I don't see
>> any
>> reason it should.
>
> Agree. I will try to address this issue. I think we can have a separate
> run time/log directory for OVN. ovn-controller needs to talk to the
> local ovsdb-server
> and br-int.mgmt and other related socket interfaces, so it needs access
> to
> the /var/run/openvswitch/ folder. I think we can solve this.
>
> Thanks
> Numan
>
>> >> Can you give a more concrete example? I believe logrotate is
>> running
>> >> under root and should be able to rotate everything?
>> >
>> > The logrotate configuration for openvswitch logs has a 'su'
>> directive
>> > to run under the openvswitch user, precisely to prevent something
>> > similar to the above. So it wont be able to rotate root owned logs
>> in
>> > the openvswitch directory. How it fails precisely depends on global
>> > logrotate configuration. For example it will fail to create the new
>> log
>> > file after rotation if the 'create' directive is used. Or it may
>> fail
>> > to compress the rotated log file as it wont be able to read it.
>> >
>> > BR
>> > Jaime
>> >
>> > -----Original Message-----
>> > From: Ansis Atteka <ansisatteka at gmail.com>
>> > To: jcaamano at suse.de
>> > Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
>> > Subject: Re: [ovs-discuss] Handling conf.db ownership on
>> OVS_USER_ID
>> > changes
>> > Date: Wed, 24 Apr 2019 19:07:24 -0700
>> >
>> > On Tue, 23 Apr 2019 at 09:30, Jaime Caamaño Ruiz <jcaamano at suse.de>
>> > wrote:
>> >> 
>> >> Hello
>> >> 
>> >> So the non root owned log directory (and run directory) is shared
>> >> between non root OVS processes and root OVN processes. Doesn't
>> this
>> >> raise some security concerns?
>> >
>> > As a "security concern" you mean something among the lines where
>> one
>> > of ovs-* processes running under openvswitch user would go ahead
>> and
>> > create a file with its owner that later one of ovn processes would
>> > blindly reuse without checking that it actually belongs to
>> root:root?
>> >
>> >
>> >> 
>> >> Also, since logrotate rotates the logs with the OVS user, it may
>> fail
>> >> to some extent to rotate the root owned OVN log files.
>> Consequences
>> >> vary but in it's current state some logging might be lost. This
>> could
>> >> be improved but I would guess that the approprioate thing to do is
>> to
>> >> etiher use a different log directory for OVN or make its processes
>> >> run
>> >> with the OVS user also. Has any of this been considered?
>> >
>> > Can you give a more concrete example? I believe logrotate is
>> running
>> > under root and should be able to rotate everything?
>> >
>> >
>> >> 
>> >> BR
>> >> Jaime.
>> >> 
>> >> 
>> >> -----Original Message-----
>> >> From: Jaime Caamaño Ruiz <jcaamano at suse.de>
>> >> Reply-to: jcaamano at suse.com
>> >> To: Ansis Atteka <ansisatteka at gmail.com>
>> >> Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
>> >> Subject: Re: [ovs-discuss] Handling conf.db ownership on
>> OVS_USER_ID
>> >> changes
>> >> Date: Wed, 17 Apr 2019 12:52:32 +0200
>> >> 
>> >> > You also need to chown /var/log/openvswitch.*.log files.
>> >> 
>> >> OVS seems to be already handling this. I dont know the details but
>> I
>> >> guess that before dropping capabilities, OVS chowns these by
>> itself.
>> >> 
>> >> > However,
>> >> > what about other daemons, like ovn? Do they share run time dir?
>> >> > Haven't looked into this in a while and don't have a setup to
>> check
>> >> > myself.
>> >> 
>> >> The permisions of the openvswitch run directory are already being
>> >> handled by the service unit file. There is read access to files
>> >> created
>> >> there by OVS and AFAIK what it ultimately makes it a no problem
>> for
>> >> OVN
>> >> is that it still runs as root with the provided unit files. If
>> anyone
>> >> changes this to a user different than the openvswitch user, then
>> that
>> >> is an already existing issue. Am I missing something here?
>> >> 
>> >> Agree with you other considerations.
>> >> 
>> >> BR
>> >> Jaime.
>> >> 
>> >> -----Original Message-----
>> >> From: Ansis Atteka <ansisatteka at gmail.com>
>> >> To: jcaamano at suse.de
>> >> Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
>> >> Subject: Re: [ovs-discuss] Handling conf.db ownership on
>> OVS_USER_ID
>> >> changes
>> >> Date: Tue, 16 Apr 2019 18:21:48 -0700
>> >> 
>> >> On Tue, 16 Apr 2019 at 17:20, Jaime Caamaño Ruiz <jcaamano at suse.de
>> >
>> >> wrote:
>> >> > 
>> >> > My intention was doing it at systemd unit (prestart) for file
>> >> > conf.db
>> >> > (and .conf.db.~lock~ that stays around) only.
>> >> 
>> >> You also need to chown /var/log/openvswitch.*.log files.
>> >> > 
>> >> > I was not thinking on absolutely fool proof mechanism, among
>> other
>> >> > things because the admin might have customized the location for
>> the
>> >> > database. Also, updating and restarting the service are things
>> that
>> >> > would usually be monitored. So best effort.
>> >> 
>> >> make sure that best effort solution informs admin in case of error
>> so
>> >> that he does not have to go file by file to find the offending
>> one.
>> >> 
>> >> > 
>> >> > At systemd unit prestart the service is stopped and OVS nor
>> nobody
>> >> > should really be messing with the database files owned by OVS.
>> If
>> >> > the
>> >> > spec file is changing things back to root unappropriately then
>> that
>> >> > should be changed too.
>> >> 
>> >> Yes.
>> >> 
>> >> > 
>> >> > The runtime /run/openvswitch directory is also controlled by
>> >> > systemd
>> >> > and at least on my system is removed if the service is not
>> active,
>> >> > either by graceful or ungraceful exit.
>> >> 
>> >> I somewhat agree with this, because back in those days we were
>> using
>> >> SystemV where the init.d managed lifecycle of run directory..
>> >> However,
>> >> what about other daemons, like ovn? Do they share run time dir?
>> >> Haven't looked into this in a while and don't have a setup to
>> check
>> >> myself.
>> >> 
>> >> > 
>> >> > I guess we will need to look for specific issues. Any hint to
>> find
>> >> > that
>> >> > patch?
>> >> 
>> >> 1. test upgrade to your patched OVS and also from your patched OVS
>> to
>> >> another future version
>> >> 2. if you change existing spec files that are shared with on
>> Fedora
>> >> or
>> >> RHEL/CentOS make sure you don't break anything for them.
>> >> 3. prefer consistency w.r.t. default behavior across deb and rpm
>> >> packages (if you change user by default consider to do that for
>> all
>> >> platforms)
>> >> 4. besides database you will also need to set right access bits to
>> >> log
>> >> files. On my Ubuntu it is "-rw-r----- 1 root adm " which means:
>> >> 4.1. you have to chown it; OR
>> >> 4.2. add openvswitch user to adm group and chmod that file
>> >> 5. test logrotation. Maybe you can use a one forced logrotation
>> >> invocation to change the ownrer.
>> >> 6. check that other stuff like ovs-monitor-ipsec is not affected.
>> It
>> >> stil need to talk with StrongSwan or libreswan that best to my
>> >> knowledge still runs as root.
>> >> 7. if you use USER= then:
>> >> 7.1. For Linux datapath make sure you retain CAP_NET_ADMIN Linux
>> >> Capabilities
>> >> 7.2. I don't know if there are special privileges you need to
>> retain
>> >> for DPDK. And in future possibly AF_XDP.
>> >> 8. I haven't looked in a while but check if ovn (that has its own
>> >> Systemd service file) shares run dir with OVS
>> >> 
>> >> > 
>> >> > BR
>> >> > Jaime.
>> >> > 
>> >> > -----Original Message-----
>> >> > From: Ansis Atteka <ansisatteka at gmail.com>
>> >> > To: jcaamano at suse.de
>> >> > Cc: ovs-discuss at openvswitch.org, Ben Pfaff <blp at ovn.org>
>> >> > Subject: Re: [ovs-discuss] Handling conf.db ownership on
>> >> > OVS_USER_ID
>> >> > changes
>> >> > Date: Tue, 16 Apr 2019 16:11:44 -0700
>> >> > 
>> >> > On Tue, 16 Apr 2019 at 15:36, Jaime Caamaño Ruiz <jcaamano at suse.
>> de>
>> >> > wrote:
>> >> > > 
>> >> > > > On any given system, I would expect ovsdb-server to run as
>> the
>> >> > > > same
>> >> > > > user
>> >> > > > every time.  Is there a reason to sometimes use a different
>> >> > > > user?
>> >> > > 
>> >> > > Well, changing from root which was the only supported option
>> in
>> >> > > the
>> >> > > past and current default to a different user (like the
>> suggested
>> >> > > openvswitch), this one time, might be somewhat common. And the
>> >> > > scenario
>> >> > > that I am looking at is suporting a pianless upgrade through
>> it.
>> >> > 
>> >> > Actually debian packages still by default use "root" user. Not
>> >> > "openvswitch".
>> >> > 
>> >> > Long time ago there was an attempt to change the default user
>> >> > across
>> >> > all different flavor packages to openvswitch (you can lookup the
>> >> > patch
>> >> > on mailing list). However, as you noticed upgrades are tricky.
>> >> > Hence
>> >> > packages built from our debian/rules and
>> rhel/openvswitch.spec.in
>> >> > files still use "root" as default user. The packages built with
>> >> > rhel/openvswitch-fedora.spec.in are an exception where the user
>> >> > indeed
>> >> > is "openvswitch". Unless you passed --without libcapng flag to
>> >> > rpmbuild invocation. Then the user would still be "root".
>> >> > 
>> >> > Here are the difficulties with automating the change of file
>> >> > ownership:
>> >> > 1. from which context to change ownership? As chown must be
>> invoked
>> >> > from something that runs under root then there is not much
>> choice -
>> >> > either package installation time (%post scriptlet) or daemon
>> >> > startup
>> >> > time (assuming you are not using Systemd's USER= feature in spec
>> >> > file)
>> >> > or something that admin explicitly has to invoke from bash as
>> >> > "root".
>> >> > 1.1. if it is package installation time then you can't update
>> user
>> >> > gracefully at daemon init time. It will be possible only at
>> >> > installation time.
>> >> > 1.2. if it is installation time then you can run into weird race
>> >> > conditions where some code that was executed under root (as
>> package
>> >> > %post scriptlets) or previous instance of OVS that was still
>> >> > running
>> >> > as root could change ownership back to root for some files and
>> >> > cause
>> >> > unexpected "permission denied" issues (the patch I mentioned was
>> >> > affected with these rare race conditions)
>> >> > 2. for which files to change ownership? What happens if Unix
>> domain
>> >> > socket file remained on fileystem with "root" permissions when
>> OVS
>> >> > was
>> >> > abruptly killed? What if someone else put a file under ovs
>> >> > directories
>> >> > with non-root user - should you blindly chown that too? What if
>> >> > %post
>> >> > scriptlet that runs under root non-atomically created a file and
>> in
>> >> > next line chown() it back - is there a window for race? Should
>> you
>> >> > follow directories/symlinks recursively? Changing user for
>> conf.db
>> >> > file is not enough...
>> >> > 
>> >> > I am not saying it is impossible to change user gracefully on
>> >> > upgrades. I am saying that code may have to be reorganized quite
>> a
>> >> > bit
>> >> > to eliminate the issues I raised above.
>> >> > 
>> >> > Alternatively, for Centos, RHEL and Fedora one can use SElinux
>> to
>> >> > confine processes and leave them running as root (instead of
>> >> > running
>> >> > them under limited privilege user). I believe on SUSE (and
>> Debian)
>> >> > once could use AppArmor to achieve the same results. At least
>> for
>> >> > SElinux it was a little bit easier to make upgrade seamless than
>> >> > with
>> >> > Linux user confinement approach.
>> >> > 
>> >> > > 
>> >> > > So if that makes sense to you, I will go ahead with the patch.
>> >> > > 
>> >> > > BR
>> >> > > Jaime.
>> >> > > 
>> >> > > -----Original Message-----
>> >> > > From: Ben Pfaff <blp at ovn.org>
>> >> > > To: jcaamano at suse.de
>> >> > > Cc: ovs-discuss at openvswitch.org
>> >> > > Subject: Re: [ovs-discuss] Handling conf.db ownership on
>> >> > > OVS_USER_ID
>> >> > > changes
>> >> > > Date: Tue, 16 Apr 2019 15:25:24 -0700
>> >> > > 
>> >> > > On Tue, Apr 16, 2019 at 11:32:21PM +0200, Jaime Caamaño Ruiz
>> >> > > wrote:
>> >> > > > When sysconfig OVS_USER_ID is changed to a different user,
>> it
>> >> > > > requires
>> >> > > > a manual ownership change of the OVS conf.db database if
>> >> > > > existing
>> >> > > > or
>> >> > > > otherwise ovsdb-server will fail to (re)start. I was
>> wondering
>> >> > > > if
>> >> > > > I
>> >> > > > am
>> >> > > > missing any particular reason why this is change of
>> ownership
>> >> > > > is
>> >> > > > not
>> >> > > > automatically being handled through the service unit file as
>> it
>> >> > > > is
>> >> > > > being done with other items.
>> >> > > 
>> >> > > On any given system, I would expect ovsdb-server to run as the
>> >> > > same
>> >> > > user
>> >> > > every time.  Is there a reason to sometimes use a different
>> user?
>> >> > > 
>> >> > > Or, perhaps you are saying that, just before invoking ovsdb-
>> >> > > server,
>> >> > > the
>> >> > > service unit should chown (or whatever) the database file to
>> the
>> >> > > user
>> >> > > that it is going to use to invoke ovsdb-server.  That might be
>> >> > > reasonable, but no one has thought to do it yet.  Do you want
>> to
>> >> > > submit
>> >> > > a patch?
>> >> > > 
>> >> > > Thanks,
>> >> > > 
>> >> > > Ben.
>> >> > > 
>> >> > > _______________________________________________
>> >> > > discuss mailing list
>> >> > > discuss at openvswitch.org
>> >> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>> >> > 
>> >> > 
>> >> 
>> >> 
>> >
>> >
>> > _______________________________________________
>> > discuss mailing list
>> > discuss at openvswitch.org
>> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


More information about the discuss mailing list