[ovs-dev] filename encoding in Python--any Python programmers in the house?

[Ben Pfaff]

Thanks for all the information.  After some experimentation this
morning, I think I'm pretty close to a solution good enough to make
the tests pass.

In general string encoding in Python seems to be a big mess.

On Mon, Oct 21, 2013 at 07:37:12PM -0700, Reid Price wrote:
> Apologies for the long answer.  Skip to the last two paragraphs for
> suggestions.
> 
> > "%s: %s %s" % (self.name, title, msg)
> 
> I've narrowed this down a bit.  The minimal reproduction is:
> 
>  # python -c '"%s%s" % ("\xc2", u"")'
>  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0:
> ordinal not in range(128)
> 
> Which is really the same as
> 
>  # python -c 'unicode("\xc2")'
> 
> This behavior results because there is a unicode string, which forces
> the entire string to be converted to unicode - it is actually 'msg' above
> causing the problem, which looks like
> 
>  u'request, method="exit", params=[], id=0'
> 
> via
> 
>  ['request', 'method="exit"', u'params=[]', u'id=0']
> 
> when I debugged 0369 (testing unixctl ovs-vswitchd exit).
> 
> I assume the "proper fix" involves converting everything everywhere to
> unicode.  Note that json decodes to unicode by default, so this will
> come up frequently.
> 
> If you converted the path name to utf-8 or the other portion to a string
> that would also allow the test to succeed, but I'm not sure how fragile
> that is.
> 
>    "\xc2\xab".decode('utf-8')
> 
> I tested a variety of fixes, but got lost at the ipsec tests, they seem
> to have their own version of interface-reconfigure hiding somewhere.
> If you want to fix these on a case-by-case basis, the easiest fix
> seems likely to be doing a string cast in all the places that unicode
> is used in combination with the path.
> 
> A slightly better fix might be doing the decode when the path is
> passed in or loaded (is a noop for normal strings), so that things
> are moving towards unicode.
> 
>   -Reid
> 
> 
> On Mon, Oct 21, 2013 at 10:26 AM, Ben Pfaff <blp at nicira.com> wrote:
> 
> > The Debian builds of Open vSwitch have suffered mysterious (to me)
> > errors for some time, e.g.:
> >
> > https://buildd.debian.org/status/fetch.php?pkg=openvswitch&arch=amd64&ver=1.9.3%2Bgit20131021-1&stamp=1382373784&file=log
> >
> > The root of these problems seems to be in filename encoding.  The Debian
> > buildds appear to build in directories named with gratuitous UTF-8, such
> > as:
> >         /???BUILDDIR???/openvswitch-1.9.3+git20131021/python/ovs/util.py
> >
> > When I try "make check" in a directory with a similar name on my own
> > system, I get pretty clear encoding-related failures in the same tests,
> > e.g.:
> >
> >     ../../tests/unixctl-py.at:10: $PYTHON $srcdir/appctl.py --timeout 20
> > -t ovs-vswitchd exit
> >     --- /dev/null   2013-07-07 19:52:44.697519595 -0700
> >     +++
> > /home/blp/nicira/???ovs???/_build/tests/testsuite.dir/at-groups/370/stderr
> >  2013-10-21 09:59:11.000000000 -0700
> >     @@ -0,0 +1,14 @@
> >     +Traceback (most recent call last):
> >     +  File "../../../../tests/appctl.py", line 75, in <module>
> >     +    main()
> >     +  File "../../../../tests/appctl.py", line 60, in main
> >     +    err_no, error, result = client.transact(args.command, args.argv)
> >     +  File "/home/blp/nicira/???ovs???/python/ovs/unixctl/client.py", line
> > 41, in transact
> >     +    error, reply = self._conn.transact_block(request)
> >     +  File "/home/blp/nicira/???ovs???/python/ovs/jsonrpc.py", line 306, in
> > transact_block
> >     +    error = self.send(request)
> >     +  File "/home/blp/nicira/???ovs???/python/ovs/jsonrpc.py", line 235, in
> > send
> >     +    self.__log_msg("send", msg)
> >     +  File "/home/blp/nicira/???ovs???/python/ovs/jsonrpc.py", line 229, in
> > __log_msg
> >     +    vlog.dbg("%s: %s %s" % (self.name, title, msg))
> >     +UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
> > 22: ordinal not in range(128)
> >
> > However I can't figure out the root cause of these problems.  In simple
> > tests with Python I don't get similar errors (note that the tests set
> > LC_ALL=C):
> >
> >     blp at blp:~/nicira/???ovs???/_build(0)$ LC_ALL=C python
> >     Python 2.7.3 (default, Jan  2 2013, 16:53:07)
> >     [GCC 4.7.2] on linux2
> >     Type "help", "copyright", "credits" or "license" for more information.
> >     >>> import os
> >     >>> 'prefix %s postfix' % os.getcwd()
> >     'prefix /home/blp/nicira/\xc2\xabovs\xc2\xbb/_build postfix'
> >     >>> print 'prefix %s postfix' % os.getcwd()
> >     prefix /home/blp/nicira/???ovs???/_build postfix
> >     >>>
> >
> > So I'm pretty puzzled.  Can anyone help me out?
> >
> > Thanks,
> >
> > Ben.
> >
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > http://openvswitch.org/mailman/listinfo/dev
> >
> >