[ovs-dev] filename encoding in Python--any Python programmers in the house?

[Reid Price]

Apologies for the long answer.  Skip to the last two paragraphs for
suggestions.

> "%s: %s %s" % (self.name, title, msg)

I've narrowed this down a bit.  The minimal reproduction is:

 # python -c '"%s%s" % ("\xc2", u"")'
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0:
ordinal not in range(128)

Which is really the same as

 # python -c 'unicode("\xc2")'

This behavior results because there is a unicode string, which forces
the entire string to be converted to unicode - it is actually 'msg' above
causing the problem, which looks like

 u'request, method="exit", params=[], id=0'

via

 ['request', 'method="exit"', u'params=[]', u'id=0']

when I debugged 0369 (testing unixctl ovs-vswitchd exit).

I assume the "proper fix" involves converting everything everywhere to
unicode.  Note that json decodes to unicode by default, so this will
come up frequently.

If you converted the path name to utf-8 or the other portion to a string
that would also allow the test to succeed, but I'm not sure how fragile
that is.

   "\xc2\xab".decode('utf-8')

I tested a variety of fixes, but got lost at the ipsec tests, they seem
to have their own version of interface-reconfigure hiding somewhere.
If you want to fix these on a case-by-case basis, the easiest fix
seems likely to be doing a string cast in all the places that unicode
is used in combination with the path.

A slightly better fix might be doing the decode when the path is
passed in or loaded (is a noop for normal strings), so that things
are moving towards unicode.

  -Reid


On Mon, Oct 21, 2013 at 10:26 AM, Ben Pfaff <blp at nicira.com> wrote:

> The Debian builds of Open vSwitch have suffered mysterious (to me)
> errors for some time, e.g.:
>
> https://buildd.debian.org/status/fetch.php?pkg=openvswitch&arch=amd64&ver=1.9.3%2Bgit20131021-1&stamp=1382373784&file=log
>
> The root of these problems seems to be in filename encoding.  The Debian
> buildds appear to build in directories named with gratuitous UTF-8, such
> as:
>         /芦BUILDDIR禄/openvswitch-1.9.3+git20131021/python/ovs/util.py
>
> When I try "make check" in a directory with a similar name on my own
> system, I get pretty clear encoding-related failures in the same tests,
> e.g.:
>
>     ../../tests/unixctl-py.at:10: $PYTHON $srcdir/appctl.py --timeout 20
> -t ovs-vswitchd exit
>     --- /dev/null   2013-07-07 19:52:44.697519595 -0700
>     +++
> /home/blp/nicira/芦ovs禄/_build/tests/testsuite.dir/at-groups/370/stderr
>  2013-10-21 09:59:11.000000000 -0700
>     @@ -0,0 +1,14 @@
>     +Traceback (most recent call last):
>     +  File "../../../../tests/appctl.py", line 75, in <module>
>     +    main()
>     +  File "../../../../tests/appctl.py", line 60, in main
>     +    err_no, error, result = client.transact(args.command, args.argv)
>     +  File "/home/blp/nicira/芦ovs禄/python/ovs/unixctl/client.py", line
> 41, in transact
>     +    error, reply = self._conn.transact_block(request)
>     +  File "/home/blp/nicira/芦ovs禄/python/ovs/jsonrpc.py", line 306, in
> transact_block
>     +    error = self.send(request)
>     +  File "/home/blp/nicira/芦ovs禄/python/ovs/jsonrpc.py", line 235, in
> send
>     +    self.__log_msg("send", msg)
>     +  File "/home/blp/nicira/芦ovs禄/python/ovs/jsonrpc.py", line 229, in
> __log_msg
>     +    vlog.dbg("%s: %s %s" % (self.name, title, msg))
>     +UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
> 22: ordinal not in range(128)
>
> However I can't figure out the root cause of these problems.  In simple
> tests with Python I don't get similar errors (note that the tests set
> LC_ALL=C):
>
>     blp at blp:~/nicira/芦ovs禄/_build(0)$ LC_ALL=C python
>     Python 2.7.3 (default, Jan  2 2013, 16:53:07)
>     [GCC 4.7.2] on linux2
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> import os
>     >>> 'prefix %s postfix' % os.getcwd()
>     'prefix /home/blp/nicira/\xc2\xabovs\xc2\xbb/_build postfix'
>     >>> print 'prefix %s postfix' % os.getcwd()
>     prefix /home/blp/nicira/芦ovs禄/_build postfix
>     >>>
>
> So I'm pretty puzzled.  Can anyone help me out?
>
> Thanks,
>
> Ben.
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> http://openvswitch.org/mailman/listinfo/dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-dev/attachments/20131021/c4eb5d1c/attachment-0003.html>