为什么PyYAML/ruamel在Python2上组合转义的代理？

Question

对于在YAML文件中指定了一些测试用例的Python项目，我想在YAML测试用例中指定UTF-8编码的字节字符串，包括无效的UTF-8序列，例如不成对的代理或交换的高/低代理，目的是测试Python项目对它们的处理。

我在测试用例中使用双引号标量YAML (因为这是在YAML中指定转义序列的唯一方法)和16位unicode转义(例如"\u03B5")来指定代理代码点。

我们的Python项目使用PyYAML加载测试用例YAML文件，但我验证了以下结果也适用于ruamel。下面是一个使用PyYAML的示例(除非另有说明，否则在Python2.7和3.7上，以及在使用CPython的macOS上)：

>>> import yaml  # PyYAML

# some valid Unicode character (lower case greek epsilon, fwiw)
>>> yaml.safe_load('x: "\\u03B5"')  
{'x': u'\u03b5'}

# invalid unpaired surrogate
>>> yaml.safe_load('x: "\\uD800"')  
{'x': u'\ud800'}

# invalid swapped high/low surrogates
>>> yaml.safe_load('x: "\\uDC00\\uD800"')
{'x': u'\udc00\ud800'}

# valid surrogates, but using surrogates is illegal in UTF-8
>>> yaml.safe_load('x: "\\uD800\\uDC00"')  
{'x': '\ud800\udc00'}  # on Python 3.7
{'x': u'\U00010000'}   # on Python 2.7  <-- why??

上面最后一种情况在Python2.7上的行为使得我们不可能将这种情况传递给我们的Python项目，因为它被PyYAML或Python本身“更正”了。

这种“修正”发生在哪里，可以禁用它吗？

Answer 1

我不确定你使用的是什么版本，但我不能重现这个。似乎只有libyaml对您的输入有问题：

from __future__ import print_function

import sys
print('Python:     ', sys.version_info[:2])
import yaml as pyyaml
print('PyYAML:     ', pyyaml.__version__)
import ruamel.yaml
print('ruamel.yaml:', ruamel.yaml.version_info)

yaml_str = 'x: "\\uD800\\uDC00"'
print('yaml_str', yaml_str)

print(repr(pyyaml.safe_load(yaml_str)['x']))



for typ, pure in [('rt', True), ('safe', True), ('safe', False)]:
    yaml = ruamel.yaml.YAML(typ=typ, pure=pure)
    data = yaml.load(yaml_str)
    print(repr(data['x']))

提供：

Python:      (2, 7)
PyYAML:      5.1.2
ruamel.yaml: (0, 16, 6, u'dev')
yaml_str x: "\uD800\uDC00"
u'\ud800\udc00'
u'\ud800\udc00'
u'\ud800\udc00'
Traceback (most recent call last):
  File "/tmp/ryd-of-anthon/ryd-111/tmp_0.py", line 20, in <module>
    data = yaml.load(yaml_str)
  File "/home/anthon/.venv/27/lib/python2.7/site-packages/ruamel/yaml/main.py", line 341, in load
    return constructor.get_single_data()
  File "/home/anthon/.venv/27/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
    node = self.composer.get_single_node()
  File "_ruamel_yaml.pyx", line 706, in _ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 724, in _ruamel_yaml.CParser._compose_document
  File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
  File "_ruamel_yaml.pyx", line 731, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event
ruamel.yaml.scanner.ScannerError: while parsing a quoted scalar
  in "<byte string>", line 1, column 4
found invalid Unicode character escape code
  in "<byte string>", line 1, column 7