使用shlex进行Fortigate设备字符串解析和拆分

Question

考虑Fortigate配置，如

config firewall policy
    edit 168
        set name "policy 168"
        set uuid 14435052-3097-4d70-98c7-1dd2d60e229f
        set srcintf "jimmylin__1688"
        set dstintf "port1"
        set srcaddr "address__jimmylin__10.100.168.11/32"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "\"customer\": \"Jimmy Lin\""
        set nat enable
        set ippool enable
        set poolname "ippool__jimmylin__168.100.168.11"
    next
end

(这是一个.conf，类似于纯文本文件)

假设文件位于/conf/中，名为firewall_policy.conf。我们想把它放到python中，这样我们就可以完成围绕它的所有数据处理工作，所以我们就这样做了。

>>> with open(file='/conf/firewall_policy.conf', mode='r', encoding='utf-8') as f:
>>>     lines = f.read().splitlines()

现在，让我们更深入地研究它。

>>> lines[2]
'        set name "policy 168"'
>>> print(lines[2])
        set name "policy 168"
>>> lines[10]
'        set service "ALL"'
>>> print(lines[10])
        set service "ALL"
>>> lines[11]
'        set comments "\\"customer\\": \\"Jimmy Lin\\""'
>>> print(lines[11])
        set comments "\"customer\": \"Jimmy Lin\""

我们想要的东西是

>>> def parse(string):
...     # some string-parsing things
...     return string
...
>>> parse(lines[2])
['set', 'name', '"policy 168"']
>>> parse(lines[10])
['set', 'service', '"ALL"']
>>> parse(lines[11])
['set', 'comments', '"\\"customer\\": \\"Jimmy Lin\\""'']

如果我们只是用

>>> def parse(string):
...    return string.split()
...

那我们就能

>>> parse(lines[2])
['set', 'name', '"policy', '168"']

显然这不是我们想要的。

如果我们尝试使用shlex -简单的词法分析，Python标准库中的一个模块。

>>> from shlex import split as shlex_split
>>> def parse(string):
...    return shlex_split(string)
...

那我们就能

>>> parse(lines[2])
['set', 'name', 'policy 168']

太好了，我们可以把他们重新联系起来

>>> from shlex import join as shlex_join
>>> shlex_join(parse(lines[2]))
"set name 'policy 168'"

这几乎与原始字符串相同，但是围绕Policy168的双引号变成了单引号。不过这是可以接受的。

但说到lines10

>>> parse(lines[10])
['set', 'name', 'ALL']
>>> shlex_join(parse(lines[10]))
'set service ALL'

双引号就会消失，因为ALL不是一个包含空格的字符串，而shlex不认为需要在它周围添加引号。

我查阅了shlex文档，它是一个名为posix的参数。

>>> from shlex import split as shlex_split
>>> def parse(string):
...    return shlex_split(string, posix=False)
...
>>> parse(lines[2])
['set', 'service', '"ALL"']

我们可以简单地使用内置的join将其返回。

>>> ' '.join(parse(lines[2]))
'set service "ALL"'

但就lines11而言，情况并非如此。

>>> ' '.join(parse(lines[11]))
['set', 'comments', '"\\"', 'customer\\":', '\\"Jimmy', 'Lin\\""']

我认为这是解析Forti设备和shlex模块之间字符串的不同方法。Forti设备看到的是

'set comments \'"customer": "Jimmy Lin"\''

“客户”：“林吉米”

而shlex看到的是

'set comments "\\"customer\\":  \\"Jimmy Lin\\""'

如果我们使用.replace(‘\“，’)，它看起来会更好

>>> lines[11].replace('\\"', '\'')
'        set comments "\'customer\': \'Jimmy Lin\'"'
>>> parse(lines[11].replace('\\"', '\''))
['set', 'comments', '"\'customer\': \'Jimmy Lin\'"']

但是这是一种肮脏的方式，如果有更多的转义字符或更复杂的嵌套事物，我认为它可能会失败，而且join部件也不正确。

>>> ' '.join(parse(lines[11].replace('\\"', '\'')))
'set comments "\'customer\': \'Jimmy Lin\'"'

是否有一个解决方案使这些解析过程正确和干净。我是不是遗漏了什么？

Answer 1

我想这就是你想要的。对不起，我不是为每一行做任何函数，而是解析所有的一个for循环。如果需要，可以将for循环插入到函数中。代码注释了进一步的信息。

# coding: utf-8

import shlex

with open(file='firewall_policy.conf', mode='r', encoding='utf-8') as f:
    lines = f.read().splitlines()

for i in range(len(lines)):
    line = lines[i].strip() # stripping each line because there are some spaces
    if line.startswith('set'): # checking whether it starts with the 'set' word
        if '\\' in line:
            lexer1 = shlex.shlex(line, posix=True) # posix stays true
            print(list(lexer1))
        else:
            lexer2 = shlex.shlex(line) # else another lexer
            lexer2.quotes = '"' # quotes are preserved
            lexer2.whitespace_split = True # using whitespaces to split
            print(list(lexer2))
    else:
        pass

输出：

['set', 'name', '"policy 168"']
['set', 'uuid', '14435052-3097-4d70-98c7-1dd2d60e229f']
['set', 'srcintf', '"jimmylin__1688"']
['set', 'dstintf', '"port1"']
['set', 'srcaddr', '"address__jimmylin__10.100.168.11/32"']
['set', 'dstaddr', '"all"']
['set', 'action', 'accept']
['set', 'schedule', '"always"']
['set', 'service', '"ALL"']
['set', 'comments', '"customer": "Jimmy Lin"']
['set', 'nat', 'enable']
['set', 'ippool', 'enable']
['set', 'poolname', '"ippool__jimmylin__168.100.168.11"']

这里唯一的问题是\消失了。但是，所有其他的事情都是好的。这是我使用shlex所能解决的问题。显然，参数posix = True必须在‘Python3.8.x中传递。

只使用Regex的替代工作解决方案。

import re


def quote_preserved_split(line):
    """The Regex preserve outer quotes and also the escape characters"""
    return re.findall("(?:\".*?[^\\\]\"|\S)+", line)

with open(file='firewall_policy.conf', mode='r', encoding='utf-8') as f:
    lines = f.read().splitlines()

for i in range(len(lines)):
    line = lines[i].strip()
    if line.startswith('set'):
        print(quote_preserved_split(line))

输出：

['set', 'name', '"policy 168"']
['set', 'uuid', '14435052-3097-4d70-98c7-1dd2d60e229f']
['set', 'srcintf', '"jimmylin__1688"']
['set', 'dstintf', '"port1"']
['set', 'srcaddr', '"address__jimmylin__10.100.168.11/32"']
['set', 'dstaddr', '"all"']
['set', 'action', 'accept']
['set', 'schedule', '"always"']
['set', 'service', '"ALL"']
['set', 'comments', '"\\"customer\\": \\"Jimmy Lin\\""']
['set', 'nat', 'enable']
['set', 'ippool', 'enable']
['set', 'poolname', '"ippool__jimmylin__168.100.168.11"']

通过使用join()连接每一行，您将得到：

set name "policy 168"
set uuid 14435052-3097-4d70-98c7-1dd2d60e229f
set srcintf "jimmylin__1688"
set dstintf "port1"
set srcaddr "address__jimmylin__10.100.168.11/32"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set comments "\"customer\": \"Jimmy Lin\""
set nat enable
set ippool enable
set poolname "ippool__jimmylin__168.100.168.11"

Answer 2

如果目标是："..to把它放到python中，这样我们就可以完成围绕它的所有数据处理工作“，并且您不介意使用其他TTP库，那么您可能想看看TTP和下面的代码示例：

from ttp import ttp
import pprint

data = """
config firewall policy
    edit 168
        set name "policy 168"
        set uuid 14435052-3097-4d70-98c7-1dd2d60e229f
        set srcintf "jimmylin__1688"
        set dstintf "port1"
        set srcaddr "address__jimmylin__10.100.168.11/32"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "\"customer\": \"Jimmy Lin\""
        set nat enable
        set ippool enable
        set poolname "ippool__jimmylin__168.100.168.11"
    next
    edit 200
        set name "policy 200"
        set uuid 14435052-3097-4d70-98c7-1dd2d60e2200
        set srcintf "jimmylin__16"
        set dstintf "port2"
    next
end
"""

template = """
<group name="policies**">
config firewall policy {{ _start_ }}
    <group name="{{ name }}**">
    edit {{ id }}
        set name "{{ name | ORPHRASE }}"
        set uuid {{ uuid }}
        set srcintf "{{ srcintf }}"
        set dstintf "{{ dstintf }}"
        set srcaddr "{{ srcaddr }}"
        set dstaddr "{{ dstaddr }}"
        set action {{ action }}
        set schedule "{{ schedule}}"
        set service "{{ service}}"
        set comments {{ comments | ORPHRASE | replace('"', '') }}
        set nat enable {{ nat_enabled | set(True) }}
        set ippool enable {{ ippool_enabled | set(True) }}
        set poolname "{{ poolname }}"    
    next {{ _end_ }}
    </group>
end {{ _end_ }}
</group>
"""

parser = ttp(data, template)
parser.parse()
res = parser.result()
pprint.pprint(res, width=100)

# will print:
# [[{'policies': {'policy 168': {'action': 'accept',
#                                'comments': 'customer: Jimmy Lin',
#                                'dstaddr': 'all',
#                                'dstintf': 'port1',
#                                'id': '168',
#                                'ippool_enabled': True,
#                                'nat_enabled': True,
#                                'poolname': 'ippool__jimmylin__168.100.168.11',
#                                'schedule': 'always',
#                                'service': 'ALL',
#                                'srcaddr': 'address__jimmylin__10.100.168.11/32',
#                                'srcintf': 'jimmylin__1688',
#                                'uuid': '14435052-3097-4d70-98c7-1dd2d60e229f'},
#                 'policy 200': {'dstintf': 'port2',
#                                'id': '200',
#                                'srcintf': 'jimmylin__16',
#                                'uuid': '14435052-3097-4d70-98c7-1dd2d60e2200'}}}]]