如何使用python从json文件中删除重复条目？

Question

如何使用python从JSON文件中删除重复条目？

我有一个JSON文件，如下所示：

感激一些人可以帮助提供一个解决方案来修复它。

json_data = [
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9099",
                    "aks9098",
                    "aks9100",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9103",
                    "aks9103"
                    
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chinq.n@example.com"
    },
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9098",
                    "aks9098",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9102",
                    "aks9103"
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chinw.d@example.com"
    },
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9098",
                    "aks9100",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9102",
                    "aks9103"
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chins.b@example.com"
    }
]

我想从列表中删除重复条目，预期结果如下：

感谢您可以帮助提供一个解决方案来修复它。

json_data = [
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9098",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9103"
                    
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chinq.n@example.com"
    },
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9098",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9103"
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chinw.d@example.com"
    },
    {
        "authType": "ldap",
        "password": "",
        "permissions": [
            {
                "collections": [
                    "aks9099",
                    "aks9098",
                    "aks9100",
                    "aks9101",
                    "aks9102",
                    "aks9103"
                ],
                "project": "Central Project"
            }
        ],
        "role": "devSecOps",
        "username": "chins.b@example.com"
    }
]

Answer 1

下面的方法能解决你的问题吗？

new_list=[]
for i in json_data:
    if not i in new_list:
        new_list.append(i)
print(new_list)

Answer 2

尽管OP要求使用Python来完成这一任务，但在jq中可以很容易地使用函数unique完成这一任务，只需一个更新分配：

$ jq '.[].permissions[].collections |= unique' json.txt 
[
  {
    "authType": "ldap",
    "password": "",
    "permissions": [
      {
        "collections": [
          "aks9098",
          "aks9099",
          "aks9100",
          "aks9101",
          "aks9102",
          "aks9103"
        ],
        "project": "Central Project"
      }
    ],
    "role": "devSecOps",
    "username": "chinq.n@example.com"
  },
  {
    "authType": "ldap",
    "password": "",
    "permissions": [
      {
        "collections": [
          "aks9098",
          "aks9099",
          "aks9100",
          "aks9101",
          "aks9102",
          "aks9103"
        ],
        "project": "Central Project"
      }
    ],
    "role": "devSecOps",
    "username": "chinw.d@example.com"
  },
  {
    "authType": "ldap",
    "password": "",
    "permissions": [
      {
        "collections": [
          "aks9098",
          "aks9099",
          "aks9100",
          "aks9101",
          "aks9102",
          "aks9103"
        ],
        "project": "Central Project"
      }
    ],
    "role": "devSecOps",
    "username": "chins.b@example.com"
  }
]

要在Python中调用它，可以这样做：

import subprocess
return_obj = subprocess.run(["jq", ".[].permissions[].collections |= unique","json.txt"], stdout=subprocess.PIPE)
json_data = return_obj.stdout.decode()