基于位置的Shodan查询结果过滤

Question

我试图解析Shodan查询结果，并编写一个新的JSON文件，其中只包含与我设置的条件相匹配的结果。

示例JSON条目：

{
  "matches": [
    {
      "product": "Microsoft IIS httpd",
      "hostnames": [],
      "hash": -1722221328,
      "ip": 1261462342,
      "isp": "AT&T Internet Services",
      "transport": "tcp",
      "cpe": [
        "cpe:/a:microsoft:iis:7.5",
        "cpe:/o:microsoft:windows"
      ],
      "data": "",
      "asn": "AS7018",
      "port": 631,
      "version": "7.5",
      "link": "Ethernet or modem",
      "location": {
        "city": null,
        "region_code": null,
        "area_code": null,
        "longitude": -97.822,
        "country_code3": "USA",
        "latitude": 37.751000000000005,
        "postal_code": null,
        "dma_code": null,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:55:24.306344",
      "domains": [],
      "org": "AT&T Internet Services",
      "os": null,
      "_shodan": {
        "crawler": "122dd688b363c3b45b0e7582622da1e725444808",
        "id": null,
        "module": "http-simple-new",
        "options": {}
      },
      "ip_str": "75.48.99.70"
    },
    {
      "hash": 605323305,
      "ip": 1757819678,
      "isp": "Google Cloud",
      "transport": "tcp",
      "data": "",
      "asn": "AS15169",
      "port": 9000,
      "hostnames": [
        "30.51.198.104.bc.googleusercontent.com"
      ],
      "location": {
        "city": "Mountain View",
        "region_code": "CA",
        "area_code": 650,
        "longitude": -122.0574,
        "country_code3": "USA",
        "latitude": 37.41919999999999,
        "postal_code": "94043",
        "dma_code": 807,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:51:35.997036",
      "domains": [
        "googleusercontent.com"
      ],
      "org": "Google Cloud",
      "os": null,
      "_shodan": {
        "crawler": "545144fc95e7a7ef13ece5dbceb98ee386b37950",
        "id": null,
        "module": "https-simple-new",
        "options": {}
      },
      "ip_str": "104.198.51.30"
    }
  ],
  "total": 2
}

我希望加载JSON文件并遍历一组元素，如果元素不符合"US“位置country_code的条件，则删除它。

我拥有的代码(由https://gist.github.com/madonnelly和Iterate over JsonObject properties提供)如下：

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Map;
import java.util.Set;

import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;

public class ParseJSON {

    public static void main(String[] args) {
        JsonObject shodanJSON = convertFileToJSON("<Path to JSON file>");

        Set<Map.Entry<String,JsonElement>> queryResults = shodanJSON.entrySet();

        for (Map.Entry<String, JsonElement> queryResult : queryResults) {
            JsonArray locArray = queryResult.getValue().getAsJsonObject().getAsJsonArray("location");
            for (JsonElement locData : locArray) {
                if (locData.getAsJsonObject().getAsJsonPrimitive("country_code").equals("US")) {
                    System.out.println(locData.getAsString());
                }
            }
        }
    }

    public static JsonObject convertFileToJSON(String fileName) {

        // Read from File to String
        JsonObject jsonObject = new JsonObject();

        try {
            JsonParser parser = new JsonParser();
            JsonElement jsonElement = parser.parse(new FileReader(fileName));
            jsonObject = jsonElement.getAsJsonObject();
        } catch (FileNotFoundException e) {

        }
        return jsonObject;
    }
}

当我运行我的代码时，我会收到错误。

线程"main“中的异常:不是JSON对象：[{”产品“：”Microsoft“，”主机名“：[]，”散列“：-1722221328，"ip":1261462342，"isp":" at &T因特网服务”，“传输”：.}]在com.google.gson.JsonElement.getAsJsonObject(JsonElement.java:90) at com.cti.shodan.ParseJSON.main(ParseJSON.java:22)

我确信我犯了很多错误，我希望有人能指出我所犯的错误。提前感谢！

Answer 1

对于具体的JSON文档解析，您有一些不符合其实际结构的假设。我假设您将显示匹配的结果($.matches)，筛选出它们的子属性值($.matches.*.location.country_code)。

    for ( final Entry<String, JsonElement> queryResult : shodanJsonObject.entrySet() ) {
        final JsonElement value = queryResult.getValue();
        // This is necessary to skip <"total": 2>
        if ( value.isJsonArray() ) {
            // Here comes an array, and should be iterated, rather than taken as an object
            for ( final JsonElement match : value.getAsJsonArray() ) {
                // This was the root cause, not an array
                final JsonObject location = match.getAsJsonObject().getAsJsonObject("location");
                // Previously jsonPrimitive.equals("US") -- convert the JSON primitive to a string first
                if ( location.getAsJsonPrimitive("country_code").getAsString().equals("US") ) {
                    // Previously getAsString() -- it requires a JSON string literal, just remove it
                    System.out.println(match);
                }
            }
        }
    }

对于Java 8，它可能要稍微简单一些：

shodanJsonObject.entrySet()
        .stream()
        .map(Entry::getValue)
        .filter(JsonElement::isJsonArray)
        .map(JsonElement::getAsJsonArray)
        .flatMap(jsonElements -> StreamSupport.stream(jsonElements.spliterator(), false))
        .peek(System.out::println)
        .map(JsonElement::getAsJsonObject)
        .map(jsonObject -> jsonObject.getAsJsonObject("location"))
        .filter(location -> location.getAsJsonPrimitive("country_code").getAsString().equals("US"))
        .forEach(jsonObject -> {
        }); // forEach is a terminal operation and it "pushes" the entire chain above

如果有可能使用像JsonPath这样的查询库，那么可能是最有表现力的方式：

final JsonPath jsonPath = JsonPath.compile("$.matches.*[?(@.location.country_code=='US')]");
for ( final Object match : jsonPath.<JSONArray>read(JSON) ) {
    System.out.println(match);
}