Java crawler获取所有第一方和第三方cookies

Question

我想在Java中建立一个爬虫，让我从一个网站的所有cookie。这个爬虫被认为是自动抓取一个网站列表(很明显是下面的)。

我在我的计划中使用了jSoup和Selenium。

package com.mycompany.app;

import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.openqa.selenium.Cookie;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.*;

public class BasicWebCrawler {
    private static Set<String> uniqueURL = new HashSet<String>();
    private static List<String> link_list = new ArrayList<String>();

    private static Set<String> uniqueCookies = new HashSet<String>();

    private static void get_links(String url) {
        Connection connection = null;
        Connection.Response response = null;
        String this_link = null;

        try {
            connection = Jsoup.connect(url);
            response = connection.execute();

            //cookies_http = response.cookies();

            // fetch the document over HTTP
            Document doc = response.parse();

            // get all links in page
            Elements links = doc.select("a[href]");

            if(links.isEmpty()) {
                return;
            }

            for (Element link : links) {
                this_link = link.attr("href");

                boolean add = uniqueURL.add(this_link);

                System.out.println("\n" + this_link + "\n" + "title: " + doc.title());

                if (add && (this_link.contains(url))) {
                    System.out.println("\n" + this_link + "\n" + "title: " + doc.title());

                    link_list.add(this_link);

                    get_links(this_link);
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) {
        get_links("https://de.wikipedia.org/wiki/Wikipedia");

        /**
         * Hier kommt Selenium ins Spiel
         */
        WebDriver driver;

        System.setProperty("webdriver.chrome.driver", "D:\\crawler\\driver\\chromedriver.exe");

        driver = new ChromeDriver();

        // create file named Cookies to store Login Information
        File file = new File("Cookies.data");

        FileWriter fileWrite = null;
        BufferedWriter Bwrite = null;

        try {
            // Delete old file if exists
            file.delete();
            file.createNewFile();

            fileWrite = new FileWriter(file);
            Bwrite = new BufferedWriter(fileWrite);
            // loop for getting the cookie information
        } catch (Exception ex) {
            ex.printStackTrace();
        }


        for(String link : link_list) {
            System.out.println("Open Link: " + link);

            driver.get(link);

            try {
                // loop for getting the cookie information
                for (Cookie ck : driver.manage().getCookies()) {
                    String tmp = (ck.getName() + ";" + ck.getValue() + ";" + ck.getDomain() + ";" + ck.getPath() + ";" + ck.getExpiry() + ";" + ck.isSecure());

                    if(uniqueCookies.add(tmp)) {
                        Bwrite.write("Link: " + link + "\n" + (ck.getName() + ";" + ck.getValue() + ";" + ck.getDomain() + ";" + ck.getPath() + ";" + ck.getExpiry() + ";" + ck.isSecure())+ "\n\n");
                        Bwrite.newLine();
                    }
                }
            } catch (Exception ex) {
                ex.printStackTrace();
            }
        }

        try {
            Bwrite.close();
            fileWrite.close();

            driver.close();
            driver.quit();
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

我在wikipedia页面上测试了这段代码，并将结果与一个名为CookieMetrix的cookie扫描器进行了比较。

我的代码只显示了四个cookie：

Link: https://de.wikipedia.org/wiki/Wikipedia:Lizenzbestimmungen_Commons_Attribution-ShareAlike_3.0_Unported
GeoIP;DE:NW:M__nster:51.95:7.54:v4;.wikipedia.org;/;null;true


Link: https://de.wikipedia.org/wiki/Wikipedia:Lizenzbestimmungen_Commons_Attribution-ShareAlike_3.0_Unported
WMF-Last-Access-Global;13-May-2019;.wikipedia.org;/;Mon Jan 19 02:28:33 CET 1970;true


Link: https://de.wikipedia.org/wiki/Wikipedia:Lizenzbestimmungen_Commons_Attribution-ShareAlike_3.0_Unported
WMF-Last-Access;13-May-2019;de.wikipedia.org;/;Mon Jan 19 02:28:33 CET 1970;true


Link: https://de.wikipedia.org/wiki/Wikipedia:Lizenzbestimmungen_Commons_Attribution-ShareAlike_3.0_Unported
mwPhp7Seed;55e;de.wikipedia.org;/;Mon Jan 19 03:09:08 CET 1970;false

但曲奇扫描仪显示有七个。我不知道为什么我的代码比CookieMetrix显示的要少。你能帮帮我吗?

Answer 1

用于java.util.Set getCookies()的JavaDoc

获取当前域的所有cookies。这相当于调用"document.cookie"并解析结果的等价物

1. document.cookie不会返回 HttpOnly cookie，原因很简单，因为JavaScript不允许这样做。
2. 还注意到，“CookieMetrix”似乎列出了来自不同域的cookie。

解决方案：

• 要获得诸如“CookieMetrix”(1+2)之类的列表，您可以在浏览器后添加一个代理并嗅探请求。
• 如果你想获取当前域名的所有cookies，包括HttpOnly (1)，你可以尝试直接访问Chrome(当然，它也会返回HttpOnly cookies)