旁通炉墙硒

Question

我想刮从荷兰的招聘列表网站上的职位清单。然而，当我尝试用selenium打开页面时，我遇到了一个高墙(新的GDPR规则)。我该怎么绕过厨房墙？

import selenium 

#launch url
url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO"

# create a new Firefox session
driver = webdriver.Firefox()
driver.implicitly_wait(30)
driver.get(url)

编辑我尝试了一些

import selenium 
import pickle

url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO"

driver = webdriver.Firefox()
driver.set_page_load_timeout(20)
driver.get(start_url)

pickle.dump(driver.get_cookies() , open("NVBCookies.pkl","wb"))

在加载之后，cookie不起作用了。

for cookie in pickle.load(open("NVBCookies.pkl", "rb")):
    driver.add_cookie(cookie)

InvalidCookieDomainException: Message: Cookies may only be set for the current domain (cookiewall.vnumediaonline.nl)

我好像没有从厨房的墙上拿到饼干，对吧？

Answer 1

driver.find_element_by_xpath('//*[@id="form_save"]').click()

好的，我让selenium点击接受按钮。我也很好。不知道以后我会不会碰到厨房墙

Answer 2

与其绕过它，不如编写代码来检查它是否存在，然后接受它，否则继续下一个操作。请查找下面的代码以获得更多详细信息。

import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys


class PythonOrgSearch(unittest.TestCase):

    def setUp(self):

        self.driver = webdriver.Chrome(executable_path="C:\\Users\\USER\\Downloads\\New folder (2)\\chromedriver_win32\\chromedriver.exe")

    def test_search_in_python_org(self):
        driver = self.driver
        driver.get("https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO")

        elem = driver.find_element_by_xpath("//div[@class='article__button']//button[@id='form_save']")
        elem.click()

    def tearDown(self):
        self.driver.close()

if __name__ == "__main__":
    unittest.main()