刮玻璃门返回重复条目

Question

因此，我正在尝试使用请求，美丽汤和硒从Glassdoor上抓取工作岗位。除了从30页中抓取数据后，大多数条目都是重复的(其中几乎80%是重复的)，整个代码都能工作。这不是一个无头刮刀，所以我知道它将在每一个新的页。为什么会有这么多重复的条目？它可能是玻璃门使用的某种防刮工具，还是在我的代码中出现了问题？

结果是870个条目，其中一个惊人的690是重复的！

我的代码：

def glassdoor_scraper(url):
    
    driver = webdriver.Chrome()
    driver.get(url)
    time.sleep(10)
    
    # Getting to the page where we want to start scraping
    jobs_search_title = driver.find_element(By.ID, 'KeywordSearch')
    jobs_search_title.send_keys('Data Analyst')
    jobs_search_location = driver.find_element(By.ID, 'LocationSearch')
    
    time.sleep(1)
    
    jobs_search_location.clear()
    jobs_search_location.send_keys('United States')
    click_search = driver.find_element(By.ID, 'HeroSearchButton')
    click_search.click()
    
    for page_num in range(1,10):
        time.sleep(10)
        
        res = requests.get(driver.current_url)
        soup = BeautifulSoup(res.text,'html.parser')
        
        time.sleep(2)

        companies = soup.select('.css-l2wjgv.e1n63ojh0.jobLink')
        for company in companies:
            companies_list.append(company.text)
    
        positions = soup.select('.jobLink.css-1rd3saf.eigr9kq2')
        for position in positions:
            positions_list.append(position.text)
    
        locations = soup.select('.css-l2fjlt.pr-xxsm.css-iii9i8.e1rrn5ka0')
        for location in locations:
            locations_list.append(location.text)
    
        job_post = soup.select('.eigr9kq3')
        for job in job_post:
            salary_info = job.select('.e1wijj242')
            if len(salary_info) > 0:
                for salary in salary_info:
                    salaries_list.append(salary.text)
            else:
                salaries_list.append('Salary Not Found')
    
        ratings = soup.select('.e1rrn5ka3')
        for index, rating in enumerate(ratings):
            if len(rating.text) > 0:
                ratings_list.append(rating.text)
            else:
                ratings_list.append('Rating Not Found')
        
        
        next_page = driver.find_elements(By.CLASS_NAME, 'e13qs2073')[1]
        next_page.click()
        time.sleep(5)
        try:
            close_jobalert_popup = driver.find_element(By.CLASS_NAME, 'modal_closeIcon')
        except:
            pass
        else:
            time.sleep(1)
            close_jobalert_popup.click()        
        continue
    
    #driver.close()
    print(f'{len(companies_list)} jobs found for you!')
    
    global glassdoor_dataset
    
    glassdoor_dataset = pd.DataFrame(
    {'Company Name': companies_list,
     'Company Rating': ratings_list,
     'Position Title': positions_list,
     'Location' : locations_list,
     'Est. Salary' : salaries_list
    })
    
    glassdoor_dataset.to_csv(r'glassdoor_jobs_scraped.csv')

Answer 1

你开得太快了。你需要做些等待。

我看你已经把含蓄的。试着用明显的等待代替。

就像这样：

)把你自己的条件说出来。您也可以尝试使用不可见元素。就像某些东西是不可见的，然后是可见的，以确保你现在在下一页)，如果没有，那么增加你的time.sleep()

WebDriverWait(driver, 40).until(expected_conditions.visibility_of_element_located(
    (By.XPATH, '//*[@id="wrapper"]/section/div/div/div[2]/button[2]')))

Answer 2

我不认为重复是由于一个代码问题-我认为玻璃门只是开始循环结果过一段时间。[如果感兴趣的话，请看一些数据的要点。 --基本上，从第7页左右开始，第一页的大部分结果似乎都显示在每一页之前。我手动做了一个小测试-只有5个清单，通过id，甚至直接在一个非自动浏览器上，他们开始重复一段时间后.]

我的建议是在循环到下一页之前只对它们进行过滤--每个围绕清单的li都有一个li属性，这似乎是一个唯一的标识符。如果将其添加到其他列的列表中，则只能开始收集未收集的列表；如果只编辑for page_num循环以：

    for page_num in range(1, 10):
        time.sleep(10)

        scrapedUrls.append(driver.current_url)

        res = requests.get(driver.current_url)
        soup = BeautifulSoup(res.text, 'html.parser')
        # soup = BeautifulSoup(driver.page_source, 'html.parser') # no noticable improvement

        time.sleep(2)

        filteredListings = [
            di for di in soup.select('li[data-id]') if
            di.get('data-id') not in datId_list
        ]
        datId_list += [di.get('data-id') for di in filteredListings] 

        companies_list += [
            t.select_one('.css-l2wjgv.e1n63ojh0.jobLink').get_text(strip=True)
            if t.select_one('.css-l2wjgv.e1n63ojh0.jobLink')
            else None for t in filteredListings
        ]

        positions_list += [
            t.select_one('.jobLink.css-1rd3saf.eigr9kq2').get_text(strip=True)
            if t.select_one('.jobLink.css-1rd3saf.eigr9kq2')
            else None for t in filteredListings
        ]

        locations_list += [
            t.select_one(
                '.css-l2fjlt.pr-xxsm.css-iii9i8.e1rrn5ka0').get_text(strip=True)
            if t.select_one('.css-l2fjlt.pr-xxsm.css-iii9i8.e1rrn5ka0')
            else None for t in filteredListings
        ]

        job_post = [
            t.select('.eigr9kq3 .e1wijj242') for t in filteredListings
        ]
        salaries_list += [
            'Salary Not Found' if not j else
            (j[0].text if len(j) == 1 else [s.text for s in j])
            for j in job_post
        ]

        ratings_list += [
            t.select_one('.e1rrn5ka3').get_text(strip=True)
            if t.select_one('.e1rrn5ka3')
            else 'Rating Not Found' for t in filteredListings
        ]

而且，如果将datId_list添加到dataframe中，它可以作为一个有意义的索引。

    dfDict = {'Data-Id': datId_list,
              'Company Name': companies_list,
              'Company Rating': ratings_list,
              'Position Title': positions_list,
              'Location': locations_list,
              'Est. Salary': salaries_list
              }
    for k in dfDict:
        print(k, len(dfDict[k]))
    glassdoor_dataset = pd.DataFrame(dfDict)
    glassdoor_dataset.set_index('Data-Id', drop=True)

    glassdoor_dataset.to_csv(r'glassdoor_jobs_scraped.csv')