擦拭概览

Question

我不知道为什么我不能浏览一下这个公司的概况。举个例子，我想刮一下沃尔玛的规模，那就是10000+的员工。下面是我的代码，不知道为什么我要找的信息不在那里.

import requests
from bs4 import BeautifulSoup
import pandas as pd 


headers = {'user-agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.80 Safari/537.36'}
url = f'https://www.glassdoor.com/Overview/Working-at-Walmart-EI_IE715.11,18.htm'
    # f'https://www.glassdoor.com/Reviews/Google-Engineering-Reviews-EI_IE9079.0,6_DEPT1007_IP{pg}.htm?sort.sortType=RD&sort.ascending=false&filter.iso3Language=eng'

r = requests.get(url, headers)
soup = BeautifulSoup(r.content, 'html.parser')

我非常感谢任何帮助刮这个“大小”因素在公司网页上。

Answer 1

以下是一个可能的解决方案：

import re
import json
import requests
from bs4 import BeautifulSoup


headers = {
    'user-agent': 'Mozilla/5.0'
}

with requests.Session() as session:
    session.headers.update(headers)
    raw_data = session.get(f'https://www.glassdoor.com/Overview/Working-at-Walmart-EI_IE715.htm').text
    
    script = [s.text for s in BeautifulSoup(raw_data, "lxml").find_all("script") if "window.appCache" in s.text][0]
    json_data = json.loads(re.findall(r'(\"Employer:\d+\":)(.+)(,\"ROOT_QUERY\")', script)[0][1])

    data = {
        "id": json_data["id"],
        "shortName": json_data["shortName"],
        "website": json_data["website"],
        "type": json_data["type"],
        "revenue": json_data["revenue"],
        "headquarters": json_data["headquarters"],
        "size": json_data["size"],
        "yearFounded": json_data["yearFounded"]
    }
    
    print(data)

输出：

{
  'id': 715,
  'shortName': 'Walmart',
  'website': 'careers.walmart.com',
  'type': 'Company - Public',
  'revenue': '$10+ billion (USD)',
  'headquarters': 'Bentonville, AR',
  'size': '10000+ Employees',
  'yearFounded': 1962
}

如果您只需要“大小”，那么只需使用例如size = json_data["size"]