首页
学习
活动
专区
圈层
工具
发布
社区首页 >问答首页 >如何使用BeautifulSoup从职务发布中提取隐藏的表数据?

问如何使用BeautifulSoup从职务发布中提取隐藏的表数据?
EN

Stack Overflow用户
提问于 2022-09-07 01:18:02
回答 2查看 65关注 0票数 2

嗨,我在做python课程,今天的任务之一,我们要在:https://remoteok.com/remote-python-jobs上提取工作清单

下面是一个有关html的屏幕截图:python f12

到目前为止,我写的是:

代码语言:javascript
复制
import requests
from bs4 import BeautifulSoup

def extract(term):
  url = f"https://remoteok.com/remote-{term}-jobs"
  request = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
  if request.status_code == 200:
    soup = BeautifulSoup(request.text, 'html.parser')
    table = soup.find_all('table', id="jobsboard")
    print(len(table))
    for tbody in table:
      tbody.find_all('tbody')
      print(len(tbody))
  else:
    print("can't request website")

extract("python")

print(len(table))给我1,print(len(tbody))给我131。

很明显,我在某个地方犯了一个错误,但我很难找出原因。

我怀疑,当我请求html文本并使用BeautifulSoup解析它时,我并没有得到完整的网页。但除此之外,我真的不知道我在这里做错了什么。

EN

回答 2

Stack Overflow用户

发布于 2022-09-07 08:55:13

Requests不像浏览器那样操作或呈现网站,它只提供静态HTML内容由JavaScript动态生成,将一些JSON数据转换为结构。

使用这些来提取数据:

代码语言:javascript
复制
[json.loads(e.text.strip()) for e in soup.select('table tr.job [type="application/ld+json"]')]

结果:

代码语言:javascript
复制
/*
* 提示:该行代码过长,系统自动注释不进行高亮。一键复制会移除系统注释 
* [{'@context': 'http://schema.org', '@type': 'JobPosting', 'datePosted': '2022-09-04T05:21:13+00:00', 'description': 'About the Team\n\nThe Design Infrastructure team designs, builds, and ships the Design System foundations and UI components used in all of DoorDash’s products, on all platforms. Specifically, the iOS team works closely with designers and product engineering teams across the company to help shape the Design System, and owns the shared UI library for iOS – developed for both SwiftUI and UIKit.\nAbout the Role\n\nWe are looking for a lead iOS engineer who has a strong passion for UI components and working very closely with design. As part of the role you will be leading the iOS initiative for our Design System, which will include working closely with designers and iOS engineers on product teams to align, develop, maintain, and evolve the library of foundations and UI components; which is adopted in all our products.\n\nYou will report into the Lead Design Technologist for Mobile on our Design Infrastructure team in our Product Design organization. This role is 100% flexible, and can b\n Apply now and work remotely at DoorDash', 'baseSalary': {'@type': 'MonetaryAmount', 'currency': 'USD', 'value': {'@type': 'QuantitativeValue', 'minValue': 70000, 'maxValue': 120000, 'unitText': 'YEAR'}}, 'employmentType': 'FULL_TIME', 'directApply': 'http://schema.org/False', 'industry': 'Startups', 'jobLocationType': 'TELECOMMUTE', 'jobLocation': [{'address': {'@type': 'PostalAddress', 'addressCountry': 'United States', 'addressRegion': 'Anywhere', 'streetAddress': 'Anywhere', 'postalCode': 'Anywhere', 'addressLocality': 'Anywhere'}}], 'applicantLocationRequirements': [{'@type': 'Country', 'name': 'United States'}], 'title': 'Lead Design Technologist iOS', 'image': 'https://remoteok.com/assets/img/jobs/f2f1ab68227768717536a0ab7e2578ab1662268873.png', 'occupationalCategory': 'Lead Design Technologist iOS', 'workHours': 'Flexible', 'validThrough': '2022-12-03T05:21:13+00:00', 'hiringOrganization': {'@type': 'Organization', 'name': 'DoorDash', 'url': 'https://remoteok.com/doordash', 'sameAs': 'https://remoteok.com/doordash', 'logo': {'@type': 'ImageObject', 'url': 'https://remoteok.com/assets/img/jobs/f2f1ab68227768717536a0ab7e2578ab1662268873.png'}}}, {'@context': 'http://schema.org', '@type': 'JobPosting', 'datePosted': '2022-09-03T00:00:09+00:00', 'description': "We’re seeking a senior core, distributed systems engineers to build dev tools. At [Iterative](https://iterative.ai) we build [DVC](https://dvc.org) (9000+ ⭐on GitHub) and [CML](https://cml.dev) (2000+ ⭐ on GitHub) and a few other projects that are not released yet. It's a great opportunity if you love open source, dev tools, systems programming, and remote work. Join our well-funded remote-first team to build developer tools to see how your code is used by thousands of developers every day!\n\nABOUT YOU\n\n- Excellent communication skills and a positive mindset \n- No prior deep knowledge of ML is required\n- At least one year of experience with file systems, concurrency, multithreading, and server architectures\n- Passionate about building highly reliable system software\n- Python knowledge and excellent coding culture (standards, unit test, docs, etc) are required.\n- Initiative to help shape the engineering practices, products, and culture of a young startup\n-&nbsp;R\n Apply now and work remotely at Iterative", 'baseSalary': {'@type': 'MonetaryAmount', 'currency': 'USD', 'value': {'@type': 'QuantitativeValue', 'minValue': 50000, 'maxValue': 180000, 'unitText': 'YEAR'}}, 'employmentType': 'FULL_TIME', 'directApply': 'http://schema.org/False', 'industry': 'Startups', 'jobLocationType': 'TELECOMMUTE', 'applicantLocationRequirements': {'@type': 'Country', 'name': 'Anywhere'}, 'jobLocation': {'address': {'@type': 'PostalAddress', 'addressCountry': 'Anywhere', 'addressRegion': 'Anywhere', 'streetAddress': 'Anywhere', 'postalCode': 'Anywhere', 'addressLocality': 'Anywhere'}}, 'title': 'Senior Software Engineer', 'image': 'https://remoteOK.com/assets/img/jobs/cb9a279f231a5312283e6d935bba3be91636086324.png', 'occupationalCategory': 'Senior Software Engineer', 'workHours': 'Flexible', 'validThrough': '2022-12-02T00:00:09+00:00', 'hiringOrganization': {'@type': 'Organization', 'name': 'Iterative', 'url': 'https://remoteok.com/iterative', 'sameAs': 'https://remoteok.com/iterative', 'logo': {'@type': 'ImageObject', 'url': 'https://remoteOK.com/assets/img/jobs/cb9a279f231a5312283e6d935bba3be91636086324.png'}}}, {'@context': 'http://schema.org', '@type': 'JobPosting', 'datePosted': '2022-09-06T09:10:04+00:00', 'description': '<p dir="ltr">Get a remote job that you will love with better compensation and career growth.<strong></strong></p><p dir="ltr">We’re <a href="https://lemon.io/">Lemon.io</a> — a marketplace where we match you with hand-picked startups from the US and Europe.\xa0<strong></strong></p><p dir="ltr"><br /></p><p dir="ltr"><strong>Why work with us:</strong></p><ul><li dir="ltr"><p dir="ltr">We’ll find you a team that respects you. No time-trackers or any micromanagement stuff</p></li><li dir="ltr"><p dir="ltr">Our engineers earn $5k - $9k / month. We’ve already paid out over $10M.</p></li><li dir="ltr"><p dir="ltr">Choose your schedule. We have both full- and part-time projects.</p></li><li dir="ltr"><p dir="ltr">No project managers in the middle — only direct communications with clients, most of whom have a technical background</p></li><li dir="ltr"><p dir="ltr">Our customer success team provides life support to help you resolve anything.</p></li><li dir="ltr"><p dir="ltr">You don’\n Apply now and work remotely at lemon.io', 'baseSalary': {'@type': 'MonetaryAmount', 'currency': 'USD', 'value': {'@type': 'QuantitativeValue', 'minValue': 60000, 'maxValue': 110000, 'unitText': 'YEAR'}}, 'employmentType': 'FULL_TIME', 'directApply': 'http://schema.org/False', 'industry': 'Startups', 'jobLocationType': 'TELECOMMUTE', 'applicantLocationRequirements': {'@type': 'Country', 'name': 'Anywhere'}, 'jobLocation': {'address': {'@type': 'PostalAddress', 'addressCountry': 'Anywhere', 'addressRegion': 'Anywhere', 'streetAddress': 'Anywhere', 'postalCode': 'Anywhere', 'addressLocality': 'Anywhere'}}, 'title': 'DevOps Engineer', 'image': 'https://remoteOK.com/assets/img/jobs/b31a9584a903e655bd2f67a2d7f584781662455404.png', 'occupationalCategory': 'DevOps Engineer', 'workHours': 'Flexible', 'validThrough': '2022-12-05T09:10:04+00:00', 'hiringOrganization': {'@type': 'Organization', 'name': 'lemon.io', 'url': 'https://remoteok.com/lemon-io', 'sameAs': 'https://remoteok.com/lemon-io', 'logo': {'@type': 'ImageObject', 'url': 'https://remoteOK.com/assets/img/jobs/b31a9584a903e655bd2f67a2d7f584781662455404.png'}}},...]
*/

示例

代码语言:javascript
复制
import requests,json
from bs4 import BeautifulSoup

def extract(term):
    url = f"https://remoteok.com/remote-{term}-jobs"
    request = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
    if request.status_code == 200:
        soup = BeautifulSoup(request.text, 'html.parser')
        table = soup.find_all('table', id="jobsboard")
        print(len(table))
        for tbody in table:
            tbody.find_all('tbody')
            print(len(tbody))
    else:
        print("can't request website")
    
    data = [json.loads(e.text.strip()) for e in soup.select('table tr.job [type="application/ld+json"]')]
    return data

for post in extract("python"):
    print(post['hiringOrganization']['name'])

输出

代码语言:javascript
复制
DoorDash
Iterative
lemon.io
Angaza
Angaza
Kandji
Kandji
Kandji
Kandji
Kandji
Great Minds
Jobber
Udacity
...
票数 2
EN

Stack Overflow用户

发布于 2022-09-07 08:41:06

tbody确实会出现在网页上,但是不会被漂亮的汤拉到表变量中。

我以前也遇到过这个。解决方案是直接从selenium获取标记。

但是网页上只有一个工作板和一个tbody,所以您可以跳过tbody,寻找一个更有用的标记。

我用谷歌Chrome。它有免费的扩展ChroPath,这使得它非常容易识别选择器。我只需右键单击浏览器中的文本,并选择“检查”(有时是两次),并突出显示正确的HTML标记。

PyCharm允许您轻松地查看每个变量的内容。

此代码将允许您在文本文件中查看网页HTML源代码:

代码语言:javascript
复制
outputFile = r"C:\Users\user\Documents\HP Laptop\Documents\Documents\Jobs\DIT\IDMB\OutputZ.txt"


def update_output_file(pageSource: str):
    with open(outputFile, 'w', encoding='utf-8') as f:
        f.write(pageSource)
    f.close()
票数 1
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/73629133

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档