Linkedin scraper提取技能

Question

我试图从人们的公共档案中获取某些角色最常用的技能。我可以提取电子邮件，公司，姓名，职位等，但我不能获得技能。我使用的是parsel中的Selector。我尝试了许多方法，但显然我的目标是错误的类，我可能应该遍历技能。到目前为止，我的代码如下：

def linkedin_scrape(linkedin_urls):

profiles = []

for url in linkedin_urls:

    _DRIVER_CHROME.get(url)
    sleep(5)

    selector = Selector(text=_DRIVER_CHROME.page_source)

    # Use xpath to extract the exact class containing the profile name
    name = selector.xpath('//*[starts-with(@class, "inline")]/text()').extract_first()
    if name:
        name = name.strip()

    # Use xpath to extract the exact class containing the profile position
    position = selector.xpath('//*[starts-with(@class, "mt1")]/text()').extract_first()

    if position:
        position = position.strip()
        position = position[0:position.find(' at ')]

    # Use xpath to extract the exact class containing the profile company
    company = selector.xpath('//*[starts-with(@class, "text-align-left")]/text()').extract_first()

    if company:
        company = company.strip()

    # Use xpath to extract skills

    skills = selector.xpath('//*[starts-with(@class, "pv-skill")]/text()').extract_first()

    if skills:
        skills = skills.strip()


    profiles.append([name, position, company, url])
    print(f'{len(profiles)}: {name}, {position}, {company}, {url}, {skills}')

return profiles

Answer 1

为了捕获所有技能，您需要首先展开skills部分，使其显示所有技能，然后使用以“pv-skill-category-entity__ name -text”开头的名称确定类。

这对我来说是有效的，直到今天。

#locate link to expand skills
show_more_skills_button = driver.find_element_by_class_name("pv-skills-section__chevron-icon")
#expand
show_more_skills_button.click()

skills = driver.find_elements_by_xpath("//*[starts-with(@class,'pv-skill-category-entity__name-text')]")

#create skills set
skill_set = []
for skill in skills:
    skill_set.append(skill.text)