TypeError:当web抓取以将列表/值追加到csv文件中的列时，无法调用“列表”对象

Question

我有这个TypeError：'list‘对象在调用我前面定义的函数时不能在for循环中调用。我希望将/注入函数添加到for循环中，以便按列自动填充csv行

import requests
import bs4
from bs4 import BeautifulSoup
import pandas as pd
import time
import csv

# copy and paste the url from indeed using your search term
URL = 'https://www.indeed.com/jobs?q=data+science+summer+intern&l=New+York'

# conducting a request of the stated URL above:
page = requests.get(URL)

# specifying a desired format of “page” using the html parser - this allows python to read the various components of the page, rather than treating it as one long string.
soup = BeautifulSoup(page.text, 'html.parser')

# printing soup in a more structured tree format that makes for easier reading
print(soup.prettify())

这个extract_job_title_from_result()函数接受“作业标题”，并附加到“作业”列表中。

def extract_job_title_from_result(soup): 
    jobs = []
    for div in soup.find_all(name='div', attrs={'class':'row'}):
        for a in div.find_all(name='a', attrs={'data-tn-element':'jobTitle'}):
            jobs.append(a['title'])
    return(jobs)

extract_job_title_from_result = extract_job_title_from_result(soup)
print('extract_job_title_from_result is: ', extract_job_title_from_result)

产出：“数据工程师暑期实习生”、“数据科学暑期实习生”、“数据科学家暑期实习生”、“统计研究和数据科学实习生”、“数据科学家/数据分析实习生-2019年夏季实习”、“2019暑期实习-数据科学实习”、“棒球数据”、“数据科学夏季实习”、“实习生、数据科学”、“数据科学实习生(社会媒体分析)”、“数据科学实习生”

# Set max result per city
max_results_per_city = 100
city_set = ['New+York','Chicago','San+Francisco', 'Austin', 'Seattle', 'Los+Angeles', 'Philadelphia', 'Dallas', 'Pittsburgh', 'Denver', 'Miami', 'Washington+DC','Jersey+City', 'Princeton']
columns = ['city', 'job_title', 'company_name', 'location', 'summary', 'salary']
sample_df = pd.DataFrame(columns = columns)
sample_df

输出sample_df作为列标题，但没有数据，但仍=城市job_title company_name位置汇总薪资

现在，我正试图在网络上刮取数据，并从中提取数据。我已经编写了可以工作的函数，并且可以使用这些函数在csv中逐列保存/附加到列中。

我想在每个城市抓取100个结果，并使用我写到csv文件的函数保存/追加这些数据。

for city in city_set:
    for start in range(0, max_results_per_city, 10):
        #ensuring at least 1 second between page grabs
        time.sleep(1)
        #soup = BeautifulSoup(page.text, 'lxml', from_encoding='utf-8')
        sample_df['job_title'] = extract_job_title_from_result(soup)

        ### Ignore the below functions. They worked individually but not here in this for loop. I'm using a function to try to make it work first before appending all functions to csv by column name

        #extract_company_from_result(soup)
        #extract_location_from_result(soup)
        #extract_salary_from_result(soup)
        #extract_summary_from_result(soup)
        #sample_df.loc[num] = job_post

sample_df.to_csv('/Users/KingKong1/AnacondaProjects/testing1.csv', encoding='utf-8')  

我从**sample_df['job_title'] = extract_job_title_from_result(soup)**那里得到了"TypeError：'list‘对象不可调用“

Answer 1

在extract_job_title_from_result = extract_job_title_from_result(soup)中，您已经将函数extract_job_title_from_result替换为其结果，这是一个列表。

因此，下次尝试调用它时，extract_job_title_from_result不再是函数的名称，而是引用这个列表。

使用不同的名称，例如：

job_title = extract_job_title_from_result(soup)
print('job_title is: ', job_title)