为什么scrapy在这个例子中不能工作？

Question

我试图从一个网站抓取数据，如文章枚举，定价和股票，并将其导出到excel工作表。

以下脚本成功登录。未登录时，仅可见articl枚举器。我测试了刮刀，它成功地抓取了文章编号。在下面的示例中，我尝试将登录和抓取数据结合起来，但它不起作用。

我做错了什么？

import scrapy
import pandas as pd
from scrapy import FormRequest
import os

artkl_list = []
price_list = []
stock_list = []
link_site = []

class PostsSpider(scrapy.Spider):
    name = "posts"

    start_urls = [
        'https://dealerportal.exertis.nl/action/products/catalog/?producttypeID=5'
       ]

    def parseAfterLogin(self, response):
        # i am not sure if all the syntax below is correct. I can supply the HTML for you to check.
        for i in response.css('div.productlistblock.row'):
            artkl = i.css('div.articlenumber::text').extract()
            price = i.css('span.unitfourprice::text').get().strip()   #i am not sure if the syntax here is correct
            stock = i.css('div.right.stockStatusTitle::text').extract()   #i am not sure if the syntax here is correct
            link = i.css('a.product_img_link::attr(href)').get()   #i am not sure if the syntax here is correct

            # put it names in list
            artkl_list.append(artkl)
            price_list.append(price)
            stock_list.append(stock)
            link_site.append(link)

            # display information when you scraped from website
            print('artkl              = ', artkl)
            print('price              = ', price)
            print('stock              = ', stock)
            print('link_site          = ', link)

            print('\n --------------------------------------------- \n')
            print('artkl             = ', len(artkl_list))
            print('price             = ', len(price_list))
            print('stock             = ', len(stock_list))
            print('link              = ', len(link_site))
            print('\n --------------------------------------------- \n')

        # move to next page
        next_page = response.css('a.next::attr(href)').get()
        if next_page:
            yield response.follow('' + str(next_page))

        # put it in dataframe
        df = pd.DataFrame({
            'artkl': artkl_list,
            'price': price_list,
            'stock': stock_list,
            'link_site': link_site
        })
        # save in excel
        df.to_excel('exertis.xlsx', index=False)

    def parse(self, response):
        os.environ['my_em'] = 'thisismyusername'
        os.environ['my_pw'] = 'thisismypassword'
        self.em = os.environ.get('my_em')
        self.pw = os.environ.get('my_pw')

        self.login_url = "https://dealerportal.exertis.nl/action/frontusers/login"

        dataLogin = {
            'username': self.em,
            'password': self.pw,
            'login': 'Inloggen'
        }
        print(self.login_url)
        print('--------')
        print(dataLogin)
        print('--------')
        yield FormRequest(url=self.login_url, formdata=dataLogin, callback=self.parseAfterLogin)

Answer 1

如果它是一个股票网站，我高度认为它使用JavaScript。因此，如果你想抓取网站，你必须使用Splash和Scrapy。一旦JS被呈现，你应该能够抓取你想要的东西。是否可以提供该网站的链接？