这是抓取只有4个链接,并返回15个抓取项目/行,但我需要抓取20个链接与35+抓取项目。
如果你们谁来帮我,那将对我很有帮助
from scrapy import Spider
from scrapy.http import Request
class TastySpider(Spider):
name = 'tasty'
allowed_domains = ['tasty.co']
start_urls = ['https://tasty.co/topic/game-day']
def parse(self, response):
lists=response.css('a.feed-item::attr(href)').extract()
for link in lists:
link=response.urljoin(link)
if link:
yield Request(link, callback=self.parse_page)
def parse_page(self, response):
Recipe_URL=response.url
Title=response.xpath('//h1//text()').extract_first()
instruction=response.css('ol.prep-steps li.xs-mb2::text').extract()
Cooking_time=response.xpath('//*[@class="recipe-time-container xs-flex xs-mx2 xs-relative xs-b0 md-b2 xs-mt2 md-mt0"]//p//text()').extract_first()
Image_video_URL=response.xpath('//*[@property="og:image"]//@content').extract_first()
servings=response.xpath('//h2[contains(text(),"Ingredients")]//following-sibling::p//text()').extract_first()
Us_measurement=response.xpath('//*[@class="measurement us-measurement xs-inline-block"]//text()').extract()
Ingredient=response.xpath('//*[@class="xs-mb1 xs-mt0"]//section//following-sibling::text()').extract()
Non_us_measurement=response.xpath('//*[@class="measurement non-us-measurement xs-inline-block"]//text()').extract()
for Us_measur, Ingred, Non_us in zip(Us_measurement, Ingredient, Non_us_measurement):
Us_measur=Us_measur.strip()
Ingred=Ingred.strip()
Non_us=Non_us.strip()
yield {
'Us-measurement':Us_measur,
'Ingredient':Ingred,
'Non-us-measurement':Non_us,
'Recipe_URL':Recipe_URL,
'Title':Title,
'instruction':instruction,
'Cooking_time':Cooking_time,
'Servings':servings,
'Image-video_URL':Image_video_URL
}发布于 2018-12-18 23:16:42
Scrapy确实抓取了你想抓取的20个链接。
但是,只有/recipe/链接包含您试图抓取的数据。
/compilation/链接只包含指向其他食谱的链接。
实现这一点的最简单方法可能是使用CrawlSpider,并为配方和编译链接创建单独的规则。
https://stackoverflow.com/questions/53835499
复制相似问题