爬虫数据表单网站使用Scrapy1.5.0-Python

Question

我尝试用Scrapy (1.5.0)- Python从网站上抓取数据。

项目目录：

stack/
    scrapy.cfg           

    stack/            
        __init__.py

        items.py          

        pipelines.py      

        settings.py       

        spiders/         
            __init__.py
              stack_spider.py

这是我的items.py

import scrapy

class StackItem(scrapy.Item):
    title = scrapy.Field()

这是stack_spider.py

from scrapy import Spider
from scrapy.selector import Selector

from stack.items import StackItem

class StackSpider(Spider):
    name = "stack"
    allowed_domains = ["batdongsan.com.vn"]
    start_urls = [
        "https://batdongsan.com.vn/nha-dat-ban",
    ]

    def parse(self, response):
        questions = Selector(response).xpath('//div[@class="p-title"]/h3')

        for question in questions:
            item = StackItem()
            item['title'] = question.xpath(
                'a/text()').extract()[0]

            yield item

我不知道为什么我不能抓取数据，我真的需要你的帮助。谢谢

Answer 1

设置用户代理

去你的刮擦项目settings.py

把这个贴进去，

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'

Answer 2

如果你只想抓取网站并获得源代码，这可能会有所帮助。

import urllib.request as req

def imLS():
    url = "https://batdongsan.com.vn/nha-dat-ban"
    data = req.Request(url)
    resp = req.urlopen(data)
    respData = resp.read()
    print(respData)
imLS()

Answer 3

找到答案：http://edmundmartin.com/random-user-agent-requests-python/需要设置头部以通过防止爬行