Python 3、Beautifulsoup和urlopen web scraping

Question

我在这个website网站上搜寻有关产品的信息，如产品描述和价格。该网站使用分页来加载前20个产品，而当我进行抓取时，我的程序只获得前20个产品。我怎么才能买到所有这些产品。以下是我的代码

 from bs4 import BeautifulSoup
 from urllib.request import urlopen
 import csv
 import random


myfile = open('example.csv', 'w')

def scrape():
    page = urlopen('https://www.olx.com.ng/')
    soup = BeautifulSoup(page.read(), 'html.parser')


    for price, description  in zip(soup.findAll('p', 
    {'class':'_13OzP'}),soup.findAll('p',{'class':'_2uySz'})):
    info = price.text + ' -------------------------- ' + description .text
    print(info)

`这是一个html结构的示例图像image 检查我在图像上用红色标记的区域

Answer 1

您可以直接访问该API。您需要读取第一个请求中的“游标”，并将其传递给下一个请求，以获得下一组结果。本例使用了请求，但您可以对其进行调整，以便更容易地使用urllib。

import requests
import json
cursor = "0"
for i in range (30):
    r = requests.get("https://www.olx.com.ng/api/items?query=%7B%22sorting%22%3A%22desc-creation%22%7D&cursor=" + cursor)
    j = r.json()
    cursor = j['metadata']['cursor']
    for d in j['data']:
        print ("Title: {}, Price {}".format(d['title'].encode("utf-8"), d['price']['value']['raw']))

输出：

Title: Brand New 2.5HP Electric Motorized Treadmill With Mp3 Player Exercise, Price 250000
Title: Tokunbo Toyota Venza 2012 White, Price 6500000
Title: Clean Registered 2007 Toyota Tundra, Price 3550000
Title: Ladies Bike Hajour Lucky, Price 80000
Title: 110k p/a two bedroom to let in Agbede, Price 110000
...