Web抓取动态HTML页面结构

Question

我正在做一个大规模的网页抓取项目，每个网页的HTML结构都是不同的。我想从网页上抓取产品描述，现在我使用的是BeautifulSoup包。

例如，我试图抓取的产品描述存储在HTML结构中：

​

<div class="product-description">
  <p> "Title" </p>
  <p> "Some content" </p>
  <p> "Product description" </p>
</div>


<div class="product-description">
  <p> "Title" </p>
  <p> "Product description" </p>
</div>

<div class="product-description">
  <p> "Title" </p>
  <p> "Some content" </p>
  <p> "Some content" </p>
  <p> "Product description" </p>
</div>


<div class="product-description">
  <p> "Title" </p>
  <p> "Some-content" </p>
  <p> "Some-content" </p>
  <p> "Some-content" </p>
  <p> "Product description" </p>
</div>

​

我已经编写了一个for循环，它根据页面结构从div类"product-description“获取数据。我的示例代码片段：

requests = (grequests.get(url) for url in urls)
responses = grequests.imap(requests, grequests.Pool(1000))

for response in responses:

        html_soup = BeautifulSoup(response.text, 'html.parser')

        if html_soup.find('div',class_='product_description').next_element.next_sibling.next_sibling.next_sibling.next_sibling:
                product_description = html_soup.find('div',class_='product_description').next_element.next_sibling.next_sibling.next_sibling.next_sibling.text

        elif html_soup.find('div', class_='product-description').next_element.next_sibling.next_sibling.next_sibling:
                product_description = html_soup.find(
                  'div', class_='product_description').next_element.next_sibling.next_sibling.next_sibling.text

        elif html_soup.find('div', class_='product-description').next_element.next_sibling.next_sibling:
                product_description = html_soup.find(
                  'div', class_='product_description').next_element.next_sibling.next_sibling.text

        else:
                product_description = html_soup.find(
                  'div', class_='product_description').next_element.next_sibling.text

我希望if条件能够检查当前级别的HTML中是否有同级元素，如果没有则检查后续条件。然而，在3000次迭代之后，我得到了一个写着Nonetype object has no attribute next_sibling的Attribute error。截图附图如下：

​

​

我知道一定有其他更简单的方法来处理这种动态页面结构。任何帮助都将不胜感激。提前感谢！

Answer 1

试试这个：

for i in soup.find_all('div',class_="product-description"):
    try:
        print(i.find_all('p')[-1].text)
    except:
        pass

这里的汤是：

<div class="product-description">
  <p> "Title" </p>
  <p> "Some content" </p>
  <p> "Product description" </p>
</div>


<div class="product-description">
  <p> "Title" </p>
  <p> "Product description" </p>
</div>

<div class="product-description">
  <p> "Title" </p>
  <p> "Some content" </p>
  <p> "Some content" </p>
  <p> "Product description" </p>
</div>


<div class="product-description">
  <p> "Title" </p>
  <p> "Some-content" </p>
  <p> "Some-content" </p>
  <p> "Some-content" </p>
  <p> "Product description" </p>
</div>