使用BeautifulSoup按id选择html

Question

我正在尝试从网页中选择html的一个部分

我使用代码：

import urllib.request, urllib.parse, urllib.error
from lxml import html
from bs4 import BeautifulSoup

html = urllib.request.urlopen("http://archiveofourown.org/works/1758087?view_full_work=true").read()
soup = BeautifulSoup(html, 'lxml')

for tag in soup.find_all(id="children"):
    print(tag.get('href'))

为了得到这个：

<div id="children" class="children module">
  <h3 class="heading">Works inspired by this one:</h3>
  <ul>
    <li>
        <a href="/works/2219289">not an origin story</a> by <a rel="author" href="/users/eiirene/pseuds/eiirene">eiirene</a>
    </li>
  </ul>
</div>

然后提取数字(2219289)和名称(eiirene)。

当我在url = "work=true“上运行相同的代码尝试选择以下内容时：

<div id="children" class="children module">
  <h3 class="heading">Works inspired by this one:</h3>
  <ul>
    <li>
        <a href="/works/240745">Cover-  Performance in A Leading Role by mad_lori</a> by <a rel="author" href="/users/blanketforyourshock/pseuds/blanketforyourshock">blanketforyourshock</a>
    </li>
    <li>
        <a href="/works/341326">Movie still: To A Stranger</a> by <a rel="author" href="/users/RedOrchid/pseuds/RedOrchid">RedOrchid</a>
    </li>
    <li>
        <a href="/works/787850">Future Yesterday</a> by <a rel="author" href="/users/ladymac111/pseuds/ladymac111">ladymac111</a>
    </li>
  </ul>
</div>

它返回“无”。如果我使用find_all，它将返回一个空列表[]。

有什么想法吗？

Answer 1

我将两个urls都包含在一个列表中，并将它们循环起来。此外，为了简单起见，我使用了请求库。

代码：

import requests, re
from bs4 import BeautifulSoup

urls = ["http://archiveofourown.org/works/1758087?view_full_work=true", "http://archiveofourown.org/works/225563?view_full_work=true"]
for url in urls:
    print("***********************")
    html = requests.get(url).text
    soup = BeautifulSoup(html, 'html.parser')
    part = soup.find("div", id="children")
    soup = BeautifulSoup(str(part), 'html.parser')
    nums = soup.findAll('a', href=re.compile('/works/'))
    names = soup.findAll('a', href=re.compile('/users/'))
    for i in range(0, len(nums)):
        print(str(nums[i]['href']).split('/works/')[1] + " : " + names[i].text)

输出：

***********************
2219289 : eiirene
***********************
240745 : blanketforyourshock
341326 : RedOrchid
787850 : ladymac111