为什么有些html标签在抓取时是看不见的？

Question

我正在尝试从here中抓取文本以直接输入到excel工作表中，而不是复制和粘贴。该网站使用Html来包含有关原始字体的信息。这是一个如何在页面上编码一行文本的示例：

<div class="line">
    <span class="milestone_wrap"> </span>
    <a id="tln-2212" href="index.html#tln-2212" class="milestone tln invisible" title="TLN: 2212">2212</a>
    <span class="milestone_wrap">When </span>
    <span class="typeform" data-setting="ſ">s</span>
    <span class="milestone_wrap">uch ill dealing mu</span>
    <span class="ligature" data-precomposed="ſt">
        <span class="typeform" data-setting="ſ">s</span>
        <span class="milestone_wrap">t</span>
    </span>
    <span class="milestone_wrap"> be </span>
    <span class="typeform" data-setting="ſ">s</span>
    <span class="milestone_wrap">eene in thought. </span>
    <span class="sd exit">
        <span class="space" style="padding-right:1em;" xml:space="preserve"></span>
        <i>Exit</i>
        <span class="milestone_wrap">.</span>
    </span>
</div>

我尝试过使用find_all方法

import requests
from bs4 import BeautifulSoup as bs
url = 'https://internetshakespeare.uvic.ca/doc/R3_F1/scene/3.6/index.html'
page = requests.get(url)
text = bs(page.text, 'html.parser')

divs = text.find_all('div', class_="line")
for div in divs:
    for item in div.contents: print(item)

这是我得到的结果：

When 
<span class="typeform" data-setting="ſ">s</span>
uch ill dealing mu
<span class="ligature" data-precomposed="ſt"><span class="typeform" data-setting="ſ">s</span>t</span>
 be 
<span class="typeform" data-setting="ſ">s</span>
eene in thought. 
<span class="sd exit"><span class="space" style="padding-right:1em;" xml:space="preserve"> </span><i>Exit</i>.</span>

所有带有<span class="milestone_wrap">标记的内容都没有标记:因此，当我使用.find_all表示“span”时，这些字符串就不会出现，所以我只剩下随机的字母。这个类没有出现有什么原因吗？

Answer 1

在line类级别工作，但分解a标记，以便删除行号；除非您确实需要它们，在这种情况下，我会在它们和下面的文本之间添加空格：

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://internetshakespeare.uvic.ca/doc/R3_F1/scene/3.6/index.html')
soup = bs(r.content, 'lxml')

for line in soup.select('.line'):
    line.select_one('a').decompose()
    print(line.text)

Answer 2

在执行代码时稍作调整(必须导入requests模块)，您应该会得到站点的内容。

from bs4 import BeautifulSoup as bs
import requests

url = 'https://internetshakespeare.uvic.ca/doc/R3_F1/scene/3.6/index.html'
page = requests.get(url)
text = bs(page.text, 'html.parser')

divs = text.find_all('div', class_="line")
for div in divs:
    for item in div.contents: print(item)

文本可以在<span class="milestone_wrap">标记中找到。你可以用浏览器的检查器来检查。文本以一小部分标签的形式递送，例如“其中在一个”。您应该能够提取文本。