如何从网页中获取所有下一页的链接？

Question

我已经用python编写了一些脚本来获取指向下一页的所有链接。然而，它只在一定程度上工作得很好。下一页链接的最大数量为255。运行我的脚本，我得到了前23个链接和最后一个页面链接，但它们之间缺少24到254个链接。我怎样才能得到所有的它们呢？这是我正在尝试的：

import requests
from lxml import html

page_link = "https://www.yify-torrent.org/search/1080p/"
b_link = "https://www.yify-torrent.org"

def get_links(main_link):
    links = []
    response = requests.get(main_link).text
    tree = html.fromstring(response)
    for item in tree.cssselect('div.pager a'):
        if item.attrib["href"] not in links:
            links.append(item.attrib["href"])
    for link in links:
        print(b_link + link)

get_links(page_link)

下一页链接中的元素位于：

<div class="pager"><a href="/search/1080p/" class="current">1</a> <a href="/search/1080p/t-2/">2</a> <a href="/search/1080p/t-3/">3</a> <a href="/search/1080p/t-4/">4</a> <a href="/search/1080p/t-5/">5</a> <a href="/search/1080p/t-6/">6</a> <a href="/search/1080p/t-7/">7</a> <a href="/search/1080p/t-8/">8</a> <a href="/search/1080p/t-9/">9</a> <a href="/search/1080p/t-10/">10</a> <a href="/search/1080p/t-11/">11</a> <a href="/search/1080p/t-12/">12</a> <a href="/search/1080p/t-13/">13</a> <a href="/search/1080p/t-14/">14</a> <a href="/search/1080p/t-15/">15</a> <a href="/search/1080p/t-16/">16</a> <a href="/search/1080p/t-17/">17</a> <a href="/search/1080p/t-18/">18</a> <a href="/search/1080p/t-19/">19</a> <a href="/search/1080p/t-20/">20</a> <a href="/search/1080p/t-21/">21</a> <a href="/search/1080p/t-22/">22</a> <a href="/search/1080p/t-23/">23</a> <a href="/search/1080p/t-2/">Next</a> <a href="/search/1080p/t-255/">Last</a> </div>

我得到的结果就像是缩减到了最后五个链接

https://www.yify-torrent.org/search/1080p/t-20/
https://www.yify-torrent.org/search/1080p/t-21/
https://www.yify-torrent.org/search/1080p/t-22/
https://www.yify-torrent.org/search/1080p/t-23/
https://www.yify-torrent.org/search/1080p/t-255/

Answer 1

显然，Answer provided by @kaze应该返回255页，但如果您需要动态获取所有链接而不对总页数进行硬编码，您可以尝试

r = requests.get("https://www.yify-torrent.org/search/1080p/")
tree = html.fromstring(r.content)
page_number = tree.xpath("//div[@class='pager']/a[.='Last']/@href")[0].split("/")[-2].replace("t-", "")

for page in range(int(page_number) + 1):
    requests.get("https://www.yify-torrent.org/search/1080p/t-%s/" % page)

Answer 2

如果链接结构不可推断，你将不得不“遍历站点”，但在这里，你最好自己生成链接，如下所示：

for i in range(1,256):
    print('https://www.yify-torrent.org/search/1080p/t-%s/' % i)

Answer 3

您的脚本看起来是正确的。查看该页面的HTML，我看到了以下内容：

​

<a href="/search/1080p/t-2/">2</a> 
<a href="/search/1080p/t-3/">3</a> 
<a href="/search/1080p/t-4/">4</a> 
<a href="/search/1080p/t-5/">5</a> 
<a href="/search/1080p/t-6/">6</a> 
<a href="/search/1080p/t-7/">7</a> 
<a href="/search/1080p/t-8/">8</a> 
<a href="/search/1080p/t-9/">9</a> 
<a href="/search/1080p/t-10/">10</a> 
<a href="/search/1080p/t-11/">11</a> 
<a href="/search/1080p/t-12/">12</a> 
<a href="/search/1080p/t-13/">13</a> 
<a href="/search/1080p/t-14/">14</a> 
<a href="/search/1080p/t-15/">15</a> 
<a href="/search/1080p/t-16/">16</a> 
<a href="/search/1080p/t-17/">17</a> 
<a href="/search/1080p/t-18/">18</a> 
<a href="/search/1080p/t-19/">19</a> 
<a href="/search/1080p/t-20/">20</a> 
<a href="/search/1080p/t-21/">21</a> 
<a href="/search/1080p/t-22/">22</a> 
<a href="/search/1080p/t-23/">23</a> 
<a href="/search/1080p/t-2/">Next</a> 
<a href="/search/1080p/t-255/">Last</a> 

​

t-2似乎是指向Next页面的指针，该页面将包含其余的链接。