从网页中提取URL链接

Question

我对网络抓取是个新手。我正在尝试使用python从https://www.clinicaltrialsregister.eu中提取数据，关键字为“急性髓性白血病”、“慢性粒细胞白血病”、“急性淋巴细胞白血病”，以提取以下信息-EudraCT编号、试验状态、试验全称、发起人名称、国家、正在调查的医疗条件、参与试验的研究人员网络。

我试图从每个链接收集URL，然后转到每个页面并提取信息，但我没有得到一个适当的链接。我想要的网址像"https://www.clinicaltrialsregister.eu/ctr-search/trial/2014-000526-37/DE“，但得到

 '/ctr-search/trial/2014-000526-37/DE',
 '/ctr-search/trial/2006-001777-19/NL',
 '/ctr-search/trial/2006-001777-19/BE',
 '/ctr-search/trial/2007-000273-35/IT',
 '/ctr-search/trial/2011-005934-20/FR',
 '/ctr-search/trial/2006-004950-25/GB',
 '/ctr-search/trial/2009-017347-33/DE',
 '/ctr-search/trial/2012-000334-19/IT',
 '/ctr-search/trial/2012-001594-93/FR',
 '/ctr-search/trial/2012-001594-93/results',
 '/ctr-search/trial/2007-003103-12/DE',
 '/ctr-search/trial/2006-004517-17/FR',
 '/ctr-search/trial/2013-003421-28/DE',
 '/ctr-search/trial/2008-002986-30/FR',
 '/ctr-search/trial/2008-002986-30/results',
 '/ctr-search/trial/2013-000238-37/NL',
 '/ctr-search/trial/2010-018418-53/FR',
 '/ctr-search/trial/2010-018418-53/NL',
 '/ctr-search/trial/2010-018418-53/HU',
 '/ctr-search/trial/2010-018418-53/DE',
 '/ctr-search/trial/2010-018418-53/results',
 '/ctr-search/trial/2006-006852-37/DE',
 '/ctr-search/trial/2006-006852-37/ES',
 '/ctr-search/trial/2006-006852-37/AT',
 '/ctr-search/trial/2006-006852-37/CZ',
 '/ctr-search/trial/2006-006852-37/NL',
 '/ctr-search/trial/2006-006852-37/SK',
 '/ctr-search/trial/2006-006852-37/HU',
 '/ctr-search/trial/2006-006852-37/BE',
 '/ctr-search/trial/2006-006852-37/IT',
 '/ctr-search/trial/2006-006852-37/FR',
 '/ctr-search/trial/2006-006852-37/GB',
 '/ctr-search/trial/2008-000664-16/IT',
 '/ctr-search/trial/2005-005321-63/IT',
 '/ctr-search/trial/2005-005321-63/results',
 '/ctr-search/trial/2011-005023-40/GB',
 '/ctr-search/trial/2010-022446-24/DE',
 '/ctr-search/trial/2010-019710-24/IT',

尝试的代码-

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.clinicaltrialsregister.eu/ctr-search/search?query=acute+myeloid+leukemia&page=1')
soup = BeautifulSoup(page.text, 'html.parser')

#links = [a['href'] for a in soup.find_all('a', href=True) if a.text]


#links_with_text = []
#for a in soup.find_all('a', href=True): 
#    if a.text: 
#        links_with_text.append(a['href'])

links = [a['href'] for a in soup.find_all('a', href=True)]

输出-

'/help.html',
 '/ctr-search/search',
 '/joiningtrial.html',
 '/contacts.html',
 '/about.html',
 '/about.html',
 '/whatsNew.html',
 '/dataquality.html',
 '/doc/Sponsor_Contact_Information_EUCTR.pdf',
 '/natauthorities.html',
 '/links.html',
 '/about.html',
 '/doc/How_to_Search_EU_CTR.pdf#zoom=100,0,0',
 'javascript:void(0)',
 'javascript:void(0)',
 'javascript:void(0)',
 'javascript:void();',
 '#tabs-1',
 '#tabs-2',
 '&page=2',
 '&page=3',
 '&page=4',
 '&page=5',
 '&page=6',
 '&page=7',
 '&page=8',
 '&page=9',
 '&page=2',
 '&page=19',
 '/ctr-search/trial/2014-000526-37/DE',
 '/ctr-search/trial/2006-001777-19/NL',
 '/ctr-search/trial/2006-001777-19/BE',
 '/ctr-search/trial/2007-000273-35/IT',
 '/ctr-search/trial/2011-005934-20/FR',
 '/ctr-search/trial/2006-004950-25/GB',
 '/ctr-search/trial/2009-017347-33/DE',
 '/ctr-search/trial/2012-000334-19/IT',
 '/ctr-search/trial/2012-001594-93/FR',
 '/ctr-search/trial/2012-001594-93/results',
 '/ctr-search/trial/2007-003103-12/DE',
 '/ctr-search/trial/2006-004517-17/FR',
 '/ctr-search/trial/2013-003421-28/DE',
 '/ctr-search/trial/2008-002986-30/FR',
 '/ctr-search/trial/2008-002986-30/results',
 '/ctr-search/trial/2013-000238-37/NL',
 '/ctr-search/trial/2010-018418-53/FR',
 '/ctr-search/trial/2010-018418-53/NL',
 '/ctr-search/trial/2010-018418-53/HU',
 '/ctr-search/trial/2010-018418-53/DE',
 '/ctr-search/trial/2010-018418-53/results',
 '/ctr-search/trial/2006-006852-37/DE',
 '/ctr-search/trial/2006-006852-37/ES',
 '/ctr-search/trial/2006-006852-37/AT',
 '/ctr-search/trial/2006-006852-37/CZ',
 '/ctr-search/trial/2006-006852-37/NL',
 '/ctr-search/trial/2006-006852-37/SK',
 '/ctr-search/trial/2006-006852-37/HU',
 '/ctr-search/trial/2006-006852-37/BE',
 '/ctr-search/trial/2006-006852-37/IT',
 '/ctr-search/trial/2006-006852-37/FR',
 '/ctr-search/trial/2006-006852-37/GB',
 '/ctr-search/trial/2008-000664-16/IT',
 '/ctr-search/trial/2005-005321-63/IT',
 '/ctr-search/trial/2005-005321-63/results',
 '/ctr-search/trial/2011-005023-40/GB',
 '/ctr-search/trial/2010-022446-24/DE',
 '/ctr-search/trial/2010-019710-24/IT',
 'javascript:void(0)',
 '&page=2',
 '&page=3',
 '&page=4',
 '&page=5',
 '&page=6',
 '&page=7',
 '&page=8',
 '&page=9',
 '&page=2',
 '&page=19',
 'https://servicedesk.ema.europa.eu',
 '/disclaimer.html',
 'http://www.ema.europa.eu',
 'http://www.hma.eu'

Answer 1

正如我所说的，你可以通过将url的必要部分连接到每个结果来实现这一点。

尝试以下代码：

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.clinicaltrialsregister.eu/ctr-search/search?query=acute+myeloid+leukemia&page=1')
soup = BeautifulSoup(page.text, 'html.parser')

links =  ["https://www.clinicaltrialsregister.eu" + a['href'] for a in soup.find_all('a', href=True)]

Answer 2

此脚本将遍历搜索结果的所有页面，并尝试查找相关信息。

有必要添加完整的url，而不仅仅是https://www.clinicaltrialsregister.eu。

import requests
from bs4 import BeautifulSoup

base_url = 'https://www.clinicaltrialsregister.eu/ctr-search/search?query=acute+myeloid+leukemia'

url = base_url + '&page=1'
soup = BeautifulSoup(requests.get(url).text, 'lxml')

page = 1
while True:
    print('Page no.{}'.format(page))
    print('-' * 160)
    print()

    for table in soup.select('table.result'):

        print('EudraCT Number:    ', end='')
        for span in table.select('td:contains("EudraCT Number:")'):
            print(span.get_text(strip=True).split(':')[1])

        print('Full Title:        ', end='')
        for td in table.select('td:contains("Full Title:")'):
            print(td.get_text(strip=True).split(':')[1])

        print('Sponsor Name:      ', end='')
        for td in table.select('td:contains("Sponsor Name:")'):
            print(td.get_text(strip=True).split(':')[1])

        print('Trial protocol:    ', end='')
        for a in table.select('td:contains("Trial protocol:") a'):
            print(a.get_text(strip=True), end=' ')
        print()

        print('Medical condition: ', end='')
        for td in table.select('td:contains("Medical condition:")'):
            print(td.get_text(strip=True).split(':')[1])

        print('-' * 160)

    next_page = soup.select_one('a:contains("Next»")')
    if next_page:
        soup = BeautifulSoup(requests.get(base_url + next_page['href']).text, 'lxml')
        page += 1
    else:
        break

打印：

Page no.1
----------------------------------------------------------------------------------------------------------------------------------------------------------------

EudraCT Number:    2014-000526-37
Full Title:        An Investigator-Initiated Study To Evaluate Ara-C and Idarubicin in Combination with the Selective Inhibitor Of Nuclear Export (SINE) 
Selinexor (KPT-330) in Patients with Relapsed Or Refractory A...
Sponsor Name:      GSO Global Clinical Research B.V.
Trial protocol:    DE 
Medical condition: Patients with relapsed/refractory Acute Myeloid Leukemia (AML)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
EudraCT Number:    2006-001777-19
Full Title:        A Phase II multicenter study to assess the tolerability and efficacy of the addition of Bevacizumab to standard induction therapy in AML and 
high risk MDS above 60 years.
Sponsor Name:      HOVON foundation
Trial protocol:    NL BE 
Medical condition: Acute myeloid leukaemia (AML), AML FAB M0-M2 or M4-M7;
diagnosis with refractory anemia with excess of blasts (RAEB) or refractory anemia with excess of blasts in transformation (RAEB-T) with an IP...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
EudraCT Number:    2007-000273-35
Full Title:        A Phase II, Open-Label, Multi-centre, 2-part study to assess the Safety, Tolerability, and  Efficacy of Tipifarnib Plus Bortezomib in the Treatment of Newly Diagnosed Acute  Myeloid Leukemia  AML  ...
Sponsor Name:      AZIENDA OSPEDALIERA DI BOLOGNA POLICLINICO S. ORSOLA M. MALPIGHI
Trial protocol:    IT 
Medical condition: Acute Myeloid Leukemia
----------------------------------------------------------------------------------------------------------------------------------------------------------------

...and so on.