Web刮除html加油站

Question

鉴于这个链接，http://www.nj.gov/treasury/administration/statewide-support/motor-fuel-locations.shtml，我想刮每个加油站和它的信息

import requests
from bs4 import BeautifulSoup
page=requests.get("http://www.nj.gov/treasury/administration/statewide-support/motor-fuel-locations.shtml")
soup=BeautifulSoup(page.content,'html.parser')
for x in soup.find_all('p'):
    print x

提取正确数据的下一步是什么？

Answer 1

对于给定的HTML，您几乎没有什么可供选择的。所以你可以取消选择你不想要的。您想要第6个表，就不需要带有"colspan“属性或任何”宽度“属性的td元素。但是td元素必须有数值顶部。

import requests
from bs4 import BeautifulSoup
page=requests.get("http://www.nj.gov/treasury/administration/statewide-support/motor-fuel-locations.shtml")
soup=BeautifulSoup(page.content,'html.parser')
table = soup.find_all('table')[5]
for x in table.find_all('td', {'valign': 'top', 'colspan': None, 'width':None}):
    print (x.text)
    print ('#############')

响应评论更新了

import requests
from bs4 import BeautifulSoup
import pprint
import re

page=requests.get("http://www.nj.gov/treasury/administration/statewide-support/motor-fuel-locations.shtml")
soup=BeautifulSoup(page.content,'html.parser')
table = soup.find_all('table')[5]

tmps = [i.text for i in table.find_all('td', {'valign': 'top', 'colspan': None, 'width':None})]
my_dict = dict()
for tmp in tmps:
    if len(tmp.strip()) != 0:
        # Clean up the output and add to dictionary.
        my_dict[re.sub( '\s+', ' ', tmp.split('\n', 1)[0] ).strip()] = re.sub( '\s+', ' ', tmp ).strip()

pp = pprint.PrettyPrinter(indent=4)
pp.pprint (my_dict)

产出：

{   'Bayside Facility': 'Bayside Facility 4294 Rt. 47 Leesburg Open 24 Hrs '
                        'Unleaded / Diesel 856-785-0040 X-5429 Latitude: '
                        '39.23339997 Longitude: -74.96568202',
    'Bedminster DOT': 'Bedminster DOT 455 Rt. 202/206 South Pluckemin Open 24 '
                      'Hrs Unleaded / Diesel 908-234-2130 Latitude: '
                      '40.65123677 Longitude: -74.64499021',
    'Berlin DOT': 'Berlin DOT 36 Walker Ave. Berlin Open 24 Hrs Unleaded / '
                  'Diesel 856-767-7717 Latitude: 39.80369329 Longitude: '
                  '-74.93442722',
    'Bloomsbury DOT': 'Bloomsbury DOT 1000 Rt. 173 Bloomsbury Open 24 Hrs '
                      'Unleaded / Diesel 908-479-4851 Latitude: 40.66078600 '
                      'Longitude: -75.06664165',
    'Bordentown DOT': 'Bordentown DOT Dunns Mill Rd. -off Rt. 130 Bordentown '
                      'Unleaded -Open 24 Hrs Diesel – 7:30 am -3:45 pm M-F '
                      '609-298-2980 Latitude: 40.13178135 Longitude: '
                      '-74.71658907',
...

但它错过了两个加油站：

<td valign="top" width="235"><p>Elizabeth DOT<br />
<td align="top" width="264">Summit DOT<br />

由于它们具有宽度属性，因此在HTML中也没有选择这些属性的方法。