抓取网页的策略，最大化收集的信息

Question

以下是问题所在：

用户注册一个网站，可以从8个工作类别中选择一个，或者选择跳过这一步。我想根据电子邮件地址中的域名，将跳过这一步的用户分类为工作类别。

当前设置：

使用Beautiful Soup和nltk的组合，我抓取主页并查找站点上包含单词"about“的页面的链接。我也刮掉了那一页。我在这篇文章的末尾复制了一小段代码来进行抓取。

问题是：

我没有得到足够的数据来建立一个好的学习程序。我想知道我的抓取算法是否为成功而设置--换句话说，我的逻辑中是否有任何漏洞，或者有什么更好的方法来确保我有一个很好的文本块来描述公司所做的工作？

(相关)代码：

import bs4 as bs
import httplib2 as http
import nltk


# Only these characters are valid in a url
ALLOWED_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]@!$&'()*+,;="


class WebPage(object):
    def __init__(self, domain):
        """
            Constructor

            :param domain: URL to look at
            :type domain: str
        """
        self.url = 'http://www.' + domain

        try:
            self._get_homepage()
        except: # Catch specific here?
            self.homepage = None

        try:
            self._get_about_us()
        except:
            self.about_us = None

    def _get_homepage(self):
        """
            Open the home page, looking for redirects
        """
        import re

        web = http.Http()
        response, pg = web.request(self.url)

        # Check for redirects:
        if int(response.get('content-length',251)) < 250:
            new_url = re.findall(r'(https?://\S+)', pg)[0]
            if len(new_url): # otherwise there's not much I can do...
                self.url = ''.join(x for x in new_url if x in ALLOWED_CHARS)
                response, pg = web.request(self.url)

        self.homepage = self._parse_html(nltk.clean_html(pg))
        self._raw_homepage = pg

    def _get_about_us(self):
        """
            Soup-ify the home page, find the "About us" page, and store its contents in a
            string
        """
        soup = bs.BeautifulSoup(self._raw_homepage)
        links = [x for x in soup.findAll('a') if x.get('href', None) is not None]
        about = [x.get('href') for x in links if 'about' in x.get('href', '').lower()]

        # need to find about or about-us
        about_us_page = None
        for a in about:
            bits = a.strip('/').split('/')
            if len(bits) == 1:
                about_us_page = bits[0]
            elif 'about' in bits[-1].lower():
                about_us_page = bits[-1]

        # otherwise assume shortest string is top-level about pg.
        if about_us_page is None and len(about):
            about_us_page = min(about, key=len)

        self.about_us = None
        if about_us_page is not None:
            self.about_us_url = self.url + '/' + about_us_page
            web = http.Http()
            response, pg = web.request(self.about_us_url)
            if int(response.get('content-length', 251)) > 250:
                self.about_us = self._parse_html(nltk.clean_html(pg))

    def _parse_html(self, raw_text):
        """
            Clean html coming from a web page. Gets rid of
                - all '\n' and '\r' characters
                - all zero length words
                - all unicode characters that aren't ascii (i.e., &...)
        """
        lines = [x.strip() for x in raw_text.splitlines()]
        all_text = ' '.join([x for x in lines if len(x)]) # zero length strings
        return [x for x in all_text.split(' ') if len(x) and x[0] != '&']

Answer 1

这超出了您的要求，但我会考虑调用已经收集了此信息的外部数据源。查找此类服务的好地方是在Programmable Web上(例如Mergent Company Fundamentals)。并非Programmable Web上的所有数据都是最新的，但似乎有很多API提供程序。