首页
学习
活动
专区
圈层
工具
发布
社区首页 >问答首页 >有没有办法让我的Python程序运行得更快?

问有没有办法让我的Python程序运行得更快?
EN

Stack Overflow用户
提问于 2011-12-31 00:43:31
回答 2查看 291关注 0票数 0

我用python写了一个计算机程序,但它的运行速度比我想要的要慢得多。

代码如下:

代码语言:javascript
复制
from gzip import GzipFile
from cStringIO import StringIO
import re
import webbrowser
import time
from difflib import SequenceMatcher
import os
import sys
from BeautifulSoup import BeautifulSoup
import eventlet
from eventlet.green import urllib2
import urllib
import urllib2
import cookielib

TITLE_MATCH = re.compile(r'(.*) \(\d{1,10}.{1,100}\)$')
ADDRESS_MATCH = re.compile(r'.{1,100}\((.*), .{4,14}, United States\)$')
LOCATION_LISTING = re.compile(r'http://www\.locationary\.com/place/en/US/.{1,50}/.{1,50}/.{1,100}\.jsp')

def download(url):
    print "Downloading:", url
    s = urllib2.urlopen(url).read()
    if s[:2] == '\x1f\x8b': # assume it's gzipped data
        ifh = GzipFile(mode='rb', fileobj=StringIO(s))
        s = ifh.read()
    print "Downloaded: ", url
    return s

def replace_chars(text, replacements):
    return ''.join(replacements.get(x,x) for x in text)

def handle_listing(listing_url):
    listing_document = BeautifulSoup(download(listing_url))

    # ignore pages that link to yellowpages
    if not listing_document.find("a", href=re.compile(re.escape("http://www.yellowpages.com/") + ".*")):
        listing_title = listing_document.title.text
        reps = {' ':'-', ',':'', '\'':'', '[':'', ']':''}
        if TITLE_MATCH.match(listing_title) is not None:
            title, = TITLE_MATCH.match(listing_title).groups()
            address, = ADDRESS_MATCH.match(listing_title).groups()

            yellow_page_url = "http://www.yellowpages.com/%s/%s?order=distance" % (
                replace_chars(address, reps),
                replace_chars(title, reps),
            )

            yellow_page = BeautifulSoup(download(yellow_page_url))

            page_url = yellow_page.find("h3", {"class" : "business-name fn org"})
            if page_url:
                page_url = page_url.a["href"]

                business_name = title[:title.index(",")]

                page = BeautifulSoup(download(page_url))
                yellow_page_address =  page.find("span", {"class" : "street-address"})
                if yellow_page_address:

                    if SequenceMatcher(None, address, yellow_page_address.text).ratio() >= 0.5:
                        pid, = re.search(r'p(\d{5,20})\.jsp', listing_url).groups(0)
                        page_escaped = replace_chars(page_url, {':':'%3A', '/':'%2F', '?':'%3F', '=':'%3D'})

                        final_url = "http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=%s&xxx_c_1_f_987=%s" % (
                                pid, page_escaped)
                        return final_url

def log_in(final_url):
    data = urllib.urlencode({"inUserName":"jacob.grannis@gmail.com", "inUserPass":"secretword"})
    jar = cookielib.FileCookieJar("cookies")
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar))
    opener.addheaders.append(('User-agent', 'Mozilla/4.0'))
    opener.addheaders.append(('Referer', 'http://www.locationary.com/'))
    opener.addheaders.append(('Cookie','site_version=REGULAR; __utma=47547066.912030359.1322003402.1324959960.1325009956.58; __utmz=47547066.1324655802.52.13.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=cache:dr23PN5fUj4J:www.locationary.com/%20locationary; nickname=jacob501; jforumUserId=1; PMS=1; locaCountry=1033; locaState=1786; locaCity=Vancouver; JSESSIONID=5CDDA2D527C20A6CDD04936115DE3FA2; PSESSIONID=c677beb4e6b8d58f1443d9b9585b225f579ef29a; Locacookie=enable; __utmb=47547066.1.10.1325009956; __utmc=47547066'))
    opener.addheaders.append(('Cookie','Cookie: site_version=REGULAR; __utma=47547066.912030359.1322003402.1324959960.1325009956.58; __utmz=47547066.1324655802.52.13.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=cache:dr23PN5fUj4J:www.locationary.com/%20locationary; nickname=jacob501; jforumUserId=1; PMS=1; locaCountry=1033; locaState=1786; locaCity=Vancouver; JSESSIONID=5CDDA2D527C20A6CDD04936115DE3FA2; PSESSIONID=c677beb4e6b8d58f1443d9b9585b225f579ef29a; Locacookie=enable; __utmb=47547066.4.10.1325009956; __utmc=47547066'))
    request = urllib2.Request("https://www.locationary.com/index.jsp?ACTION_TOKEN=tile_loginBar_jsp$JspView$LoginAction", data)
    response = opener.open(request) 
    url = str(final_url)
    anything = opener.open(url)
    page = anything.read()

States = [#'Alabama',
          #'Alaska',
          'Arizona',
          'Arkansas',
          'California',
          'Colorado',
          'Connecticut',
          'Delaware',
          'Florida',
          'Georgia',
          'Hawaii',
          'Idaho',
          'Illinois',
          'Indiana',
          'Iowa',
          'Kansas',
          'Kentucky',
          'Louisiana',
          'Maine',
          'Maryland',
          'Massachusetts',
          'Michigan',
          'Minnesota',
          'Mississippi',
          'Missouri',
          'Montana',
          'Nebraska',
          'Nevada',
          'New_Hampshire',
          'New_Jersey',
          'New_Mexico',
          'New_York',
          'North_Carolina',
          'North_Dakota',
          'Ohio',
          'Oklahoma',
          'Oregon',
          'Pennsylvania',
          'Rhode_Island',
          'South_Carolina',
          'South_Dakota',
          'Tennessee',
          'Texas',
          'Utah',
          'Vermont',
          'Virginia',
          'Washington',
          'West_Virginia',
          'Wisconsin',
          'Wyoming']

Cities = []

def find_cities(state):
    state_url = 'http://www.locationary.com/place/en/US/' + str(state)
    state_document = download(str(state_url))
    findCities = re.compile('<b>(.*)</b>')
    getCities = re.findall(findCities,state_document)

    for City in getCities:
        reps = {' ':'_'}
        City = replace_chars(City, reps)
        Cities.append(str(City))

bestworst = ['0','1']

def main():
    for state in States:
        find_cities(state)
        for city in Cities:
            for num in range(0,1):
                for pagenum in range(15,16):
                    print '------------------------------------------------------------------------------------------------------------------------------------------------------------'
                    print '------------------------------------------------------------------------------------------------------------------------------------------------------------'
                    if str(num) == '0':
                        print str(state) + ', ' + str(city) + ', ' + 'Best Profiles' + ', ' + 'Page ' + str(pagenum)
                    else:
                        print str(state) + ', ' + str(city) + ', ' + 'Worst Profiles' + ', ' + 'Page ' + str(pagenum)
                    START_URL = 'http://www.locationary.com/place/en/US/' + str(state) + '/' + city + '-page' + str(pagenum) + '/?ACTION_TOKEN=NumericAction&order=' + str(num)
                    pool = eventlet.GreenPool()
                    listings_document = BeautifulSoup(download(START_URL))
                    listings = listings_document.findAll("a", href = LOCATION_LISTING)
                    listings = [listing['href'] for listing in listings]

                    count_listings = 0

                    for final_url in pool.imap(handle_listing, listings):
                        print final_url
                        if final_url is not None:
                            log_in(final_url)

if __name__ == '__main__':
    main()

有没有办法让它更快,或者是不可能的?它必须从互联网下载网址的很多,但我很确定我不能使我的互联网连接速度10到50倍,已经is...And我的电脑不是很slow...so,有什么方法可以让我的程序,比如说,10-50倍快?我知道这听起来很可笑,但是专业程序员怎么才能让他们的程序更快呢?

EN

回答 2

Stack Overflow用户

发布于 2011-12-31 00:49:03

加速任何程序的第一步是理解为什么它很慢--也就是,时间去哪里了?程序员用来做这件事的工具叫做剖析器。标准Python包括其中的几个:您可以通过here.了解它们

一旦你学会了使用分析器,就在你的程序上运行它来识别热点,或者程序花费时间最多的地方。然后尝试使用以下两种方法之一来加速程序:

  1. 尝试缩短作用点的执行时间;或者
  2. 尝试缩短作用点的执行次数。

通常情况下,#2更有成效。选择更好或更合适的算法可以减少执行的代码量。

不要浪费时间去猜测为什么程序很慢;测量它,然后投入你的精力去解决真正的问题。众所周知,程序员不善于猜测性能问题出在哪里。

票数 6
EN

Stack Overflow用户

发布于 2011-12-31 00:52:09

程序员优化代码的方式是使用探查器,python提供了几种可用的方法。Here is a great article 帮助您入门。

您可以从命令行调用timeit:

代码语言:javascript
复制
python -m timeit myprogram.py

上面的链接有一大堆使用timeit的例子。一旦你找出了瓶颈所在,你就可以想办法解决它们了。如果您的程序在download()函数中花费了过多的时间,那么您可以考虑引入某种并发性,并在后台下载内容,同时您的程序继续使用BeautifulSoup从已下载的内容中解析提取信息。

这里的关键是要看到:

  1. 是你的程序花费时间最多的地方。
  2. 步骤1中的位置,在那里你可以优化最简单的

举一个假设的例子,如果你的正则表达式写得特别糟糕,它们可能会花费很长时间,然后你就可以进行优化了。我之所以说“假设”,是因为在实践中,您的正则表达式不太可能成为一个重要的瓶颈,除非您要执行它们数百万次或类似的奇怪事情。

票数 2
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/8681766

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档