Web-crawler循环

Question

我在我的网络爬虫中写了下面的循环。

它在几秒钟后就会用完。我不知道为什么。

def crawlweb(seed):
    crawled = []
    tocrawl = [seed]
    page = tocrawl[0]
    while tocrawl:
        if page not in crawled:
            tocrawl = tocrawl[1:] + (get_links(get_page(page)))
            crawled.append(page)
    return crawled, tocrawl

Answer 1

def crawl_web(seed)
tocrawl = [seed]
crawled = []
while tocrawl:
    page = tocrawl.pop()
    if page not in crawled:
        union(tocrawl, get_all_links(get_page(page)))
        crawled.append(page)
return crawled