Web Crawler -从2000多个网页中获取数据(TED网站示例)

Question

我正在编写一个php crone作业脚本，每周运行一次。

这个脚本的主要目的是从TED网站上获得所有TED演讲的细节(例如，让这个问题更容易理解)。

这个脚本将花费大约70分钟的时间运行，它将超过2000网页。

我的问题是：

( 1)是否有一种更好/更快的方法，每次都能得到网页，使用以下功能：

file_get_contents_curl($url)

2)把所有的会谈都安排成一个数组(可能会变得相当大)，这是一个好做法吗？

3)一般来说，是否有更好的方法从网站上获取所有ted的详细信息？在TED网站上“爬行”获得所有演讲的最佳方式是什么？

**我检查了使用rss提要的选项，但缺少了一些我需要的细节。

谢谢

<?php
define("START_ID", 1);
define("STOP_TED_QUERY",20);
define ("VALID_PAGE","TED | Talks");
/**
 * this script will run as a cron job and will go over all pages
 * on TED http://www.ted.com/talks/view/id/ 
 * from id 1 till there are no more pages
 */

/**
 * function get a file using curl (fast)
 * @param $url - url which we want to get its content
 * @return the data of the file
 * @author XXXXX
 */
function file_get_contents_curl($url)
{
    $ch = curl_init();

    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

    $data = curl_exec($ch);
    curl_close($ch);

    return $data;
}

//will hold all talks in array
$tedTalks = array();

//id to start the query from
$id=START_ID;

//will indicate when needed to stop the query beacuse reached the end id's on TED website
$endOFQuery=0;

//get the time
$time_start = microtime(true);

//start the query on TED website
//if we will query 20 pages in a row that do not exsist we will stop the querys and assume there are no more
while ($endOFQuery < STOP_TED_QUERY){

    //get the page of the talk
    $html = file_get_contents_curl("http://www.ted.com/talks/view/id/$id");

    //parsing begins here:
    $doc = new DOMDocument();
    @$doc->loadHTML($html);
    $nodes = $doc->getElementsByTagName('title');

    //get and display what you need:
    $title = $nodes->item(0)->nodeValue;


    //check if this a valid page
    if (! strcmp ($title , VALID_PAGE ))
        //this is a removed ted talk or the end of the query so raise a flag (if we get anough of these in a row we will stop)
        $endOFQuery++;
    else {
        //this is a valid TED talk get its details

        //reset the flag for end of query
        $endOFQuery = 0;

        //get meta tags
        $metas = $doc->getElementsByTagName('meta');

        //get the tag we need (keywords)
        for ($i = 0; $i < $metas->length; $i++)
        {
            $meta = $metas->item($i);
            if($meta->getAttribute('name') == 'keywords')
                $keywords = $meta->getAttribute('content');
        }

        //create new talk object and populate it
        $talk = new Talk();
        //set its ted id from ted web site
        $talk->setID($id);
        //parse the name (name has un-needed char's in the end)
        $talk->setName( substr($title, 0, strpos( $title, '|')) );

        //parse the String of tags to array
        $keywords = explode(",", $keywords);
        //remove un-needed items from it
        $keywords=array_diff($keywords, array("TED","Talks"));

        //add the filters tags to the talk
        $talk->setTags($keywords);

        //add to the total talks array
        $tedTalks[]=$talk;
    }

    //move to the next ted talk ID to query
    $id++;
} //end of the while

$time_end = microtime(true);
$execution_time = ($time_end - $time_start);
echo "this took (sec) : ".$execution_time;

?>

Answer 1

在github.com上获得了一个web爬虫php示例

如果有人在寻找它是如何工作的

https://github.com/Nimrod007/TED-talks-details-from-TED.com-and-youtube

我已经在实现这个脚本https://market.mashape.com/bestapi/ted的Mashape上发布了一个freemium api

享受吧！