Web::Scraper和Perl

Question

我有以下脚本，它抓取我的学校CS部门以获得所有课程的列表。我想能够提取CRN (课程号)和其他重要信息放在一个数据库中，我可以让用户通过网络应用程序浏览。

下面是一个示例网址：http://courses.illinois.edu/cis/2011/spring/schedule/CS/411.html

我想从这样的页面中提取信息。抓取器的第一层只是从所有课程的列表中构建单独的站点。一旦我到达了课程特定的目录页面，我就会使用第二个抓取器来尝试获取所有我想要的信息。出于某种原因，尽管CRN和课程讲师都是“td”元素。我的刮刀在刮的时候好像没有返回任何东西。我尝试专门抓取“div”，但我得到了每个相关页面的一堆信息。因此，不知何故，我没有得到'td‘元素，但我从正确的页面抓取。

  my $tweets = scraper {
      # Parse all LIs with the class "status", store them into a resulting
      # array 'tweets'.  We embed another scraper for each tweet.
     # process "h4.ws-ds-name.detail-title", "array[]" => 'TEXT';
      process "div.ws-row", "array[]" => 'TEXT';
      };

my $res = $tweets->scrape( URI-    >new("http://courses.illinois.edu/cis/2011/spring/schedule/CS/index.html?skinId=2169") );

foreach my $elem (@{$res->{array}}){

my $coursenum = substr($elem,2,4);

my $secondLevel = scraper{
process "td.ws-row", "array2[]" => 'TEXT';
};

my $res2 = $secondLevel->scrape(URI-    >new("http://courses.illinois.edu/cis/2011/spring/schedule/CS/$coursenum.html"));
my $num = @{$res2->{array2}};
print $num;

print "---------------------", "\n";
my @curr = @{$res2->{array2}};
foreach my $elem2 (@curr){
$num++;
print $elem2, "    ", "\n";
}
print "---------------------", "\n";
}

有什么想法吗？

谢谢

Answer 1

在我看来就像

my $coursenum = substr($elem,2,4)

应该是

my $coursenum = substr($elem,3,3)

Answer 2

在这种情况下，最简单的方法是使用

HTML::TableExtract

以防您仅从表中查找数据。

Answer 3

我玩弄了一下你的问题。您可以在initial scraper中获取课程id、标题和各个课程页面的链接：

my $courses = scraper {
    process 'div.ws-row',
        'course[]' => scraper {
            process 'div.ws-course-number',  'id'    => 'TEXT';
            process 'div.ws-course-title',   'title' => 'TEXT';
            process 'div.ws-course-title a', 'link'  => '@href';
        };
    result 'course';
};

抓取的结果是具有如下hashrefs的arrayref：

{   id    => "CS 103",
    title => "Introduction to Programming",
    link  => bless(do{\(my $o = "http://courses.illinois.edu/cis/2011/spring/schedule/CS/103.html?skinId=2169")}, "URI::http"),
},
....

然后，您可以从各个页面对每个课程进行额外的抓取，并将这些信息添加到原始结构中：

for my $course (@$res) {
    my $crs_scraper = scraper {
        process 'div.ws-description', 'desc' => 'TEXT';
        # ... add more items here
    };
    my $additional_data = $crs_scraper->scrape(URI->new($course->{link}));

    # slice assignment to add them into course definition
    @{$course}{ keys %$additional_data } = values %$additional_data;
}

组合在一起的源码如下：

use strict; use warnings;
use URI;
use Web::Scraper;
use Data::Dump qw(dump);

my $url = 'http://courses.illinois.edu/cis/2011/spring/schedule/CS/index.html?skinId=2169';

my $courses = scraper {
    process 'div.ws-row',
        'course[]' => scraper {
            process 'div.ws-course-number',  'id'    => 'TEXT';
            process 'div.ws-course-title',   'title' => 'TEXT';
            process 'div.ws-course-title a', 'link'  => '@href';
        };
    result 'course';
};

my $res = $courses->scrape(URI->new($url));

for my $course (@$res) {
    my $crs_scraper = scraper {
        process 'div.ws-description', 'desc' => 'TEXT';
        # ... add more items here
    };
    my $additional_data = $crs_scraper->scrape(URI->new($course->{link}));

    # slice assignment to add them into course definition
    @{$course}{ keys %$additional_data } = values %$additional_data;
}

dump $res;