从雅虎批量下载数据

Question

我想使用Perl批量从Yahoo下载数据。

• 首先，我需要将股票代码从txt文件读取到数组中。
• 第二，我需要循环30个代码来下载。
• 第三，我需要将结果打印到txt文件中。

我做了如下的事情

#!D:/Perl/perl/bin/perl.exe

use strict;
use warnings;

use JSON;
use Data::Dumper;
use LWP::Simple;

my $rootDir = 'D:/Perl/data';

# First read stockcodes from txt file into array (Need advise)
my $file = $rootDir . '/HKGListYH.txt';
open(FH, '<' . $file) or die("cannot open " . $file);
while (my $record = <FH>) {
  chomp($record);
  $record =~ s/\s*$//;
  my @content = split(/\t/, $record);
}
close(FH);

# Second I need to loop 30 codes to download (Need advise)
my $livePriceLink  = 'http://download.finance.yahoo.com/d/quotes.csv?s=';
my $livePriceLink2 = '&f=snl1p2a2p5p6rydj1s6j4t8s7m3m8m4m6kk5jk5&e=.csv';

# some for loop needed
my $dataRef = get($livePriceLink . join(',', @content) . join('', $livePriceLink2));

# Third I need to print the results into txt file (Need advise)
my @header = ("t", "name", "l", "cp", "av", "p/s", "p/b", "p/e", "d/p", "d", "mc");
open(FH, '>' . $rootDir . '\terryHKGResults.txt');
print FH join("\t", @header) . "\n";
print FH $dataRef;
close(FH);

示例股票代码文本文件

0001.HK
0002.HK
0003.HK
0004.HK
0005.HK
0006.HK
0007.HK
0008.HK
0009.HK
0010.HK
0011.HK
0012.HK
0013.HK
0014.HK
0015.HK

示例下载雅虎数据

http://download.finance.yahoo.com/d/quotes.csv?s=1288.HK,0753.HK,0914.HK,3988.HK,3328.HK,1211.HK,0939.HK,1359.HK,1898.HK,1800.HK,2628.HK,0916.HK,2883.HK,0390.HK,1088.HK,0728.HK,2202.HK,0998.HK,6030.HK,3968.HK,3323.HK,2601.HK,0489.HK,2238.HK,2333.HK,6837.HK,0902.HK,1398.HK,0358.HK,1988.HK,1336.HK,0857.HK,1339.HK,2328.HK,2318.HK,0386.HK,1099.HK,0168.HK,2338.HK,1066.HK&f=snl1p2a2p5p6rydj1s6j4m8m6k5j6&e=.csv

Answer 1

从数组开始到为空，您需要splice所需的项数。

while ( @contents ) {
    my @chunk = splice @contents, 0, 30;
    my $url = $livePriceLink . join(',', @chunk) . $livePriceLink2;
    my $dataRef = get($url);
    ...
}

更新

下面是一个例子，说明它在上下文中如何工作。您必须扩展print $data，因为您需要处理可能是JSON数据的内容，以提取您需要的字段

请注意，我使用了autodie，这样就不必检查open和chdir调用的状态；我对根目录执行了一个chdir，以避免在连接路径字符串时出现混乱；我已经将所有常量字符串声明为程序顶部的常量；我使用URI模块来更方便地处理URL；删除了不必要的(在Windows上) shebang行

use strict;
use warnings;
use 5.010;
use autodie;

use URI;
use URI::QueryParam;
use LWP::Simple;
use JSON;

use constant ROOT_DIR       => 'D:/Perl/data';
use constant STOCK_CODES    => 'HKGListYH.txt';
use constant RESULTS        => 'terryHKGResults.txt';
use constant LIVE_PRICE_URL => 'http://download.finance.yahoo.com/d/quotes.csv';

chdir ROOT_DIR;

open my $stock_fh, '<', STOCK_CODES;
my @content = <$stock_fh>;
close $stock_fh;
s/\s+\z// for @content;

my $url = URI->new(LIVE_PRICE_URL);
$url->query_form(
  f => 'snl1p2a2p5p6rydj1s6j4t8s7m3m8m4m6kk5jk5',
  e => '.csv',
);

my @header = qw{  t  name  l  cp  av  p/s  p/b  p/e  d/p  d  mc  };
open my $results_fh, '>', RESULTS;
select $results_fh;
print join("\t", @header), "\n";

while ( @content ) {
    my @chunk = splice @content, 0, 30;
    $url->query_param(s => join(',', @chunk));
    say $url;

    my $data = get($url);

    print $data;  # Needs expanding
}

close $results_fh;