从URL同时下载文件

Question

我们有一个perl脚本，可以从不同的服务器下载文件。目前要花上几个小时。在哪里可以找到一种同时下载多个文件的方法。谢谢你谢尔盖。

我仍然试图从开发团队获得全部细节，但我正在寻找一些提示。

sub getConfData {
    my $mech = WWW::Mechanize->new( autocheck => 1 );

    print "USER NAME :- $Inputs::conf_user\n";
    print "USER PASS :- $Inputs::conf_pass\n";

    $mech->credentials( "$Inputs::conf_user" => "$Inputs::conf_pass" );
    logs( $Inputs::logpath, "Opening the URL of confD : $Inputs::conf_url" );
    print "Opening the URL of confD : $Inputs::conf_url\n";
    
    $mech->mirror( $Inputs::conf_url, $Inputs::conf_arch );

    my $next = Archive::Tar->iter( $Inputs::conf_arch, 1, { filter => qr// } );
    my $confdName = $next->()->name;
    logs( $Inputs::logpath, "confD Downloading Filename is  : $confdName" );
    print "confD Downloading Filename is  : $confdName\n";
    my $tar = Archive::Tar->new();
    $tar->read($Inputs::conf_arch) or die logs( $Inputs::errorpath, " Unable to read the TAR file." );
    $tar->extract();

    my $destination = "$Inputs::download_path" . "$Inputs::confddb";
    rmtree($destination);
    my $cwd = getcwd();
    move_reliable( "$confdName", "$destination" ) or logs( $Inputs::errorpath, " unable to move folder to $destination." );
    logs( $Inputs::logpath, "Moving confD $confdName to $Inputs::download_path" . "$Inputs::confddb" );
    print "Moving confD $confdName to $Inputs::download_path $Inputs::confddb\n";

    if ($@) {
        my $message = "Failed to pull confd data and write it to $Inputs::download_path :: $@";
        send_mail( $Inputs::from, $Inputs::to, $Inputs::error_subject, $message );
        logs( $Inputs::errorpath, " $message" );
        die("$message");
    }
    else {
        unlink $Inputs::conf_arch or die logs( $Inputs::errorpath,"COULD NOT UNLINK $Inputs::conf_arch" );
        unlink $destination;
       }
   }


 
    foreach my $server (@productList) {
           my $pid;
           if ( defined( $pid = fork ) ) {
               if ( !$pid ) {
                   exec("$main_file $server &");
                   die "Error executing command: $!\n";
               }
           }
           else {
               die "Error in fork: $!\n";
              }
           }
 
       logs( $Inputs::logpath, "Downloading config started at :" . datetimes( 'dtime', 'db' ) );
        &getConfData();
       logs( $Inputs::logpath, "Downloading config completed at :" . datetimes( 'dtime', 'db' ) );

        print "Downloading config Completed at : "`your text`. datetimes( 'dtime', 'normal' ) . "\n";
        logs( $Inputs::logpath, "Downloading config completed at :" . datetimes( 'dtime', 'db' ) );

Answer 1

将Mojo::UserAgent与接受承诺的方法一起使用。有关扩展示例，请参见你答应给我打电话。

您也可以在Perl中使用一些其他并发性，但是由于Mojolicious内置了它，所以我将使用它。

或者，您可以非常喜欢使用仆从。每个链接都被插入到作业队列中。这样，您可以很容易地标记哪些您需要重试，等等。