Perl LWP::UserAgent模拟浏览器

Question

我试图自动获取网页使用LWP::UserAgent，但我得到403禁止的错误，而如果我从控制台使用wget，https://dreaminislam.com/a/或curl，我得到的页面正常。如何为LWP::UserAgent设置正确的选项以获取该页面，并类似于模拟真实浏览器。以下是示例代码。

use HTTP::CookieJar::LWP ();
use LWP::UserAgent;
use LWP::Simple;

    my $url = qq{https://dreaminislam.com/a/};
    my $content = getUrl($url);
    exit;

sub getUrl {

    my $url = shift;
    my $jar = HTTP::CookieJar::LWP->new;
    my $ua = LWP::UserAgent->new(timeout => 180,  cookie_jar => $jar, protocols_allowed => ['http', 'https']);
    $ua->agent(qq{Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0});

    my $response = $ua->get($url);

    if ($response->is_success) {
        my $content = $response->decoded_content;
        return $content;
    } else {
        my $content = $response->decoded_content;
        printf "Get url error [%d] %s.\n", $response->code, $response->message;
    }
}

Answer 1

看起来在这个站点上安装了一些反机器人保护。它似乎至少需要一个User-Agent和一个Accept报头：

use LWP::UserAgent;
use HTTP::Request;

my $ua = LWP::UserAgent->new();

my $req = HTTP::Request->new(GET => 'https://dreaminislam.com/a/');
$req->header('User-Agent' => 'Mozilla/5.0');
$req->header('Accept' => '*/*');
my $response = $ua->request($req);

die $response->code if ! $response->is_success;
print $response->decoded_content;