LWP::Simple & LWP::UserAgent不返回完整的网页内容

Question

我有一个内部本地机器，托管分析，我需要处理。如果您有URL，分析将以JSON格式返回。例如,

http://1.2.3.4:8081/analytics/mydata/myNodeData?flat

它会给出JSON文件。现在，我通过LWP::Simple (也尝试过LWP::UserAgent)将JSON文件放入一个变量，然后解析它。它适用于我的大多数URL。问题是，对于某些URL，它不会将全部数据读取到我的变量中，而只是部分读取。

use LWP::Simple;                # From CPAN
use LWP::UserAgent;
use JSON qw( decode_json );     # From CPAN
use Data::Dumper;               # Perl core module
use Data::Diver qw{ Dive };         # for diving in the Hash
use strict;                     # Good practice
use warnings;                   # Good practice

#using LWP Simple    
my $trendsurl = 'http://1.2.3.4:8081/analytics/mydata/myNodeData?flat';
my $json = get( $trendsurl );
die "Could not get $trendsurl!" unless defined $json;
my $decoded_json = decode_json( $json );

#using LWP::UserAgent
my $ua = LWP::UserAgent->new();
my $req = new HTTP::Request GET => $trendsurl;
my $res = $ua->request($req);
my $content = $res->content;

如果在这里有用的是内容( decoded_json)的调试信息的显示是相同的。

  DB<1> p $content
{"NodeStatus": {"deleted": false, "disk_usage_info": [{"partition_space_availabl
e_1k": 791475728, "partition_space_used_1k": 171611096, "partition_name": "/dev/
mapper/os-root", "partition_type": "ext4"}, {"partition_space_available_1k": 151
200, "partition_space_used_1k": 39244, "partition_name": "/dev/vda3", "partition
_type": "ext2"}], "process_info": [{"process_name": "XX-api:0", "process_s
tate": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_count": 2, "core_
file_list": [], "last_start_time": "1461822985334246", "stop_count": 0, "last_ex
it_time": null, "exit_count": 0}, {"process_name": "XX-config-nodemgr", "p
rocess_state": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_count": 2
, "core_file_list": [], "last_start_time": "1461822979324868", "stop_count": 0,
"last_exit_time": null, "exit_count": 0}, {"process_name": "XX-discovery:0
", "process_state": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_coun
t": 2, "core_file_list": [], "last_start_time": "1461822983332516", "stop_count"
: 0, "last_exit_time": null, "exit_count": 0}, {"process_name": "XX-svc-m

  DB<2>

您会看到日志文件突然终止了。如果我通过我的浏览器获得，它是完整的..

编辑：

在获取URL然后解码JSON时，我添加了以下eval语句。检索没有错误，但JSON解码失败，因为它不是一个完整的JSON。

eval { $decoded_json = parse_json( $json ) };
if ($@) {
    warn "func raised an exception: $@";
}
func raised an exception: JSON error at line 1: Unexpected end of input parsing 
string starting from byte 1105 at Tester.pl line 28.

编辑2:我只是尝试了另一种方式...

#!/usr/bin/perl  

 use LWP::UserAgent;  
 use HTTP::Request;  

 my $URL = 'http://1.2.3.4:8081/analytics/mydata/myNodeData?flat';  

 my $ua = LWP::UserAgent->new();  
 my $header = HTTP::Request->new(GET => $URL);  
 my $request = HTTP::Request->new('GET', $URL, $header);  
 my $response = $ua->request($request);  

 if ($response->is_success){  
     print "URL:$URL\nHeaders:\n";  
     print $response->headers_as_string;  
 }elsif ($response->is_error){  
     print "Error:$URL\n";  
     print $response->error_as_HTML;  
 } 

调试显示选择了$response-> is _success块，下面是输出。

URL:http://1.2.3.4:8081/analytics/mydata/myNodeData?flat
Headers:
Connection: close
Date: Tue, 10 May 2016 19:49:40 GMT
Content-Type: application/json
Client-Aborted: die
Client-Date: Tue, 10 May 2016 19:49:38 GMT
Client-Peer: 1.2.3.4:8081
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
X-Died: read failed: An existing connection was forcibly closed by the remote ho
st. at C:/Perl64/lib/LWP/Protocol/http.pm line 465.

Answer 1

我使用这个代码或者类似的东西，这个代码使用POST作为params，我得到了一个很大的JSON，它工作得很好

use LWP::UserAgent;
use Data::Dumper;
use JSON;
use JSON::Parse;

my $params = 
{(
'flat' => 1,
)};

    my $server_endpoint = "http://1.2.3.4:8081/analytics/mydata/myNodeData";
    my $ua = LWP::UserAgent->new;
    my $req = HTTP::Request->new();
    my $response = $ua->post( $server_endpoint, $params );
    my $result  = $response->decoded_content();

    $result = from_json($result);
    print Dumper $result;