PHP Error[2]：fopen(fopen

Question

我正在尝试下载robots.txt文件的内容

我最初的问题链接：PHP file_exists() for URL/robots.txt returns false

这是第22行：$f = fopen($file, 'r');

我得到了这个错误：

PHP Error[2]: fopen(http://www1.macys.com/robots.txt): failed to open stream: Redirection limit reached, aborting
    in file /host/chapache/host/apache/www/home/flaviuspogacian/proiecte/Mickey_ClosetAffair_Discovery/webroot/protected/modules/crawler/components/Robots.php at line 22
#0 /host/chapache/host/apache/www/home/flaviuspogacian/proiecte/Mickey_ClosetAffair_Discovery/webroot/protected/modules/crawler/components/Robots.php(22): fopen()

对于此代码，其中$website_id是一个数字，而$website类似于http://www.domain.com/

public function read_website_save_2_db($website_id, $website) {
    $slashes = 0;
    for ($i = 0; $i < strlen($website); $i++)
        if ($website[$i] == '/')
            $slashes++;
    if ($slashes == 2)
        $file = $website . '/robots.txt';
    else
        $file = $website . 'robots.txt';
    echo $website_id . ' ' . $file . PHP_EOL;
    try {
        $f = fopen($file, 'r');
        if (($f) || (strpos(get_headers($file, 1), "404") !== FALSE)) {
            fclose($f);
            echo 'exists' . PHP_EOL;
            $curl_tool = new CurlTool();
            $content = $curl_tool->downloadFile($file, ROBOTS_TXT_FILES . 'robots_' . $website_id . '.txt');
            //if the file exists on local disk, delete it
            if (file_exists(ROBOTS_TXT_FILES . 'robots_' . $website_id . '.txt'))
                unlink(ROBOTS_TXT_FILES . 'robots_' . $website_id . '.txt');
            echo ROBOTS_TXT_FILES . 'robots_' . $website_id . '.txt', $content . PHP_EOL;
            file_put_contents(ROBOTS_TXT_FILES . 'robots_' . $website_id . '.txt', $content);
        }
    else {
        echo 'maybe it\'s not there' . PHP_EOL;
    }
} catch (Exception $e) {
    echo 'EXCEPTION ' . $e . PHP_EOL;
}

}

Answer 1

你的部分代码看起来很混乱。我会这样做(当然不是从函数内部回显，只是举个例子)

public function read_website_save_2_db($website_id, $website) {
  $url = rtrim($website, '/') . '/robots.txt';
  $content = @file_get_contents($url);
  $status = 0;
  $success = false;
  if( !empty($http_response_header) ) {
    foreach($http_response_header as $header) {
      if(substr($header, 0, 6) == 'HTTP/1') {
        $status = trim(substr($header, strpos($header, ' '), strlen($header)));
        $success = strnatcasecmp($status, '200 OK') === 0;
        break;
      }
    }
  }
  if(!$success) {
    echo 'Request failed with status '.$status;
  }
  elseif(!$content) {
    echo 'Website responded with empty robots.txt';
  }
  else {
    file_put_contents(ROBOTS_TXT_FILES . 'robots_' . $website_id . '.txt', $content);
    echo 'Wii, we have downloaded a copy of '.$url;
  }
}