如何在不关闭共享磁盘的情况下避免抓取它？

Question

我正在使用Nutch。我打算抓取共享磁盘，而不是互联网网站。

我担心爬行它会使磁盘变得非常慢。如何在不关闭共享磁盘的情况下避免抓取它？

Answer 1

您可以在conf/nutch-site.xml中设置线程数和请求之间的等待时间。

尝试覆盖这些属性，并将它们设置为您喜欢的值：

<property>
  <name>fetcher.threads.fetch</name>
  <value>10</value>
  <description>The number of FetcherThreads the fetcher should use.
  This is also determines the maximum number of requests that are
  made at once (each FetcherThread handles one connection). The total
  number of threads running in distributed mode will be the number of
  fetcher threads * number of nodes as fetcher has one map task per node.
  </description>
</property>

<property>
  <name>fetcher.threads.per.queue</name>
  <value>1</value>
  <description>This number is the maximum number of threads that
    should be allowed to access a queue at one time.
   </description>
</property>