Nutch2.1 distributed fetching

On the basis of this article http://www.linuxidc.com/Linux/2014-01/95796.htm.

1 Prepare the environment ：Hadoop colony 、java、mysql database , The code can be in eclipse Run in , You can insert data into... In stand-alone mode mysql database .

2 Modify the configuration file nutch-site.xml：

<property>

  <name>plugin.folders</name>

  <value>./plugins</value>

  <description>Directories where nutch plugins are located.  Each

  element may be a relative or absolute path.  If absolute, it is used

  as is.  If relative, it is searched for on the classpath.</description>

</property>

stay eclipse Choose buil.xml,run as ant, function runtime, A successful run will result in a folder runtime.

3 hold runtime Upload the folder to hadoop colony Medium master The server （ No other validation The server Is it possible to ）, The location after I upload is :/home/hadoop/nutch/runtime, Set the environment variable ：

stay /etc/profile in ：export NUTCH_HOME=/home/hadoop/nutch/runtime/local  source /etc/profile Make changes work .

4 It should be url Upload the seed file to hadoop. My seed file never succeeded , Skip this step .

5 stay /home/hadoop/nutch/runtime/deploy Run in directory ：

./bin/nutch crawl  -dir crawl -depth 2 -threads 4 -topN 50

A little experience ：nutch2 There is no need to change the configuration file （conf） Distribute to colony Each machine in the , But you need to use it again after modifying the configuration file ant pack , Configuration can take effect .