Para crear el sistema de crawling se ha de especificar ciertos aspectos del sistema. [Ejemplo] (https://github.com/Shathe/101CrawlersWeb/blob/master/source-java/Crawler/conf_tutorial.yml)

En lo que se refiere al contenedor docker, se ha de especificar el sistema operativo sobre el que va a correr el sistema de crawling. Del OS se debe especificar el nombre y la versión de esta forma:

#Docker container Operating System
dockerOS:
name: ubuntu
version: 14.04

Respecto al sistema de crawling, se deben especificar aspectos sobre qué sistema es y sobre su configuración que se aplicará en el momento en el que este se empiece a ejecutar.

1. Se debe especificar el nombre y la versión del sistema de crawling.

2. Las semillas que servirán como base al sistema

3. El numero de rondas/iteraciones que el sistema permanecerá ejecutándose (A menos que se le fuerce a parar)

4. Los plugins extras que se utilizarán

5. El numero de bytes máximos para un documento parseado

6. El número de links máximos que se puedan extraer de una sola url

7. Si se admiten urls del dominio propio

8. Tiempo máximo admitido para el fetch de una url

9. El modo de colas que se utilizará (Default value is 'byHost', also takes 'byDomain' or 'byIP' )

10. El modo de extracción que se utilizará, posibles valores [round | finish ]

11. Configuraciones sobre el control de timeouts

#COMPULSORY CONFIGURATIONS (MUST BE SPECIFIED) #Docker container Operating System dockerOS: name: ubuntu version: 14.04

#Crawling System crawlSystem: name: nutch version: 1.9 #seeds are separated by line break and preceded by '- ' seeds: - https://dzone.com/articles/using-yaml-java-application - https://dzone.com/articles/ #number of rounds the crawler is going to iterate rounds: 2 # whether the extraction is going to be made at every round or only at the finished of the crawl # Possibles values [round | finish ] extraction: round #the structure of the crawled info, you can choose between text or hmtl infoCrawled: text

Determines how to put URLs into queues. Default value is 'byHost', also takes 'byDomain' or 'byIP'.

queueMode: byHost # fetcher.queue.mode

#OPTIONAL CONFIGURATIONS (CAN BE OMITTED) #Plugins are not specified in the DSL, they have to be in a folder named 'plugins' in the same level of the #DSL file. This folder must have one folder for every plugin you want to add and that folder must have the #compiled plugin ( plugin.xml and its jar files)

#The length limit for downloaded content  (-1 for o limit)
maxFileLength: 65536 # file.content.limit  and http.content.limit
# If the Crawl-Delay in robots.txt is set to greater than this value (in seconds)
# then the fetcher will skip this page, generating an error report. If set to -1
# the fetcher will never skip such pages and will wait the amount of time retrieved from robots.txt Crawl-Delay, however long that might be.
maxCrawlDelay: 50 # fetcher.max.crawl.delay
#Maximum number of Inlinks per URL to be kept in LinkDb.
linksLimitURL: 10000 # db.max.inlinks
timeouts:
  #	Timeout in seconds for the parsing of a document
  parser: 30 # parser.timeout
  #The number of times a thread will delay when trying to fetch a page
  fetchTimes: 100 # http.timeout
  #The default network timeout, in millisecond
  network: 10000 # http.timeout

En el caso de que haya algún problema, se indicará el primer error encontrado.