Functions Importer WebCrawlerImporter - SmartEnviroSystem/SmartDataPorter GitHub Wiki

The WebCrawlerImporter supports crawling of webpages and extact information from them to add to databases.

Currently tested supports:

• Crawling Images included on page
• Crawling to sub documents.

Configuation

Parameter	Description	Default	Optional
crawler.baseurl	Base url of the webpage	-	no
crawler.targetFolder	Folder where to store downloaded files	-	no
crawler.allowedExtensions	List of allowed file extensions (e.B. jpg,avif)	-	no
crawler.allowedMimeTypes	Allowed mime types (e.g. image/jpg)	-	no
crawler.maxDepth	Max. depth of link follow	-	no
crawler.followParameterizedLinks	If true parametrized links are followed	-	no
crawler.datepath	Creates a subpath from date (e.g. "/{yyyy}/{MM}/{dd}/")	-	no

Example:

{
	"server":"http://localhost:8080/", 
	"smartdata":"SmartData", 
	"storage":"storage_name", 
	"collection":"collection_name",
	"resultReciptants": "[email protected]",
	"usethreading": true,
	
	"importer":"WebCrawlerImporter", 
	"crawler.baseurl":"https://www.mydata.de/webcam/archive_wolkencam/2026/05/25/",
	"crawler.targetFolder": "../docroot/files_helios/images/",
	"crawler.allowedExtensions": "jpg",
	"crawler.allowedMimeTypes": "*",
	"crawler.maxDepth": 1,
	
	"parser":"FileAnalyserParser"
}