Overall architecture - adarrivi/hidden-data GitHub Wiki

• Hidden Data is a Maven multi-module project.
• It uses Liquibase for the Database creation.
• Spring Framework widely used across them (spring context, web, core, orm, etc...), as well as several utility libraries from Apache Commons.
• Also uses Multitier architecture.
• From the testing point of view, JUnit and database Integration Tests with are used across the modules.
• There is a module to start the different processes developed in Swing and using Spring AOP.
• Uses two databases: MySql and MongoDb (+1 for HsqlDb)

Following the dependency diagram between modules:

Workflow

The process starts with Data Loader loading text files (books) into a Relational Database Sql DB. For a given set of pre-loaded patterns, the Data Producer goes through all of them packing consecutive lines together and sending them to Data Filter through Data Queue, were only the packets matching the given pattern will be accepted and stored in a NoSql Database NoSql Db. Another process, Data Aggregator, will recover these packets, aggregate some extra data from Sql DB and store the aggregated data into NoSql Db. This information is ready now to be investigated. In order to facilitate it, the last module Statistics Web will gather the information and show it into a Web Application. Data Monitor manages and monitors all this processes.