Overview

Welcome to the Aleph2-examples wiki!

This repo contains example plugins for the Aleph2 platform.

Some of the modules in here will eventually be promoted to production "contrib" components, others are intended to be used as templates, or are simply "throwaways" for demos. Therefore nothing in here should be treated as production quality code unless otherwise stated.

Manifest

Harvesters - example and production

• aleph2_empty_examples: currently contains a harvester that simply replies to each request with a "success" indicator and the name of the method called. Used for testing
• aleph2_external_harvest_example: a useful template for a harvester that needs to launch an external application (in particular one that runs in Java, though this isn't necessary)
• aleph2_storm_samples: a fairly functional generic Storm-based harvester that lets users launch and control Storm topologies, together with a set of "throwaway" examples of using it
• aleph2_flume_harvester: an ongoing development that will likely end with a production component ... enables users to build and control Flume components and plugin code (documentation)
• aleph2_logstash_harvester: an ongoing development that will likely end with a production component ... enables users to build and control multiple Logstash flows independently (documentation)
• aleph2_script_harvester: an ongoing development that will likely end with a production component ... enables users to run scripts on demand as part of Aleph2 workflows (documentation)

Streaming enrichment production and example modules

• aleph2_storm_script_topo: an ongoing development that will likely end with a production component .. runs a simple map/fold topology controlled by user javascript that can be uploaded or specified per-bucket (documentation)

Batch enrichment production and example modules

• aleph2_enrichment_utils: a collection of generic batch enrichment utilities to perform common activities:
  • JsScriptEngineService: (will likely end with a production component) provides a JS engine scripting engine that can be accessed via the JSON config (or by uploading ".js" files as shared libraries) (documentation)
  • SimpleRegexFilterService: (more of a sample for developers to follow) filters out objects that don't match the provided regexes
  • LookupService: (planned) will enable the incoming data to be joined with data from other buckets via cached lookups (instead of outer joins)
    • (will just be an overlay on top of the DeduplicationService, will likely get moved into aleph2_analytics_context_library to be co-located with that service)
  • FileDecodingService: (planned) will leverage Tika to convert arbitrary binary and text file formats into JSON

Analytic job production and example modules

• aleph2_spark_script_engine: Runs inside the Spark Analytic/Enrichment Engine and allows Scala scripts with an Aleph2 API to be excecuted on the specified inputs. (documentation)

Applications and application examples

• aleph2_example_client: a dummy/template standalone application
• aleph2_example_webapp_kafka: a sample webapp (currently not integrated with SSO) illustrating hooking Kafka up to a streaming HTTP interface
• aleph2_web_login: (will likely end up with a production component) LDAP login component for v1/v2 hybrid deployments
• aleph2_web_sso: (will likely end up with a production component) SSO login component for v1/v2 hybrid deployments supporting LDAP and SAML (documentation)
• aleph2_web_utils: (will likely end up with a production component) utility library for v2 webapps