(Detailed Design) Aleph2 filesystem layout - IKANOW/Aleph2 GitHub Wiki

TODO in progress

Distributed filesystem layout

Library

TODO descriptions

  • /app/aleph2/
    • library/
      • access/
      • analytics/
        • tech/
        • module/
      • import/
        • harvest/
          • tech/
          • module/
        • enrichment/
          • tech/
          • module/

Shared configuration

TODO descriptions

  • /app/aleph2/
    • config/
      • services/
        • <service_name>/
      • library/
        • <technology_name>/
        • <module_name>/
      • managers/
        • <manager_name>/

Buckets and data

TODO descriptions

  • /app/aleph2/
    • data/
      • <misc path>/
        • <bucket_name>/
          • (other folder tree which can themselves contain buckets)
          • managed_bucket/
            • logs/
              • harvest/
                • TODO timeslices
              • enrichment/
                • TODO timeslices
              • storage/
                • TODO timeslices
            • library/
              • (see "global" library, above)
            • assets/
              • TODO docs, spreadsheets, knowledge graph,
            • import/
              • temp/
              • stored/
                • raw/
                  • TODO timeslices
                • json/
                  • TODO timeslices
                • processed/
                  • TODO timeslices
              • ready/

Local filesystem layout

  • /opt/aleph2-home - the root (can be overridden from "globals" configuration)
    • config/ - Mirror of distributed config directory structure, see above
    • lib/ - All the core and contrib classes
    • yarn-config/ - Location of all *-site.xml files for Hadoop ecosystem configuration (can be separately overridden from "globals" configuration)
    • cached-jars/ - Location of cached library JARs (can be separately overridden from "globals" configuration)
⚠️ **GitHub.com Fallback** ⚠️