Future Implementations - TinkerStorm/channel-backup GitHub Wiki

• Core Architecture
• Plugin Authoring
• Endpoint / Destination Adapters
  • Discord (partial support)
    https://discord.com/api/webhooks/:id/:token
    • Webhook
    • Embed
    • Embed Limits
  • Slack (unknown)
  • Office (Teams) (unknown)
  • rocket.chat (unknown)
  • Guilded (new, unknown)

The initial plan was to deploy the project for multiple languages, either by a monorepo that handles all base principles at once or through multiple repositories (each with their own environments). It becomes evident that the latter would be far more effective than trying to collate all libraries of varying styles under one roof.

This also brings up the opportunity to publish a module and CLI export to NPM so that automated updates can take place (as well as the vague idea to implement a plugin architecture under the channel-backup-plugin-* namespace). Further brainstorming suggests that caching of previous runs may also be efficient and mutually benefit the possibility of pushing this to GitHub Actions (with use of @actions/cache). Some options will have to be arbitrary and custom for the action itself to work, but most of the options available to the core functionality should be used by its action counterpart.

For plugins specifically, a Mono repository will most likely do the trick for all plugins that TinkerStorm or RocketDragon maintain, all that remains is which organization will host that repository and how will it be done.

Including data types, template engines and data transfer protocols.

If a template engine has native html processing that cannot be disabled, it cannot be used here (i.e. Vue, React, Marko, Pug, etc.).

I should note that the Markdown content parser is just turning the string content into an object { content: fileContents }, and that Markdown only has support for what Discord supports (i.e. headers, footnote links, checkmarks, codeblocks by indent, etc. aren't supported; spoilers are, but you won't know until it's posted).

Mentions are also handled differently with the allowed mentions change the config and intended behaviour of how a webhook can target users and roles. If disabled, mentions for these entities will become regular strings with an @ prefixed (much like how you would write it).

A message reference can also be provided, but knowing of such message must be done in the file itself or by logic through a template.

• Define logging strategy

  • Pre-existing prefixes (upper case)
    • resolve
    • query
    • warn
    • sent/{index} where index is the loop index from the generated file listing
  • Currently declared with format [{upper($prefix)}] {message} which can vary based on the progress of the sequence.

• Move away from discord.js to lightweight / standalone alternative.

  • Library currently handles:
    • Webhook ratelimit handling
    • Embed colour resolution
    • File attachment handling
    • Native loop delay

• Glob star matching

  • Attempt to minimise the file listing by making use of glob.hasMagic(pattern) instead of requiring file type. Feature implementations below would likely prevent this from happening at this stage of development.
  • Add option to reverse a glob result. action / sequence - OneOf(reverse, first, last, etc.)
  • Allow foreign files to be requested from the web.
    These would be retrieved without knowing the file contents, nor how to process it.

• Use of GitHub Learning Lab to provide interactive tutorials?

• GitHub Action for CI / CD use

  • References to:
    • dorny/paths-filter for job workflow filtering
    • actions/issues for glob matching against label keys

• Plugins should be able to turn off certain aspects of the core functions on initial run through.

  • i.e. Resolve Author should be able to disable this to allow the forced reposting of messages incase the content differs from last run.
  • There may be an alternative to save the entire payload to the cache, but I do not see this as a viable option if the content can change if another end-user runs the script on their own system.

• {CLI} channel-backup create [dir] - create a new configuration in the current (or specified) folder, create it if it doesn't exist.

  • Option: --plugin - A list of plugins without channel-backup-plugin- key prefix included to be installed upon creation.

• Project Structure (unknown / planning)

  • Presently deciding if this should only be coded in nodejs or allow multiple languages to be existing at the same time.
  • Allowing for multiple languages of the same architecture would need a centralized specification to decide if something should be implemented for a specific language and then to decide if any others should follow.
  • The idea being that it would allow anyone to write their own plugins in the language of their choice and not be limited nor hindered by their own system limitations.
  • Have plugin base structures embedded into the core package or have it as another package.
    Would be named something like channel-backup-plugin-base, but then this would have problems with the key matching if plugins are autoloaded in.
  • Repository Structure (A) - Split across 2 orgs, multi language
    Core modules would be on TinkerStorm, repo / project templates would be on RocketDragon.
    • channel-backup - Project Hub
      • Home to discussions, spec issues, proposals, project-wide RFCs, etc.
    • channel-backup-spec?
    • channel-backup-node - NodeJS Hub
      • Chance of being a mono repo using lerna package config.
    • channel-backup-node-action - GitHub Action using the nodejs implementation
    • channel-backup-node-plugins - Node environment plugins
    • channel-backup-python / channel-backup-py - Python Project Hub
    • channel-backup-python-action - GitHub Action using the python implementation
    • channel-backup-{lang}
    • channel-backup-{lang}-action
    • channel-backup-{lang}-template
    • channel-backup-{lang}-plugins
    • channel-backup-clean template - Clean environment
  • Repository Structure (B) - Split across 2 orgs, only nodejs Same Core and Templates split as before.
    • channel-backup - Project Hub / NodeJS Hub
    • channel-backup-action
    • channel-backup-plugins
    • channel-backup-template
  • Repository Structure (C) - either A or B, but unified into one organization (either contained within an existing one, or an entirely new one dedicated to the project itself).

• {CLI / GH Actions} Allow a preconstructed message from the pipeline to be processed and sent forward to a webhook. Make use of a standard override to prevent interactions with the cache (like a 'dry run', but refuse all file system modifications).

  Would be useful for announcements on servers.

• {GH Actions} Use steps.*.output to provide:

  • an array of sent files
  • an array of resulting message IDs
  • etc.

• If messages aren't found before last available message, break out of edit and delete messages until it can recover (use of partial repost queue).

  [
    "msg1", // ✅
    "msg2", // ✅
    "msg3", // ❌
    "msg4"  // ✅
  ]
  

  2nd option would be to shift the messages over and repost what remains.

  Likely to become a problem if username / avatar differs as mentioned with the Resolver Author plugin idea.

• {CLI} Post a single message in a particular workflow.
  channel-backup post <workflow> [index/file]
  Unlikely to work as intended if templates are involved (unless the cache is used correctly).

  Unsure how it would work with a pattern, but it is likely that it can be applied across a range by checking matches[0] against the full array of files found in the particular workflow and then match that against the cache.

• Add a "Credits" header for packages used in the project or those that have contributed to the project itself (directly or indirectly).

  Use of all contributors would be useful for those that contribute directly.

• Create a "Who's using this?" page / section.

  I'm aware that I might be the only one using it right now (either for myself, those that I help or those that I am contracted to). It would be interesting to see how many individuals, communities or organizations use it over time, but a set of guidelines should be formed to ensure that all listed entities comply with (i.e. terms of service).

• Caching would be done by storing message IDs with the webhook used to send them.

  {
    "webhook-id": [
      "message-id-1",
      "message-id-2"
    ]
  }
  

  Storing by workflow name would cause multiple problems including 404s due to the messages not being posted by the particular webhook or seeing that the webhook would have been sent in another channel and then it has since been changed.

• {GH Actions} Make use of environments to declare different workflows. (Would also need to detect if any files have changed for a particular workflow by using the array of files retrieved by glob.)

• Make use of a task runner on the command line? grunt / gulp

• {CLI} CLI tool to handle plugins / multiple workflows?

  • channel-backup
  • channel-backup plugins - list / manage plugins
  • channel-backup run [key] - run [specific (key)] workflow
  • --dry-run - run without making requests
  • --no-cache - post without using the cache

    consider not saving either, but would probably be counterproductive to the overall flow of statements

  • --verbose - run without refreshing the terminal
  • --web - use log friendly format for CI

    possibly detect if a github action environment is being used instead.

• Payload validation

  One of the more common issues is when the payload gets too large for Discord to accept (namely embed limits and message limits).

• Query: Update Promotion - built in or plugin? :::info If it's a plugin, more plugin base architecture may be needed Jekyll Plugins. :::

• Using responsive templates :::danger Blocker: only certain extensions should be accepted, and the file will need to be run through the loop again. :::

• Multi config protocol.

  Proving to be difficult because when thinking about actions, the only way to get it running would be to use a matrix but then keeping the webhook urls secret is another concern by itself.

  Having a keyed secret per webhook would be quite annoying to manage (and become an increasing concern for larger repositories).

    runs-on: ubuntu-latest
    matrix:
      include:
        - webhook: ${{ secrets.INFO_CHANNEL_WEBHOOK }}
          workflow: info # if not specified use root, though may be changed
          dry-run: true
          verbose: true
  

  Another possibility would be to map out the matrix of configurations actions would use and then process it in action... but then that destroys the underlying functions of the matrix itself.

• Yeoman Generator Repo: generator-channel-backup / generator-channel-backup-plugin

  Command: yo channel-backup / yo channel-backup-plugin

• Test suite? ava, mocha, tap, etc.

• Linter: xo, eslint, etc.

• i18n support (default to English)

• Option: --dry-run - If provided, no mutually destructive methods should be invoked. Useful for verbose logging.

• Option: --verbose - Log all minor steps, possibly include rendered data before parsing into the environment.

• If originating from CLI, attempt to retrieve webhook from config file, then look in the environment using WEBHOOK_URL. Unsure which to prioritise, ENV sounds like it would give more priority, but config is more accessable. Consider use of dotenv for windows systems.

  Log it as a debug that the webhook was not found in config. Force fail the program if a valid webhook is not found in ENV either. (Make use of GET request to make sure it's valid.)

type EntityMap = { [author: string]: string }

• ~.key implies it is from the root of the constructed / generated payload.
• ~.iterable.* or ~.iterable.*.key implies it is an Iterator at the root ^, and to iterate over all the values. If another key is provided after, then also access that key during the loop.

See Object filters for GitHub Actions as reference.

embeds:
  - title: ":date: Weekend Schedule"
    color: "#F44106"

  - title: Friday
    color: "#10B7FD"
    fields:
      - name: Time
        inline: true
        value: |
          :clock530: `17:30`
          :clock6: `18:00`
          :clock630: `18:30`
          :clock7: `19:00`
          :clock730: `19:30`
          :clock11: `23:00`

      - name: ​ # 200B - Zero Width Space
        inline: true
        value: |
          :tada: Pre Jam mingle on Discord
          :projector: Keynote and Theme
          :brain: Brainstorming
          :teacher: Pitching and Teamforming (optional)
          :technologist: Jamming
          :city_sunset: Closing for night

  - title: Saturday
    color: "#3142C5"
    fields:
      - name: Time
        inline: true
        value: |
          :clock9: `09:00`
          :clock12: `12:00`
          :clock1: `13:00`
          :clock7: `19:00`
          :clock11: `23:00`

      - name: ​ # 200B - Zero Width Space
        inline: true
        value: |
          :city_sunrise: Jam Re-opening
          :flying_disc: Sharing of Progress
          :sandwich: Lunch time mingle
          :boomerang: Sharing of Progress
          :city_sunset: Closing for night

  - title: Sunday
    color: "#6A00DD"
    fields:
      - name: Time
        inline: true
        value: |
          :clock9: `09:00`
          :clock12: `12:00`
          :clock4: `16:00`
          :clock6: `18:00`

      - name: ​ # 200B - Zero Width Space
        inline: true
        value: |
          :city_sunrise: Jam Re-starts
          :flying_disc: Sharing of Progress
          :package: Start of game uploads to GGJ Site
          :tada: Jam Finishes

Results in the following post being rendered:

The template below was coded in nunjucks but wasn't used because the template engine wasn't fully implemented. So I have no idea if this works. 😂

{% set numberMap = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine"] %}
{% macro numberToEmoji(n, shift) %}{% for digit in n.toString().padLeft("0", shift).split("") %}{{ numberMap[+digit] }}{% endfor %}{% endmacro %}

username: Falmouth Global Game Jam

embeds:
  - description: | {% set shift = messages.length // 10 %}
      {% for item in messages %}
      {{ numberToEmojis(loop.index, shift) }} **[{{ msg.embeds[0].title }}](https://discord.com/channels/{{ msg.guild.id }}/{{ msg.channel.id }}/{{ msg.id }})**
      {% endfor %}

• GH Actions input names and types
  • load_from: enum(args, env)
  • sequence: string
  • webhook_url: string from ${{ secrets.* }}

• Jekyll Hooks
• GitHub Actions