Technology outline - jhhl/AUMI-Together GitHub Wiki

The AUMI engine itself is:

  • a standard for setups [ json]
    Setup object
  • standards for instruments
    Instrument object
    • AUMI Sings-like attack - sustain (loop/grain) - release
    • possible round robining of them
    • compatible with .sfz, might use "all attack" , or "all release" to play one shots
  • standards for "layout"
    Layout object
    • layout is a collection of quads registered with the instrument
      • a quad has its coordinates, state colors and/or state images, associated sound (but not sample), quad code that defines behavior * hard to know if this data should be static or in a configuration script *
    • presets layout quads in that collection,
  • a protocol for trackers. Trackers generate tracker events from cursors and their interaction with quads quads can overlay other quads.
    • on screen quads that transmits OSC events like:
      • entered with speed
      • exited with speed
      • inside at ZO uv coordinate [continuous?]
      • defined gesture inside quad: uses logic from mouse gesture tracking
        • dwell
        • Horiz Zig RLR or LRL
        • Vert Zag TBT or BTB
        • V/H run and stop
        • circle
        • shake (fast multiple zig-zags) < good for undo
        • combos of those
      • quad code like AUMI Sings was supposed to have
        • declares color and image options including state feedback
        • declares sound resources
        • direct events to the sound engine
        • direct events to the control engine

Skins Skin object

  • manages UI characteristics:
    • colors,
    • typefaces and sizes,
    • control layout and enabling.
  • manages play screen UI as well (as created by the layout engine)
    • cursors, sound boxes, background

Sound Engine AUMI Synth

  • receives events
    • enter event puts a sound in start state
  • maintains a list of playing sounds so they can be removed, transposed or otherwise modulated
  • runs sounds, MIDI, MPE (?), OSC(?)

Control Engine

  • selector Wizard
  • selector for Instruments
  • selector for Layouts
  • selector for Trackers
  • selector for Timing
  • selector for Setups
  • selector for Session Control

Camera Module Camera object

  • video source choices
    • cameras and virtual cameras
      • virtual cameras could include networked video sources and generated sources
      • camera and resolution choices
    • video files and gifs, not just for testing, but mostly for that!
  • affine processing, clipping
  • contrast, saturation, hue
  • presentation transforms: letterboxing, fitting

possible Tracker video transformations

Operated between the camera and the tracker, and also from the tracker to the screen to undo transformations like fish-eyeing.

This MAY be how we manage relative movement, the idea of which is to use the relative motion to move a single cursor, and when idle to return the cursor to the center. That way, any tracker can work "relatively". Color tracker or other true trackers will only use its relative info, not its absolute. Relative MAY need parameters to adjust sensitivity toward jitter, and the return to center time curve.

  • plate registration and subtraction
  • tracker region of interest: screen sees all video, tracker only sees ROI, which MAY actually track content
  • "center is interesting" blur edges

Layout "Looks" Engine Looks object

Macros can lay out areas which can then be manually moved and sized. Setups store the complete customized parms for each area. It understands the mapping of an area onscreen to a sound id.

  • sound areas are quads(with bound boxes)
  • layout uses "macros" and also manual layout to lay out the screen in familiar patters quickly
  • each macro maps the sound order to sound areas
  • layout form macros are:
    • horizontal
    • vertical
    • circular (now using quads)
    • grid
    • semi circle
    • spiral
    • bullseye
    • triangles in a hex grid
    • penrose tiles
    • diatonic overlay (?)
    • areas defined by the tracker
  • layout sound area qualities (related to Skins)
    • selected and unselected variants
    • even and odd variants
    • color
    • edge width
    • edge color
    • POSSIBLE imagery:
      • full color+alpha OR template to be colored
      • image mapped to inner bound box OR warped with quad.
    • padding width
  • melodics understand the form macros:
    • color, edge color, and edge width use even only, even and odd by "octave", even and odd by note
    • sound order now understands "octaves"
  • manual placement:
    • multi select for transformations
      • affine
      • quad edge placement

Sound order (organization) SoundOrder object

The UI will need to be more sophisticated than currently, with sound previews and the like.

  • this is an ordered set of sound resources into "slots"
  • now has a concept of sound pool and serially ordered slots. The pool is the list of samples, built in silences, or generated sounds (!) which may be taken from an AUMIInst or better, a sforzando file. I'm hoping to transition AUMIInsts to sforzando. The sounds may themselves be round -robinned. It may also be MIDI notes (channel + number) with no audio data. Note number is not longer done my naming convention. ideally, the sample would have a frequency associated with it that we convert to MIDI note and channel.
  • even more radically, sounds from different "instruments" can now share the screen.
  • POOLS: SoundPool object
  • sound POOL may be "silences" that are used for MIDI
    • POSSIBLE image for each sound [see: layout]
    • image could be pixel or svg vector?
    • POSSIBLE: melodic sound pool generates missing 12EDO notes. That might actually be part of an instrument builder app.
  • SLOTS: SoundSlot object
  • a slot is associated with a MIDI Note [which might actually be extended beyond 0-16 channel 0-127 Note Number]
  • ordered with scale-building macros, taking base note and scale type.
  • can import from multiple sound sources
  • POSSIBLE image for each sound or slot [see: layout]
    • image could be pixel or svg vector?
    • AUMI Instrument, uses naming conventions and metadata to understand it
    • Sforzando
  • manually ordered, scrambled, disabled
  • Silence to space sound areas out might not be needed since we can move around sound boxes, but it COULD play a silence to mute a sound
  • duplicates allowed
  • not in this scale (disabled by scale making macro)
  • POSSIBLE implicit volume and pan
  • POSSIBLE looping? might be a global parameter, like cutoff.

Timing Timing object

Timing is really the synthesizer action queue. From a gestural event, the queue actions are given times and parameters

Right now, a timer goes off on Sound Per Beat, Beat per minute times and checks if there are actions scheduled for then.

Technically, a rhythm could be imagined as a chord that is played over time, but these macros are treated as different. They can combine to put chords into rhythms.

  • Rhythmic controls:
  • Chordal controls:

Wiki Links: Technology Philosophy