Native Tables - openantz/antz GitHub Wiki

• Overview
• Tables
• Table Field Constants
• Table Explained
  • Tree Hierarchy & Links
  • Coordinates
• CSV Conventions
• MySQL Conventions
• Changelog

*We plan to add JSON support.

Our Native Scene Graph can be stored and loaded as a set of tables located in the 'usr' sub-folders (relative to the app). The primary table is the 'np_node' table, which defines node attributes and tree hierarchy. Auxiliary (file) content such as texture maps and 3D models are referenced by the np_node table using ID's (texture_id, geometry_id). Metadata tables (np_texture, np_models) store the pathname and format info. There is also a np_globals table that stores the user environment and network connections (ie: MySQL, OSC). Animation channel is set per node (np_ch_in_id) and is mapped to a combo of tracks (np_track) using a channel map table (np_ch).

A good way to get familiar with the table structure is to create a simple scene with one or two nodes, save it (press 'K' key) and then try changing a few attributes by editing the file 'antznodeYYMMDDHHMMSS.csv' (timestamp appended) using a spreadsheet program. Then load your modified node table ('L' key) to see your changes.

The np_node table (ie: antznode0001.csv) is a snapshot of the entire scene geometry. It contains all information needed to represent the spatial characteristics of the scene at the specific moment the file is generated. However, some global parameters are not stored in the file, such as window position, Alpha Mode (transparency type), and grid spacing....

The np_node table also stores velocity rates that result in animation. However, the data that changes over time (per cycle) is stored in the np_track tables and mapped by the np_ch table, wich tracks assigned using np_ch_in_id and np_ch_output fields.

It is possible to merge scenes from multiple tables. This can be done by the user at runtime or at launch by passing in multiple file names.

Our Scene Graph data and user environment is stored using either MySQL DB tables or CSV files.

• np_globals - User environment (fullscreen, menu mode, etc.) & network configs (MySQL, OSC).
• np_node - Node parameters (hierarchy, color, topology, etc.), cameras and grids.
• np_tag - Node text tags (referenced in the node table using 'tag_id').
• np_chmap - Node animation by mapping raw track data to node attributes ('ch_in').
• np_track - Raw track data.

Planned Tables (not yet implemented):

• np_models - File paths for loading 3D models ('np_geometry_id') with textures & metadata.
• np_texture - File paths of texture map images ('np_texture_id') with metadata.
• np_tables - To list all tables belonging to a dataset (and where they are).
• np_format - To convert between glyphs and external records (perhaps a set of tables).
• np_palette - To allow for loading custom color palettes.
• np_topo - To describe built-in and custom topologies.
• np_log - To record console output.

Example rows for globals table:

1,"np_gl",1,"alpha_mode","i",0,"","","1"
...
4,"np_gl",1,"window_size_xy","ii",0,"","","964,565"
...
19,"np_browser",1,"url","s",0,"","","http://openantz.com/docs/id.html?id="

Related Issues:

• #197 - Add 'tag_id' to node table, so that unique tags can share a record
• #347 - Grid texture alpha color control

For usage and (title) formatting see Text Tags.

Related Issues:

• #287 - Set groups of tags
• #197 - Add 'tag_id' to node table, so that unique tags can share a record

The log is designed to capture all user actions, status events and errors generated by the app. Essentially the same information that is currently displayed in the system console. Includes a time_stamp with each entry.

• np_node table
  • type
  • topo
  • geometry
• np_globals
• np_ch

• np_globals details
• Nodes
  • Tree Hierarchy & Links
    • np_node_id
    • parent_id
    • child_id
    • branch_level
  • Coordinates
    • Translate
    • Rotate
    • Camera Position
  • [Color]((Native-Tables#color)
    • color_r/g/b/a
    • color_id
    • palette_id
  • [Other Attributes]
    • topo
    • geometry_id
    • ratio
• Text Tags
  • np_tag_id
  • np_table_id
  • record_id
  • title
• Channels & Tracks
  • np_chmap
  • np_track
• External Assets
  • np_texture
  • np_models

Here is an example of the globals table (as of 2020-07-21):

1,"np_gl",1,"alpha_mode","i",0,"","","1"
2,"np_gl",1,"background_rgba","ffff",0,"","","0.000000,0.000000,0.000000,0.000000"
3,"np_gl",1,"fullscreen","i",0,"","","0"
4,"np_gl",1,"window_size_xy","ii",0,"","","964,565"
5,"np_gl",1,"position_xy","ii",0,"","","312,324"
6,"np_gl",1,"hud_level","i",0,"","","4"
7,"np_gl",1,"subsample","i",0,"","","1"
8,"np_mouse",1,"tool","i",0,"","","6"
9,"np_mouse",1,"cam_mode","i",0,"","","0"
10,"np_mouse",1,"pick_mode","i",0,"","","3"
11,"np_db",1,"host_ip","s",0,"","","127.0.0.1"
12,"np_db",1,"user","s",0,"","","root"
13,"np_db",1,"password","s",0,"","","admin"
14,"np_db",1,"db_type","s",0,"","","mysql"
15,"np_osc",1,"tx_ip","s",0,"","","127.0.0.1"
16,"np_osc",1,"rx_ip","s",0,"","","127.0.0.1"
17,"np_osc",1,"tx_port","i",0,"","","8000"
18,"np_osc",1,"rx_port","i",0,"","","9000"
19,"np_browser",1,"url","s",0,"","","http://openantz.com/docs/id.html?id="
20,"np_git",1,"repo","s",0,"","","openantz/antz"
21,"np_key",1,"layout","i",0,"","","0"
22,"np_globals",1,"item_count","i",1,"","","22"

Tree hierarchy is defined in the np_node table using the 'parent_id' field which corresponts to the parents np_node_id (primary key) in the first column.

Links have two ends, the A-end is defined by parent_id and the B-end is defined by child_id.

The table read process is agnostic to record order. Orphan nodes (child nodes that precede their parent) are sorted and attached after reading the entire file (or discarded if no parent exists).

The np_node_id field is used to build the topo tree hierarchy and is unique within the table (and scene).

In general, (most) nodes create a new object on import and are re-assigned ID's. Two exceptions are the primary main camera and main grid which are never re-created, but are updated. All others nodes are assigned session unique IDs. Which means that you can merge multiple datasets.

Note that if you save a file, then read it back in, delete and/or add some nodes, then re-save, you will get a different set of IDs. To keep track of nodes over multiple sessions, use record_id (with np_table_id).

The parent ID determines the tree hierarchy. Root-Nodes are set to zero, all child nodes must specify the 'parent_id', which is the only parameter used to build the hierarchy.

The child_id has two uses:

• It specifies which branch was last traversed (for user navigation behavior).
• For links, it defines the B-end, for details see node type = link.

Root nodes have 'branch_level' = 1. The level increases with each additional child branch, ie: the primary torus (on a default pin) is set to level = 2. All grids (regardless of location in a hierarchy) have branch_level = 0, which resets their attached root nodes to level 1.

Note that the branch_level is automatically set during the import process. Which is to say that the branch_level field is simply for convieance when 3rd party tools are used to process the tables.

The following table describes the coordinate system for a KML based Sphere in comparison to a Torus. The range for the two are the same, except for translate_y where the Torus has a range of -180 to 180 latitude vs KML -90 to +90 deg. Cube and Grid topo coordinates are the same as a Torus. Latitude values greater then +/- 90 degrees are wrapped onto the Sphere, (causes upside down object.) For detail see 'Topologies & Coordinates' section.

Position is based on translation, rotation and scale (of the parent.) Root nodes are relative to the global origin. Child node coordinates are relative to (or inherent) there parents position and topo type.

'topo' types include default, cube, sphere, torus, surface, pin and grid. Each defines a coordinate system with a unique set of operations that transform the position, orientation and scale of any attached child nodes.

Some topo types, (such as a 'cube') have multiple local coordinate systems. The child node 'facet' value tracks which system, (or cube side to use.) See 'Cube - Key Table' appendix and 'Coordinates & Topologies' section.

Radial offset is 'translate_z' for a torus.

Translational velocity is set by 'translate_rate_x/y/z' and is the delta distance applied per cycle, (typically 60 cycles per second.) Nodes may be restricted (or wrapped) by both local node limits imposed and/or the parent topo type limits.

Typically you do not assign a 'translate_rate' to a root-node as it will continue to move until it disappears by leaving the scene boundaries, (at the clipping planes.) However, child nodes will typically limit or wrap there coordinates when boundaries are hit, (depends on parent topo type.)

The rotate_x/y/z sets the node orientation with units in degrees. Order of operation is:

1st rotate about z-axis by the Heading, rotate_y 2nd rotate about x-axis by the Tilt, rotate_x 3rd rotate about z-axis(again) Roll, rotate_z

Camera 'Tilt' differs from all other objects in that the 'rotate_x' axis is inverted. A value of 0 results in a view looking straight down the -z axis, basically looking straight down on a map. At x = 90 deg the cam will rotate CCW about the x-axis and look at the horizon towards North (with Heading rotate_y = 0.) Effectively this is a 'Right-Handed' coordinate system for rotation, (with an inverted up vector.) Translation is still a 'Left-Handed' coordinate system. Compatible with the google KML '' standard.

Objects (other then the camera) use a 'Left-Handed' coordinate system for rotation (and translation.) An object with zero Tilt will be upright. At rotate_x = 45 deg the object will lean towards North, (with Heading = 0).

rotate_y Heading range is 0 to 360 deg, North = 0, E=90, S=180, W=270. rotate_z Roll range is -180 to +180 deg.

'x' axis of a child node can be used to set the spacing of toroids around there parent. 'y' axis will spin the torus, not very noticeable unless it has a texture map or is assigned a different primitive.

Rotational velocity is set by 'rotate_rate_x/y/z' and is applied (added) to the 'rotate' orientation every cycle.

Orientation and position of nodes, (other then the camera,) are compatible with the google KML '' specification. The altitude units are not the same. Instead of feet (or meters) altitude is specified relative to the circumference, altitude units are 1 to 1 with translate_x at the equator.

Cameras use the '' KML coordinate system which differs in that the 'Tilt' x-axis is inverted, (and camera up vector.) See 'Camera Position” in this appendix for further detail.

Position is set using 'translate_x/y/z' and 'rotate_x/y/z' for orientation.

Tilt is 'rotate_x', straight down is 0.0 deg, horizon is 90 deg, up is 180. Heading is 'rotate_y', North is 0.0 deg, E 90, S 180, W 270. Roll is 'rotate_z', CW is 0 to 180 deg and CCW is 0 to -180.0 deg.

Note that the x-axis 'tilt' is essentially a flipped axis, and is therefore a right-handed coordinate system. This is an exception, the rest of the coordinates are all left-handed systems.

Note that the 'rotateVec_x/y/z/s' parameter should be ignored, it is a unit vector calculated from 'rotate' and is automatically updated.

There are multiple cameras available to the user. There is a single root camera that all other cameras are attached to as child nodes. New child camera's may be created or existing child cams may be deleted. The Root-Camera is locked and cannot be deleted. By default there are four child cams attached to the Root-Camera.

The coordinate system is designed to be compatible with the Google KML standard for a '', not to be confused with ''. For detail see 'Topology & Coordinates' section in this document.

Also see the '' specification in the Google KML docs:

http://code.google.com/apis/kml/documentation/kmlreference.html

*At this time, all cameras coordinates are relative to the World Origin, which is flat cartesian space, (non-spherical coordinates.) Future updates will allow for attaching a camera to any node. Attaching a camera to a parent with a Sphere topo type would result in polar coordinates.

Object size is set by 'scale_x/y/z'. A scale value of 0.5 is equivalent to a 50% reduction, default is 1.0 which results in no change in size. Valid scale values can be very small or large. A negative value results in an inversion of the geometry, including all child nodes of the current node. For uniform scaling the x, y and z values must be identical. Individual axes may have different scale values. This results in non-uniform scaling which causes an object to appear stretched or squished.

In general 'scale_rate_x/y/z' is used internally by the application but is not practical if set to a value other then zero. Otherwise continuos scaling will occur and an object will typically become far too large.

The 'ratio' sets the inner and outer radius of a torus. Range is between 0.01 and 1.0, where 1.0 is a donut with no hole and 0.01 is a very thin donut. The default ratio is 0.1 which is equivalent to 10% of the outer radius. Nodes attached to a parent of Torus topo type are positioned relative to the surface, so scale and ratio (inner radius) effect the position.

*Ratio may effect other characteristics depending on geometry and topo type.

Sets the RGBA of a node and is usually based on the 'color_index' that maps to a palette of colors, default palette has 20 colors, larger index numbers are mapped into the palette range. T

Sets the alpha transparency and is not set by the 'color_index'. Texture maps can be RGBA with per pixel alpha values.

Note that you may set the color to any custom RGBA you choose, but if the user changes the color, the custom color will be lost.

Because the node 'id' is subject to automatic re-assignment by the app, the 'record_id' is provided for the user to keep track of the orginal (internal or external) record that the node represents. It is retained by the node and is not modified by the application. This may be used in conjuction with np_table_id to reference which external DB table the record originates.

Both the np_table_id and np_record_id are used to attach tags to nodes. So they need to match across tables (np_tag & np_node).

In the future we plan to switch to using the np_tag_id (adding it to the np_node table) to attach tags to nodes, as this will allow for multiple tags to belong to a single external record.

The tag title can be plain (ASCII) text or formatted as a URL path. For details see: Text Tags usage.

Channels map track data to node attributes to modulate (animate) objects position, scale, color, etc. Each node has an input channel (np_ch_input_id) that corresponds to the np_chmap table (channel_id). Multiple nodes can share the same channel input, but only one node can use a (np_ch_output_id).

A single channel can map multiple tracks to a single node. In example, move a node using GPS data by mapping 3 tracks to: translate_x, translate_y & translate_z.

*Tip: You can load an image sequence and play it back by creating an np_track table with the texture_id sequence and assign it to a node with np_ch_input defined in the np_ch table.

Specifies which texture map to apply. When texture_id = 0 the texture map is disabled, zero is the default for most objects. The root-grid has a default texture_id = 1 that is loaded from 'map00001.jpg'. For detail see 'File - Texture Maps' section.

We use the FreeImage library to load textures, which supports a large variety of image formats. The file loading routine first scans the folder 'usr/images' for files named antzXXXX.jpg and then loads files (in the same folder) with any other name. It will also load textures when loading a 3D model (as specified in the model file) and any files specified in a 'np_texture' table.

Textures are automatically loaded from the '../usr/image' folder at launch and may cause a significant delay if they are numerous and/or large.

By default the main grid has a default np_texture_id = 1, which corresponds to 'map00001.jpg'.

The GPU memory determines the total size of the textures loaded. Maximum texture size depends on GPU hardware, however the user does not need to be concerned as the loading routine will down-convert large textures (if required by the hardware.) A typical GPU today will support a maximum size between 4096x4096 to 16384x16384. Power of 2 sizes are also no longer required in most hardware, the texture loader will handle any necessary scaling.

All textures are converted to 8-bits per channel. Both RGB and RGBA with alpha channel is supported.

Tip: You can re-generate the (bounding box) scale and (geometric) center by setting the x/y/z scale to zero (in np_models table). Note that at draw time, grid objects are scaled to fit the grid extents (and it is assumed the geometry is normalized).

Future planned import flags are "GTKSNMPGLCAT" for: Geometry, Textures, Kinematics, Shading, Nurbs, Morphing, Particles, Geolocation, Lights, Cameras, (track) Animation and Tags. At this time, we only import geometry and its texture map, so "GTXXXXXXXXXX" are the current default flags.

Comma Separated Values (CSV) files are easily imported from spreadsheet programs and databases, but they lack standardization! So here is our convention:

• Comma (,) is used for the field seperator.
• CRLF line endings (DOS style, which works with Linux, OSX & MSW).
• First row is the field names (for readability).
• First cell (upper-left) is the 'id' column (and table name), ie: np_node_id.
• The first record (data) starts on the 2nd row.
  • For convienance, we often start with a null record (unused by scene) to show default values.
• Numbers (float, int, uchar, etc.)
  • Numbers do NOT need to be quoted (ie: 42,0.1,777,3.14).
  • By default, numbers are single precision 32bit (signed) int or float.
  • Some exceptions include color components (8bit) and record_id (64bit).
• Text Strings:
  • Double quotes designate (ASCII) text strings (ie: 42,"My Text",...).
  • A quote within a text string must use two double quotes.
  • example: "My ""quoted text""" -> My "quoted text"
  • Fields containing a line-break, double-quote and/or commas must be quoted.
• Each record must terminate with a line ending (CRLF).
• The file is expected to have one extra line break at the end, (the last line is blank.) .

Note that DB tables use the np_[table] naming convention (ie: np_node), where as the CSV files use antz[table][dataset].csv, where 'dataset' is typically a 4 digit number or timestamp (YYMMDDHHMMSS), ie: antznode0001.csv or antznode2007191317.

*Reading is tolerant to different line endings CRLF(MSW), CR(OSX) and LF(Linux). Writes using CRLF.

For a general discussion on CSV formats see "Towards standardization" on wikipedia CSV page:

• http://en.wikipedia.org/wiki/Comma-separated_values

2020-06-25

np_node table:

• New (Glyph) Grid topo = 8 (kNPtopoGrid):
  • This allows a grid anywhere in a hyperglyph tree.
  • New grids are now node type = 5 (kNPnodeGlyph) with topo = 8 (kNPtopoGrid).
    • Currently, one exception is the main root grid which is still type = 6 (*may change to 5 soon).
  • Grid spacing is stored using aux_a_x/y/z.

*Near Future plan is to have all grids (including sub-grids) have branch_level = 0 and the attached glyphs will have a (root) branch_level = 1. This will allow compositing multiple datasets in a hierarchy (of grids) that retain their original branch_level.

2017-06-01

np_node table:

• *Changed field name 'id' to 'np_node_id' for better table identification.
• *Replaced 'format_id' with 'np_tag_id' to allow multiple unique tags per record.
• Implemented 'shader' column for two-sided rendering option:
  • shader = 0 for single-sided rendering (default for nodes).
  • shader = 1 for two-sided rendering (default for sub-grids).

*Not yet implemented (as of 2020-06-25).