Philosophy:Serialization - hpgDesigns/hpgdesigns-dev.io GitHub Wiki

IDEs in the ENIGMA family need to work with a number of formats, and have a number of additional requirements on top of that. The crux of this issue is that not all formats which will ever need supported can be known a priori. We may know up front that we want to support a good handful of formats, but that is by no means all formats. We may know up front that we want a good number of resources and features to be stored in our format, but we could never possibly write the format to accommodate all items that may ever need saved. For the same reason that an IDE should support many plugins, it should use an extensible abstraction layer for exporting data to each format. The serializer classes should therefore provided all needed mechanisms to handle these situations:

• Case 1: The most basic case: Joe Shmoe wants to work on his game using format X. He does all saving and loading in format X, and never uses any features that the format does not offer.
• Case 2: Joe needs to save his game in format Y so that his friend can use it. Format Y is a relatively complete format, but lacks certain features of format X. Some features can be emulated, while others cannot.
• Case 3: Joe's friend, Kevin, has a problem with one of the features of his game, and asks Joe to take a look at it. Kevin uses the same IDE, and the same format X, but format X is a very extensible format, and Kevin has several plugins installed which save additional fields to resources in the game. Joe's IDE does not have these plugins installed, but he does not need them to see the problem. Joe fixes the problem, saves the game, and sends it back to Kevin.
• Case 4: Joe has been working on his game for some time now, and it has reached a total resource count of over 600, and a total size of over 120 megabytes. Joe chose to use format X, which is a modular archiving format and does not require re-writing all data to insert resources.

In Case 1, we have no problem. The IDE loads some types of data, the IDE saves the same types of data. GMK was a perfect format if we only consider this case.

In Case 2, now we have a problem. The logical solution is to warn that the destination format lacks support for the current game contents, and prompt the user for confirmation to proceed, anyway. A great example of this inconsistency is in GM6's lack of support for alpha channels in its images, and GMK's corresponding lack of a transparency pixel option. LateralGM handles this by generating an alpha channel from the transparency pixel when saving as GMK, and filling the background with flamingo pink when saving to GM6. The user should still be warned, probably. This raises other questions: How do we support an arbitrary number of formats? See #Supporting Arbitrary Formats.

In Case 3, we encounter a new type of problem. Both IDEs use a modular, fully-functional format, but Joe doesn't have the plugins to interpret all the data used in Kevin's game. There are a couple options here. The IDE can take the GM approach and fail to load if there is any unknown data. It can take an even more bastardly approach, and just discard extra data on save. It can take a less bastardly approach and warn when saving and loading such data. Or it can take a guess-and-pray approach by keeping note of which attributes were read but not processed, and simply write those attributes back to the game, unmodified, if that resource is saved. This could also be improved by standardizing the way plugins add information to existing resources. See #Standardizing Extensions.

A solution to Case 4 helps ameliorate the issue in Case 3, as well. The idea behind case 4 is that most games have a good deal of data that simply doesn't need to be loaded in memory at all times. The IDE doesn't have to save an entire EGM or GMX game each time the user updates one resource; each resource has its own file on disk which is flexible in length. Because of this, in case 3, Joe would be unhindered by problems with missing plugins unless those plugins specifically altered the resources he changed. In that case, the IDE would handle the issue as usual.

The idea behind supporting an arbitrary set of very different formats is to have each resource in the IDE serialize itself into a document-object-like scheme to be re-interpreted by any of a number of classes implementing a serializer interface. In the data abstraction class, hereafter SerialData, resources are populated using generic, but unique, names used only by the IDE. It is the job of individual serializers to translate this name to their correct format. For example, let's say that the IDE, or separable, modular components therein (such as individual ResourceKind classes), use the name "sprite" to denote a sprite resource, the name "origin_x" to denote its x offset, the name "id" to denote its ID, and the name "name" to denote its assigned identifier. Let's say that we are interested in writing three serializer classes, one for GMK, one for GMX, and one for EGM.

The GMK serializer will scan the given serial data for "sprite" objects, and order them in itself by their "id" properties. It will then look up the "name", "origin_x", etc. properties, and put those into the binary format used by GMK. Once it is done, it compresses and encrypts the data, and streams it into the file. It does this for each resource in the correct order, and then generates errors or warnings for any attributes that could not be used in the format.

The GMX serializer would take a more dynamic approach, keeping a dictionary of each usable name ("sprite", "id", "name", "origin_x", ...) to an appropriate translation, which in this case would need to be a tuple of the equivalent identifier in GMX's format, and a boolean representing whether it is stored as an attribute or an element in the XML tree (ie, stored as xorigin="16" or <xorigin>16</xorigin>). Using that dictionary, the serializer translates the given SerialData to GMX-flavored XML, all the while bitching about attributes not in the dictionary.

The EGM serializer would be very similar to the GMX serializer, only it should not give a fuck whether properties in the SerialData class are attributes or elements, because (1) it all looks the same serialized, and (2) YAML doesn't do that.

So, how to best handle issues in Case 3? The best option seems to be standardizing the way plugins extend the format, and then having the IDE track unknown (un-processed) extensions.

  <sprite id="0" name="sprite0">
    <xoffset>16</xoffset>
    <yoffset>16</yoffset>
    <extension plugin="Sprite Mipmap Levels v1.0">
      <mipmaps>4</mipmaps>
    </extension>
  </sprite>

  -sprite:
    id: 0
    name: sprite0
    x-offset: 16
    y-offset: 16
    extensions:
      -Sprite Mipmap Levels v1.0:
        mipmaps: 4

The best way to facilitate this cleanly may be to have a special SerialData object just for extension methods, so the IDE can make informed decisions about what to ignore. Another idea that may need involved is giving the extension element/property a nested element/property to denote how safely it can be ignored. This would remove the "guess and check" aspect and allow the IDE to present notices only when they are needed. This element might denote that it is safe to re-save the (unmodified) extension data with the (modified) resource data (ie, there are no ties between the two data sets), it might denote that it is safe to strip the data (eg, if it's a "convenience" thing that has some kind of dependency on the data as it was), or it might indicate that it is unsafe to save that resource with modifications without having the plugin installed (and might give a recommendation of stripping the data vs saving it anyway). A similar attribute might be constructed to denote whether the object is safe to compile without having the plugin installed (or if it represents an important feature without which the game will not function properly).