Container files - TheVNConnoisseur/MICo_Guide GitHub Wiki

Main characteristics

Purpose of these files

Protect and encrypt the assets, to ensure that those cannot be tampered with.

What engine revisions have it?

Version number	Format used	Is it possible to avoid using it?
1	.odn	:x:
2	.dat	:white_check_mark:

ODN files

Sadly, due to the spaghetti code mess IDA generates when analyzing the latest game that features this format, it is safe to assume that there might be edge cases not taken into consideration here. Knowing that, it is better to analyze each possible type of ODN file, because some of them have their own special quirks to them. To make it easier to distinguish most of these differences, here is a table that puts every file in comparison:

Path	Type of files	Encrypted	Differences with standard index structure
data00.odn	Images (Without alpha layer)	:x:	-
data01.odn	Images (With alpha layer)	:x:	Offset is not relative to raw data
data02.odn	Scripts (System configuration)	:white_check_mark:	-
data03.odn	Scripts (In-game story)	:white_check_mark:	-
data04.odn	Audios (SE)	:x:	Offset is not relative to raw data, last entry contains no data
data05.odn	Audios (BGM)	:x:	Offset is not relative to raw data, last entry contains no data
data06.odn	Scripts (Menus)	:white_check_mark:	-
data07.odn	Images (Mask)	:x:	-
data08.odn	Images (Character names in dialog window)	:x:	Offset is not relative to raw data
snd/`{first ODN files}`	Audios (Voice)	:x:	Offset is not relative to raw data, last entry contains no data

It is worth mention that inside the snd folder, there are some ODN files that are straight out MPEG-1 and WMV3 files, those simply have their extension changed to odn, probably to confuse modders.

File structure

In order to avoid cluttering the page with duplicated information, for the edge cases, only the differences with the standard index will be mentioned.

Standard

Entries

Length: Number of Entries * (Entry name + Entry offset) bytes
Type of value: -
Purpose: give the most basic information for obtaining and utilizing each of the files inside the container.

Entry name

Length: 8 bytes
Type of value: ASCII string
Purpose: unused, probably just kept there for helping the developers debug any malformed ODN file. None of them include any extension, just a description-less name.

Entry offset

Length: 8 bytes
Type of value: ASCII string (unsigned integer when decoded)
Purpose: indicate the position where the actual data for said entry starts, omitting the bytes that conform the entire index of entries.

Entries raw data

Length: -
Type of value: unsigned bytes
Purpose: hold the actual data for each entry.

Entries offsets not relative to raw data

Entry offset

Length: 8 bytes
Type of value: ASCII string (unsigned integer when decoded)
Purpose: indicate the position where the actual data for said entry starts, without omitting any byte from the ODN file.

Container with audio files

Entries

Length: Number of Entries * (Entry name + Entry offset + Entry PCM length) bytes
Type of value: -
Purpose: give the most basic information for obtaining and utilizing each of the files inside the container.

Entry PCM length

Length: 8 bytes
Type of value: ASCII string (unsigned integer when decoded)
Purpose: indicate the size in bytes of the file when decoded to a PCM stream, which lets the game know how long the audio file is.

How the engine extracts each file?

The game will act depending on the type of files stored in said container, which the game assumes based on the container's name. All of the steps will be listed in order so that it covers all edge cases at once. If any step number gets repeated, it means that for the type of ODN currently being mentioned it replaces the previously explained step with that same number.

Encrypted files

1. XOR all of its bytes following this little loop:

DataODN_CurrentOffset = 0;
for ( i = 0xFF; DataODN_CurrentOffset < DataODN_Size; --i )
  *((_BYTE *)DataODN_Buffer + DataODN_CurrentOffset++) = i ^ DataODN_Buffer[DataODN_CurrentOffset];

Container with audio files

2. When loading any function that requires reading any file, they always contain as a parameter the entry number. Thus, the game will jump to the offset of the desired entry.

3. Since all of the information of each entry is stored as an ASCII string, the game converts the value inside that string to its hexadecimal equivalent, which will be the offset of that entry.

4. After obtaining the offset of desired entry, the game repeats step 3 for the next 8 bytes, which include the PCM length for that entry.

5. To obtain the file size of the current entry, it needs to check the offset of the following entry. For that, the game repeat the exact same process detailed in step 3, but for the next entry. After finally obtaining said offset, the actual file size of the desired file will be obtained like so:

EntrySize = NextEntry_Offset - DesiredEntry_Offset

6. Once obtained the entry's size, since it does not need to know where the index of entries ends, the game can already jump to the offset where the actual data for the desired entry resides. To know that, it simply jumps to the offset indicated in the entry's offset.

Entries offsets not relative to raw data

The same as with the previous case, but skipping step 4.

Standard case

Step 4 from the second case gets skipped.

6. Once obtained the entry's size, it needs to know where the actual data for all entries starts. To know that, it will start skipping all entries until it reaches one that has an offset of ffffffff (in ASCII). Once reached that offset, the game will skip the remaining bytes left to read from the current entry's metadata.

7. Finally, to get to the start of the entry's data, starting from the offset reached in step 5, the game skips the number of bytes indicated in the entry's offset (the one obtained during step 3).

DAT files

History

Encryption version	Initial game
-	Kangokujou -Ingyaku no Shihai-
1	Masou no Ka no Snow White: Snow White and the Seven Emotions
2	Bitch Switch ☆ Revolution -Houkago Gyaru Bitch Sakusei Playlist-

File structure

Encryption version

Length: 4 bytes
Type of value: unsigned integer
Purpose: indicates the version of the encryption used.

Encryption version	Value used
-	Value not present
1	0
2	1

Index size

Length: 4 bytes
Type of value: unsigned integer
Purpose: indicates the size in bytes of the region where the metadata of all the files included the DAT file.

Entries metadata

Length: Index size * 268 bytes
Purpose: indexes all of the files that are in the container, and includes all of the necessary information to extract their corresponding data.

Entry name

Length: 260 bytes
Type of value: string
Purpose: indicates the name with extension of the file in that entry. A null byte signifies the end of the actual name.

Entry offset

Length: 4 bytes
Type of value: unsigned integer
Purpose: indicates the offset where the actual data of that entry starts, assuming that the first byte starts where the metadata of all entries ends.

Entry size

Length: 4 bytes
Type of value: unsigned integer
Purpose: indicates the size in bytes of that entry.

Entries data

Length: sum of all entries sizes
Type of value: unsigned bytes
Purpose: the actual data of all entries. They are in the same order that the metadata index holds each entry.

How the engine extracts each file?

1. First of all, the game will try to obtain the array of all the metadata that corresponds to the desired entry. That is accomplished by comparing the name of the file in that entry with the name of the wanted file, going one by one from the file.

2. Once the correct entry has been found, the game will obtain the offset and finally, the size of said entry.

3. The corresponding game function that was called will utilize the parameters obtained in the previous step to extract the actual raw data of the requested file from the .dat container, and load it into memory.

DAT decryption

The way the security is implemented in these files is incremental, which means that a game that supports a specific revision of the encryption scheme will support the previous ones too, including files that offer no protection.

Version 1

1. Now, the game will try to obtain the array of all the metadata that corresponds to the desired entry. That is accomplished by comparing the name of the file in that entry with the name of the wanted file, going one by one from the file.

2. Once the correct entry has been found, the game will obtain the offset and finally, the size of said entry.

3.1. Before sending these values out, the game needs to obtain the actual offset and size of the entry, since those are encrypted. At boot up, the game performs the following:

WinMain

DatEntries_Key_Seed = 0;
Dat_Key_Length = strlen(Dat_Key);
v12 = 0;
if ( Dat_Key_Length > 0 )
{
  do
  {
   *((_BYTE *)&DatEntries_Key_Seed + DatEntries_Key_Seed_CurrentIndex++) += Dat_Key[Dat_Key_Length];
   if ( DatEntries_Key_Seed_CurrentIndex == 4 )
     DatEntries_Key_Seed_CurrentIndex = 0;
   --Dat_Key_Length;
  }
  while ( Dat_Key_Length );
}

3.2. If we assume that the DAT key is 93 F1 94 FC 95 50 82 CC 83 7D 83 8A 83 49 83 6C 83 62 83 67 81 60 8D 82 8B 4D 82 C8 82 E9 8E 6F 96 85 82 CC 91 B8 8C B5 95 F6 89 F3 81 60, its length would be 46. It is important to remember that DAT key is 260 bytes long, so the rest of its bytes are 0x00 and thus, it means that Dat_Key[Dat_Key_Length] is 0x00.

3.3. DatEntries_Key_Seed is an integer value, meaning that its initial value would be 00 00 00 00. Knowing that, what the game does is add Dat_Key[Dat_Key_Length] with the current value DatEntries_Key_Seed[0] and write the resulting value in the latter, then it substracts 1 to the current value of Dat_Key_Length. Also, the game increases by 1 the value of DatEntries_Key_Seed_CurrentIndex. This will be repeated until the value of Dat_Key_Length is 0, meaning that Dat_Key[0] is never processed.

Following the example, the first iteration of this loop would go like this:

DatEntries_Key_Seed[DatEntries_Key_Seed_CurrentIndex] += Dat_Key[Dat_Key_Length] becomes DatEntries_Key_Seed[0] += Dat_Key[44], which means that DatEntries_Key_Seed[0] = 0x00 + 0x00 = 0x00
DatEntries_Key_Seed[0] += Dat_Key[44] becomes DatEntries_Key_Seed[1] += Dat_Key[43] for the next loop.

The second loop would look like this:

DatEntries_Key_Seed[1] += Dat_Key[43] = 0x00 + 0x60 = 0x60
DatEntries_Key_Seed[1] += Dat_Key[43] becomes DatEntries_Key_Seed[2] += Dat_Key[42] for the next loop.

This process gets repeated until DatEntries_Key_Seed_CurrentIndex reaches the value 4 (because an integer value only has 4 bytes), and in this case, said value gets set back to 0. Once finished the entire loop, the resulting value for DatEntries_Key_Seed is 0x52E992D3 = D3 92 E9 52.

4. The entire process indicated in step 4 gets repeated, but with different parameters:

GetFileFromDAT

CurrentEntry_Key = 0;
Current_FileName_Length = strlen(Current_FileName);
CurrentEntry_Key_CurrentIndex = 0;
do
{
  *((_BYTE *)&CurrentEntry_Key + CurrentEntry_Key_CurrentIndex++) += Current_FileName[Current_FileName_Length];
  if ( CurrentEntry_Key_CurrentIndex == 4 )
    CurrentEntry_Key_CurrentIndex = 0;
  --Current_FileName_Length;
 }
while ( Current_FileName_Length );

Step 3	Step 4	Implications of the changes
DatEntries_Key_Seed	CurrentEntry_Key	The key is only useful for the current entry, cannot be reused
DatEntries_Key_Seed_CurrentIndex	CurrentEntry_Key_CurrentIndex	-
Dat_Key_Length	Current_FileName_Length	Calculated from here

Following the example from before, let's pick an entry that looks like this:

73 79 73 5F 4E 6F 53 6F 75 6E 64 2E 6F 67 67 00
67 00 00 00 01 00 00 00 0A 00 00 00 50 01 00 01
A8 7D F4 02 88 7B F4 02 98 99 17 01 11 00 00 00
88 7B F4 02 44 00 00 00 1D 00 00 1D 7F 00 00 00
00 00 00 00 28 3F 11 00 00 00 01 00 11 00 00 00
90 7B F4 02 DA 00 00 00 11 00 00 00 11 00 00 00
02 00 00 00 08 00 00 00 11 00 00 00 C0 00 F4 02
03 00 00 00 70 02 F4 02 02 00 00 00 00 00 00 00
84 F8 37 01 BC F8 37 01 00 23 7A 77 59 9F 75 FE
FE FF FF FF 98 F8 37 01 8B 29 7B 77 14 02 00 00
20 02 00 00 9C 65 C3 74 BD 2A 7B 77 14 02 00 00
00 00 F4 02 14 02 00 00 44 00 00 00 20 02 00 00
00 00 00 00 10 00 00 00 03 00 00 00 00 00 00 00
14 02 00 00 E0 92 1F 77 CC F8 37 01 99 F3 75 77
08 00 00 00 7D D8 C1 88 14 02 00 00 01 00 00 00
B0 ED 92 75 00 00 00 00 00 00 00 00 00 00 00 00
D0 F8 37 01 13 60 74 05 E0 74 4F 05

In this case, we can see that the metadata to obtain from here is as follows:

Entry name: 73 79 73 5F 4E 6F 53 6F 75 6E 64 2E 6F 67 67, the name being sys_NoSound.ogg.
Entry offset: 13 60 74 05
Entry size: E0 74 4F 05

If we repeat the steps stated in the entire step 4, the resulting value for CurrentEntry_Key is 0x32BD91FC = FC 91 BD 32.

5. Now the game combines both keys obtained in step 4 and 5 doing the following:

GetFileFromDAT

LOBYTE(CurrentEntry_Key) += HIBYTE(DatEntries_Key_Seed);
BYTE1(CurrentEntry_Key) += BYTE2(DatEntries_Key_Seed);
BYTE2(CurrentEntry_Key) += BYTE1(DatEntries_Key_Seed);
HIBYTE(CurrentEntry_Key) += DatEntries_Key_Seed;

If we remember, our values here in memory are:

CurrentEntry_Key: 0x32BD91FC = FC 91 BD 32
DatEntries_Key_Seed: 0x52E992D3 = D3 92 E9 52

Therefore, since the game is developed for x86 (a little endian architecture), this function transforms to:

LOBYTE(CurrentEntry_Key) += 0xD3 = 0x32 + 0xD3 = 0x105; -> 0x105 mod 256 = 0x05
BYTE1(CurrentEntry_Key) += 0x92 = 0XBD + 0x92 = 0x14F; -> 0x14F mod 256 = 0x4F
BYTE2(CurrentEntry_Key) += 0xE9 = 0x91 + 0xE9 = 0x17A; -> 0x17A mod 256 = 0x7A
HIBYTE(CurrentEntry_Key) += 0x52 = 0xFC + 0x52 = 0x14E; -> 0x14E mod 256 = 0x4E

In the end, we obtain the final value for CurrentEntry_Key, which is 0x54F7A4E.

6. Finally, the game will decrypt both the offset and size of the given file. To do that, the game does the following:

DatFileV2_EncryptedOffsetRequestedFile = *((_DWORD *)Current_FileName + 65);
DatFileV2_SizeRequestedFile = DatFile_CurrentOffset;
*DatFileV2_OffsetRequestedFile = DatFile_IndexSize + (CurrentEntry_Key ^ DatFileV2_EncryptedOffsetRequestedFile) + 4;
*DatFileV2_SizeRequestedFile = CurrentEntry_Key ^ *((_DWORD *)Current_FileName + 66);
*DatFileV2_OffsetRequestedFile += 4;

Remembering the information obtained previously, the offset is 13 60 74 05 = 0x5746013 and the size is E0 74 4F 05 = 0x54F74E0. Now, in order to decrypt them successfully, the game will do this (in this example, DatFile_IndexSize is 0x58FC bytes long):

Size: CurrentEntry_Key ^ DatFileV2_EncryptedSizeRequestedFile = 0x54F7A4E ^ 0x54F74E0 = 0xEAE
Offset: DatFile_IndexSize + (CurrentEntry_Key ^ DatFileV2_EncryptedOffsetRequestedFile) + 4 + 4 = 0x58FC + (0x54F7A4E ^ 0x5746013) + 8 = 0x3B1A5D

7. The corresponding game function that was called will utilize the parameters obtained in the previous step to extract the actual raw data of the requested file from the .dat container, and load it into memory.