Processing Tasks - Nanook/NKit GitHub Wiki

NKit is able to perform a various tasks when processing an image. Currently there's only limited functionality available:

Process an entire image and output an xml nkit scan. Scans are a logical hybrid map of the image and filesystems. All the scan data and more is exposed to the nkit engine as it processes images. They also serve as a fingerprint of an image. As they're xml they can be compared easily to determine differences between similar images.

Convert an image to another format.

When converting to ISO, NKit will swap to the Expand Task.

RVZ is a fresh implementation from the spec. It's well tested and fully compatible with Dolphin. It is not binary equal to the RVZ created by Dolphin due to NKit inserting the source checksums as XXHash64, MD5, SHA-1 and CRC32. There are also minor (non breaking) differences with the RVZ encoding and zstd implementation.

WBFS and CISO have an option to be lossless or lossy (scrubbed). NKit is the first app to create lossless WBFS and CISO meaning that when expanding back to full ISO no data is lost, the output ISO is exactly the same that was converted to lossless WBFS/CISO.

Verifying lossy WBFS/CISO is not possible due to gap data being removed. If verify is specified it will fall back to DatLookup mode where it will attempt to look up the output image checksum in a dat. For this to yield success you would require a dat of lossy wbfs/ciso files. Otherwise, do not use verify with lossy conversions.

NKit.ISO/GCZ expands to a temp iso before converting to WBFS, CISO or RVZ.

When converting to ISO, NKit will swap to the Expand Task.

GDI <=> Cue + Bin is planned, but not implemented.

Encrypted (iso) => Decrypted (dec.iso). With 3K3y header support.

Expands images to their full state. Currently only implemented for:

Extract file systems from images. A folder is created based on the source image name and files from the filesystem are extracted to it.

Gamecube files are extracted with system files. The typical structure is as follows:

/sys
  apploader.img
  bi2.bin
  boot.bin
  fst.bin
  main.dol         # if not present then there is normally a default.dol in ../files
/files
  opening.bnr
  ...              # all other directories / files

Wii files are extracted with system files. There will be one folder per partition, typically UPDATE and DATA (sometimes there are others). The typical structure is as follows:

/DATA
  /disc
    header.bin
    region.bin
  /files
  ...              # all other directories / files
  /sys
    apploader.img
    bi2.bin
    boot.bin
    fst.bin
    main.dol       # if not present then there is normally a default.dol in ../files
  cert.bin
  h3.bin
  ticket.bin
  tmd.bin
/UPDATE
  ...              # Same as /DATA - /disc, /files, /sys etc.
/CHANNEL
  ...              # Same as /DATA - /disc, /files, /sys etc.
/P?                # ? being the partition type ID when < 255
  ...              # Same as /DATA - /disc, /files, /sys etc.
/P-????            # ???? being the 4 character ID
  ...              # Same as /DATA - /disc, /files, /sys etc.

WiiU extraction will include the data from all partitions. The typical structure is as follows:

/SI
  /02
    title.cert
    title.tik
    title.tmd
  /03              # included when there's a supplementary GM partition
    ...            # same as above
/UP0000000000000000000000000000
  /??????          # date
    ...            # update files
/GM0005000000000000000000000000     # Title ID will differ - sometimes more than one
  /code
    app.xml
    ????.rpx       # title specific boot
    cos.xml
  /content
    ...            # files
  /meta
    ...            # files

The extracted files from these images will match the content of the image as if mounted / extracted by other tools. PS3 will be decrypted if the key is present.

ISO9660 Form2 sectors used by PS1 / CD-i that have interleaved audio etc are not fully supported. This will be addressed soon. The Current behaviour is no different from a generic ISO tool.

Fix works in different ways depending on the system. Generally the FixInfo and FixFiles should be set, along with a dat to enable NKit to verify the image.

GameCube Fix can repair the following edits made to a GameCube image to match a Dat file:-

• Scrubbing (blanking of junk data)
• Moved Files (compacted / rearranged)
• Modified AppLdr.bin
• Modified Fst.bin
• Region changes
• Title changes
• Other header edits (inserted comments etc)

To facilitate image repair to match a dat, GameCube FixFiles are required. These can be extracted using the Task FixExtract (or Extract using the UI), NKitv1 is able to extract them also. Without these files only unscrubbing is possible.

Place the FixFiles in a folder e.g.

/FixFiles
  appldr[20030916][1A2B3C4D].bin
  fst[IDXX120000][1A2B3C4D][11AA22BB][FFEE0099].bin

Adding a dat will allow NKit to attempt many combinations of fixes (in-memory) in an attempt to brute-force a match.

Wii Fix can repair the following common changes made to images:-

• Scrubbing (blanking of junk data)
• Update partition removal
• Channel removal

Place the FixFiles in a folder e.g.

/FixFiles
  11AA22BB33CC44DD55EE66FF77AA88BB99CC00DD_N_FFEE0099
  IDXX120000_01_ABCD_N_A1B2C3D4
  IDXX120000_02_ABCD_N_F1E2D3C4
  ...

NKit Fix will use all files in the folder to work out a combination to get a Dat match and write them to the image. These can be extracted using the Task FixExtract (or Extract using the UI), NKitv1 is able to extract them also. Without these files only unscrubbing is possible.

PS3 Fix will scan a folder of files for the PS3_DISC.SFB. Once located, NKit will attempt to open the PARAM.SFO and read the values from them both. The root PS3 FixFiles folder is then searched for the best fit IRD that can be used to build the image.

When building the image, NKit will check the files in the filesystem to ensure they match those listed in the IRD. If files are missing or invalid they are searched for in the PS3 FixFiles Subdirectories by Size then matching MD5. Example of a PS3 FixFiles folder

/FixFiles             - FileFiles root
  Game1.ird           - Various irds, names are not important
  Game2.ird
  Game3.ird
  Game4.ird
  /LookupFixFiles1    - any directory of files - can contain subdirectories
    File1.bin
    File2.bin
  /MiscFiles          - another directory
    File3.bin
    File4.bin
  ...

Non-Contiguous / interleaved files are supported. Other IRD tools do not currently support them. When fixing images containing non-contiguous files, ensure the files have been extracted properly.

When using this fix task in batch mode, point the -in path at the directory containing all the extracted game folders. Ensure recursive search is on to allow nkit to find the SFB file (NKit needs a recursion limit to prevent it searching all sub folders - it will be added soon). Recursion can be turned off if -in points to the directory containing the SFB.

Archives of a PS3 extracted image are also supported.

FixExtract writes out files that can be used to repair/build an image matching a dat item. Files created with FixExtract are to be used with the Fix task.

Note: In the UI FIxExtract is located within the options for the Extract task.

Processing a GameCube image with FixExtract will produce the follow file types:-

• appldr[<AppLdrDate>][<AppLdrCrc>].bin - Many images contain the same appldrs
• fst[<GameId>][<AppLdrCrc>][<FstCrc>][<HeaderCrc>].bin - Unique per image

** HeaderCrc is a CRC calculated from the start of the image to the end of the FST.

The FST file contains some image header items as well as the image FST. This file's name points to the appldr to use by the CRC filename item. Using the data in the file pair, the Fix task is able to repair most images processed by older shrink tools that would move files to save space. Various region hacks and header modifications can also be reverted.

Duplicate files skipped and are not extracted.

Processing a Wii image will extract all non-game partitions. The various partitions are named:-

• <ContentSHA>_<N/K>_<FullPartitionCrc> - Update Partitions
• <PartitionId>_<ChannelNo>_<GameId>_<N/K>_<FullPartitionCrc> - Channels / Virtual Console

** FullPartitionCrc is from the start of the partition to next partition or end of image. The extracted files may not match this CRC if blank space has been removed.

These files were originally created and used by Wii Ultimate Unscrubber (pre NKit / SwiiT)

These files are required to repair images processed by tools that can optionally remove them to save space e.g.:-

• Wii Backup Manager
• Wit / Wiim's Tools
• NKit v1

FixExtract will create IRD's, a format used by existing tools for building Redump Images from extract files. The filename format is:

• <ImageName> [<GameTitle>] [ImaceCrc].ird

IRD files can be used with the Fix task. See PS3 Fix task documentation for more information.

The Verify task will attempt to test the image using various methods. There are 2 modes:-

• y: The default for the Verify Task (even when v is n)
• datLookup: Look up the image checksums in a Dat file. If no Dat is specified an error is thrown

When not in datLookup mode NKit will use the following methods to verify the image (in order of preference)

• Internal Source Image Checksums - NKit will display the methods and checksums
• DatMatch - checksums located in the dat where the source image name matches a Dat Item
• NKitScan - matches a .nkit scan. Used when inScan is set and a scan has the same name as the source image
• DatLookup - use the checksums from the scanned image and look up a match in the Dat

NKit will display the method and any verify checksums being used.

See the Verify page for detailed information.

Dedupe takes a set of images and extracts all the files from multiple images in a set puts them in a single file structure. Other non-recreatable data is also stored. This greatly reduces the size of a full set of images.

An NKit scan (xml) is produced for each image that is deduped. The scans can then be used as images by nkit. NKit can process the image using the scan and the deduped file set allowing all NKit Tasks and features to be used.

There is also a currently unreleased tool that can mount the NKit scans to a Virtual File System (VFS).

This task is experimental as there are changes still to be made. Deduped sets also bring more complicated management scenarios that need to be addressed, E.g:

• Removing unused files if a scan is removed
• Uplifting the deduped file stores when there's breaking changes
• Validating all the files are still present and valid

There is still work on the file store to be completed. When using this task, DO NOT DELETE YOUR SOURCE IMAGES as it is currently an experimental feature.