1. Mirth Connect
2. Home
3. Examples and Tutorials

Created by Gerald Bortis, last modified on Dec 14, 2010

Moving large files (greater than 10MB) between systems using Mirth Connect can be a challenge since the entire file must be read into memory before it is sent to the destination. This will often result in "java.lang.OutOfMemoryError: Java heap space" exceptions after several files have been processed. One approach to this problem is to "chunk" the file, or break it into smaller files that can be readily transmitted and reassembled into the original file at the destination. This article describes how you can use the file chunker library along with a few channels to transfer large files.

You will need two channels running in Mirth Connect on the source system: the File Chunker Channel and the Chunk Transmitter Channel.

The first channel you will need to create is the chunker channel. The purpose of this channel is to read large files and create chunks that can be transmitted. Since the file chunker is still experimental, you must use the JavaScript Reader source connector to call the file chunker library. Below is an example of how to use the library:

importPackage(Packages.com.mirth.connect.chunker);
importPackage(Packages.java.io);
importPackage(Packages.java.util);

var chunker = new FileChunker();

// this is the chunk size in bytes
var chunkSize = 1000; 
// this is the directory to scan for large files
var sourceDir = new File("/source"); 
// this is where the chunk files should be written to
var chunksDir = new File("/chunks");
// this is where the large file will be moved to once it has been "chunked"
var processedDir = new File("/processed");

// this scans the source directory for new files, chunks them, and moves the original file to the processed directory
for each (var file in sourceDir.listFiles()) {
    if (chunker.chunkFile(chunksDir, file, chunkSize)) {
        file.renameTo(new File(processedDir, file.getName()));
    }
}

// all source connectors must return a list, in this case empty
return new ArrayList();

Icon

You must use forward slashes "/" in your file paths.

Icon

While the source connector for this channel is scanning for files and creating chunks, you will not see the message counts increase because the channel is not actually processing messages. All of the chunking action is performed in the source.

The second channel you will need is a chunk transmitter. The goal of the chunk transmitter is to transfer chunks produced by the chunker channel to the destination system. You can create this channel by using the File Reader pointed at the chunks directory (see previous channel) and any destination connector.

You will need two channels running in Mirth Connect on the destination system: the File Chunk Receiver Channel and the File Chunk Assembler Channel.

The third channel you will need is a chunk receiver. This channel complements the chunk transmitter channel by receiving the individual chunks as binary data and writing it out to files on the file system.

The final channel you will need is a chunk assembler. This channel will be deployed to a Mirth Connect instance at the destination and will receive the individual chunk files. Like the File Chunker Channel, the File Assembler Channel must be configured using the JavaScript Reader. The source connector will scan the specified directory for any chunk files and assemble them into the original file once all of the chunks have been received. Below is an example of how to use the library:

importPackage(Packages.com.mirth.connect.chunker);
importPackage(Packages.java.io);
importPackage(Packages.java.util);

var chunker = new FileChunker();

// this is the directory to scan for incoming chunks
var chunksDir = new File("/chunks");
// this is where the final assembled file will be placed once all the chunks are received
var finalDir = new File("/final");

// if all the chunks have been received, this will assemble them into the original file
chunker.scanAndAssemble(chunksDir, finalDir);

// all source connectors must return a list, in this case empty
return new ArrayList();

Icon

To monitor the chunker and assembler activity, you must enable logging for the chunking library. To do this, edit the $MIRTH_HOME/conf/log4j.properties file and add the following entry to the end of the file:

log4j.logger.com.mirth.connect.chunker=DEBUG

The library will be updated to also transmit an MD5 hash of the original file to allow for verification on the destination.

Document generated by Confluence on Nov 11, 2019 08:40

Atlassian