How to transfer huge volume of data - GenuineChannels/GenuineChannels GitHub Wiki

You need to send lengthy file from one host to another host or want to send an object that takes more than 350Kb being serialized.

By Dmitry Belikov

It is a common problem. You need to send a large file from one host to another. Or want to send an object that takes more than 350Kb being serialized.

First, never try to send it directly at one call.

Serialization is a really weak place in the entire .NET Remoting building. Serialization of large objects could be inadmissible for a server servicing large number of clients.
The transport level could be suppressed by sending or receiving such a huge block. For example, you use the HTTP protocol and an HTTP proxy denies all incoming requests exceeding the appropriate size.
Another issue is Garbage Collection. Each time you try to send an array larger than 2Mb, two copies are created. The first one is the source MBV array. The second one is the boxed array that will go to the real proxy for serialization. I hope you know that .NET uses a different algorithm for processing chunks larger than 2 Mb. And take a look at Q322975.

I found these reasons serious enough to choose another approach for sending huge data chunks in my applications.

The general idea is very simple. You should send an MBR object instead of the data. And it is up to the remote host to fetch the data in small chunks. The sample is here.

The known layer contains the IContentDownloader interface (to initiate file transfer) and the IDownloadableContent interface. The IDownloadConfirmation interface is used to show download progress. Generally, you need such an interface if you want either to get to know when downloading is finished or to show the progress bar to the end user.

/// <summary>
/// Downloader interface.
/// </summary>
public interface IContentDownloader
{
  /// <summary>
  /// Requests a server to download a content.
  /// </summary>
  /// <param name="content">The content to upload.</param>
  /// <param name="totalSize">Total content size.</param>
  /// <param name="confirmation">Confirmation agent.</param>
  void DownloadContent(IDownloadableContent content, int totalSize,
             IDownloadConfirmation confirmation);
}

/// <summary>
/// Represents downloadable content.
/// </summary>
public interface IDownloadableContent
{
  /// <summary>
  /// Is used to get the next portion of the data being downloaded.
  /// </summary>
  /// <param name="portionSize">Size of the requested portion.</param>
  /// <returns>Portion.</returns>
  byte[] GetNextPortion(int portionSize);
}

/// <summary>
/// Is used by the server to inform about download progress.
/// </summary>
public interface IDownloadConfirmation
{
  /// <summary>
  /// Is called by the remote host to inform of the download progress.
  /// </summary>
  /// <param name="percent">Percent downloaded.</param>
  void Progress(int percent);

  /// <summary>
  /// Is called by the remote host to inform of the download progress.
  /// </summary>
  /// <param name="percent">Percent downloaded.</param>
  [OneWay]
  void OneWayProgress(int percent);

  /// <summary>
  /// Is called after downloading is finished.
  /// </summary>
  void Downloaded();
}

The content provider (a client application in my sample) needs to create an object implementing the IDownloadableContent interface and give a download event receiver (IDownloadConfirmation) for receiving the download progress marks.

// CONTENT PROVIDER (CLIENT) CODE

/// <summary>
/// ProviderBasedOnStream.
/// </summary>
public class ProviderBasedOnStream : MarshalByRefObject, 
                                     IDownloadableContent
{
  /// <summary>
  /// Constructs ProviderBasedOnStream instance.
  /// </summary>
  /// <param name="stream">Stream.</param>
  public ProviderBasedOnStream(Stream stream)
  {
    this.Stream = stream;
  }

  public Stream Stream;

  /// <summary>
  /// Is used to get the next portion of the data being downloaded.
  /// </summary>
  /// <param name="portionSize">Size of the requested portion.</param>
  /// <returns>Portion.</returns>
  public byte[] GetNextPortion(int portionSize)
  {
   // here I try to calculate result size and allocate such chunk.
   portionSize = Math.Min((int) (this.Stream.Length - 
                      this.Stream.Position), portionSize);
   byte[] chunk = new byte[portionSize];

   // read from the stream
   portionSize = this.Stream.Read(chunk, 0, chunk.Length);    

   if (portionSize != chunk.Length)
   {
     // in my case it's impossible, but probably this code will 
     // fit to your situation
     byte[] secondChunk = new byte[portionSize];
     Buffer.BlockCopy(chunk, 0, secondChunk, 0, portionSize);
     return secondChunk;
   }

   return chunk;
  }
}

How the sender initiates uploading:

// CONTENT PROVIDER (CLIENT) CODE

IContentDownloader iContentDownloader = (IContentDownloader)
         Activator.GetObject(typeof(IContentDownloader), 
         ConfigurationSettings.AppSettings["RemoteHostUri"] + 
         "ContentDownloader.rem");

// let user to choose the file to upload
OpenFileDialog openFileDialog = new OpenFileDialog();
if (openFileDialog.ShowDialog() == DialogResult.OK)
{
   Stream fileStream = openFileDialog.OpenFile();
   iContentDownloader.DownloadContent(new ProviderBasedOnStream
        (fileStream), (int) fileStream.Length, new Class1());
   Console.WriteLine("Download initiated.");
}

_fileDownloaded.WaitOne();

The receiver receives the data in small portions. In order to inform the client about download progress, the receiver makes an asynchoronous or one-way call.

// CONTENT RECEIVER (SERVER) CODE

string fileName = DestinationDirectory + Guid.NewGuid().ToString("N");
Stream outputStream = File.Create(fileName);

for ( int bytesDownloaded = 0; bytesDownloaded < this.TotalSize; )
{
   // request the next portion
   int sizeToRead = Math.Min(PORTION_SIZE, 
                        this.TotalSize - bytesDownloaded);

   // This synchronous call will fail if content provider
   // does not reply for the specified time-out 
   // (120 secs by default)
   byte[] content = 
           this.IDownloadableContent.GetNextPortion(sizeToRead);

   // save it into the file
   outputStream.Write(content, 0, content.Length);      

   // update counters
   bytesDownloaded += content.Length;

   int percent = bytesDownloaded * 100 / this.TotalSize;

   //I would recommend to use one-way method for notifying of progress
   // unless you need ClientSession or host info.
   this.IDownloadConfirmation.OneWayProgress(percent);
}

this.IDownloadConfirmation.Downloaded();

Please note, there is no need to register a sponsor on the provided MBR object. The lease will be extended each time you request the next portion.

The second issue that should be raised here is Stream and other MBR providers that require the byte[] buffer to be sent as a parameter. In the previous edition of this article I advised to use Stream as a content provider. The primary problem here is that each time you call the Stream.Read method, you send useless content to the remote host. So you just double the traffic. As you can see, my sample returns the byte[] buffer, it does not receive it. You can declare it as a reference or output parameter (ref or out in C#) and set its initial value to null to avoid transferring useless content to the content provider.

How to transfer huge volume of data - GenuineChannels/GenuineChannels GitHub Wiki

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️