Saving to server over HTTP 2 - chung-leong/zigar GitHub Wiki

This example demonstrates how you can stream data from a web browser to a HTTP server using Zig. It makes use of WebAssembly threads so be sure that you have first applied the the necessary patch to the standard library.

Create the app

For ease-of-use sake we're going to rely on Vite on the browser side. In a terminal window, run the following command to create a boilerplate app:

npm create vite@latest

Enter stream-upload as name, then select React and JavaScript + SWC:

Need to install the following packages:
[email protected]
Ok to proceed? (y) y
✔ Project name: … stream-upload
✔ Select a framework: › React
✔ Select a variant: › JavaScript + SWC

Once the project is created, go into its directory and install the necessary files:

cd stream-upload
npm install

Then install the Zigar plugin:

npm install --save-dev rollup-plugin-zigar

In order to test our code we're going to need an HTTP server. We're going use Fastify for this purpose:

npm install --save-dev fastify @fastify/cors

CORS handling is neccessary since our test server will be listening on a port different from the one used by Vite's dev server.

We have to employ HTTP-2 because streaming from the client side isn't possible in HTTP-1. As secured connections are mandatory for HTTP-2, we need to create a self-signed certificate for our test server:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=CommonNameOrHostname"

In the src sub-directory, add server.js:

import CORS from '@fastify/cors';
import Fastify from 'fastify';
import { createWriteStream } from 'node:fs';
import { mkdir, readFile } from 'node:fs/promises';
import { dirname, join, resolve } from 'node:path';
import { pipeline } from 'node:stream/promises';
import { fileURLToPath } from 'node:url';

const srcDir = dirname(fileURLToPath(import.meta.url));
const fastify = Fastify({
  http2: true,
  https: {
    key: await readFile(resolve(srcDir, '../key.pem')),
    cert: await readFile(resolve(srcDir, '../cert.pem')),
  },
});
fastify.register(CORS, { methods: [ 'GET', 'PUT' ] });
fastify.get('/', async () => 'Hello world');
fastify.put('/uploads/:filename', async (request) => {
  const uploadDir = resolve(srcDir, '../uploads');
  await mkdir(uploadDir, { recursive: true });
  const filePath = join(uploadDir, request.params.filename);
  const fileStream = createWriteStream(filePath);
  await pipeline(request.raw.stream, fileStream);
  return `Received ${fileStream.bytesWritten} bytes`;
});
fastify.listen({ port: 8080 });

The PUT handler simply saves the incoming data stream to a file in /uploads. The GET is there only so that you can tell the browser to accept the self-signed cert.

Start the server with the following command:

node src/server.js

Then open https://localhost:8080. When the browser complains about the insecurity of the site, tell it to proceed nonetheless:

Connection warning

Once that's done, open App.jsx. Throw out the boilerplate code and replace it with the following:

import { useActionState } from 'react'
import './App.css'

async function sendData() {
  try {
    const url = 'https://localhost:8080/uploads/test1.txt';
    const transform = new TransformStream(undefined, { highWaterMark: 1024 * 16 });
    const writer = transform.writable.getWriter();
    const encoder = new TextEncoder();
    const buffer = encoder.encode('Hello world!\n');
    await writer.write(buffer);
    await writer.close();
    const response = await fetch(url, {
      method: 'PUT',
      body: transform.readable,
      duplex: 'half',
    });
    return await response.text()
  } catch (err) {
    return err.message;
  }
}

function App() {
  const [ message, formAction, isPending] = useActionState(sendData, '');
  return (
    <form action={formAction}>
      <button disabled={isPending}>Send data</button>
      <div>{message}</div>
    </form>
  )
}

export default App

All we have is a form consisting of a single button. When it's pressed, sendData() gets called. For now, it only sends "Hello world!" to the server. Once we've verified that our server-side code is functioning properly, we'll transfer the write operation to the Zig side.

We're using TransformStream here with undefined as the transform. This create an identity transform stream: what goes into transform.writable comes out of transform.readable. The highWaterMark setting allows the stream to buffer a certain amount of data before it stalls the writer.

Start Vite's dev server with the following command:

npm run dev

Then open the link shown on-screen. After clicking on the button you should see a message informing you that 12 bytes were received. You should also see an uploads directory. Within it there should be a file called test1.txt with the expected content.

We're ready to move the write operation to Zig. Open vite.config.js and add roll-plugin-zigar:

import react from '@vitejs/plugin-react-swc';
import zigar from 'rollup-plugin-zigar';
import { defineConfig } from 'vite';

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [react(), zigar({ topLevelAwait: false, multithreaded: true, optimize: 'ReleaseSmall' })],
  server: {
    host: true,
    headers: {
      'Cross-Origin-Opener-Policy': 'same-origin',
      'Cross-Origin-Embedder-Policy': 'require-corp',
    }
  },
})

Because threading is only possible when crossOriginIsolated is true, we have to tell Vite's dev server to send those HTTP headers.

In your code editor, create save.zig in the zig sub-directory:

const std = @import("std");
const zigar = @import("zigar");

var work_queue: zigar.thread.WorkQueue(thread_ns) = .{};

pub fn startup() !void {
    try work_queue.init(.{ .allocator = std.heap.wasm_allocator });
}

pub fn shutdown(promise: zigar.function.Promise(void)) void {
    work_queue.deinitAsync(promise);
}

pub fn save(
    writer: std.io.AnyWriter,
    promise: zigar.function.PromiseOf(thread_ns.save),
) !void {
    try work_queue.push(thread_ns.save, .{writer}, promise);
}

const thread_ns = struct {
    pub fn save(writer: std.io.AnyWriter) !void {
        var buffer = std.io.bufferedWriter(writer);
        var buffered_writer = buffer.writer();
        for (0..100000) |i| {
            try buffered_writer.print("Hello world {d}\n", .{i});
        }
        try buffer.flush();
    }
};

save() in the namespace thread_ns is the function that does the heavy-lifting. We use std.io.bufferedWriter() to buffer the web stream due the substantial latency going from Zig to JavaScript. Data transfer would be painfully slow if only a few bytes are written at a time. Zigar runtime actively checks for this condition. It'll throw an error if it receives on average less than 8 bytes per call after a hundred calls.

We then import the Zig functions into App.jsx and modify sendData():

import { startup, shutdown, save } from '../zig/save.zig';

async function sendData() {
  try {
    startup();
    const url = 'https://localhost:8080/uploads/test2.txt';
    const transform = new TransformStream(undefined, { highWaterMark: 1024 * 16 });
    const writer = transform.writable.getWriter();
    save(writer).then(() => writer.close());
    const response = await fetch(url, {
      method: 'PUT',
      body: transform.readable,
      duplex: 'half',
    });
    return await response.text()
  } catch (err) {
    return err.message;
  } finally {
    await shutdown();
  }
}

When you press the button now, the server should respond with a message saying 1788890 bytes was received. You should also see test2/txt in the uploads directory.

Source code

You can find the complete source code for this example here.

Saving to server over HTTP 2 - chung-leong/zigar GitHub Wiki

Create the app

Source code

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️