Additional info in private wiki.

All code changes should be done in a separate branch and submitted as a pull request, before they are incorporated into the main production branch.

We use a custom commit style, loosely based on but not adherent to conventional commits.

The top line (subject) should be prefixed with a [<tag1>, <tag2>, ...], where a tag can be one of:

• fix (bug fix)
• add (add files/code)
• del (delete/remove files/code)
• mod (modify variable/function/file/directory/etc names)
• opt (refactor/optimize code)
• clean (clean up, e.g removing unused imports)
• install install new npm package(s)
• remove remove npm package(s)
• merge or rebase

The subject line (after the tags) should also start with a capital letter, and should be short (50 chars or less). Do not end the subject line with a period. Use an imperative mood in the subject, i.e. written as though issuing a command or instruction. This is opposed to indicative mood, i.e. written as though reporting facts.

Good:

• [add] Add cropping to the preprocessing pipeline
• [opt, clean] Speed up interleaving of channels

Bad:

• cropping was added to the image preprocess pipeline. also clean up a few lines of cruft
  • not tagged
  • not capitalized
  • several sentences
  • indicative mood
  • more than 50 chars
• [FIX] More fixes for broken stuff
  • vague
  • indicative mood
  • tags should be lowercase

A rule of thumb for getting the mood right, excluding the tag and the capitalization, the subject line should be able to complete the following sentence: "If applied, this commit will your subject line here"

Separate subject and body with a blank line.

The content of the body should be a description of the motivation behind the change, and what is achieved by it, not how the changes were made. Explain the problem being solved. The code explains the how.

If relevant tag related issues, or resolved issues. For example:

• resolves: #321
• see also: #456, #457

Last updated: 12/1/2021

Below is a list of packages used by Piximi along with their general use. Where necessary, the motivation behind including a given package has been elaborated, and its integration within Piximi described.

Piximi was initially created with create-react-app with typescript and redux toolkit. Additional packages were installed and incorporated as needed.

In general, the packages which play the largest roles are Typescript, React, Redux (as well as the Redux Toolkit wrapper), Saga, MUI, and Tensorflow.js. Proficiency is needed with each in order to understand and develop on the codebase.

• react
  • React is a library for creating user interfaces.
  • The react package contains only the functionality necessary to define React components.
  • It is used together with a React renderer, react-dom, which targets the web.
  • Piximi: Used throughout the app.
• react-dom
  • Entry point to the DOM and server renderers for React.
  • Piximi: Used throughout the app.
• react-scripts
  • Includes scripts and configuration used by Create React App.
• react-router-dom
  • Client-side routing for React.
  • Annotator is a separate route from the main view in Piximi.
  • Classifier and Segmenter will be nested routes within the main view of Piximi.
• @react-hook/resize-observer
  • A React hook that fires a callback whenever ResizeObserver detects a change to its size.
• react-async-hook
  • A React hook that handles fetches, mutations, cancellations, and API race conditions
• react-color
  • A Collection of 13 Color Pickers (e.g. Sketch, Photoshop, Chrome, Github, Twitter, Material Design)
  • Piximi: used to pick colors for categories (classification labels)
• React DND
  • react-dnd
    • Set of utilities for implementing Drag and drop interfaces in React.
  • react-dnd-html5-backend
    • Enables react-dnd to access the HTML 5 Drag and Drop API.
  • Piximi: used for dragging and dropping files into and within the annotator.
  • TODO: integrate app-wide (e.g. dragging and dropping images into and within within the image grid).
• react-i18next
  • React integration with i18next.
  • Piximi: not currently used to any meaningful degree.
  • TODO: REMOVE (for now), or work to integrate it app-wide.
• (canvas) react-konva
  • Canvas oriented library for drawing complex canvas graphics using React.
  • It provides declarative and reactive bindings to the Konva Framework.
  • Is a peer dependency of the konva package.
  • Piximi: Used by the annotator for working with HTML5 canvas library.
• (canvas) react-konva-utils
  • Components and hooks for react-konva apps.
  • Has peer dependencies of konva and react-konva.
  • TODO: REMOVE, not used.
• use-dropbox-chooser
  • React hook for dropbox file chooser.
  • Piximi: Used in UploadButton component for uploading images directly from Dropbox.
• use-image
  • Custom React Hook for loading images.
  • Piximi: Used to create DOM image elements with passed in url, and pass the resulting element to Konva.
• use-sound
  • Custom React Hook for sound effects.
  • Piximi: Used in the annotator for sound effects. Loads mp3s and plays them when creating/deleting annotations.
  • TODO: Implement app-wide (e.g. training complete).
• @reduxjs/toolkit
  • The official, opinionated, batteries-included toolset for efficient Redux development.
  • Wrapper around Redux, which offers an opinionated, batteries-included toolset and API.
  • Helps slim down boilerplate redux code, and simplify configuration of the store.
  • Piximi: Used throughout the app.
  • NOTE: Redux toolkit comes with thunk, which has little to no use within Piximi. Instead, the use of saga is favored.
• react-redux
  • React bindings for Redux.
  • Has peer dependency of @reduxjs/toolkit.
  • Its purpose is to subscribe to the store, check to see if the data a component wants has changed, and re-render the component if so.
• redux-logger
  • Logger middleware for Redux actions.
  • Note: Must be last middleware in chain.
  • Piximi: Used throughout the app to log action dispatches to console.
• redux-saga
  • Redux middleware which acts as a side effect manager, for asynchronous actions and more.
  • Piximi: Used by the classifier. See Sagas.

• typescript
  • Static type analysis for JavaScript.

• @types/file-saver
  • Typescript type definitions for FileSaver.js
  • For use with file-saver package.
• @types/jest
  • Typescript type definitions for Jest
• @types/node
  • Typescript type definitions for Node.js
• @types/numeral
  • Typescript type definitions for Numeral.js
  • For use with numeral package.
• @types/react-dom
  • Typescript type definitions for React.
  • For use with the react-dom package.
• @types/react
  • Typescript type definitions for React.
  • For use with the react package.
• @types/react-color
  • Typescript type definitions for the react-color package.
• @types/react-redux
  • Typescript type definitions for the react-redux package.
• @types/react-vis
  • Typescript type definitions for the react-vis package.
• @types/redux-logger
  • Typescript type definitions for the redux-logger package.
• @types/lodash
  • Typescript type definitions for Lo-Dash.
  • For use with the lodash package.
• @types/uuid
  • Typescript type definitions for the uuid package.

• io-ts
  • Type checking at runtime for I/O data (e.g. project files); works well with static Typescript
• fp-ts
  • Peer dependency of io-ts providing Typescript oriented functional programming capabilities

• @tensorflow/tfjs
  • TensorFlow for the browser.
  • WebGL Hardware-accelerated JavaScript library for training and deploying machine learning models.
  • The tfjs monorepo contains the logic and scripts combining several packages, including @tensorflow/tfjs and @tensorflow/tfjs-vis.
  • Piximi: Used by the classifier and segmenter to train models, and perform inference.
• @tensorflow/tfjs-vis
  • Library for in browser visualization intended for use with Tensorflow.js.
  • Piximi: Used by the classifier and segmenter to visualize model performance in training.
• (dev) @tesnorflow/tfjs-node
  • CPU TensorFlow.Js for node.js
  • Use in testing only.

Testing in Piximi currently has low code coverage. The short-term plan is to begin implementing unit tests on components; while the longer term plan is to move to continuous integration and test-driven development.

• @testing-library/dom
  • Companion library for Testing-Library.
  • A DOM Testing Library for testing DOM nodes.
• @testing-library/jest-dom
  • Companion library for Testing-Library.
  • Allows the use of Jest to write tests that assert various things about the state of a DOM.
  • Provides custom DOM element (jest) matchers used to extend jest.
• @testing-library/react
  • Companion library for Testing-Library.
  • A React Testing Library providing testing utilities for the React DOM.
• @testing-library/user-event
  • Companion library for Testing-Library.
  • user-event tries to simulate real events that would happen in the browser as the user interacts with it.
• (dev) jest-canvas-mock
  • Mock canvas when running unit test cases with jest.

• @sentry/react
  • Sentry provides error tracking and performance monitoring.
  • @sentry/react is an SDK enabling automatic reporting of errors and exceptions.
  • Wrapper around @sentry/browser with added functionality related to React.
  • TODO: @sentry/tracing should be added?
• (dev) husky
  • Git hooks.
  • Used to lint commit messages, run tests, lint code, etc. when committing and pushing.
• (dev) lint-staged
  • Run linters against staged git files.
• (dev) prettier
  • Opinionated code formatter, enforcing consistent code styling by parsing the code and re-printing it with its own rules.

Pixmi makes heavy use of MUI for styling.

• @mui/material
  • Component library.
  • Usually integrated with Material Design, but custom design systems also supported.
  • Has peer dependencies of @emotion/react and @emotion/styled
  • Piximi: Used application-wide for various components (e.g. buttons, dialogs, inputs, etc.), and for defining styling themes for them.
• @emotion/react
  • Simple styling in React.
  • Sub-packages of larger Emotion library.
  • Needed because it's a peer dependency of @mui/material.
• @emotion/styled
  • Emotion's Styled API for @emotion/react.
  • Emotion is the default styling solution for MUI.
  • Well be used in favor of @mui/styles (below) by Piximi for defining the styles (CSS) for most components.
• @mui/styles
  • Legacy styling solution for MUI.
  • Not compatible with React.StrictMode or React 18.
  • Piximi: Although it is considered Legacy by MUI, Piximi currently makes heavy use of @mui/styles for defining the styles (CSS) that the themes from @mui/material rely on.
  • Will be replaced with MUI's styled, which makes use of @emotion/styled (above).
• @mui/lab
  • Laboratory for new MUI components, not yet ready to move to MUI core.
  • Pixmi: Used for various segmenter related components, non of which are yet integrated into the application.
• material-ui-popup-state
  • Integrates with MUI.
  • Takes care of the boilerplate for common Menu, Popover and Popper use cases.
  • Provides a Custom React Hook that keeps track of the local state for a single popup, and functions to connect trigger, toggle, and popover/menu/popper components to the state.
  • Pixmi: used throughout application.
• @mui/icons-material
  • Provides Google Material icons packaged as a set of MUI SvgIcon components.
  • Has peer dependency of @mui/material.
  • Pixmi: Used throughout application for various icons.

• konva
  • HTML5 2d Canvas library for desktop and mobile applications.
  • Used alongside react-konva.
  • TODO: Evaluate draggable: true against React DND.
  • Pixmi: Used heavily by the annotator for various type definitions needed for the use of react-konva.

• lodash
  • A utility library.
  • Piximi: Used throughout application, particularly in the annotator.
• numeral
  • Format and manipulate numbers.
  • Piximi: Used for some minor string formatting.
• uuid
  • For the creation of RFC4122 UUIDs
  • Piximi: Used throughout the application to assign categories and images unique ids.
• i18next
  • Internationalization-framework.
  • TODO: REMOVE, or work to integrate it.

• file-saver
  • Package for FileSaver.js
  • Saving generated files on the client-side.
  • Piximi: Used by the annotator to save annotations and project files.
• jszip
  • A library for creating, reading and editing .zip files with JavaScript.
  • Piximi: Used by the annotator for exporting annotations.

• image-js
  • Advanced image processing and manipulation in JavaScript.
  • Piximi: Used by the annotator for the bulk of its image processing.

• ngraph
  • Set of graph related algorithms.
  • ngraph.graph
    • Graph data structure.
  • ngraph.path
    • Fast path finding for arbitrary graphs.
  • Piximi: Used in the annotator for the magnetic annotation tool.
• ts-priority-queue
  • Provides an O(log n) approach to priority queue insertions and removals.
  • Written for use with Typescript.
  • Pixmi: Used in the annotator.

• web-vitals
  • The web-vitals library is a tiny (~1K), modular library for measuring all the Web Vitals metrics on real users, in a way that accurately matches how they're measured by Chrome and reported to other Google tools.

When working within a file or creating a new file, the order of imports should be consistent.

• The use of named imports should be prioritized when possible.
• When importing components from the same component subdirectory (i.e. src/components/annotator) use absolute imports. Otherwise use relative imports.

The order of imports should move from external to internal in the following manner...

1. Built-in

import fs from "fs";

2. External (grouped by module beginning with "react" and then "react-redux" and ending with @mui -- use named imports when possible)

import React from "react";
import {} from "react-redux";
...
import {} from "@mui/material";
import {} from "@mui/material-icon";

3. Internal hooks

import {} from "hooks";

4. Internal components (ordered from least to most related)

import {} from "components/common";
import {} from "../Parent"
import {} from "./Sibling"
import {} from "./"

5. Redux Store (no particular order)

import {} from "store/classifier";
import {} from "store/segmenter";
...

6. Types

import {} from "types";

7. Utils

import {} from "utils/"

8. Etc.

import {} from "data";
import {} from "colorPalette";
etc...

Although the createSlice functionality of redux toolkit neatly places all reducers in one easy to see place, there are additional side effects of action dispatches. This is due to the use of the saga middleware. src/store/sagas/rootSaga.ts combines various listeners which watch for actions, all of which are currently related to the classifier model. On certain action dispatches, these watchers are notified and in turn call appropriate functions in order to perform some work. The benefit of this is that the work is done asynchronously. Therefore, concurrent to the normal redux flow where the reducer responsible for the action is triggered, some additional work is concurrently performed by the saga middleware.

src/index.tsx

The main index.tsx file wraps the Application with a Provider from react-redux, passing in productionStore.

  ReactDOM.render(
  <Provider store={productionStore}>
    ...
    <Application />
    ...
  </Provider>,
  document.getElementById("root")
);

src/store/stores/productionStore.ts

productionStore is a redux store that is configured with a reducers, and a set of Middleware which includes the saga middleware. The reducer is a typical redux reducer produced with redux toolkit. Saga is then run on rootSaga (explained below).

  export const saga = createSagaMiddleware();

  const middleware: Middleware[] = [logger, saga, thunk];

  const options = {
    devTools: true,
    enhancers: enhancers,
    middleware: middleware,
    preloadedState: preloadedState,
    reducer: reducer,
  };

  export const productionStore: EnhancedStore = configureStore(options);

  saga.run(rootSaga);

src/store/reducer/reducer.ts

The reducer passed in to productionStore is a combination of reducers from various store slices created with redux toolkit. Of particular interest to the fit flow is the classifierSlice.

  const reducers = {
    classifier: classifierSlice.reducer,
    imageViewer: imageViewerSlice.reducer,
    project: projectSlice.reducer,
    settings: applicationSlice.reducer,
    toolOptions: toolOptionsSlice.reducer,
  };

  export const reducer = combineReducers(reducers);

src/store/classifier/classifierSlice.ts

Redux toolkit is used to create the classifierSlice, which has a set of reducers, including fit. Again, this is the typical redux flow. When an action is dispatched with type "classfier/fit", the fit reducer receives it and the current state. Here it seems as though the state is modified directly, which is not allowed by redux since state should be immutable, but redux toolkit is using Immer in the background, so the state is not actually mutated. Instead the new state is returned.

  export const classifierSlice = createSlice({
    name: "classifier",
    initialState: initialState,
    reducers: {
      ...
      fit(
        state,
        action: PayloadAction<{
          onEpochEnd: any;
        }>
      ) {
        state.fitting = true;
      },
      ...
    }
}

src/store/sagas/rootSaga.ts

The fit reducer in classifierSlice is not the only part of the code interested in "classifier/fit" actions. Here we have the rootSaga which is a combination of various sagas that watch for actions, including the watchFitSaga.

  export function* rootSaga() {
    const effects = [
      ...
      fork(watchFitSaga),
      ...
    ];

    yield all(effects);
  }

src/store/classifier/sagas/watchFitSaga.ts

The watchFitSaga listens for classfierSlice.actions.fit.type, which is the action of type "classfier/fit". When this action is dispatched fitSaga is called.

  export function* watchFitSaga(): any {
    yield takeEvery(classifierSlice.actions.fit.type, fitSaga);
  }

src/store/classifier/sagas/fitSaga.ts

fitSaga performs work related to fitting the classifier. As part of the work it calls fit.

  export function* fitSaga(action: any): any {
    ...
    const { fitted, status } = yield fit(compiled, data, options, onEpochEnd);
    yield put(classifierSlice.actions.updateFitted({fitted, status}));

src/store/classifier/coroutines/fit.ts

fit is defined in src/store/classifier/coroutines. Notice that it is an async function that returns a Promise. This is the benefit of using Saga. While classifierSlice.reducers.fit does its work synchronously, the fitSaga is run asynchronously. This means that we can keep interacting with the UI while the model is fitting in the background.

  export const fit = async (
    compiled: LayersModel,
    data: {
      val: tensorflow.data.Dataset<{
        xs: tensorflow.Tensor;
        ys: tensorflow.Tensor;
      }>;
      train: tensorflow.data.Dataset<{
        xs: tensorflow.Tensor;
        ys: tensorflow.Tensor;
      }>;
    },
    options: FitOptions,
    onEpochEnd: any
  ): Promise<{ fitted: LayersModel; status: History }> => {
    ...
    return { fitted: compiled, status: status };
  }

src/components/classifier/FitClassifierDialog/FitClassifierDialog.tsx

In the UI, the FitClassifierDialogAppBar takes the asynchronous onFit function as a prop. This function is responsible for dispatching the "classifier/fit" action.

  export const FitClassifierDialog = (props: FitClassifierDialogProps) => {
    const onFit = async () => {
      ...
      dispatch(
        classifierSlice.actions.fit({
          onEpochEnd: (epoch: number, logs: any) => {
            console.info(logs);
            console.info(epoch + ":" + logs.loss);
          },
        })
      );
    };

    return (
      <Dialog
        disableEscapeKeyDown
        fullScreen
        onClose={closeDialog}
        open={openedDialog}
        TransitionComponent={DialogTransition}
        style={{ zIndex: 1203 }}
      >
        <FitClassifierDialogAppBar
          closeDialog={closeDialog}
          fit={onFit}
          openedDrawer={openedDrawer}
          disableFitting={noCategorizedImagesAlert}
        />
        ...
        <DialogContent>
          ...
        </DialogContent>
      </Dialog>
    );
  };

src/components/classifier/FitClassifierDialogAppBar/FitClassifierDialogAppBar.tsx

The IconButton component is given the onFit function as a prop. When the button is clicked in the UI the function is called and the action is dispatched. The action is received by the reducer, as in the normal redux flow, and by watchFitSaga registered in the Saga middleware.

  export const FitClassifierDialogAppBar = (props: any) => {
    ...

    return (
      <AppBar className={classes.appBar}>
        <Toolbar>
          ...
          <Tooltip
            title={
              disableFitting
                ? "Plaese label images before fitting a model."
                : "Fit the model"
            }
            placement="bottom"
          >
            <span>
              <IconButton
                className={classes.button}
                onClick={fit}
                href={""}
                disabled={disableFitting}
              >
                <PlayCircleOutline />
              </IconButton>
            </span>
          </Tooltip>
          ...
        </Toolbar>
      </AppBar>
    );
  };

Basic instructions on how to use Piximi (i.e. labeling and annotating images) are described in the help dialogs in the Classifier and Annotator. Follow this short description on how to edit the Help content:
Clone the Piximi repository and create a new branch by running git checkout -b <someBranchName> origin/master. If you already have a local copy of the repository on your machine run git fetch --all to update your local copy before creating a new branch.

The content of the HelpDialogs can be found in src/components/common/Help/HelpContent/AnnotatorHelpContent.json and src/components/common/Help/HelpContent/ClassifierHelpContent.json.
The .json files contain a list of topics (e.g. "Open images") that are further divided into subtopics (e.g. "Deleting images"). Each subtopic has a property "subtitle" and a list of "descriptions". The descriptions in the Help dialogs should be short and precise. To add a new topic or subtopic, simply add a new object to the corresponding list. Apart from these .json files no other files need to be changed for editing the HelpDialog content.

Push your changes to the documentation by executing the following commands:
git commit -a -m "<some descriptions of what you changed>"
git push origin HEAD
Go to the Piximi GitHub page and create a pull request for your changes.

Keyboard shortcuts are implemented using the a custom hotkey hook library. Each hotkey needs to be registered with a "hotkeyView" which lives in a stack in the applicationSlice. Each view type is stored in the HotkeyView Enum in the Settings.ts type file. The following code creates a hotkeyView and registers shift+c as a keyboard shortcut to select the ColorAnnotation tool in the Annotator:

//Settings.ts

export enum HotkeyView {
  Annotator
}

...

//AnnotatorView.ts

import { useHotkeys } from "hooks";
import { HotkeyView } from "types";

import { registerHotkeyView, unregisterHotkeyView } from "store/application";

... 

useHotkeys("shift+c", () => {
    dispatch(setOperation({ operation: ToolType.ColorAnnotation }));
  },
  HotkeyView.Annotator
);

useEffect(() => {
  dispatch(registerHotkeyView({ hotkeyView: HotkeyView.Annotator }));
  return () => {
    dispatch(unregisterHotkeyView({}));
  };
}, [dispatch]);

...

The code in the useEffect pushes the view onto the hotKeyStack on component mount, and pops it on unmount.

User should be able to cancel dialogs by pressing esc and confirm their input by pressing enter. By setting the onClose prop in a dialog component <Dialog onClose={onClose} open={open}> pressing esc automatically closes the dialog. Setting hotkeys on dialogs is different from ordinary components, since dialogs are aways mounted. To use a dialog with hotkeys you must use useDialogHotkey() and pass in the HotkeyView, instead of using the useDialog() hook. The following code creates a hotkey dialog and registers enter as a keyboard shortcut for the CreateCategoryDialog:

//CreateCategoryItem.tsx

import {useDialogHotkey} from "hooks";
import { HotkeyView } from "types";

...

export const CreateCategoryItem = (props: CreateCategoryItemProps) => {
  
  ...

  const { onClose, onOpen, open } = useDialogHotkey(
    HotkeyView.CreateCategoryDialog
  );

  return (
    <>
     ...

      <CreateCategoryDialog
        categoryType={categoryType}
        onClose={onClose}
        open={open}
      />
    </>
  );
};

//CreateCategoryDialog.tsx

import {useHotkeys} from "hooks";
import { HotkeyView } from "types";

...

export const CreateCategoryDialog = ({
  categoryType,
  onClose,
  open,
}: CreateCategoryDialogProps) => {

  useHotkeys(
    "enter",
    () => {
      onCreate();
    },
    HotkeyView.CreateCategoryDialog,
    { enableOnTags: ["INPUT"] },
    [onCreate]
  );

 ...

};

The {enableOnTags: ["INPUT"] } argument enables the use of the hotkey when an input tag is focused.