Contribute - piximi/prototype Wiki

Packages

Last updated: 12/1/2021

Below is a list of packages used by Piximi along with their general use. Where necessary, the motivation behind including a given package has been elaborated, and its integration within Piximi described.

Primary Packages

Piximi was initially created with create-react-app with typescript and redux toolkit. Additional packages were installed and incorporated as needed.

In general, the packages which play the largest roles are Typescript, React, Redux (as well as the Redux Toolkit wrapper), Saga, MUI, and Tensorflow.js. Proficiency is needed with each in order to understand and develop on the codebase.

React + Redux

Typescript

Types

Tensorflow

Testing

Testing in Piximi currently has low code coverage. The short-term plan is to begin implementing unit tests on components; while the longer term plan is to move to continuous integration and test-driven development.

Development Tools

Styling

Pixmi makes heavy use of MUI for styling.

Canvas

Utilities

File Utilities

Image Utilities

Algorithm Utilities

Misc

Code Structure

Sagas

Although the createSlice functionality of redux toolkit neatly places all reducers in one easy to see place, there are additional side effects of action dispatches. This is due to the use of the saga middleware. src/store/sagas/rootSaga.ts combines various listeners which watch for actions, all of which are currently related to the classifier model. On certain action dispatches, these watchers are notified and in turn call appropriate functions in order to perform some work. The benefit of this is that the work is done asynchronously. Therefore, concurrent to the normal redux flow where the reducer responsible for the action is triggered, some additional work is concurrently performed by the saga middleware.

Saga Example - fit

src/index.tsx

The main index.tsx file wraps the Application with a Provider from react-redux, passing in productionStore.

  ReactDOM.render(
  <Provider store={productionStore}>
    ...
    <Application />
    ...
  </Provider>,
  document.getElementById("root")
);

src/store/stores/productionStore.ts

productionStore is a redux store that is configured with a reducers, and a set of Middleware which includes the saga middleware. The reducer is a typical redux reducer produced with redux toolkit. Saga is then run on rootSaga (explained below).

  export const saga = createSagaMiddleware();

  const middleware: Middleware[] = [logger, saga, thunk];

  const options = {
    devTools: true,
    enhancers: enhancers,
    middleware: middleware,
    preloadedState: preloadedState,
    reducer: reducer,
  };

  export const productionStore: EnhancedStore = configureStore(options);

  saga.run(rootSaga);

src/store/reducer/reducer.ts

The reducer passed in to productionStore is a combination of reducers from various store slices created with redux toolkit. Of particular interest to the fit flow is the classifierSlice.

  const reducers = {
    classifier: classifierSlice.reducer,
    imageViewer: imageViewerSlice.reducer,
    project: projectSlice.reducer,
    settings: applicationSlice.reducer,
    toolOptions: toolOptionsSlice.reducer,
  };

  export const reducer = combineReducers(reducers);

src/store/slices/classifierSlice.ts

Redux toolkit is used to create the classifierSlice, which has a set of reducers, including fit. Again, this is the typical redux flow. When an action is dispatched with type "classfier/fit", the fit reducer receives it and the current state. Here it seems as though the state is modified directly, which is not allowed by redux since state should be immutable, but redux toolkit is using Immer in the background, so the state is not actually mutated. Instead the new state is returned.

  export const classifierSlice = createSlice({
    name: "classifier",
    initialState: initialState,
    reducers: {
      ...
      fit(
        state,
        action: PayloadAction<{
          onEpochEnd: any;
        }>
      ) {
        state.fitting = true;
      },
      ...
    }
}

src/store/sagas/rootSaga.ts

The fit reducer in classifierSlice is not the only part of the code interested in "classifier/fit" actions. Here we have the rootSaga which is a combination of various sagas that watch for actions, including the watchFitSaga.

  export function* rootSaga() {
    const effects = [
      ...
      fork(watchFitSaga),
      ...
    ];

    yield all(effects);
  }

src/store/sagas/classifier/watchFitSaga.ts

The watchFitSaga listens for classfierSlice.actions.fit.type, which is the action of type "classfier/fit". When this action is dispatched fitSaga is called.

  export function* watchFitSaga(): any {
    yield takeEvery(classifierSlice.actions.fit.type, fitSaga);
  }

src/store/sagas/classifier/fitSaga.ts

fitSaga performs work related to fitting the classifier. As part of the work it calls fit.

  export function* fitSaga(action: any): any {
    ...
    const { fitted, status } = yield fit(compiled, data, options, onEpochEnd);
    yield put(classifierSlice.actions.updateFitted({fitted, status}));

src/store/coroutines/classifier/fit.ts

fit is defined in src/store/coroutines. Notice that it is an async function that returns a Promise. This is the benefit of using Saga. While classifierSlice.reducers.fit does its work synchronously, the fitSaga is run asynchronously. This means that we can keep interacting with the UI while the model is fitting in the background.

  export const fit = async (
    compiled: LayersModel,
    data: {
      val: tensorflow.data.Dataset<{
        xs: tensorflow.Tensor;
        ys: tensorflow.Tensor;
      }>;
      train: tensorflow.data.Dataset<{
        xs: tensorflow.Tensor;
        ys: tensorflow.Tensor;
      }>;
    },
    options: FitOptions,
    onEpochEnd: any
  ): Promise<{ fitted: LayersModel; status: History }> => {
    ...
    return { fitted: compiled, status: status };
  }

src/components/FitClassifierDialog/FitClassifierDialog/FitClassifierDialog.tsx

In the UI, the FitClassifierDialogAppBar takes the asynchronous onFit function as a prop. This function is responsible for dispatching the "classifier/fit" action.

  export const FitClassifierDialog = (props: FitClassifierDialogProps) => {
    const onFit = async () => {
      ...
      dispatch(
        classifierSlice.actions.fit({
          onEpochEnd: (epoch: number, logs: any) => {
            console.info(logs);
            console.info(epoch + ":" + logs.loss);
          },
        })
      );
    };

    return (
      <Dialog
        disableEscapeKeyDown
        fullScreen
        onClose={closeDialog}
        open={openedDialog}
        TransitionComponent={DialogTransition}
        style={{ zIndex: 1203 }}
      >
        <FitClassifierDialogAppBar
          closeDialog={closeDialog}
          fit={onFit}
          openedDrawer={openedDrawer}
          disableFitting={noCategorizedImagesAlert}
        />
        ...
        <DialogContent>
          ...
        </DialogContent>
      </Dialog>
    );
  };

src/components/FitClassifierDialog/FitClassifierDialogAppBar/FitClassifierDialogAppBar.tsx

The IconButton component is given the onFit function as a prop. When the button is clicked in the UI the function is called and the action is dispatched. The action is received by the reducer, as in the normal redux flow, and by watchFitSaga registered in the Saga middleware.

  export const FitClassifierDialogAppBar = (props: any) => {
    ...

    return (
      <AppBar className={classes.appBar}>
        <Toolbar>
          ...
          <Tooltip
            title={
              disableFitting
                ? "Plaese label images before fitting a model."
                : "Fit the model"
            }
            placement="bottom"
          >
            <span>
              <IconButton
                className={classes.button}
                onClick={fit}
                href={""}
                disabled={disableFitting}
              >
                <PlayCircleOutline />
              </IconButton>
            </span>
          </Tooltip>
          ...
        </Toolbar>
      </AppBar>
    );
  };