Design - Gmadges/fluidPainter GitHub Wiki

Main design of the project

Inspiration and idea:

  1. My inspiration for this project came from seeing a paper called "WetBrush"[1] which detailed a painting system that uses fluid dynamics to achieve a realistic painting experience.
  2. I decided that I wanted to make a project that was cross platform and easy to share. For this reason I decided to make it browser based.

Tool choice:

Emscripten:

Emscripten is a tool that can compile C/C++ code into javascript, specifically asm.js. Asm.js is a dialect of javascript that has the potential to out perform conventional javascript. I chose emscripten because it allowed me to have a small amount of familiarity using C++ and would hopefully enable me to have greater performance than writing standard javascript.

Typescript:

I chose to use Typescript for most of my javascript side coding. Although this adds an extra build step, I think typescripts type system and other language features results in more robust code.

Nodejs + Npm:

I use nodejs and npm for my build system and testing just because of its flexibility and ease of use with web applications.

Mocha and Chai:

Mocha is the testing framework I decided to use. I chose it because it seemed simple and flexible to get up and running. Chai is an assertion library that i used for testing.

JQuery:

I used JQuery for most of the code that required reacting to DOM events and any DOM manipulation tasks.

Styling:
  • Bootstrap V4 - styling and responsive layouts.
  • Fontawesome - icons.
  • Bootstrap-ColorPicker - Colorpicker.

Key Design points:

Basic Premise

octocat

API design:

My main idea was to make the emscripten portion of the code act like an API. That way anyone could use it as a third party library and create their own fluid simulation. The style of the library was partly inspired by OpenCV. I like the method of having a main data structure, in openCV's case a "Mat" in mine a "buffer", that could be passed along different processes in a pipeline fashion and then display the result.

Buffers:

The system works around the idea of manipulating 'buffers' of data. I used 2D textures bound to a framebuffer as my data structure. I wanted to use WebGL as a method of improving performance, therefore textures were my only choice to manipulate data. These are all 4 channel due to webGL not supporting any less channels.

Using Emscripten:

I Decided to let the main loop be controlled by the javascript portion for maximum flexibility for the user. I used Emscripten's own binding library "Embind" as my tool of choice to create javascript bindings. This worked well, although it did cause linker issues resulting in the use of .hpp files.

Fluid solving:

GPU Gems[2] was a main source of info for the fluid simulation. Most could be converted to work with webGL with minor changes due to webGL's minimal capabilities. I used a grid based system because of its stability, reliability and speed.

User input:

User interaction proved to be quite tricky. Interaction between the javascript layer and Emscripten proved to be a point of difficulty. I kept most user input code on the javascript side to give the greatest flexibilty when designing. the biggest problem was receiving javascript code on the C++ side. Obviously C++ is a very strictly typed language and it needs things to be tightly defined unlike javascript. At one point I needed to pass an array filled with custom data to the Emscripten portion of the code. My solution was to use an array style structure defined in c++ and add a wrapper around it for inputting data and resetting. This allowed me to easily write code using this data structure in the C++ portion of the code.

Drawing Lines:

Drawing lines proved to be more difficult than I had expected. To draw I record the position of the mouse over the canvas when the mouseMove event is fired. The event does not fire often enough to get smooth lines. To try and get around this, I store the last 3 positions of the mouse and generate two quads in a line. The reason I use two quads is to account for any bends so they appear smoother. This could still use some improvement. Artifacts still appear with the texture either being stretched or strange lines appearing throughout.

Undo/Redo:

I wrote a generic class to handle my undo buffers. The task suited itself to TDD(Test Driven development) so I defined some tests based on the functionality I needed and worked from there. The problem I had over using conventional arrays was that I wanted to instantiate a set amount of buffers and not have to recreate them. the class acts as a wrapper moving the arrays index around and shifting the buffers based on the undo/redo action.

Trade Offs:

  • Using the embind functionality also resulted in linker errors when using .h and .cpp files, therefore I made the decision to use .hpp files.
  • Using Emscripten to handle webGL makes it very difficult to use pass data at run time. One example of how limiting this can be is that I wanted to implement the ability to load an image that the user can manipulate. you can not pass file data directly to Emscripten, therefore I thought I could work around this by loading the image to a texture in javascript and then passing the texture ID to be bound by the Emscripten code. I could not get this to work because texture ID's are not used with webGL and the IDs being generated in Emscripten were being performed by Emscripten itself not webGL. Maybe this is possible with a lot of hacking but I doubt it is worth it.

Future Improvements:

  • Improved user input. There are still artifices when the brush moves quickly across the canvas.
  • Better visualization of the fluid. I think the best improvement would be to add a level of depth to the image. One idea I had was to use pressure buffer as a bump/normal map because it would give and idea of the density of the fluid.
  • Support Mac OSX. I was unable to solve the problems found on Mac machines. I think it is something related to the webGL drivers.
  • Boundary checks on the fluid solver should be implemented.
  • Additional user control and expansion. e.g. the ability load own brush textures and canvas images. It would also be cool from a developers standpoint to support custom shaders being used with buffers.
  • Reduce the amount of resets needed when changing settings. e.g. when resizing, copy the existing image to the new buffer.

References:

  1. Chen, Z., Kim, B., Ito, D. and Wang, H., 2015. Wetbrush: Gpu-based 3d painting simulation at the bristle level. ACM Transactions on Graphics (TOG), 34(6), p.200.
  2. Harris, M.J., 2005, July. Fast fluid dynamics simulation on the GPU. In SIGGRAPH Courses (p. 220).