OpenCV GSoC 2020 Proposal - huningxin/opencv GitHub Wiki
IDEA: OpenCV.js: WASM SIMD optimization 2.0
Description: Through GSoC 2019 project Improve the performance of JavaScript version of OpenCV (OpenCV.js), the performance of some kernels, e.g. threshold, are improved significantly by applying WebAssembly (WASM) SIMD intrinsics and WebWorker/SharedArrayBuffer threads. However as WASM SIMD spec was still under active development in 2019, there were some remaining issues that include low performance for some kernels, e.g. cvtColor and resize, and lack of 64bit intrinsics support. And due to limited GSoC duration, only 3 kernels were tested and optimized specifically. With WASM SIMD spec will be in stable in 2020, it would be useful to improve the WASM SIMD optimization of OpenCV.js by fixing the remaining issues and targeting more kernels.
Expected Outcomes:
Implement 64bit intrinsics
Improve the performance of cvtColor and resize kernels
Refactory the js perf test framework for better code reuse
Add more perf tests for kernels of imgproc, dnn and objectdetect modules
Fix any potential perf issues due to new perf tests
Create sample code of OpenCV.js loader that can detect browser features and load according OpenCV.js build, e.g. SIMD-enabled or SIMD-disabled.
Skills Required: C++, JavaScript, HTML, WebAssembly, performance optimization, multi-threading and SIMD. Fluent in written English.
Mentors: Vitaly Tuzov, Ningxin Hu
Difficulty: Medium to Hard
IDEA: Develop OpenCV.js DNN modules for promising web use cases together with their tutorials
Description: OpenCV.js has JavaScript API of dnn module that allows web apps to do deep learning model inference in browser. However, except a face identification tutorial, there are lack of comprehensive tutorials about the deployment and inference of models in different formats, for promising usages like image classification and object detection with interactive web content like HTML5 video and WebRTC video streams.
Expected Outcomes:
Develop comprehensive tutorials about the end-to-end deployment and inference of deep learning models on the Web with OpenCV.js. The model formats include ONNX, TensorFlow, Caffe, Darknet and Pytorch.
Develop reusable sample code and helper functions for processing visual contents on Web including image, video (HTML5 video) and camera capture (WebRTC video stream) with dnn JS API.
Develop interactive samples for the deep learning usages including image classification, object detection, segmentation, pose estimation and image processing.
Skills Required: C++, JavaScript, HTML, image, canvas, video, WebRTC and experience with deep neural networks. Fluent in written English.
Possible Mentors: Ningxin Hu
Difficulty: Medium
IDEA: GPU-enabled OpenCV.js
Description: Today's OpenCV.js compiles kernels into WebAssembly format, so Web browsers can execute them on CPU efficiently. With the recent development of WebGPU and Web Shading Language(WSL), Web browser is adding the capability to run GPU compute shader for general purpose computing. Some web-based frameworks, like TensorFlow.js, are adopting WebGPU support for better performance. OpenCV has sophisticated GPU backends include Cuda, OpenCL, Vulkan and Halide. It would be useful to start prototyping OpenCV.js GPU backend with WebGPU by investigating different solutions.
Expected Outcomes:
Create a prototype of OpenCV.js GPU backend with WebGPU. Explore the possible solutions include (but not limited):
Translating Vulkan compute shader (SPIR-V) of OpenCV.js Vulkan backend to WSL and execute by WebGPU API.
Supporting WebGPU in Halide, so the OpenCV.js Halide backend can generate WSL for WebGPU execution.
Create a prototype of OpenCV.js I/O module with WebGPU support for more efficient buffer sharing between video/image and OpenCV.js WebGPU kernels.
Develop a benchmark to compare OpenCV.js CPU (WASM) and GPU (WebGPU) backends.