Engine usage, import and build structures - PROCEED-Labs/proceed GitHub Wiki
This page is about the different ways, the PROCEED engine can be bundled and included in other systems (e.g. in the Management System).
Be sure to have a look at the Git Repo Structure Page and the Cross Platform Framework before reading any further, in order to be familiar with the separation between native and universal parts. It contains a graphic with the structure of the repository. If you are interested in using different native modules, then have a look at this page.
Since development usually happens with the JavaScript native implementation, this guide will mostly focus on that. If you develop a native part for a different system, then the development and bundling will be depending on that ecosystem, case to case.
In this section we describe how to use the engine as its own program without including it in another program.
We can execute the JavaScript native part, which starts the universal part, in NodeJS via the command yarn dev (after we've installed all dependencies with yarn install). This starts the NodeJS file located at /src/engine/native/node/index.js. This files contains all the native modules we want to use and also initializes the start up of the universal engine part. For an explanation of this file as well as how to use own custom modules for the native part, have a look here. As you can see in the image below, the start of the universal part happens via a simple require() if it should run in the same process, and fork() if the Universal Part should be started as a child process.
The injector represents the code that is defined in the native parts but should be included in the runtime process of the universal part. One example for this is the vm2 module for the script execution, which needs to run in the universal part process but contains NodeJS-specific code and thus cannot be a part of the universal engine codebase. Thus, code like that gets injected from the native part into the universal part.
For production purposes, we would like to have a version of the engine, that can be run without requiring any additional setup or installs. For that reason, you can build the engine with yarn build, it uses webpack to create the bundled build files. They contain all used JavaScript files bundled into optimized output as well as their dependencies. After the build process, there are three files inside the /build/engine/ folder, native.js (using the webpack.native.config.js file), injector.js (using the webpack.injector.config.js) and one file universal.js (using the webpack.config.js file). This files contain all the code of the respective sub-folders bundled in one file. All webpack config files are explained in more detail in the following section.
The injected code needs to run in the same process as the universal part, which is why the eventual fork() is used on it and it requires the universal part itself. Since the universal part, as its name suggests, should not contain any NodeJS-specific code, the injector is not part of the univeral part directly, but rather its own file. This also enables us to exchange just the universal part between different implementations and versions.
After running yarn build, which calls webpack, you will get the bundled package with the following structure:
The green files and folders will be dynamically created at runtime and the orange config.js file can be created by the user to configure the engine.
The description is about the structure in the monorepo of the PROCEED project, since we provide executables for production that don't require any installations. If you customize the engine by making use of the published npm packages, then the webpack files could still be useful but might need to be adjusted to fit the new environment.
Click this to collapse/fold and see the details about the webpack file.
To bundle the universal code part of the monorepo (that is all the universal JavaScript of the PROCEED engine that runs unchanged on every supported system), we use the file webpack.universal.config.js at the location /src/engine/universal/ in the monorepo. This file tells webpack how to bundle the engine as a universal JavaScript file. It currently looks like this:
const path = require('path');
module.exports = {
entry: './core/src/module.js',
mode: 'production',
output: {
path: path.resolve(__dirname, '../../../build/engine'),
filename: 'universal.js',
libraryTarget: 'umd',
library: 'PROCEED',
globalObject: 'this',
},
node: {
process: false,
},
resolve: {
alias: {
'./separator.js$': './separator.webpack.js',
},
},
};The most important settings are:
entry, which tells webpack where to begin the bundling process.
output.libraryTarget, which enables us to use the bundled single-file in a web-context via a script tag as well as in node via require() by exposing the library in a number of different ways.
output.globalObject, which otherwise would cause an error in NodeJS since window is used as default.
node.process, which makes sure the process global is not polyfilled. (Note: Since the universal part, as its name suggests, does not use any built-in NodeJS modules we can ignore any target setting, in fact, the bundled code should be able to run on all targets, but we do check the process global internally so we need to explicitly exclude it here).
Click this to collapse/fold and see the details about the webpack file.
The corresponding webpack configuration file for the native part is called webpack.native.config.js.
const path = require('path');
module.exports = {
target: 'node',
entry: './index.js',
mode: 'production',
output: {
path: path.resolve(__dirname, '../../../../build/engine'),
filename: 'native.js',
},
externals: {
'@proceed/core': 'Function',
},
module: {
rules: [
{
test: /\.m?js$/,
exclude: /(node_modules|bower_components)/,
use: {
loader: 'babel-loader',
options: {
presets: [
[
'@babel/preset-env',
{
targets: {
node: 'current',
},
},
],
],
plugins: ['@babel/plugin-proposal-class-properties'],
},
},
},
],
},
node: {
__dirname: false,
},
};Here the most importing settings are:
target, which we set to node since it is the NodeJS native part and not universal code.
externals, which we use to exclude the universal part from the bundling process (since it is bundled separately). This setting swaps every require('@proceed/core') with an arbitrary object, here Function, since we don't want to include the whole universal part in the native bundle. (If the bundled version is started, is is looking for the separately bundled file universal.js.) In non-webpack environments (e.g. when running yarn dev) this obviously has no effect and we can make use of the require('@proceed/core').
We include Babel for transpiling the new way of writing properties in classes ( classX{ x=1; } ) to the old way of defining properties inside the constructor, because Node can not handle that yet.
Usually Webpack automatically creates polyfills for some objects. We disable this for node.__dirname, because the __dirname polyfill would insert a hard coded path and would not give us the currect directory. (We don't know why Webpack polyfills this at all, because we actually set the target to node)
Click this to collapse/fold and see the details about the webpack file.
The following is the webpack configuration file for the injector code of the native part, that is called webpack.injector.config.js.
// Detect all injected modules
let dependencies = [];
module.exports = {
registerModule: (mod) => {
if (mod.id && mod.onAfterEngineLoaded) {
dependencies.push(require.resolve(mod.id));
}
},
startEngine: () => {},
};
// We overwrite the @proceed/native module, to detect all registered modules and
// filter the ones which should be injected. This has to be of type Module, so
// we just pass the module object of this file.
require.cache[require.resolve('@proceed/native')] = module;
require('./index.js');
const path = require('path');
module.exports = {
target: 'node',
entry: [require.resolve('./native/src/injector.js')].concat(dependencies),
mode: 'development',
output: {
// eslint-disable-next-line no-undef
path: path.resolve(__dirname, '../../../../build/engine'),
filename: 'injector.js',
},
externals: {
'@proceed/core': 'Function',
},
node: {
__dirname: false,
},
};This config file actually does a bit more, since we have to dynamically determine, which native modules to include in the bundle. This is necessary, since we still want to have only one native/node/index.js file for the developer, in which he or she states the native modules to use for the native implementation. But not all native modules inject code into the Universal Part (for details see the Native Modules Guide). So, we have to determine at runtime (before we bundle the files), which native modules inject code into the universal part process and only bundle those.
If the engine should be used as part of a different software project, there are two options to do that.
- Using the prebundled versions that we provide (native.js, injector.js, universal.js), e.g. with
require()orimport(there are functions to start the engine withstartEngine(), stop(), etc.) - By writing an own index.js for the engine's native part and defining the (custom) native modules to use.
The 2. option works well if you don't try to bundle the whole software.
But for software that you want or need to bundle there is actually a small problem, because in our software development process the Native Part expects an already bundled and available universal.js.
(We create this file with a second webpack execution. Reason: so, we are able to start it as a child process. This is hard-coded in the file src/engine/native/node/native/src/index.js)
To solve this issue you can either work in Electron or set the environment flag SERVER.
Both is detected by software code and is then dynamically requiring the Injector Part and Universal Part, so that it can be included in the bundle.
(Note, however, that due to the nature of this operation, this prohibits the possibility of using a child process for the universal part.)
Otherwise (without the Flag or Electron), it assumes it is bundled for the standalone version and such prepares usage for the three separate files (engine.js, injector.js, universal.js).