Guide for Developers - Texera/texera Wiki

Import the Amber project into IntelliJ

Before you import the project, you need to have "Scala", "SBT Executor" and "Scalafmt" plugins installed in Intellij.

1. In Intellij, open File -> New -> Project From Existing Source, then choose the amber folder inside core.
2. In the next window, select Import Project from external model, then select sbt.
3. In the next window, make sure Project JDK is set. Click OK.
4. IntelliJ should import and build this Scala project. Wait until the indexing and importing is completed.
5. When IntelliJ prompts "Scalafmt configuration detected in this project" in the bottom right corner, select "Use scalafmt formatted". Note: you PR needs to pass the format lint check; to check format, under core/amber run command sbt scalafmtCheckAll; to fix format, run sbt scalafmtAll. If you missed the IntelliJ prompt, you can check the Event Log on the bottom right

Run the backend engine in IntelliJ

The easiest way to run the webserver is in IntelliJ. Amber is a distributed engine. We need to run the controller process, and at least one worker process.

To run the controller process, go to src/main/scala/texera/web/TexeraWebApplication.scala, then run the main function by pressing on the green run button.

Wait for the process to start up. If you can find the following messages, then the controller process is successfully started.

• [main] [akka.remote.Remoting] Remoting now listens on addresses:
• org.eclipse.jetty.server.Server: Started

To run the controller process, go to src/main/scala/texera/web/TexeraRunWorker.scala, then run the main function. Wait for it to connect to the same cluster. If you see the following prompt, then it means you have successfully started the Texera server based on the Amber engine.

---------Now we have 1 node in the cluster---------

Run the backend engine in the command line (optional)

An alternative to run the backend engine is to run it in the command line. Navigate to core folder in terminal window, run scripts/server.sh, and then open a separate tab and run scripts/worker.sh.

Testing the backend

1. The test framework is scalatest, tests are located under core/amber/src/test. You can find unit tests and e2e tests.
2. To execute it, navigate to core/amber directory in the command line and execute sbt run test.
3. If using IntelliJ to execute the test cases please make sure to change the working directory to core/amber.

Install Angular CLI

Recommend using [email protected] LTS and [email protected]. Install yarn:

npm install -g yarn

You need to install Angular CLI to build and run the new GUI.

yarn global add @angular/cli
yarn install

Ignore those warnings (warnings are usually marked in yellow color or start with WARN).

Start the frontend development server

Navigate to the core/new-gui directory and start the frontend development server:

ng serve

Wait for some time and the server will get started. Open a browser and access http://localhost:4200. You should see the Texera UI with a canvas.

Develop Frontend using VS Code

We use VS Code for frontend Development.

Follow the instructions at https://code.visualstudio.com/download to install VS Code.

In VS Code, open the new-gui folder. VS Code will recommend some plugins to install. Follow the prompt to install those plugins.

In VS Code, open the terminal and run ng serve. Every time you save the changes to the frontend code, the ng serve process will automatically refresh the browser to show the latest UI.

Testing the frontend

Before merging your code to the master branch, you need to pass the existing unit tests first.

1. Open a command line. Navigate to the core/new-gui directory.
2. Start the test:

ng test --watch=false

3. Wait for some time and the test will get started. You should also write some unit tests to cover your code. When others need to change your code, they will have to pass these unit tests so that you can keep your features safe. The unit tests should be written inside .spec.ts file.

Deploy the frontend to the production environment

Run the following command

yarn run build

This command will optimize the frontend code to make it run faster. This step will take a while. After that, start the backend engine in IntelliJ and use your browser to access http://localhost:8080.

Formatting frontend code

Run the following command

yarn format:fix

This command will fix the formatting of the frontend code.

• Install [email protected]+, it's recommended to create a virtualenv to get a new copy of Python. Note: if you are using Apple's new M1 chip, please install Python through Anaconda.
• Obtain Python executable path, for example, run which python or where python, and copy the returned path.
• Fill the Python executable path into core/amber/src/main/resources/python_udf.conf, under the path key.
• Install dependencies: pip install -r core/amber/requirements.txt.
• For formatting python files: black core/amber/src/main/python.

To enable User Dashboard

1. Install and use [email protected]+ and create texera_db with core/scripts/sql/texera_ddl.sql.
2. Edit core/conf/jdbc.conf to include your MySQL user and password that can access texera_db that you just created.
3. Edit core/new-gui/src/environments/environment.default.ts, change userSystemEnabled to true.
4. Edit core/amber/src/main/resources/application.conf, change user-sys.enabled to true.
5. Restart frontend and backend. You should see the Dashboard Button shown on the workspace, or you can access it by navigating to /dashboard.

This part is optional, you only need to do this if you are working on a specific task.

To save workflow results to google sheet

Follow these steps if you work on saving the workflow result feature on the frontend.

1. Create/choose project and enable Google Sheets API and Google Drive API.
2. In the Credentials part, create OAuth 2.0 Client ID, choose Web application type and set Authorized redirect URIs to http://localhost:8888/Callback.
3. Download the Credentials json file and edit tokenPath and credentialPath in the file core/conf/web.conf.
   • tokenPath: a token cache directory path that is relative to core/conf/. It will be created if not exist.
   • credentialPath: a path to the Google Credential JSON file, with a path relative to core/conf.
4. Edit core/new-gui/src/environments/environment.default.ts, change downloadExecutionResultEnabled to true.
5. Restart frontend and backend. You should be able to use the file download button in the toolbar in the top middle after running workflow.
6. First time saving the results to google sheet will request your authorization. This step will create a token in the tokenPath you set in step 2. Keeping that token will allow you to skip the authentication step in the future.

PS: In the Credentials part in step 2, the type Service Account should be a more suitable way in this situation since it can skip the authentication. However, the Service Account OAuth document is out of date and its code is deprecated and is not working anymore. The ServiceAccountCredentials created in new version does not work with Google Sheet API/Google Drive API.

To enable MongoDB storage for sink

1. Install MongoDB on your development environment(4.4 works for Ubuntu, 5.0 works for Mac/Windows).
2. Start MongoDB with the default configuration.
3. Edit core/amber/src/main/resources/application.conf, change cache.storage to "MongoDB".
4. Start Texera, all the results of sink operators will be saved into MongoDB if applicable.

To create a new database table and write queries using Java through Jooq

1. Create the needed new table in MySQL and update core/scripts/sql/texera_ddl.sql to include the new table.
2. Run core/util/src/main/java/edu/uci/ics/util/RunCodegen.java to generate the classes for the new table.
3. Create a helper class under core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard.

Note: Jooq creates DAO for simple operations if the requested SQL query is complex, then the developer can use the generated Table classes to implement the operation