Guide for Developers - Texera/texera GitHub Wiki

0. Development Environment Setup

Please follow the Getting Started Guide to install the required packages and obtain the codebase.

1. Setup Backend Development.

Import the Amber project into IntelliJ

Before you import the project, you need to have "Scala", "SBT Executor" and "Scalafmt" plugins installed in Intellij.

  1. In Intellij, open File -> New -> Project From Existing Source, then choose the amber folder inside core.
  2. In the next window, select Import Project from external model, then select sbt.
  3. In the next window, make sure Project JDK is set. Click OK.
  4. IntelliJ should import and build this Scala project. Wait until the indexing and importing is completed. >
  5. When IntelliJ prompts "Scalafmt configuration detected in this project" in the bottom right corner, select "Use scalafmt formatted". If you missed the IntelliJ prompt, you can check the Event Log on the bottom right

Your PR should pass scalafix check (lint) and scalafmt check.

  • To check lint, under core/amber run command sbt "scalafixAll --check"; to fix lint issues, run sbt scalafixAll.
  • To check format, under core/amber run command sbt scalafmtCheckAll; to fix format, run sbt scalafmtAll.
  • When you need to execute both, scalafmt is supposed to be executed after scalafix.

Run the backend engine in IntelliJ

The easiest way to run the webserver is in IntelliJ. Amber is a distributed engine. We need to run the controller process, and at least one worker process.

To run the controller process, go to src/main/scala/edu/uci/ics/texera/web/TexeraWebApplication.scala, then run the main function by pressing on the green run button.

Wait for the process to start up. If you can find the following messages, then the controller process is successfully started.

[main] [akka.remote.Remoting] Remoting now listens on addresses:
org.eclipse.jetty.server.Server: Started

To run the worker process, go to src/main/scala/edu/uci/ics/texera/web/TexeraRunWorker.scala, then run the main function. Wait for it to connect to the same cluster. If you see the following prompt, then it means you have successfully started the Texera server based on the Amber engine.

---------Now we have 1 node in the cluster---------

Run the backend engine in the command line (optional)

An alternative to run the backend engine is to run it in the command line. Navigate to core folder in terminal window, run scripts/server.sh, and then open a separate tab and run scripts/worker.sh.

Testing the backend

  1. The test framework is scalatest, tests are located under core/amber/src/test. You can find unit tests and e2e tests.
  2. To execute it, navigate to core/amber directory in the command line and execute sbt test.
  3. If using IntelliJ to execute the test cases please make sure to change the working directory to core/amber.

2. Develop Frontend

This is for developers that work on the frontend part of the project. This step is NOT needed if you develop the backend only.

Install Angular CLI

Recommend using nodejs@18 LTS and [email protected]. Install yarn:

npm install -g yarn

You need to install Angular CLI to build and run the new GUI.

yarn global add @angular/cli
yarn install

Ignore those warnings (warnings are usually marked in yellow color or start with WARN).

Start the frontend development server

Navigate to the core/new-gui directory and start the frontend development server:

yarn run start

Wait for some time and the server will get started. Open a browser and access http://localhost:4200. You should see the Texera UI with a canvas.

Develop Frontend using VS Code

We use VS Code for frontend Development.

Follow the instructions at https://code.visualstudio.com/download to install VS Code.

In VS Code, open the new-gui folder. VS Code will recommend some plugins to install. Follow the prompt to install those plugins.

In VS Code, open the terminal and run ng serve. Every time you save the changes to the frontend code, the ng serve process will automatically refresh the browser to show the latest UI.

Testing the frontend

Before merging your code to the master branch, you need to pass the existing unit tests first.

  1. Open a command line. Navigate to the core/new-gui directory.
  2. Start the test:
ng test --watch=false
  1. Wait for some time and the test will get started. You should also write some unit tests to cover your code. When others need to change your code, they will have to pass these unit tests so that you can keep your features safe. The unit tests should be written inside .spec.ts file.

Deploy the frontend to the production environment

Run the following command

yarn run build

This command will optimize the frontend code to make it run faster. This step will take a while. After that, start the backend engine in IntelliJ and use your browser to access http://localhost:8080.

Formatting frontend code

Run the following command

yarn format:fix

This command will fix the formatting of the frontend code.

3. Python UDF

  • Install [email protected]+, it's recommended to create a virtualenv to get a new copy of Python. Note: if you are using Apple's new M1 chip, please install Python through Anaconda.
  • Obtain Python executable path, for example, run which python or where python, and copy the returned path.
  • Fill the Python executable path into core/amber/src/main/resources/python_udf.conf, under the path key.
  • Install dependencies: pip install -r core/amber/requirements.txt -r core/amber/operator-requirements.txt.
  • For formatting python files: black core/amber/src/main/python.

4. User Dashboard (Optional)

  1. Install and use MySQL@8+ and create texera_db with core/scripts/sql/texera_ddl.sql.

  2. Edit core/new-gui/src/environments/environment.default.ts, change userSystemEnabled to true.

  3. Edit core/amber/src/main/resources/application.conf

    1. Change user-sys.enabled to true.
    2. Change jdbc.username and jdbc.password to your MySQL user and password to access texera_db that you just created.
  4. Optional: Add googleClientId to the same file to enable google login.

  5. Restart frontend and backend. You should see the homepage. You can register or login there.

5. Email Notification (Optional)

  1. Set clientId and clientSecret in core/amber/src/main/resources/application.conf.
  2. Login to Texera with an admin account.
  3. Open the Gmail dashboard under the admin tab.
  4. Authorize a Google account for sending emails.
  5. Send a test email.

6. Misc (Optional)

This part is optional, you only need to do this if you are working on a specific task.

To enable MongoDB storage for sink

  1. Install MongoDB on your development environment(4.4 works for Ubuntu, 5.0 works for Mac/Windows).
  2. Start MongoDB with the default configuration.
  3. Edit core/amber/src/main/resources/application.conf, change storage.mode to "mongodb".
  4. Start Texera, all the results of sink operators will be saved into MongoDB if applicable.

To create a new database table and write queries using Java through Jooq

  1. Create the needed new table in MySQL and update core/scripts/sql/texera_ddl.sql to include the new table.
  2. Run core/util/src/main/java/edu/uci/ics/util/RunCodegen.java to generate the classes for the new table.
  3. Create a helper class under core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard.

Note: Jooq creates DAO for simple operations if the requested SQL query is complex, then the developer can use the generated Table classes to implement the operation

Disable password login

Edit core/new-gui/src/environments/environment.default.ts, change localLogin to false.

Enforce invite only

Edit core/new-gui/src/environments/environment.default.ts, change inviteOnly to true.

Backend endpoints Role Annotation

There are two types of permissions for the backend endpoints:

  1. @RolesAllowed(Array("Role"))
  2. @PermitAll Please don't leave the permission setting blank. If the permission is missing for an endpoint, it will be @PermitAll by default.
⚠️ **GitHub.com Fallback** ⚠️