Development - UBOdin/mimir GitHub Wiki

Before anything else... DO NOT COMMIT WITHOUT FIRST RUNNING sbt test

Compiling the Code

SBT has an interactive mode that I suggest using for development purposes. Start by running sbt with no arguments. The command compile will compile the code.

Thor:Src okennedy$ sbt
[info] Set current project to Mimir (in build file:/Users/okennedy/Documents/Mimir/Src/)
> compile
[info] Compiling 1 Scala source to /Users/xthemage/Documents/Mimir/Source/target/scala-2.10/classes...
[warn] there were 1 deprecation warning(s); re-run with -deprecation for details
[warn] one warning found
[success] Total time: 1 s, completed May 25, 2015 12:58:09 PM

SBT also has an auto-run mode that watches the src directory for changes and auto-runs the relevant commands. Invoke auto-run mode by prefixing the command to be executed with a tilde:

> ~compile
[success] Total time: 0 s, completed May 25, 2015 12:57:32 PM
1. Waiting for source changes... (press enter to interrupt)
[info] Compiling 1 Scala source to /Users/xthemage/Documents/Mimir/Source/target/scala-2.10/classes...
[warn] there were 1 deprecation warning(s); re-run with -deprecation for details
[warn] one warning found
[success] Total time: 1 s, completed May 25, 2015 12:58:09 PM
2. Waiting for source changes... (press enter to interrupt)

Running Code

Mimir-Core

  • In SBT interactive mode, you can use run [arguments] to compile and run the command-line interface.
  • From the UNIX command line, you can use sbt run [arguments] to compile and run the command-line interface.
  • sbt assembly will create a standalone binary jar that you can use with ./bin/mimir [arguments]

Mimir-UI

Mimir also has a graphical user interface built using the play framework. This is a separate repository. Compile and run it using sbt run or just run from SBT interactive mode. Point your (chrome) browser at http://localhost:9000 to get to the UI.

Code Organization

Mimir is presently organized into two components:

Mimir-Core

  • /scala/mimir
    • /Mimir.java : The console UI for Mimir. The View in the MVC model.
    • /Database.java : The central hub of Mimir. The Controller in the MVC model.
    • /adaptive : Aspects of Mimir related to adaptive schemas
    • /algebra : ASTs for Relational Algebra (Operator), and Primitive-Valued Expressions (Expression)
    • /ctables : An abstraction layer for representing, rewriting, and analyzing Virtual C-Tables (see also lenses).
    • /exec : Key components of Mimir's execution engine (Compilers, Query Rewriting Systems)
    • /gprom : Integration with GProM
    • /lenses : Implementations of lens abstractions. See also /models
    • /models : Implementations of the models backing lenses
    • /optimizer : Logic for optimizing/rewriting queries and expressions
    • /parser : Logic for parsing text expressions
    • /plot : Logic for generating graphs
    • /provenance : Temporary placeholder for functionality required to track provenance (to be replaced by GProM)
    • /semistructured : Logic for working with semistructured data
    • /sql : Connectors for interfacing between traditional SQL and Mimir RA. Also code for interacting with a backend database (this will move)
    • /statistics : Logic for gathering statistics about the contents and structure of relations.
    • /util : Random utility classes.
    • /views : Logic for working with Views
  • /java/mimir : Legacy and Connector code for machine learning tools and JSqlParser

FAQ

The Scala Build Tool (SBT) is quite powerful... but that power comes hidden behind a lot of voodoo. It's hard to unwrap that voodoo to figure out how to do certain things. The goal of this section is to help you figure out how to do some of the basic things that SBT's documentation isn't particularly clear on.

How do I get SBT to do anything?

Some basic commands to know:

  • sbt run compiles and runs the project in the current directory
  • sbt compile just compiles the current project
  • sbt test runs all of the test cases
  • sbt testOnly [classname] where classname is a class in src/test/scala will run just that one test
  • sbt doc compiles a scaladoc to target/scala-2.XX/api/index.html
  • sbt package compiles the current project and creates a jar file in target/scala-2.XX/

You can also launch SBT in interactive mode by just running sbt. In interactive mode you can run any command from above (just remove the sbt in front). For example:

thor:mimircore okennedy$ sbt
[info] Set current project to Mimir (in build file:/Users/okennedy/Documents/Mimir/Src/mimircore/)
> compile
... [ compilation output ] ...
>

For easy debugging you can also put a tilde (~) in front of compile, test, testOnly, or run to have SBT keep re-running the command every time you save a file. For example sbt ~compile will compile the project every time you save a file.

How do I get SBT to run my code?

You need to test your code. This is a simple fact. As you're developing new features for Mimir you're going to want to get SBT to run your code and just your code without involving the Mimir command line or graphical UI. There are several practical ways to do this, but if you're asking this question the the correct (tm) thing to do is to write a new test case.

SBT works with a package called Specs2 to write specifications for testing code. A specification looks like:

package mimir.PACKAGENAME

import org.specs2.mutable._

object MyNewFeatureSpec
   Specification
{
  "My Fancy New Feature" should {
    "be awesome" >> {
      [ tests go here ]
    }
    "be epic" >> {
      [ more tests go here ]
    }
    "be cool" >> {
      [ still more tests ]
    }
  }
}

Each test must include one matcher. A matcher checks to see if the output of one of your functions is what you expect it to be. For example, the following matcher runs a query, extracts all rows, flattens the list of lists into a list of A values, and makes sure that there's at least one row where R.A = 1.

db.query("SELECT A FROM R").allRows().flatten() must contain(IntPrimitive(1))

I'll admit that it's a little bit annoying to bootstrap a test case, but once you do it's incredibly helpful. Once you're done coding your fancy new feature, your test case becomes part of the regular nightly unit tests. That way if someone changes code in a way that breaks one of your assumptions or if anything else goes wrong, we know about it sooner rather than later. Unit testing like this has saved my butt more than once.

How do I add new libraries

SBT uses a dependency tracking system called Maven for managing library dependencies. The way you configure dependencies in SBT is stupid, unintuitive, and a pain in the ass to figure out at first... Typically, adding a new library is as simple as adding one line to the project's build.sbt file. An example of adding a library dependency is

libraryDependencies += "ch.qos.logback" %  "logback-classic" % "1.1.7"

Here ch.qos.logback is the organization publishing the library (the GroupId), logback-classic is the library itself (the ArtifactId), and 1.17 is the version. Sometimes, nice library developers will give you this information. Otherwise, to find it, start with the Maven Repository's Search Page. Odds are you'll find it there.

There are two twists to the above. First, for scala-specific libraries, the Maven Repo may have separate copies of the library for different versions of Scala. You can tell because the ArtifactId will end in an underscore followed by a Scala version, as in scallop_2.11 or scallop_2.12.0-M4. In that case, (mumble mumble stupid UI), replace the first % with %% as in:

libraryDependencies += "org.rogach" %% "scallop" % "0.9.5"

Since Mimir-Core is using Scala Version 2.10.5, this line requests the ArtifactID: scallop_2.10.

Second, you can configure SBT to only include dependencies in certain stages by adding another % 'stagename'. In the following example, specs2 is included only in 'test' operations.

libraryDependencies += "org.specs2" %% "specs2-core" % "3.8.4" % "test"