Tutorial ~ Example (Part 4) - miniboxing/ildl-plugin GitHub Wiki

ildl logo In the last part, we've seen how the `ildl-plugin` not only transforms code correctly, but also warns us about possible suboptimal behaviour and gives us actionable points. In this part, we will play around with the transformation more, extending it to traits and objects.

Since we have finished working on the transformation, let us put it in a separate file and compile it once and for all:

Put the following content in file complex.scala:

import ildl._

object Complex {
  // "define" type complex based on integer pairs
  type Complex = (Int, Int)

  // add the addition and multiplication operation to complex numbers
  implicit class IntPairAsComplex(val p1: Complex) extends AnyVal {
    def +(p2: Complex): Complex = (p1._1 + p2._1 , p1._2 + p2._2)
    def *(p2: Complex): Complex = (p1._1 * p2._1 - p1._2 * p2._2,
                                   p1._1 * p2._2 + p1._2 * p2._1)
    // we could define other operations here as well...
  }

  // transform Complex into long integer
  object IntPairAsComplex extends TransformationDescription {
    // bitwise conversions:
    private def real(l: Long @high): Int = (l >>> 32).toInt
    private def imag(l: Long @high): Int = (l & 0xFFFFFFFF).toInt
    private def pack(re: Int, im: Int): Long @high =
      (re.toLong << 32l) | (im.toLong & 0xFFFFFFFFl)  	
    // conversions:
    def toRepr(p: (Int, Int)): Long @high = pack(p._1, p._2)
    def toHigh(l: Long @high): (Int, Int) = (real(l), imag(l))
    // constructor:
    def ctor_Tuple2(_1: Int, _2: Int): Long @high = pack(_1, _2)
    // operations:
    def implicit_IntPairAsComplex_+(c1: Long @high, c2: Long @high) =
      pack(real(c1) + real(c2), imag(c1) + imag(c2))
    def implicit_IntPairAsComplex_*(c1: Long @high, c2: Long @high) =
      pack(real(c1) * real(c2) - imag(c1) * imag(c2),
           real(c1) * imag(c2) + imag(c1) * real(c2))
    def extension_toString(c: Long @high) = 
      "(" + real(c) + "," + imag(c) + ")"
  }
}

and the following code in example.scala:

object Test {
  import ildl._
  import Complex._

  adrt(IntPairAsComplex) {
    // test the output
    def main(args: Array[String]): Unit = {
      val x1: Complex = (3, 5)
      val x2: Complex = (2, 8)
      println(x1 + x2)
      println(x1 * x2)
    }
  }
}

Now we can compile the files individually:

$ ildl-scalac complex.scala 
$ ildl-scalac example.scala 
$ ildl-scala  Test
(5,13)
(-34,34)

Okay, we're on safe ground: we separated the transformation and the transformed code and everything still compiles and runs as expected. Now, let us make a more complex example:

object Test {
  import ildl._
  import Complex._
  
  trait ComplexOperation {
    def apply(o1: Complex, o2: Complex): Complex
  }

  object ComplexAddition extends ComplexOperation {
    def apply(o1: Complex, o2: Complex): Complex = o1 + o2 
  }

  object ComplexMultiplication extends ComplexOperation {
    def apply(o1: Complex, o2: Complex): Complex = o1 * o2 
  }

  adrt(IntPairAsComplex) {
    // test the output
    def main(args: Array[String]): Unit = {
      val x1: Complex = (3, 5)
      val x2: Complex = (2, 8)
      val op: ComplexOperation =
        if (util.Random.nextBoolean())
          ComplexAddition
        else
          ComplexMultiplication

      println(op(x1, x2).toString)
    }
  }
}

We introduced the ComplexOperation trait and two implementations: ComplexAddition and ComplexMultiplication. In the test, we randomly choose one of the two and apply it. A question you may be asking yourself is whether the newly defined objects are aware of the optimized long integer representation. A quick call to -Xprint:ildl-commit will give us the answer:

$ ildl-scalac example.scala -Xprint:ildl-commit
[[syntax trees at end of               ildl-commit]] // example.scala
... 
    abstract trait ComplexOperation extends Object {
      def apply(o1: (Int, Int), o2: (Int, Int)): (Int, Int)
    };
...

No, they are not. Since the newly defined objects are not part of the adrt(IntPairAsComplex) scope, they are not transformed. But, we can include them:

object Test {
  import ildl._
  import Complex._
  
  adrt(IntPairAsComplex) {
    trait ComplexOperation {
      def apply(o1: Complex, o2: Complex): Complex
    }

    object ComplexAddition extends ComplexOperation {
      def apply(o1: Complex, o2: Complex): Complex = o1 + o2 
    }

    object ComplexMultiplication extends ComplexOperation {
      def apply(o1: Complex, o2: Complex): Complex = o1 * o2 
    }

    // test the output
    def main(args: Array[String]): Unit = {
      val x1: Complex = (3, 5)
      val x2: Complex = (2, 8)
      val op: ComplexOperation =
        if (util.Random.nextBoolean())
          ComplexAddition
        else
          ComplexMultiplication

      println(op(x1, x2).toString)
    }
  }
}

As you can see, the adrt scope can be extended to include not only methods or statements but entire classes. It behaves much like a keyword, with the only limitation that it cannot occur at the top level, due to technical limitations (we don't currently extend the Scala parser from the ildl plugin -- although it has been done before by the macro-paradise plugin, it is rather difficult and negatively impacts portability across different Scala versions).

Now, with the transformed code, let us see the -Xprint:ildl-commit output (we simplified the code to make it readable):

$ ildl-scalac example.scala -Xprint:ildl-commit
[[syntax trees at end of               ildl-commit]] // example.scala
package <empty> {
  object Test extends Object {
    abstract trait ComplexOperation extends Object {
      def apply(o1: Long, o2: Long): Long
    };
    object ComplexAddition extends Object with Test.ComplexOperation {
      def apply(o1: Long, o2: Long): Long = IntPairAsComplex.implicit_IntPairAsComplex_+(o1, o2)
    };
    object ComplexMultiplication extends Object with Test.ComplexOperation {
      def apply(o1: Long, o2: Long): Long = IntPairAsComplex.implicit_IntPairAsComplex_*(o1, o2)
    };
    def main(args: Array[String]): Unit = {
      val x1: Long = IntPairAsComplex.ctor_Tuple2(3, 5);
      val x2: Long = IntPairAsComplex.ctor_Tuple2(2, 8);
      val op: Test.ComplexOperation = 
        if (util.Random.nextBoolean())
          ComplexAddition
        else
          ComplexMultiplication;
      println(IntPairAsComplex.extension_toString(op.apply(x1, x2)))
    }
  }
}

As you can see, all objects have been transformed correctly. Now, if we run the code, we'll get correct (random) answers:

$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(-34,34)
$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(-34,34)
$ ildl-scala  Test
(5,13)

A question to ask now is what if we abuse the adrt scope and leave some of the objects outside? While we discuss such cases thoroughly later, it is interesting to experiment:

object Test {
  import ildl._
  import Complex._

  trait ComplexOperation {
    def apply(o1: Complex, o2: Complex): Complex
  }

  object ComplexAddition extends ComplexOperation {
    def apply(o1: Complex, o2: Complex): Complex = o1 + o2 
  }

  // hahaaaa, we only transformed one of the objects!
  adrt(IntPairAsComplex) {

    object ComplexMultiplication extends ComplexOperation {
      def apply(o1: Complex, o2: Complex): Complex = o1 * o2 
    }

    // test the output
    def main(args: Array[String]): Unit = {
      val x1: Complex = (3, 5)
      val x2: Complex = (2, 8)
      val op: ComplexOperation =
        if (util.Random.nextBoolean())
          ComplexAddition
        else
          ComplexMultiplication

      println(op(x1, x2).toString)
    }
  }
}

Well, if you hoped the fact that signatures are no longer compatible would break the transformation, too bad:

$ ildl-scalac example.scala
$ ildl-scala  Test
(-34,34)
$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(-34,34)
$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(5,13)

Although only one of the objects was transformed, the transformation remains consistent across both transformed and non-transformed objects, producing correct results. Doing such a transformation manually would have been more difficult, as we would have had to introduce multiple methods to correctly maintain the inheritance relation between ComplexOperation and ComplexMultiplication. With adrt, the ildl-plugin does this automatically for us, offering the guarantee of correctness.

Let us try something else now: moving the object definitions to another file. As we mentioned previously, we can't put the adrt scope at the top level (at the package level) due to technical limitations, but we can enclose the three definitions in another object. Put the following code in ops.scala:

object Ops {
  import ildl._
  import Complex._

  adrt(IntPairAsComplex) {
    trait ComplexOperation {
      def apply(o1: Complex, o2: Complex): Complex
    }

    object ComplexAddition extends ComplexOperation {
      def apply(o1: Complex, o2: Complex): Complex = o1 + o2 
    }

    object ComplexMultiplication extends ComplexOperation {
      def apply(o1: Complex, o2: Complex): Complex = o1 * o2 
    }
  }
}

And this code in example.scala:

object Test {
  import ildl._
  import Complex._
  import Ops._

  adrt(IntPairAsComplex) {
    // test the output
    def main(args: Array[String]): Unit = {
      val x1: Complex = (3, 5)
      val x2: Complex = (2, 8)
      val op: ComplexOperation =
        if (util.Random.nextBoolean())
          ComplexAddition
        else
          ComplexMultiplication

      println(op(x1, x2).toString)
    }
  }
}

Now we can recompile everything and run:

$ rm *.class
$ ildl-scalac complex.scala 
$ ildl-scalac ops.scala 
$ ildl-scalac example.scala 
$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(-34,34)
$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(-34,34)
$ ildl-scala  Test
(-34,34)

If you have versions 1.0 or 1.1 of the virtual machine and haven't updated to the latest version of the ildl plugin, you will get the following output:

$ ildl-scalac example.scala error: value is not a member of object Complex.IntPairAsComplex error: value is not a member of object Complex.IntPairAsComplex error: value is not a member of object Complex.IntPairAsComplex error: value is not a member of object Complex.IntPairAsComplex ... error: value is not a member of object Complex.IntPairAsComplex error: value is not a member of object Complex.IntPairAsComplex Exception in thread "main" java.lang.StackOverflowError at scala.collection.immutable.List.hashCode(List.scala:84) ... at ildl.plugin.transform.coerce.CoerceTreeTransformer$TreeAdapters$TreeAdapter.adapt(CoerceTreeTransformer.scala:130) at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5410) at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)

> 
> This is known bug and [has been fixed recently](https://github.com/miniboxing/ildl-plugin/commit/00619ed3fcbf6714b1ebda7669c073fc1d6013f9). To get the example compiling, you need to update the `ildl-plugin` source and recompile:
> 
> ```
$ cd ildl-plugin
$ git reset HEAD~10 --hard # sorry, we did a couple of force-pushes
$ git pull origin master
...
$ sbt package
...
$ cd sandbox
$ ildl-scalac example.scala 
$ ildl-scala  Test
(-34,34)
$ ildl-scala  Test
(5,13)
$ ildl-scala  Test
(-34,34)

Now, seeing it works correctly, does it also work optimally? If you run the last compilation step with -Xprint:ildl-commit, you will notice the methods are called using the long integer encoding, even though the ComplexOperation, ComplexAddition and ComplexMultiplication objects were compiled separately:

$ ildl-scalac example.scala -Xprint:ildl-commit
[[syntax trees at end of               ildl-commit]] // example.scala
package <empty> {
  object Test extends Object {
    def <init>(): Test.type = {
      Test.super.<init>();
      ()
    };
    def main(args: Array[String]): Unit = {
      val x1: Long = IntPairAsComplex.ctor_Tuple2(3, 5);
      val x2: Long = IntPairAsComplex.ctor_Tuple2(2, 8);
      val op: ComplexOperation = 
        if (util.Random.nextBoolean())
          ComplexAddition
        else
          ComplexMultiplication;
      println(IntPairAsComplex.extension_toString(op.apply(x1, x2)))
      //                                          \______________/
      //               operation occurs directly on long integers
      //      \___________________________________________________/
      //        toString does not require conversion to tuples :)          
    }
  }
}

This shows an important property of ildl transformations: composition across separate compilation. If we compile two parts of the code using the same transformation, they will consistently interact using the optimized representation type, instead of converting to the high-level type. Enough about scopes, you'll read about them later.

Congratulations! You have just finished reading all the introductory material on the ildl transformations. Now that you've gone through all the pain to read and understand, how about putting this to good use? The first example transformation used the IntPairAsComplex transformation we just developed to work :)

From Here:

ildl wiki The `ildl-plugin` is a meta-programming technique aimed at allowing safe, custom transformations across library boundaries. Using `ildl`-based transformations, we were able to obtain speedups in excess of 20x and have optimized code across a wide range of use-cases.
**Return to the main page** or **return to the OOPSLA Step by Step Guide**
⚠️ **GitHub.com Fallback** ⚠️