Fuzz Testing - lcompilers/lpython GitHub Wiki

This is a tracking page for a testing effort for ASR (https://github.com/rebcabin/ClojureProjects002/tree/master/asr). The idea is to generate random utterances that adhere to the ASDL grammar for ASR, and then feed them to the backends. The random utterances are called Nonsense ASR, or NASR. There are two kinds of NASR: syntactically correct nonsense ASR (SynNASR) and semantically constrained nonsense ASR (SemNASR).

• SynNASR is for testing error paths in the ASR backends. A random utterance is overwhelmingly unlikely to be semantically meaningful, but it must NEVER crash a back-end nor cause it to go into an infinite loop (spin). An example is a IntegerBinOp expression with a string and a float as arguments. ASDL allows this, but backends must reject it.

• SemNASR is for testing happy paths in the backends. SemNASR should be (meta-)semantically valid, we should be able to independently compute results, and we should be able to round-trip examples. For example, an IntegerBinOp with two IntegerConstant instances of the same kind, say i16, should generate code to compute the results, or perhaps, with optimizations turned on, compute the results at compile time.

We will write clojure specs for both SynNASR and SemNASR. clojure.spec.gen.alpha, clojure.spec.test.alpha, and clojure.test.check.generators give us ways to quickly generate large numbers of NASR strings/trees. SynNASR is the default because we can generate SynNASR directly from the ASDL grammar. SemNASR requires humans to write the specs. Our first example of SemNASR will be IntegerBinOp.