The notion of orthogonality here is analogous to the notion of orthogonality in vector spaces. There is even a similar notion of independence which may be defined.

A language L's primitives may be considered to be independent if, for any primitive p, there exists some relation which is constructible in language L which is not constructible in language L - p.

In other words, every primitive p in language L adds to language L's ability to express a variety of programs.

Orthogonality implies independence, of course, but is a much subtler property.

A language L's primitives are considered to be orthogonal if, for any primitive p, every relation declared by primitive p is not constructible in language L - p.

In order for a language to be representationally transparent, all of the language's primitives must be orthogonal. This is so that there isn't a combinatoric explosion in the number of different representations of programs due to the various ways that certain relations may be constructed.

It means that every language feature must have exclusive domain over the sorts of problems that it allows programmers to solve. It also means that, for a language to be usable, its primitives have to be both specific in domain, and general in application. It is important that any concievable composition of the language's primitives should be supported by the language's composition rules.