Insights about the host language - Lispl-Wicht/c-logo-s GitHub Wiki

Insight 1: Code that merely loads is still a draft.

Common Lisp is unusually permissive at load time:

  • undefined variables
  • wrong arity calls
  • forward references
  • duplicate definitions incomplete macro expansions

All of these are legal — by design — because Lisp values interactive development.

Compilation, however, is where Common Lisp asks the hard questions:

  • Are all names bound?
  • Are function calls well-formed?
  • Are macros expanding sanely?
  • Are types at least internally consistent?
  • Is the code stable enough to optimise?

So compile-file (and therefore ASDF) is the moment when Common Lisp says:

"Alright. Are you serious about this?"

Exploratory loading

(progn
  (ql:quickload '(parse-number str))
  (load "packages.lisp")
  (load "structs.lisp")
  (load "top-vars-consts.lisp")
  (load "primitives.lisp")
  (load "reader.lisp"))

This:

  • allows undefined variables
  • defers resolution to runtime
  • permits inconsistent intermediate states
  • does not enforce compilation hygiene

System loading via ASDF

(asdf:load-system :c-logo-s :force t)

which entails

(asdf:compile-system :c-logo-s :force t)

which in itself entails

compile-file

This

  • compiles first
  • assumes dependency correctness
  • recompiles downstream units
  • fails fast if compilation emits errors or warnings

"Loadable" ≠ "Compilable"

In SBCL (and most Common Lisps):

  • STYLE-WARNING → non-fatal, informational: design smell / optional improvement
  • WARNING → potentially serious, often correctness-related (important)
  • ERROR → fatal (must fix)
  • load works with warnings and style-warnings;
  • asdf:load-system fails when compilation signals errors, and may fail on warnings but not by default.

ASDF's default behaviour:

  • asdf:*compile-file-failure-behaviour* ; => :ERROR
  • asdf:*compile-file-warnings-behaviour* ; => :WARN

Clarification

  • load executes code with minimal validation.
  • asdf:compile-system checks code for consistency and produces compiled artifacts (FASL files).
  • asdf:load-system ensures the system is compiled (via asdf:compile-system) and then loads the resulting FASLs.
  • In SBCL, warnings are generally not fatal unless explicitly escalated by build policy.
  • FASL file = fast-load file

Practical hint

Write code so that it works when loaded cold, without any prior REPL state.

"Without any prior REPL state" means:
a freshly (re-)started Common Lisp session.

That means:

Any successful tinkering session

  • starts by defining a system using appropriate tooling
  • or by loading an existing system
  • proceeds with interactive exploration
  • ends by tidying up the sources, restarting Common Lisp, and fixing:
    • remaining errors
    • warnings
    • and reconsidering style warnings

Only then is the session considered complete.

Insight 2: Systems are not packages.

ASDF = Another System Definition Facility

  • ASDF builds systems.
  • Packages define namespaces.
  • "Libraries" are systems intended for reuse.

Different programming language ecosystems use different names for similar concepts. Common Lisp is no exception, but it is stricter in separating concerns than many other languages.

Core distinction

A system is a build-level unit.

More precisely:

A system is a directed graph of source files, dependencies, and build rules.

A system defines:

  • what files exist
  • in what order they must be compiled
  • what other systems are required
  • how the program is constructed

It is a build and loading abstraction, not a naming abstraction.

A package is a name-resolution unit.

A package defines:

  • which symbols exist in a namespace
  • how names are resolved (symbol → value/function/class)
  • how code is organised at read/evaluation time

It is a symbol management abstraction, not a build abstraction.

Community terminology ("library")

In Common Lisp practice:

An ASDF system is often informally called a "library" when it is intended for reuse.

However:

  • "library" is not a language or ASDF concept
  • it is a convention borrowed from other ecosystems (C, Python, etc.)

So:

All "libraries" are systems but not all systems are "libraries".

Examples:

See:

Key insights

  • Systems assemble code.
  • Packages organise names inside code.
  • "Libraries" are systems viewed from a reuse perspective.

Or even tighter:

  • ASDF builds programs.
  • Packages structure identifiers.
  • "Libraries" are reusable systems.

Cross-language analogy

Concept Common Lisp C/C++ Python Java JavaScript Rust Go
system system (ASDF) build target/CMake target/project project/distribution package Maven/Gradle module npm package/project crate module
package package namespace (std::) module/import namespace package ES module scope module namespace package namespace
"library" (informal Common Lisp term) ASDF system intended for reuse compiled library (.a, .so) pip-installable package reusable jar/module npm library crate reusable module

Important clarification

In Common Lisp specifically:

  • ASDF systems are not runtime objects
  • packages are runtime objects
  • "library" is purely descriptive terminology

So:

A system becomes a "library" only by intended use, not by structure.

Insight 3: Quicklisp is not a build system.

Quicklisp is best understood as a distribution and installation service, not as part of the language or the build model.

Quicklisp:

  • downloads source code
  • places it into a known directory structure
  • registers systems so ASDF can find them
  • manages versions and updates

Quicklisp does not:

  • define compilation order
  • define dependencies between files
  • define packages
  • define how code is built

That is entirely ASDF’s job.

Mental Model

  • Quicklisp answers: "Where does the code come from?"
  • ASDF answers: "How is the code built and loaded?"
  • Packages answer: "How are names resolved?"

What ql:quickload really does

When we write:

(ql:quickload :str)

Quicklisp

  1. Ensures that str and its dependencies exists locally (downloads them if needed)
  2. Registers its .asd file
  3. Delegates everything else to ASDF

After that point, Quicklisp steps aside completely.

Cross-language analogie

Concept Common Lisp C/C++ Python Java JavaScript Rust
Distribution tool Quicklisp pip npm Maven Central crates.io
Build system ASDF setuptools npm scripts Maven/Gradle Cargo

Quicklisp doesn't load the code, it only makes systems available. ASDF loads them.

Practical takeaway

If something:

  • fails during compilation → ASDF
  • fails to be found → Quicklisp
  • fails due to symbol conflicts → packages

Practical hint

If a system's source is for some reason not installed in quicklisp/quicklisp/software, or quicklisp/local-projects, of course, it can be located with:

ql:where-is-system

Example:

(ql:where-is-system :cl-charms)

Insight 4: The reader commits early.

Before Common Lisp compiles code, before ASDF builds systems, and before any evaluation takes place, there is a distinct and decisive phase:

Reading.

The Lisp reader transforms text into Lisp objects.

Once this happens, critical decisions are already fixed.

What the reader does

The reader:

  • turns characters into symbols, numbers, lists, strings, etc.
  • resolves package prefixes (foo:bar, foo::bar)
  • interns symbols into packages
  • expands reader macros (', #(...) etc.)

All of this happens before compilation and outside the control of ASDF.

What the reader does not do

The reader does not:

  • evaluate code
  • know whether a function exists
  • check arity or types
  • care whether a symbol will be bound later

It only decides what objects the code consists of.

Why this matters

Once code has been read:

  • symbol identity is fixed
  • package membership is fixed
  • macro expansion happens on already-read forms
  • compilation and evaluation can no longer "reinterpret" names

This is why many Lisp problems are reader problems, not compiler problems.

Typical reader-level errors

  • symbols read into the wrong package
  • missing or incorrect in-package
  • unintended symbol interning
  • macros failing before expansion
  • different behaviour depending on *package* at read time

These errors often appear before compilation even begins.

Key clarification

Packages are a reader concern, not a compiler concern.

By the time compile-file runs, the reader has already decided:

"These are the symbols you meant."

Mental model

Phase Question answered
Reader What objects does this text denote?
Compiler Is this code consistent and optimisable?
Evaluator What happens when it runs?

Final takeaway

  • Reading fixes identity.
  • Loading checks behaviour.
  • Compilation checks consistency.

Or, more compactly:

The reader commits early — and nothing later can undo that.

Insight 5: Function interfaces can be declared independently of their definitions.

Common Lisp allows function interfaces to be declared separately from their implementations using type declarations.

At toplevel, such declarations act as semantic promises to the compiler:

"This function exists (or will exist), and this is its calling contract."

This is similar in role to C or Pascal function headers, but more flexible and less rigid.

Declaring function existence

;; Declare that these functions exist
(declaim (ftype function
                validate-document
                validate-node
                ...))

This informs the compiler that these names refer to global functions, even if their definitions appear later.

Declaring function interfaces

(declaim
  (ftype (function (document) boolean) validate-document)
  (ftype (function (node symbol) null) validate-node)
  ...)

This specifies the argument and return types independently of the function bodies.

Implementations

(defun validate-document (document)
  ...)

(defun validate-node (node symbol)
  ...)

...

Why this matters

These declarations:

  • give the compiler freedom to optimise
  • make the intended structure explicit to human readers
  • suppress misleading or noisy style warnings
  • document intent without constraining implementation order

This is especially useful for intentional mutual recursion.

  • validate-document may call validate-node
  • validate-node may call the other validation functions
  • cycles are valid regardless of definition order.

The declarations make this structure explicit before compilation begins.

Local declarations

Inside function bodies, the same idea applies locally using declare:

(defun foo (x)
  (declare (type fixnum x)) ; Promise that X will always be of type fixnum (= integer)
  ...)

Here, declare communicates assumptions about the following code to the compiler, enabling better diagnostics and optimisation.

Caveat

These declarations are promises, not guarantees:

incorrect declarations may lead to undefined behaviour if safety is reduced.

Contrast: Common Lisp declarations vs. C forward declarations

C forward declarations exist because the language requires them.

Common Lisp declarations exist because the compiler benefits from them.

1. Purpose

C forward declarations

int validate_document(Document *doc);
  • exist primarily to satisfy the compiler
  • provide minimal information for type checking
  • are often mandatory for compilation

Common Lisp function declarations

  • inform and guide the compiler
  • are optional for correctness
  • primarily serve optimisation, documentation, and intent
  • In Lisp, code loads and runs without declarations.
  • In C, code often cannot compile without them.

2. Binding vs. Promise

C

  • a declaration introduces a required binding contract
  • linker must resolve exactly one definition
  • mismatches are compile/link errors

Common Lisp

  • a declaration is a non-binding promise
  • the function may be:
    • defined later
    • redefined interactively
    • replaced dynamically
  • violations affect optimisations or safety, not existence

C enforces structure statically.

Common Lisp optimises based on trust.

Insight 6: Declarations are compiler set screws

Declarations are not required to make code work -- they are required to make code fast, silent, and intentional.

Beyond function signatures, Common Lisp allows explicit control over compilation policy via optimize declarations.

These declarations do not change semantics -- they change how much the compiler is allowed to assume.

Global (toplevel) optimisation policy

Using declaim, you can set compilation policy of a file or system:

(declaim
  (optimize
    (speed 3)  ; fastest runtime performance  (runtime cost)
    (safety 0) ; remove runtime safety checks (runtime cost)
    (debug 0)  ; minimal debug information    (human-reasoning cost)
    (compilation-speed 0))) ; allow compiler to spend more effort on optimization
                           ;; (compile-time resource trade-off)

This tells the compiler:

"Trust the code completely. Optimise aggressively. Remove checks"

Effects may include:

  • elimination of runtime type checks
  • unsafe but faster arithmetic
  • aggressive inlining
  • potential undefined behaviour if assumptions are violated

This is comparable to -O3 -DNDEBUG in C, but explicitly controlled in the source.

Local optimisation policy

Using declare, the same controls can be applied lexically:

(defun fast-sum (vector)
  (declare (optimize (speed 3) (safety 0)))
  (loop for x across vector sum x))

Here:

  • only fast-sum is compiled under unsafe optimisation
  • the rest of the system remains safe and debuggable

Combining type and optimisation declarations

The strongest effect arises when type and optimisation cooperate:

(defun fast-add (a b)
  (declare
    (type fixnum a b)
    (optimize (speed 3) (safety 0)))
  (+ a b))

Now the compiler can:

  • assume machine-level integers
  • remove overflow checks
  • generate highly optimised code

If assumptions are violated, behaviour becomes undefined -- by design.

Conclusion

Declarations are promises that the programmers make to the compiler.

  • ftype → how functions are called
  • type → what data is assumed to be
  • optimize → how much the compiler may trust you

The compiler responds by:

  • checking less
  • warning less
  • optimising more

Common Lisp exposes a “full cost model of execution” explicitly.

Runtime performance, safety, debuggability, and compilation effort are first-class, adjustable dimensions of the language and its compiler.

Insight 6.5: Optimisation policy is not the same as compiler capability.

In Common Lisp, especially in implementations like SBCL, there is an important distinction between:

  • what the programmer asks for, and
  • what the compiler is capable of doing anyway

Optimisation policy (programmer-facing)

Declarations such as:

(declaim (optimize (speed 3) (safety 0)))

define a policy contract:

  • how much runtime safety may be removed
  • how aggressively the compiler may trade checks for speed
  • how much debugging information should be preserved

This is a guidance layer, not a switch for compiler intelligence.

Compiler capability (implementation-facing)

Independently of policy, SBCL always performs:

  • flow analysis
  • type inference
  • control-flow simplification
  • inlining and function specialization
  • intermediate representation (IR) transformations

These mechanisms are always active, even at safe settings such as:

(optimize (safety 3) (speed 1))

Key Insight

Lower safety does not make SBCL "more optimizing". It only allows the optimiser to remove more runtime checks.

The compiler’s ability to optimise code is largely independent of the optimization policy.

Practical consequence

This leads to a counterintuitive but important fact:

SBCL can generate highly optimised machine code even with high safety settings.

What changes between safety levels is mainly:

  • whether runtime type checks remain
  • whether certain errors are detected dynamically
  • how much trust is placed in type declarations

not whether the compiler performs analysis at all.

The compiler is always smart — policy only determines how free it is to act.

Insight 6.6: Failures can be localised

An assertion failure always indicts earlier phases, never the phase in which it appears.

This is about epistemology of systems:

  • what the system knows
  • where it is allowed to know it
  • how knowledge is validated

Assertions let the interpreter say "this cannot be wrong here," making errors local, structural, and honest.

So in Common Lisp:

(assert condition &optional datum &rest arguments)

means:

"I claim this must be true here. If it isn't, something upstream violated the contract."

It is not:

  • error handling
  • recovery
  • user-facing diagnostics

It is:

  • internal truth checking

Readers and parsers are notorious for producing plausible but wrong structures.

assert prevents:

  • silent structural drift
  • accidental phase bleed
  • malformed chunks that look OK but aren’t

Therefore it is very helpful during development.

Assertions often encode sentences, not conditions — they document intent.

Examples:

(defun resolve-infix (chunks)
  (dolist (c chunks)
    (assert (not (eq (chunk-kind c) :call))
            () "Infix resolver must not see :CALL chunks")))


;; Phase invariant:
(assert (every #'chunk-p chunks)
        () "Resolver must only receive CHUNK objects")

Caveat

Assertions should not attempt recovery; once triggered, execution should stop.

That keeps them from turning into ad-hoc control flow.

Insight 7: Common Lisp is about choosing where meaning lives

Lisp is often presented as a language centered on lists and functional composition. In practice, especially when building systems such as cLogos, it becomes clear that Common Lisp's central strength is something more general:

Common Lisp allows the programmer to decide where meaning resides: in structure, in state, or in identity.

Rather than enforcing a single paradigm, it provides multiple complementary models:

  • explicit state (mutation, assignment, control flow)
  • explicit identity (objects, CLOS instances, structs)
  • uniform structure (lists, trees, S-expressions)
  • transformational style (higher-order functions, recursion)

This makes it possible to design systems where different concerns are represented in different ways, rather than forcing everything into a single abstraction style.

Design consequence

In a system like cLogos, this leads to a practical distinction:

  • functional purity is used where transformations dominate (e.g. tree rewriting, chunk processing)
  • stateful design is used where temporal structure matters (e.g. parser stacks, incremental construction)

State is not an impurity -- it is a representation of time in the model.

Three guiding principles

  1. Representation matters more than elegance
    • The key question is not whether a transformation is "pure", but whether the representation preserves the distinctions needed later.
  2. State encodes temporal structure
    • A parser stack, for example, is not incidental mutation, but a record of evolving syntactic commitments.
  3. Choice of model is the real abstraction
    • The central question is not "functional or imperative", but:

      Where should meaning live so that the system remains understandable?

Example: structured vs unstructured representation

A stack frame defined as a structure:

(defstruct frame
  (kind nil :type symbol)
  (start-pos 0 :type integer)
  (chunks nil :type (or null list)))

(push (make-frame :kind kind
                  :start-pos start-pos
                  :chunks nil)
      stack)

encodes meaning explicitly:

  • fields are named
  • access is structured
  • interpretation is distributed across accessors

Whereas a list-based representation:

(push (list kind pos chunks) 
      stack)

is more flexible, but:

  • relies on positional interpretation
  • provides no enforced field semantics
  • increases risk of accidental misuse

Conclusion

Common Lisp does not enforce a single model of meaning.

Instead, it provides a palette of representational strategies, and places responsibility on the programmer to decide:

Where should meaning live: in structure, in state, or in identity?