Inclusion Methods - r3n/rebol-wiki GitHub Wiki

This is a discussion area for converging on what we want to include in R3 for inclusion methods.

Table of Contents Goal General Needs Survey of Methods DO How it satisfies the needs Tradeoffs LOAD How it satisfies the needs Tradeoffs Modules: IMPORT function How it satisfies the needs Tradeoffs Modules: IMPORTS and EXPORTS headers Tradeoffs PRebol How it satisfies the needs Tradeoffs PRebol (INCLUDE dialect) keywords Include How it satisfies the needs Tradeoffs

Our goal is to finalize what we need in Rebol to support inclusion of code components both at run-time, but also at application build time (for apps that use that method.)

Developers can:

• Edit sections below, or add new ones as needed. Just make it clear what you are proposing.
• Discuss this on R3 chat.
• For special notes that don't belong on this page, but we don't want to lose them in chat, add them via discussion link above.

Here are the primary requirements:

1. Enable programs to include (execute) other functions and code.
2. Enable programs to include (load into memory) other data, such as databases, media (images, sounds, etc.).
3. The ability to build distributable programs that include the above within their distribution unit (as a .r file or encapped executable).
4. Testing support - the ability to run a distributable program during its development cycle.
5. To support PITL encapsulating parts of the program.
6. Comfort - use a search path.
7. Comments - an ability to either keep or strip comments from the built application.
8. Standardization - provide a standard, not forcing the usage of ad hoc methods.

Standardization is a key element. Without it, there is no easy way to create public areas where code can be easily shared and used by others. It also means we'll each continue to build our own solutions that fill very common needs. E.g., a MAKE system. Finally, if we ever want to move towards true PitL, or have an IDE, we will have to address the need for some kind of project definition. Without standards for how resources are included, and systems are built and deployed, we can't do that.

It is common to use DO to include other code files in a program.

do %file1.r
do %file2.r
if something [do %file3.r]

1. Can execute other functions and code.
2. No direct support for data inclusion, data have to be wrapped is some way (ad hoc method).
3. No direct support for distributable unit building, building scripts have to exist (ad hoc method).
4. No direct support for distributable unit testing, two-step process is necessary (ad hoc method):
   1. Build the application.
   2. Test-run it.
5. No direct encapsulation support, ad hoc methods are available.
6. No direct search path support.
7. No flexible comment support.

The advantages are:

• Very simple. Easy to debug.
• Evaluates code. It can do anything, including loading other scripts, setting up environments, defining functions, etc.

The disadvantages are:

• It happens at run-time - so you must have access to all the listed files.
• It is evaluated, so must be in code format with a header. Therefore, if files contain data, they must be wrapped and assigned to a variable (or returned as a result).
• Uncomfortable - no automatic file search using a search path
• Incompatible with PRebol (ignores PRebol directives)

The LOAD function is often used instead of DO. For example, to load data files.

data: load %data.r

It is also used often to build distributions, using this simple method:

files: [
    %file1.r
    %file2.r
    ...
]
code: []
foreach file files [append code load file]
save %program.r code

It turns out, this is how the programs that run most RT tools and websites are built.

Is it scalable (PITL)? Yes, the primary Rebol releases are all built using this approach, with a few minor variations.

1. Does not execute code.
2. It can be only a part of the solution, since it does not include anything into anything. Does not support all mentioned part types.
3. Does not actually build, special code (script) is necessary.
4. Does not execute and therefore does not test.
5. Does not encapsulate.
6. Does not use a search path.
7. Does not support on-demand strippable comments.

The advantages are:

• Very simple. Easy to debug.
• Builds a single distro file as a result.

The disadvantages are:

• No control beyond that of including the file itself.
• Doesn't provide any standards. Every scenario is ad hoc.

The Rebol 3.0 method of dynamically including modules in a program at runtime.

Summary of usage:

import 'module    ; converts to .r and will use search path
import %module.r  ; will use search path
import http://www.rebol.com/module.r

Imported modules are fully encapsulated when loaded. Exported functions are controlled via an export list (see below).

The search path for finding modules can include local directories as well as HTTP URLs. In addition, version requirements can be specified.

1. Executes code.
2. Does not support fine-grained inclusion.
3. Does not build distributable applications.
4. Does not test distributable applications.
5. Encapsulates.
6. Uses search path.
7. Does not support on-demand strippable comments.

Advantages:

• A real module system for PITL.
• Proper encapsulation.
• More powerful than LOAD or DO (e.g. use of search paths).
• Supports global options (e.g. search paths and which modules to include by default).
• Supports module options specified at runtime at point of call, overriding what the module specifies.
• Can import a module by reference, rather than binding its exported words to your context.
• The import specification can be a full dialect.
• Most script programmers are familiar with this method (PERL, Python, Ruby, etc.)

Disadvantages:

• More complex and more difficult to debug.
• Dynamically imported words have to be bound at runtime, which can only be done safely under specific circumstances, or manually with BIND or IN.
• Dynamically exported words can't be determined without running the code in the module, which might not be safe. This makes the analysis and sandboxing tools difficult to make.
• The limits of dynamic import/export would be confusing to people familiar with modular languages.
• Not usable for building distributions using INCLUDE method.
• Hence, no fine-grained inclusion control (like PRebol, below).

The Rebol 3.0 method of including modules in a program at LOAD time.

Summary of usage:

Rebol [type: module imports: ['module]]   ; converts to .r and will use search path
Rebol [type: module imports: [%module.r]] ; will use search path
Rebol [type: module imports: [http://www.rebol.com/module.r]]
Rebol [type: module exports: [myfunc myobject]] ; Exported words.
Rebol [type: module exports: all] ; Export all locally set words.

Imported modules are fully encapsulated when loaded. Exported words are controlled via an export list.

The search path for finding modules can include local directories as well as HTTP URLs. In addition, version requirements can be specified.

Advantages:

• A real module system for PITL.
• Proper encapsulation - all local words can be captured.
• More powerful than LOAD or DO (e.g. use of search paths).
• LOAD and DO can handle this for you, automatically (and already do).
• Supports global options (e.g. search paths and which modules to include by default).
• Supports module options specified in the module's own header.
• The import and export headers can be a full dialect.
• Can optionally include a module by name, rather than binding its exported words to your context.
• Binding can be resolved at LOAD time, when it is safe.
• Headers can be used by distribution builders like PRebol.
• Headers can be used by analysis code without tracing.
• Can be used to sandbox Rebol code for safe execution.
• Can be backported to Rebol 2 (and mostly has been already by Gabriele).
• Most modular code programmers are familiar with this method.

Disadvantages:

• More complex. Unknown global options can make it difficult to debug.
• Dynamic management of imports and exports would need to be done by a distribution builder like PRebol, or at runtime with the IMPORT function.
• The limits of static import/export would be confusing to new programmers unfamiliar with PITL.

The Rebol SDK introduced a method called PRebol (Rebol preprocessing).

This method uses the ISSUE! datatype (which is rarely used in code itself) as a type of markup.

#include %file.r
#include %file.r

1. Does not execute code.
2. Supports fine-grained inclusion.
3. Builds distributable applications.
4. Does not execute and therefore testing has to be a two-step process:
   1. Build application.
   2. Execute the built application.
5. The approach described in INCLUDE_documentation#Context_encapsulation can be used.
6. Does not use a search path.
7. Does not support on-demand strippable comments.

Advantages:

• Finer grained inclusion than LOAD or DO.
• Greater control over what is and is not included because it can evaluate expressions at processing time.
• Familiar in concept (and even in appearance) to C/C++ and other language users (but not identical in semantics.)
• Can be extended to support modules, if imports and exports are specified in headers.

Disadvantages:

• More complex and more difficult to debug.
• Not very Rebolish. Introduces a new meta-syntax, and although it is a valid Rebol dialect, it's really outside the original symbolic intentions of the language (because it uses #issues which are series, not symbolic words.)

If the usage of #issue values as dialect keywords is undesirable, there is still a variant to use words. They just need to be somewhat "outstanding" to make them immediately visible and to suppress the probability, that the keywords will be used for other, incompatible purposes. For example, the INCLUDE function already took the comment word (since it is highly unlikely that it can cause any undesirable interference) as its keyword to support automatic stripping of comments. The following alternatives look like being available:

!include
&include
*include
+include
-include
.include
=include
?include
^include
_include
|include
`include
~include
.include.
*include*
**include

etc.

See the INCLUDE documentation.

The include method is a significant extension of PRebol created by Ladislav Mecir.

It adds new functions to PRebol (in a compatible manner), but the greatest strength/enhancement over PRebol is its compatibility with DO, supporting testing and more fine-grained inclusion than supported by PRebol (a user can specify, that a part has to be included multiple times, or just once).

It is routinely used as a standard development tool in several environments replacing PRebol as a more versatile tool. It is standard-based:

• PRebol standard
• DO standard
• LOAD standard - supporting prefixed scripts, embedded scripts, Rebol-loadable data. Having a /only refinement to yield a built block as a result similarly as LOAD
• like PRebol it supports general data too

Ladislav and others have suggested that this be considered, and perhaps even replace PRebol itself. - yes, I propose to use it in place of PRebol, since it adheres to the PRebol standard and is simpler and more versatile than PRebol.

1. Executes code.
2. Includes data.
3. Builds distributable programs.
4. Tests distributable programs.
5. The approach described in INCLUDE documentation#Context_encapsulation can be used.
6. Uses a search path, which may contain directories as well as URLs.
7. Supports on-demand strippable comments.
8. Supports user-defined commands.

Advantages:

• Fine grained inclusion (complete control for code and data) - i.e. complete support for points #1 and #2
• Direct support for #3 (evaluating code like DO) is a great advantage over PRebol.
• Compatible with PRebol - supports all its directives and functionality.
• Simpler to use than PRebol.

Disadvantages:

• More complex, harder to debug than the DO or LOAD methods. (However, all existing users have been comfortable with it.)
• (Minor) Uses the word 'include as a function name, precluding its use as a set function. (Easily renamed.)