Introduction

Memory management with zeptoforth is provided by the dictionary for each task, heap allocators, pool allocators, and temporary allocators. Ultimately all memory is alloted from the dictionary of some task, with the space for new tasks being alloted from the top of the main task's dictionary space. Words and constants compiled to RAM along with variables normally are alloted from the dictionary for the main task, even though in theory they could be alloted from the dictionary of any task. User variables are alloted from the base of the dictionaries for each task.

It should be noted that words that write to the space being compiled to such as constant, 2constant, cvariable, hvariable, variable, 2variable, buffer:, aligned-buffer:, create, <builds, value, 2value, begin-structure, end-structure, begin-module, end-module, end-module>, begin-class, end-class, begin-implement, and end-implement must not be used during a word definition or otherwise undefined behavior will result.

The Dictionary

The dictionary space of a given task in RAM may be used like a stack, with positive allotments being used to allot space and negative allotments being used to free space within a task's dictionary space. For this purpose exists the words with-allot and with-aligned-allot, both with the signature ( bytes xt -- ), which allot a space of size bytes from the dictionary space of the current task in RAM, places its address on the stack, executes xt, catching any exceptions, restores ram-here to its original address, and if an exception had been raised re-raises it; the difference between the two is that with-allot does not concern itself with alignment while with-aligned-allot ensures that the alloted block of memory it places in the stack is cell-aligned. It is highly recommended one use these words for temporarily alloting space on a task's dictionary space in RAM rather than attempting to do so manually. Note that when one is temporarily alloting space one must not compile any words, constants, variables, or buffers to the current task's dictionary space in RAM or otherwise undefined results may occur.

For an example of the use of with-aligned-allot take the following:

: test
  16 [:
    h.8 space
    16 [:
      h.8 space
      16 [:
        h.8 space
      ;] with-aligned-allot
    ;] with-aligned-allot
  ;] with-aligned-allot
  8 [:           
    h.8 space
    8 [:
      h.8 space
      8 [:
        h.8 space
      ;] with-aligned-allot
    ;] with-aligned-allot
  ;] with-aligned-allot
;

Afterwards, execute:

test 20011100 20011110 20011120 20011100 20011108 20011110  ok

Here we can see how the dictionary address put on the stack by with-aligned-allot behaves like an upward-growing stack, with the dictionary pointer increasing by the specified amount (along with alignment) for each call to with-aligned-allot and decreasing back to its original value afterwards.

Heaps

The heap allocator enables allocating, resizing, and freeing arbitrary multiples of a given block size (minus one cell to store the allocation size) within a heap. Any number of independent heaps may exist within the RAM of a system; e.g. the line editor has its own dedicated heap for storing its history. The heap allocator is in the heap module. It should be noted that the heaps are not task-safe, and if the user wishes to use a heap from multiple tasks they must protect it with a lock. Also note that the heap allocator is not deterministic in its time usage on allocating, resizing, or freeing so it should not be used within code with significant realtime considerations.

The heap allocator differs from the heap allocator in ANS Forth/Forth 2012 in that no single global heap is assumed and the user has to manually initialize any given heap they wish to use. The heap to use is passed into the words allocate, free, and resize manually. Also, instead of returning an error flag as does the heap allocator in ANS Forth/Forth 2012, an exception is raised on allocation failure instead.

Take the following example of heap allocator usage:

heap import  ok                                        
16 constant block-size  ok                             
1024 constant block-count  ok                          
block-size block-count heap-size constant my-heap-size  ok
my-heap-size buffer: my-heap  ok                       
block-size block-count my-heap init-heap  ok           
256 my-heap allocate dup h.8 constant my-data 20011104 ok
512 my-data my-heap resize dup h.8 constant my-new-data 20011104 ok
65536 my-new-data my-heap resize dup h.8 constant my-newer-data allocate failed
my-new-data my-heap free  ok
65536 my-heap allocate dup h.8 constant my-newest-data allocate failed

Here we allot (from the main task's dictionary) and initialize a heap my-heap with a block size of 16 bytes and a block count of 1024, i.e. 16384 bytes total excluding the heap's header and bitmap and the cell used at the start of each allocation to store the size of the allocation. Then we allocate a block of memory consisting of 256 bytes, excluding the length stored at the start of the allocation. Then we resize that block of memory to 512 bytes successfully. When we attempt to resize it again to 65536 bytes an exception is raised indicating we cannot allocate a block of memory of that size due to the maximum size available in the heap. Afterwards we free the block of memory. Finally, we attempt to allocate a new block of memory consisting of 65536 bytes, and an exception is raised indicating that we cannot allocate the block of memory due to insufficient available space in the heap.

Note that the size of any block of memory as allocated in memory is restricted to a multiple of the heap's block size minus one cell. Block sizes themselves are rounded up to the next cell if they are not a multiple of a cell, and have a minimum size of three cells (due to unused blocks requiring three cells for their internal management data).

Pools

The pool allocator a much lighter-weight allocation mechanism that is faster than the heap allocator and is deterministic; its primary downside compared to the heap allocator is that it may only allocate fixed sized blocks of memory at a time, so if one needs less memory that how much memory it is configured to allocate then memory will be wasted and one will not be able to allocate more memory than this fixed memory block size. Just like with the heap allocator, multiple memory pools may exist within a system, and one must pass the pool to allocate-pool, free-pool, and add-pool. Note that after a pool allocator is initialized memory must be manually added to the pool, as initializing a pool allocator only initializes its header without actually adding any space to allocate to it. It should be noted that pools are not concurrency-safe, so locks must be used if a pool is to be used from multiple tasks. (Note that this here reflects release 0.35.0; the pool module has been revamped in that release, and there was a significant bug in previous releases that prevented it from functioning properly.)

Take the following example:

pool import  ok
16 constant block-size  ok
64 constant buffer-size  ok
pool-size buffer: my-pool  ok
buffer-size buffer: my-buffer  ok
block-size my-pool init-pool  ok
my-buffer buffer-size my-pool add-pool  ok
my-pool allocate-pool dup h.8 constant block0 20011074 ok
my-pool allocate-pool dup h.8 constant block1 20011084 ok
my-pool allocate-pool dup h.8 constant block2 20011094 ok
my-pool allocate-pool dup h.8 constant block3 200110A4 ok
my-pool allocate-pool dup h.8 constant block4 allocate failed
block0 my-pool free-pool  ok
block1 my-pool free-pool  ok
block2 my-pool free-pool  ok
block3 my-pool free-pool  ok
my-pool allocate-pool dup h.8 constant block0 20011074 ok
my-pool allocate-pool dup h.8 constant block1 20011084 ok
my-pool allocate-pool dup h.8 constant block2 20011094 ok
my-pool allocate-pool dup h.8 constant block3 200110A4 ok
my-pool allocate-pool dup h.8 constant block4 allocate failed

Here we create a pool my-pool with a block size of 16 bytes and a buffer my-buffer with a size of 64 bytes. We then the buffer to the pool. We then successfully allocate four blocks from the pool, with an exception raised when we attempt to allocate a fifth block, indicating failure. Then we free all four blocks we had allocated. We repeat this process, and can see that we can allocate all the blocks we had freed again.

Temporary Allocators

Temporary allocators are mainly used for cases where one needs to transiently store data and one can ensure that less live data will exist than a total quantity. They are used for storing strings parsed at the REPL, where the size of the temporary allocator is greater than the maximum size of a single line of input on the REPL. This enables the user to use words such as s", c", s\", and c\" from the comfort of the REPL rather than having to create a non-transient word to contain such strings. Temporary allocators simply allocate space in the last-used space within their buffer, and when they reach the end of the buffer they wrap around, without any consideration of the data that may already be there. Like the other allocation constructions mentioned here, temporary allocators are not concurrency-safe, so they need to be protected with locks if they are to be used from multiple tasks.

An example of the use of temporary allocators is as follows:

temp import  ok
64 constant buffer-size  ok
buffer-size temp-size buffer: my-temp  ok
buffer-size my-temp init-temp  ok
16 my-temp allocate-temp h.8 2001102C ok
16 my-temp allocate-temp h.8 2001103C ok
16 my-temp allocate-temp h.8 2001104C ok
16 my-temp allocate-temp h.8 2001105C ok
16 my-temp allocate-temp h.8 2001102C ok

Here we allocate four blocks of memory of size 16 bytes with the temporary allocator out of a total allocator size of 64 bytes. When we allocate another 16 bytes it simply wraps around and starts allocating from the start of its buffer again. Consequently it may only be used for truly transient data where we can be certain that the data will no longer be live before its space is reallocated.