Memory Guide - Windower/packages GitHub Wiki

This guide will describe how to read from and write to memory both with the core low level ffi and scanner libraries as well as with Windower's own struct and memory helper libraries. It assumes a basic understanding of native data layouts and data types.

In this guide the word "native" is used to represent any direct memory data that is not managed by the Lua runtime, whether created by the user or Windower or FFXI itself, as well as native functions that exist within FFXI.

One thing to keep in mind when working with memory is that crashes can be quite common, especially if you're unsure about what you're poking it. Lua's regular defense mechanisms do not catch these types of errors and FFXI is likely to crash if you dereference memory you are not supposed to access.

LuaJIT provides an FFI library which is the core of all the memory interop. Its full documentation can be found here and on subsequent pages and will not be elaborated much further, although some commonly used functions will be explained again.

For this guide, the important thing to know is that LuaJIT extends regular Lua by a new type called cdata. It is an opaque type meant to represent native data, such as a block of memory, a native struct or a pointer. The FFI library is the interface to handle cdata objects in Lua code. Within Lua, cdata is just like Lua's other data types (string, boolean, number, table, etc.) with the exception that every cdata object itself has its own native type, called the ctype. Those will also be elaborated upon.

To create native objects there is one main mechanism in the FFI library, and it is ffi.new. It takes a C type and returns a cdata object of that type. Some examples:

local arr = ffi.new('int[10]')      -- Creates an array of ten integers

print(arr[3])                       -- Output: 0
arr[3] = 5
print(arr[3])                       -- Output: 5

local struct = ffi.new[[struct {
    int i;
    bool b;
    float f;
}]]                                 -- Creates a struct with the specified three members

print(struct.b)                     -- Output: false
struct.b = true
print(struct.b)                     -- Output: true

local ptr = ffi.new('int*')         -- Creates a new pointer object
                                    -- This is of very little use, since the pointer contents can be adjusted,
                                    -- but the pointer itself cannot, and LuaJIT zero-initialized all created
                                    -- values, which means 0 for numbers, false for booleans.
                                    -- So the result here is a null pointer, and dereferencing it will result
                                    -- in a FFXI crash

print(ptr[0])                       -- Crashes

Here int, bool and float are all known C types. For a full reference of available types consult the FFI docs linked above. You can create and name your own more complex structs based on those basic types using ffi.cdef:

ffi.cdef[[
    typedef struct {
        int i;
        bool b;
        float f;
    } sample_type;
]]

local struct = ffi.new('sample_type')

If you create a lot of objects this way, you can create a predefined ctype first, then pass that into ffi.new, instead of the string:

local array_type = ffi.typeof('char[0x10]')

local arr1 = ffi.new(array_type)
local arr2 = ffi.new(array_type)
local arr3 = ffi.new(array_type)

Since this is a common thing to do there is a shortcut for that, since every ctype is callable:

local array_type = ffi.typeof('char[0x10]')

local arr1 = array_type()
local arr2 = array_type()
local arr3 = array_type()

This increases performance, since it parses the C definition string only once.

In addition to creating your own native objects you can also use these types to reference in-game data. As an example, let's consider the alliance structure. It is an array of 18 structures, one for each alliance member. The individual struct looks like this:

ffi.cdef[[
    typedef struct {
        void* alliance_info;
        char _unknown_04[2];
        char name[0x10];
        char _unknown_16[2];
        uint32_t id;
        uint16_t index;
        char _unknown_1E[6];
        uint32_t hp;
        uint32_t mp;
        uint32_t tp;
        uint8_t hp_percent;
        uint8_t mp_percent;
        uint16_t zone_id;
        uint16_t _duplicate_zone_id;
        char _unknown_36[2];
        uint32_t flags;
        char _unknown_3C[38];
        uint32_t _duplicate_id;
        uint8_t _duplicate_hp_percent;
        uint8_t _duplicate_mp_percent;
        bool active;
        char _unknown_7B[1];
    } alliance_member_type;
]]

Now the C runtime knows the name alliance_member and what struct it represents. To get an array of 18 such structs we would simply write:

ffi.cdef('typedef alliance_member_type alliance_type[18];')

Now we can use the name alliance_type to refer to the struct containing the data for the entire alliance, so we know what the alliance data looks like in memory. To work with it and manipulate it we need the location in memory, called the address. This is not often easy to get, but Windower provides a signature scanner in the scanner module. We will see how to use it later, for this example we will assume that we already have the address in a variable named address.

To use this we first need to convert the address we have to a cdata object that refers to that address, but looks like the alliance_type structure. We can get this simply by casting the address with the ffi.cast function. It takes a ctype as the first parameter and the object to cast as the second. In our example that would look like this:

local alliance = ffi.cast('alliance_type*', address)

Now the alliance variable contains a cdata object referring to the memory location provided in address, and usable in a manner similar to a table defined with the fields as declared in the struct definition above:

local me = alliance[0]          -- The first array member is always the current player
                                -- Note that, unlike Lua tables, cdata arrays go from 0 to n - 1.

print(me.hp_percent)

This will print out the current HP percentage of the first member in the alliance array, i.e. the currently logged in player. Similarly this value can be adjusted:

for i = 0, 17 do
    alliance[i].hp_percent = 100
end

This will make every alliance member display a full HP bar. Obviously it won't mean they will actually have full HP, but that is what the client will think. And unless this loop is applied every time the values change it will be overwritten once new packets come in.

The raw FFI library and the underlying mechanism in LuaJIT comes with some limitations, especially when coming from a Lua background, and certain things that Lua users take for granted are much harder to do in the native world, and cdata reflect those difficulties.

As an example, it is not possible to get the length of a cdata array. So once it is created you need to be aware of its length, or store it elsewhere to not overflow it. And if it does overflow, at best you will access garbage values, but possibly also overwrite other important values or even crash if you try to access invalid memory.

Another example that is particularly relevant to Lua users is that the strings you build with char arrays come with a number of limitations. For one, they are not strings and cannot be used that way. You always have to convert them from a char array (or pointer) with ffi.string. If we go by the above example, if we wanted to get an alliance member's name we can't just do print(alliance[i].name), as it will only print out the cdata identifier. Instead we always have to do print(ffi.string(alliance[i].name)). This is made even more dangerous because ffi.string uses zero-termination as the indicator for string length, which FFXI does not always use. Alternatively we could pass the string length to the function, but since we cannot get the length of an array, as mentioned above, this also proves difficult.

There are other limitations, such as the inability to iterate both structs and arrays, which can prove difficult to work with.

To work around those limitations we provide the struct library, which is meant to be used when working with native data, especially when interfacing with FFXI data.

The struct lib was made specifically to work with native data and bridge the gap between Lua's convenience and C's low level mechanisms. Its full documentation can be found here, but this page will give a few examples of how to use it and how it can help avoid some of the pitfalls of the raw FFI mechanisms.

The basic use case is to define a type and then use struct.new to create a new native cdata object of that type. Types themselves have their own functions which return a type description called an ftype. Some examples include:

struct.bool             -- Represents a boolean value
struct.uint32           -- Represents a 4 byte unsigned integer
struct.double           -- Represents an 8 byte floating point number
struct.string(n)        -- Represents a string of size n (in bytes)
struct.array(ftype, n)  -- Represents an array of the provided type with a count of n
struct.struct(fields)   -- Represents a struct with the provided fields

There are many more types, as outlined in the struct documentation. Of particular interest for most cases is the struct.struct function, which is commonly used as a root type for defining more complex structures. An example would look like this:

local ftype = struct.struct({
    foo = {struct.int32},
    bar = {struct.bool},
    baz = {struct.double},
    moo = {struct.string(10)},
})

The ftype returned from this function is used to describe the native structure. To get an actual object of that type we can now pass it into struct.new and get a proper cdata object:

local data = struct.new(ftype)

for key, value in pairs(data) do
    print(key, value)
    -- Sample output:
    -- foo     0
    -- bar     false
    -- baz     0
    -- moo
end

data.foo = 1.23
data.bar = true
data.baz = 1.23
data.moo = 'This is a longer string'

for key, value in pairs(data) do
    print(key, value)
    -- Sample output:
    -- foo     1
    -- bar     true
    -- baz     1.23
    -- moo     This is a
end

We can see native data behavior, such as the integer value being truncated from the provided 1.23 to 1, as well as the string being truncated to fit into the provided array. Also the data object here can be iterated, unlike regular cdata objects.

There are many further convenience features of the struct lib over regular cdata objects, such as array length checks, binary data conversion, easier bit packing definitions, packed strings, etc., which are referenced in the documentation, but the mechanism is the same as shown here. Structs and arrays can be nested, so it is possible to create a struct as a member of a struct, as well as an array of structs or structs containing arrays, similar to what you would expect from regular C syntax.

There are essentially two ways to work with existing memory instead of creating your own objects. Either you copy the memory into a newly created object, or you reference it so that the cdata object itself reflects live memory.

To copy you can simply use ffi.copy, since the result of struct.new is just a cdata object. Assuming the memory you want to use is at the address saved in a variable called address, the code would look like this:

local data = struct.new(ftype)

ffi.copy(data, address, ftype.size)

Here we need to provide an explicit size so the ffi.copy functions know how much memory it needs to copy over. That information is stored in the ftype. After that function call the data variable will contain all the same memory as at that address. However, since it is a copy, changing the values will by itself not change the data in memory, you need to reverse copy it again:

local data = struct.new(ftype)

ffi.copy(data, address, ftype.size)
data.foo = data.foo + 1
ffi.copy(address, data, ftype.size)

If that is your primary reason for using native data structures, to modify memory, you can instead reference that address. The following code has exactly the same effect as the above, without the copying:

local data = struct.from_ptr(ftype, address)
data.foo = data.foo + 1

In addition it is also much faster, because two (potentially large) copies could be avoided. However, it risks corrupting game memory easier if you treat that object lightly and adjust it at will, or provide an end user interface to modify its data.

However, both the copy and reference versions explained here still have one flaw which has been inherited from the ftype as we defined above. Since Lua tables are not ordered the struct layout does not need to be in the order specified. To guarantee that, we need to specify the field positions exactly by providing the byte offsets from the beginning of the struct:

local ftype = struct.struct({
    foo = {0x00, struct.int32},
    bar = {0x04, struct.bool},
    baz = {0x08, struct.double},
    moo = {0x10, struct.string(10)},
})

While this may sound like more work, having explicitly defined locations not only helps with debugging when offsets shift, but since the struct.struct function also fills in missing values automatically we can ignore unknown values. As an example, to define the same alliance member struct as we did with the regular ffi.typeof implementation above, it would instead look like this:

local ftype = struct.struct({
    alliance_info           = {0x00, struct.ptr()},
    name                    = {0x06, struct.string(0x10)},
    id                      = {0x18, struct.uint32},
    index                   = {0x1C, struct.uint16},
    hp                      = {0x24, struct.uint32},
    mp                      = {0x28, struct.uint32},
    tp                      = {0x2C, struct.uint32},
    hp_percent              = {0x30, struct.uint8},
    mp_percent              = {0x31, struct.uint8},
    zone_id                 = {0x32, struct.uint16},
    _zone_id2               = {0x34, struct.uint16},
    flags                   = {0x38, struct.uint32},
    _id2                    = {0x74, struct.uint32},
    _hp_percent2            = {0x78, struct.uint8},
    _mp_percent2            = {0x79, struct.uint8},
    active                  = {0x7A, struct.bool},
})

And this now comes with all the conveniences the struct lib provides.

While it can guess unknown fields between the explicitly declared fields based on their size and position, it cannot know unknown fields at the end. These may be necessary padding to adjust the size of the struct. This can be accomplished by provided an optional first value to the struct.struct function which contains additional metadata for the struct:

local ftype = struct.struct({size = 0x7C}, {
    alliance_info           = {0x00, struct.ptr()},
    name                    = {0x06, struct.string(0x10)},
    id                      = {0x18, struct.uint32},
    index                   = {0x1C, struct.uint16},
    hp                      = {0x24, struct.uint32},
    mp                      = {0x28, struct.uint32},
    tp                      = {0x2C, struct.uint32},
    hp_percent              = {0x30, struct.uint8},
    mp_percent              = {0x31, struct.uint8},
    zone_id                 = {0x32, struct.uint16},
    _zone_id2               = {0x34, struct.uint16},
    flags                   = {0x38, struct.uint32},
    _id2                    = {0x74, struct.uint32},
    _hp_percent2            = {0x78, struct.uint8},
    _mp_percent2            = {0x79, struct.uint8},
    active                  = {0x7A, struct.bool},
})

The scanner lib is there to find addresses. Some applications and addons use memory addresses (also called memlocs) which usually have to be updated between FFXI updates. A more stable alternative is to use function signatures. Since obtaining signatures is not an easy thing to explain this guide will omit that. If you need one it may be best to jump onto our Discord server to contact us to help you find one, if it exists.

If you do have the signature though, you can use the scanner library to find the address:

local scanner = require('core.scanner')

local ptr = scanner.scan(signature)

The returned ptr variable points to the address of the struct relevant to that signature and can be used in all of the scenarios mentioned above.

The signature parser takes a hex string of the byte values and returns the 4 byte value at the end of the signature as a void* by default. It supports ?? as a byte wildcard. As an example:

scanner.scan('ABCD??EF')

This would search executable memory for the 0xAB 0xCD x 0xEF byte sequence, where x could be any byte, and if found, take the 4 bytes right after that sequence and return them as a void pointer.

If you want to use 4 bytes in another point of the sequence you can place a * at that point in the sequence. For example, if you wanted a 4 byte sequence in the middle of a sequence you could do this:

scanner.scan('AB*????????CDEF')

This would look for a pointer that covers the wild carded area of the signature. By default these two are equivalent:

scanner.scan('ABCDEF')
scanner.scan('ABCDEF*')

If you want to get the address of the scan result, and not the value after the asterisk, you can use & instead of *:

scanner.scan('&ABCDEF')

This will return the address of the first byte in that sequence.

And finally, it supports searching other modules by specifying them as a second parameter. By default it only searches FFXiMain.dll:

scanner.scan('ABCDEF') -- equivalent to
scanner.scan('ABCDEF', 'FFXiMain.dll')

scanner.scan('ABCDEF', 'polcore.dll')

Like the struct library is meant to abstract away some difficulties of the ffi library, so the memory library is meant to abstract away some difficulties of the scanner library.

It automates linking structs to function signatures by defining them into its own types.lua file. An example entry looks like this:

types.tell_history = struct({signature = '8B0D????????85C9740F8B15'}, {
    recipient_count         = {0x004, uint16},
    recipients              = {0x008, pc_name[8]}, -- last 8 /tell recipients
    _dupe_recipient_count   = {0x088, uint16},
    _dupe_recipients        = {0x08C, pc_name[8]},
    chatlog_open            = {0x10D, bool},
    chatmode_tell_target    = {0x10E, pc_name}, -- Only set when /chatmode tell
    senders                 = {0x11E, pc_name[8]},
})

[Note: This code uses some localized types and functions, such as struct.struct, which is just called struct here, and pc_name which is an alias for struct.string(0x10) and so on.]

As this shows, the signature is specified as part of the optional first metadata argument to the struct.struct function. This is enough for the memory library to perform searches (in all standard FFXI modules) and include that type in the base table. Which means it can now be accessed like this:

local memory = require('memory')

print(memory.tell_history.recipients[3]) -- Will print the /tell recipient at the fourth place in your saved history

So a number of things happen here. First, the signature is searched for in several modules. Once found, the address is converted to a struct as defined in the table above. Then the recipients field is accessed, which is an array of strings. Then the third index (fourth entry) is retrieved, and the automatic char array to string conversion kicks in and it will display the respective recipient's name.

This is all that is needed to use the memory library in its basic form. It has a few additional features, such as defining new structs with their own signature and adding them. If you have the signature of something that is useful to you but you don't think it might be useful to the broader community you may not want to have it in the memory library itself, instead you can just add this to your own addon:

local memory = require('memory')
local struct = require('struct')

memory.new_struct = struct.struct({signature = 'ABCDEF'}, {
    foo = {0x00, struct.int32},
    bar = {0x04, struct.bool},
    baz = {0x08, struct.double},
    moo = {0x10, struct.string(10)},
})

print(memory.new_struct.moo) -- Will print the string at the address after that signature + 0x10

Now you can access it like any other struct defined in the type.lua file.

Occasionally function signatures will not lead to the required address directly, but sometimes you need to dereference the result further. This can be achieved by specifying an offsets field. Here is an example from the types.lua file for the target structure:

types.target = struct({signature = '53568BF18B480433DB3BCB75065E33C05B59C38B0D&', offsets = {0x18, 0x00}}, {
    window                  = {0x08, ptr()},
    name                    = {0x14, npc_name},
    entity                  = {0x48, ptr(entity)},
    id                      = {0x60, entity_id},
    hp_percent              = {0x64, uint8},
})

In this case we have a signature sig that ends with & and has two offset parameters (0x18 and 0x00). What this means is that first, executable memory is searched for sig. If that is found, it takes the address of the end of the signature (since a & was specified). Next, it adds 0x18 to it and takes the four byte value at that location. Next it adds 0x00 to it and takes the four byte value at that location. Since those were all specified offsets, this is the returned value.

The result is then the struct that the game uses to hold and display the target information.