Data Modeling - SurrealTools/Documentation GitHub Wiki

Invalid column name in permission condition returns all data

https://github.com/surrealdb/surrealdb/issues/2757

Object based Id with types

DEFINE FIELD id ON something TYPE {location: string, date: datetime};

Rename a table

INSERT INTO type::table($new) (SELECT *, meta::id(id) AS id FROM type::table($old));
REMOVE TABLE type::table($old);

Make an assertion that a field points to an existing record

DEFINE TABLE permission_set SCHEMAFULL;
DEFINE FIELD permissions ON permission_set TYPE array<record>;
DEFINE FIELD permissions.* ON permission_set TYPE record<permission> ASSERT !type::is::none($input.id);
OR
DEFINE FIELD permissions.* ON permission_set TYPE record<permission> ASSERT $value = (SELECT id FROM $input)[0]['id'];

DEFINE FIELD (New Syntax)

---Old Systax
DEFINE FIELD author ON post TYPE record<user> ASSERT $value != NONE;

---New Syntax
DEFINE FIELD author ON post TYPE record<user>;
DEFINE FIELD author ON post TYPE option<record<user>>;

Data Types

When storing data, the smallest possible size is always used. So it will never take 8 bytes to store an i8, but will rather use a single byte, regardless of whether we are using i128 or i64 or anything else.

    Any,
	Array,
    Set,
	Bool,
	Bytes,
	Datetime,
	Decimal,
	Duration,
	Float,
	Int,
	Number,
	Object,
	String,
	Record(Vec<Table>),
	Geometry(Vec<String>),

Types are now defined using . For example...

DEFINE FIELD employees ON TABLE company TYPE array<record<person>>;
DEFINE FIELD age ON person TYPE int;
DEFINE FIELD enabled ON person TYPE bool | int;
DEFINE FIELD name ON person TYPE string;
DEFINE FIELD scores ON person TYPE set<float, 5>;
UPDATE person:test SET age = NONE, enabled = NONE, name = NONE, scored = [1,1,2,2,3,3,4,4,5,5];
UPDATE person:test SET age = '18', enabled = NONE, name = NONE, scored = [1,1,2,2,3,3,4,4,5,5];
UPDATE person:test SET age = '18', enabled = true, name = NONE, scored = [1,1,2,2,3,3,4,4,5,5];
UPDATE person:test SET age = '18', enabled = true, name = 'Tobie Morgan Hitchcock', scores = [1,1,2,2,3,3,4,4,5,5];

and for inline typing...

UPDATE person:test SET age = <int> NONE;
UPDATE person:test SET age = <int> '18';
UPDATE person:test SET enabled = <bool | int> NONE;
UPDATE person:test SET enabled = <bool | int> true;
UPDATE person:test SET name = <string> 'Tobie Morgan Hitchcock';
UPDATE person:test SET scores = <set<float>> [1,1,2,2,3,3,4,4,5,5];
UPDATE person:test SET scores = <array<float>> [1,1,2,2,3,3,4,4,5,5];
UPDATE person:test SET scores = <set<float, 5>> [1,1,2,2,3,3,4,4,5,5];
UPDATE person:test SET scores = <array<float, 5>> [1,1,2,2,3,3,4,4,5,5];

SCHEMAFULL mode

When in SCHEMAFULL mode you need to define EVERY field...

DEFINE TABLE person SCHEMAFULL;
DEFINE FIELD info ON person TYPE object;
DEFINE FIELD info.hello ON person TYPE number;

CREATE person:me CONTENT {
    info: {
        hello: 123
    }
};

FLEXIBLE fields on SCHEMAFULL tables

This allows you to have a field which can accept any nested data (perhaps unknown data such as log information):

DEFINE TABLE person SCHEMAFULL;
DEFINE FIELD info ON person TYPE object FLEXIBLE;

CREATE person:me CONTENT {
    info: {
        hello: 123
    }
};

IDs in Surreal

SurrealDB Record IDs support:

  • UUIDv7 as default UUID type
  • Integers: person:1, person:1000
  • Strings: person:test, person: 5349b4ddd2781d08c09890f3
  • Complex strings: person:⟨fb2cd733-eec4-4354-b070-3aba18c2a67a⟩ (can also use backticks
  • Arrays: person:['London', '2022-09-01']
  • Objects: person:{ city: 'London', date: '2022-09-01' }
  • Auto-generated IDs (these look like person:dxbebreslfp0er668y7k)
-- Use an array as the id
UPDATE config:[$section, $key] SET value = $value;
-- Use an object as the id
UPDATE config:{ section: $section, key: $key } SET value = $value;

Then to select an item...
SELECT * FROM config:{ section: $section, key: $key };

The default ID is better for URL paths, and easier to write in queries.

Although SurrealDB does parse and store UUIDs as a separate type, for record IDs they are treated as strings, and sorted accordingly with other strings. Record IDs are sorted, so you could use complex record IDs to ensure that new records (writes) always sit next to each other on the storage engine. Because of that, it is likely that the write throughput will be better when writing almost sequential ids (in a single node) - depending on your specific scenario.

Records IDs appear as strings, but are not stored as strings.However you can also use backticks...

SELECT * FROM `123`;
OR angle brackets...
SELECT * FROM123⟩;
SELECT * FROM123⟩:⟨456⟩;

The auto-generated IDs are similar to UUIDv6 in that they sort nicely, but are also URL/SQL friendly. So therefore in SurrealDB I can write

SELECT * FROM person:dxbebreslfp0er668y7k

without needing to quote the id using backticks or angular brackets...

SELECT * FROM person:⟨30d5bc81-e837-4b6d-83b3-39cd1c8892b7⟩

But for flexibility you can create a record with any text for the ID ...

SELECT * FROM person:⟨this is my record id⟩

The record id can be a uuid, but because of the - characters is needs to be enclosed within certain characters...

CREATE person:test SET name = 'test';
CREATE person:`this is an id with spaces` SET name = 'test';
CREATE person:`e030f6a5-5b11-4cde-adfb-e81d0e12d6e0` SET name = 'test';
-- You can also use angle brackets (not found on a keyboard though)...
CREATE person:test SET name = 'test';
CREATE person:⟨this is an id with spaces⟩ SET name = 'test';
CREATE person:⟨8424486b-85b3-4448-ac8d-5d51083391c7⟩ SET name = 'test';
-- Or using a function
CREATE type::thing('person', '8424486b-85b3-4448-ac8d-5d51083391c7') SET name = 'test';

Alternatively, from a client library you could run...

db.query("UPDATE type::thing('person', 'e030f6a5-5b11-4cde-adfb-e81d0e12d6e0') SET name = $name;", {
    name: 'test'
});

Complex Record IDs

For values locality, one method would be to use complex record IDs. https://surrealdb.com/releases#v1-0-0-beta-7

As an example you could do...

CREATE restaurant:{ country: 'GBR', hash: geo::hash::encode(51.509865, -0.118092) } SET name = "McDonalds";

And then perform a range query on those records...

SELECT * FROM restaurant:{ country: 'GBR', hash: 'abcde' }..{ country: 'GBR', hash: 'efghi' };

Ascending Integer Ids

you can't do ascending integer ids in SurrealDB yet. So you could specify a number, but not have that auto-increment. Maybe something like this would work though

BEGIN;
-- Set the date
LET $date = "2022-10-05";
-- Increase the counter
LET $counter = (UPDATE counter:[$date] SET value += 1);
-- Create the event
CREATE event:[$date, $counter.value];
-- Commit all changes
COMMIT;

Naming Guideline

You can name your fields how you want really. I guess it depends on what languages/backgrounds you come from.

  • Use singular nouns for main tables... person NOT people, user NOT users...This is because of record links... people:tobie does not make as much sense as person:tobie
  • Use snake_case for edge tables (if you want)...
RELATE person:tobie->purchased->product:macbook
RELATE person:tobie->bought_product->product:macbook 

In my sense, I like to nest fields... Object Types

DEFINE FIELD name ON TABLE person TYPE object;
DEFINE FIELD name.first ON TABLE person TYPE string;
DEFINE FIELD name.last ON TABLE person TYPE string;

Array Types

DEFINE FIELD hobbies ON TABLE person TYPE array;
DEFINE FIELD hobbies.* ON TABLE person TYPE object;
DEFINE FIELD hobbies.*.name ON TABLE person TYPE string;
  
-- You can define arrays of record IDs
DEFINE FIELD friends ON TABLE person TYPE array;
DEFINE FIELD friends.* ON TABLE person TYPE record (person);
-- This allows you to specify embedded objects too
DEFINE FIELD friends ON TABLE person TYPE array;
DEFINE FIELD friends.* ON TABLE person TYPE object;
DEFINE FIELD friends.*.relation ON TABLE person TYPE string;
DEFINE FIELD friends.*.who ON TABLE person TYPE record (person);
  
CREATE person:tobie SET friends = [
  { who: person:jaime, relation: 'brother' },
  { who: person:bob, relation: 'schoolfriend' },
];
  
SELECT friends[*].who.name FROM person:tobie;
-- or
SELECT friends.*.who.name FROM person:tobie;
SELECT friends[WHERE relation = 'brother'].who.name FROM person:tobie;

Enums

Note: There is no native enum type, but from v2.0.0 onwards you can use literal types via TYPE "user" | "admin" to achieve a similar behavior

DEFINE FIELD mood ON TABLE person TYPE string ASSERT $value IN ['sad', 'ok', 'happy'];
or if you want mutliple different potential types on a field...!?
DEFINE FIELD mood ON TABLE person TYPE any ASSERT $value IN ['sad', true, 'happy'];

Single and Polymorphic Types

DEFINE FIELD author ON TABLE book TYPE record (author); -- single type
DEFINE FIELD author ON TABLE book TYPE record (author, person); -- polymorphic types
DEFINE FIELD author ON TABLE book TYPE record (author) ASSERT $value != NONE; -- ensure a value is set

Record Pointers

SurrealDB, the record pointers point directly to a record. There is no table lookup, scans, or index lookups. So the 1:N, N:1, or N:M is pretty efficient. In regular SQL, you do have to have the child id on the parent record, and the parent id on the child record

DEFINE EVENT author ON TABLE book WHEN true THEN (UPDATE $after.author SET books += $this);

Number Format

In SurrealDB values are 64-but floating point numbers, with a fixed amount of precision. values do not have a fixed precision level, and therefore can store an arbitrary level of decimal places. This means that they don't lose precision... SurrealDB currently defaults non-integers to decimals not floats: whats the min and max of float min = -1.7976931348623157E+308f64 max = 1.7976931348623157E+308

The type can basically be an , , or (but not a string). However only in the way it displays in JSON does it appear as a string. With SurrealQL and with the new binary protocol (will be in 1.0.0-beta.9 and the new Rust client), this is a Decimal.

SELECT * FROM <float> 13.5719384719384719385639856394139476937756394756;
-- returns 13.571938471938473
SELECT * FROM <decimal> 13.5719384719384719385639856394139476937756394756;
-- returns 13.5719384719384719385639856394139476937756394756
SELECT * FROM <float> 0.3 + <float> 0.3 + <float> 0.3 + <float> 0.1;
-- returns 1.0000000000000002
SELECT * FROM 0.3 + 0.3 + 0.3 + 0.1;
-- returns 1.0 (because they are decimals by default)

create thing:float set num = <float> 4.2;
[{"time":"543.52µs","status":"OK","result":[{"id":"thing:float","num":4.2}]}]
create thing:decimal set num = <decimal> 4.2;
[{"time":"453.885µs","status":"OK","result":[{"id":"thing:decimal","num":"4.2"}]}]

Datetime

Datetimes are always stored in UTC (never in the local time that they were entered with), but they do need to be formatted according to ISO format with... 1. Date 2. Date and time + 'Z' 3. Date and time + tz_hours 4. Date and time + nanos + 'Z' 5. Date and time + nanos + tz_hours "2022-08-13" is already a datetime. No need to cast it to one (albeit it doesn't matter if you do)...

Computed fields

DEFINE FIELD my_new_field ON TABLE person VALUE function() {
 return 'something from javascript';
};
Or you could use futures...
DEFINE FIELD releasedate ON TABLE movie VALUE <future> { releasedate - time::now() };
DEFINE FIELD age ON TABLE person VALUE <future> { duration::years(time::now() - birthday) };
DEFINE FIELD slug ON TABLE feature TYPE string
  VALUE
    IF $value THEN
      string::slug($value)
    ELSE
      string::slug(title)
    END
;

null as a specific field type

Specify precisely whether we want to allow null values on a field

EFINE TABLE person SCHEMAFULL;
DEFINE FIELD name ON TABLE person TYPE option<string | null>;
UPDATE person:test SET name = "Tobie"; -- This works
UPDATE person:test SET name = NULL; -- This works
UPDATE person:test SET name = NONE; -- This works

Alternatively, if we wanted to ensure that the field is always present, but is allowed to be a string or a null value, then we could do:

DEFINE TABLE person SCHEMAFULL;
DEFINE FIELD name ON TABLE person TYPE string | null;
UPDATE person:test SET name = "Tobie"; -- This works
UPDATE person:test SET name = NULL; -- This works
UPDATE person:test SET name = NONE; -- This fails

Required Field

NULL is a specific value which is empty, but which is set. NONE is a value which does not exist. It effectively removes a field (or checks if a field is not set). There will actually be some changes coming with regards to SCHEMAFULL tables and required fields, but for the moment you can...

DEFINE FIELD title ON story TYPE string ASSERT $value IS NOT NONE;

Future

The is useful if you are calculating or comparing to something that changes (time::now() for instance). The benefit of values are when you are dealing with dynamically changing data (i.e comparing something to the current time like a countdown, or a specific date in the future, or something like that)... that will ALWAYS have a different result EVERY time it is selected and output. To reiterate, futures are great for constantly changing dynamic values (that are always different regardless of any changes to the record data). Computed fields are actually computed and stored on the record when inserting/updating, and are only ever changed when the record fields that they rely on are updated..

-- Re-computed every time the data is selected...
DEFINE FIELD difference ON metrics VALUE <future> { expected_sales - previous_sales };
-- Calculated only once, when the record is updated...
DEFINE FIELD difference ON metrics TYPE int VALUE $this.expected_sales - $this.previous_sales;
DEFINE FIELD difference ON metrics TYPE any VALUE <future> { expected_sales - previous_sales };
-- or just
DEFINE FIELD difference ON metrics VALUE <future> { expected_sales - previous_sales };
DEFINE FIELD difference ON metrics VALUE <future> { expected_sales - previous_sales };

That field will always be computed each time the record is updated, and either of those values are changed...And then that value is stored on the record itself, and not computed each time the record is 'output'.

Note: Always run UPDATE user (or whatever table name you want) without SET when defining a computed field to a table with records in there, to compute the existing "rows".

Assert/Validation

-- if it has to be a positive number DEFINE FIELD previous_sales ON metrics TYPE INT ASSERT $value > 0; -- or if the number should only ever increase DEFINE FIELD previous_sales ON metrics TYPE INT ASSERT $after >= $before;

you also have access to the current record/document with the $this variable. So you could do...
```sql
DEFINE FIELD previous_sales ON metrics TYPE int ASSERT
    IF $this.rank > 10 THEN
         $value > 100
    ELSE
        $value > 10 
    END
;

Every FIELD AND EVENT has:

  1. $value which refers to the current value being passed in
  2. $before which refers to the previous value before any updates were made
  3. $after which refers to the value that it has been changed to (same as $value)
  4. You can also access other fields in the document, just by specifying them by name (no need for $ or anything)...
DEFINE FIELD total_amount ON TABLE order VALUE net_price + shipping;

Linked Fields

To ensure that the field does not become an embedded object if it doesn't exist...

DEFINE FIELD role ON TABLE person TYPE record (role);

Previous or Default Value

If a value already exists, then use that value, otherwise use the current time

DEFINE FIELD created ON waitlist TYPE datetime VALUE $before OR time::now(); 

CreatedAt/UpdatedAt

DEFINE FIELD createdAt ON waitlist TYPE datetime VALUE $before OR time::now();
DEFINE FIELD updatedAt ON waitlist TYPE datetime VALUE time::now();

Read-Only Fields

-- Set a readonly field which is always a certain value
DEFINE FIELD enabled ON TABLE user VALUE true;
-- Set a field which is always a certain value and can't be updated
DEFINE FIELD created_at ON TABLE user VALUE $value OR time::now();
-- Set a field which can have a value set, but can't be changed after that
DEFINE FIELD purchased_at ON TABLE user VALUE $before OR $value;
-- Set a field which starts at 1, and allows for a new value to be set, but only if it increments
DEFINE FIELD score ON TABLE user VALUE $value OR $before OR 1 ASSERT $after >= $before;

Table Events

DEFINE EVENT email ON TABLE user WHEN $before.email != $after.email THEN http::post('https://my-api.com/update', $after);
DEFINE EVENT email ON TABLE user WHEN $event = "CREATE" AND $after.email THEN http::post('https://my-api.com/created', $after.email);
DEFINE EVENT email ON TABLE user WHEN $event = "DELETE" AND $before.email THEN http::post('https://my-api.com/deleted', $before.email);
DEFINE EVENT event_for_field ON TABLE demo WHEN $before.field != $after.field THEN (...)
DEFINE EVENT my_notification ON TABLE user WHEN $event = "UPDATE" THEN http::post('https://my-remote-endpoint.com', { action: $event, data: $this });

will send the following event...

 {
    action: 'UPDATE',
    data: {
        id: 'user:testing',
        name: 'some name',
        age: 22,
    }
}

DEFINE TABLE AS

You can't use subqueries in DEFINE TABLE ... AS ... statements. A view or foreign table in SurrealDB can only select from other tables. However, you can select from multiple tables at once.

DEFINE TABLE lifeforms AS SELECT * FROM person, dog, cat;

DEFINE TABLE @name DROP

  • Basically, the DROP keyword configures a table to ignore all writes.
  • It's useful if you have many high-frequency readings (CPU/temperature), and you actually just need to aggregate this data together in a streaming/rolling manner.
  • So what you can do is define foreign tables / views, which read off of the 'dropped' table, while the actual event data is not stored and not taking up any data storage space...
-- Drop all writes to the 'temperature' table
DEFINE TABLE temperature DROP;
DEFINE FIELD city ON temperature TYPE string;
DEFINE FIELD temp ON temperature TYPE float;
DEFINE FIELD time ON temperature TYPE datetime;
-- Create tables to aggregate temperature readings
DEFINE TABLE average_temperature_by_city_by_day AS
  SELECT
    math::mean(temp) AS average,
    time::group(time, 'day') AS day,
    city
  FROM
    temperature
  GROUP BY city, day
;
DEFINE TABLE average_temperature_by_city_by_hour AS
  SELECT
    math::mean(temp) AS average,
    time::group(time, 'hour') AS hour,
    city
  FROM
    temperature
  GROUP BY city, hour
;

Rename a field

There isn't an ability to 'rename' fields. However, if you remove a field definition, and define another field, it does not alter any records. You can choose to update all records by doing something like the following...

REMOVE FIELD userid ON user TYPE string;
DEFINE FIELD username ON user TYPE string;
-- This will remove the userid field on all documents (in schemafull mode), and add a username field.
UPDATE user;
 -- This will copy the contents of the userid field into the username field, and remove the userid field (in schemafull mode).
UPDATE user SET username = userid;

OR

You need to create another field with the new name and value of the original field, and then remove the original field:

DEFINE FIELD new_name ON my_table VALUE old_name;
REMOVE FIELD old_name ON my_table;
UPDATE my_table;

Hide a field from Selecting

DEFINE FIELD id ON TABLE profile PERMISSIONS NONE;

Remove a field

SCHEMAFULL table:

REMOVE FIELD bar ON TABLE test;
UPDATE test; -- This will remove the 'bar' field, because it is no longer defined

SCHEMALESS table:

REMOVE FIELD bar ON TABLE test;
UPDATE test SET bar = NONE; -- You have to specifically delete the field, because in a schemaless table the field can still exists, even though the field definition has been removed

Authenticated User Id as default value

DEFINE FIELD user ON test TYPE record(user) VALUE $session.sd;

DEFAULT and VALUE clause

VALUE clause only

-- Set only the VALUE clause
DEFINE FIELD timestamp ON TABLE person TYPE datetime VALUE time::now();
CREATE person;

This example fails. When the type of the field is checked, then the query fails. This is because there is no value set for the timestamp field, and yet the field is set to datetime so it requires a value. In this scenario, because the VALUE clause can be different each time it is run (because it's a dynamic function), then it can't be used automatically for the DEFAULT clause.

DEFAULT clause only

-- Set only the DEFAULT clause
DEFINE FIELD timestamp ON TABLE person TYPE datetime DEFAULT time::now();
CREATE person;

This example succeeds. The field is set initially to the current time. If a datetime is specified when the record is created, it will be used, otherwise the DEFAULT clause is used.

DEFAULT and VALUE clauses together

-- Set both the DEFAULT and VALUE clauses
DEFINE FIELD timestamp ON TABLE person TYPE datetime DEFAULT time::now() VALUE time::now();
CREATE person;

This final example succeeds. Basically the initial field value is being set to the current time using the DEFAULT clause, and then it is being set again in the VALUE clause. Then for every subsequent record update, the field value will be set to the current time.

Q And A

Question: what's the recommended way to design facebook/reddit style comments in surreal?

Answer: There is definitely flexibility in SurrealDB with regards to how you store your data as a developer. It really depends on how you want to be accessing that data.

  1. Do you want to duplicate up data (store text tags on records for example) or do you want to link to a 'tag' record in another table...

  2. How many records do you want to pull down to the user interface at once. And how big will your record be? If you have hundreds of comments, it's better to link these out with graph connections (or at the very least record pointers), so that they don't make the primary record absolutely huge.

You could do embedded comments..

CREATE post:surrealdb SET name = 'My new post about SurrealDB', comments = [];
UPDATE post:surrealDB SET comments += { created_at: time::now(), text: 'Wow this is cool' };

or record pointers to comments in another table...

CREATE post:surrealdb SET name = 'My new post about SurrealDB', comments = [];
LET $comment = (CREATE comment SET created_at = time::now(), text = 'Wow this is cool');
UPDATE post:surrealDB SET comments += $comment;

or graph connections to comments in another table...

CREATE post:surrealdb SET name = 'My new post about SurrealDB', comments = [];
LET $comment = (CREATE comment SET text = 'Wow this is cool');
RELATE $comment->commented_on->post:surrealdb SET time = time::now();
SELECT ->commented_on->comment.text AS all_comments FROM post:surrealdb;

Check if the $value is set or not.

DEFINE TABLE tbl SCHEMALESS;
DEFINE FIELD alpha ON tbl TYPE bool VALUE $value ?? true;
DEFINE FIELD beta ON tbl TYPE bool VALUE $value ?? false;

Delete TABLE

DELETE FROM type::table(tbl_page);

Rename Table

you can't rename a table in SurrealDB. The primary reason is that all record IDs store the table with the ID itself (allowing for the record pointer links, and the graph edge connections). You are able to copy records to another table...

INSERT INTO recordings_san_francisco (SELECT * FROM temperature WHERE city = 'San Francisco');

which will add all of those records, with the new table name. However it's not quite the same, as any pointer (temperature:1341039) within the database will still point at the temperature table...

⚠️ **GitHub.com Fallback** ⚠️