SurrealDB Internal design — Part 2 — Architecture

Photo by Aleksandr Popov on Unsplash

A full list of posts in the series can be found here.

In the previous post we talked about the features of SurrealDB, in this post we’ll start looking at the code itself from the top down.

We’ll start with the general architecture design and then look at the file structure of the project.

Architecture

SurrealDB is built with one of the most time-tested and well-known architectures for any project and for servers in particular — the 2-tier architecture.

2-Tier Architecture

This architecture is based on the idea that we need to separate our code into two logical parts (called layers):

1. API (Application Programming Interface)— the client facing part of the application
2. DAL (Data Access Layer)— the part of the code that handles our custom code and talks to external systems

What separates these two layers is the language they speak. What I mean by this is that each layer uses different data models:

1. The API talks in the language of the users, and translates it to the internal models
2. The Business Logic (BL) translates our models and communicates with external services using their language, returning the results back to the API layers using the internal models

Each tier only calls the tier below it and doesn’t know about the other tiers.

SurrealDB Architecture and File Structure

SurrealDB’s implementation of the 2-tier architecture is shown in it’s file structure.

|-- src            // API layer
    |-- net       
    |-- cli
    |-- rpc
    ...
|-- lib/src       // BL
   |-- sql 
   |-- fnc
   ...
   |-- kvs  
      |-- indexdb
      |-- rocksdb
      |-- tikv
      ...
   

The src folder includes all the API code, each subfolder representing a different way to access the DB (REST, websockets, CLI, etc…)

The lib folder includes both the BL.

• The kvs subfolder includes all the different storage options for the data (memory, TiKV, RocksDB, etc…)
• The rest of the subfolders are the business logic, the custom code the makes SurrealDB function

Knowing the file structure of the project and understanding how the different parts talk to each other is always the best way to getting started on a new project, as it allows us to start jumping around the code base without getting lost.

Code Examples

Let’s look at some examples from the code itself.

API Layer

As we said previously, the API is responsible for communicating with the clients, and allowing the internal code to run without knowing the origin of the request, allowing it to be easily switched out. This is the reason SurrealDB can be used in several modes, including both as a server and as an embedded database.

The heart of this function is line 15, which calls the method of the BL that runs the operations requested by the user. The rest of the code wraps around this line and translates the user language into the server’s language and back into the user response format.

Business and Data Layer

The data layer both parses runs the application code and accesses the underlying data itself.

In SurrealDB, each file in lib is responsible for both the parsing and computing of a specific functionality. For example, the select statement has it’s own file, all the code related to the geometry is in another, etc.

Let’s look for example at defining a database.

We will have a more in-depth look at the query language itself in the next post, but let’s look at the example in the context of the layer architecture.

The code to handle the database definition is divided into four parts:

1. Model — The internal model that the database knows how to handle
2. Parsing — The method that translates the user’s language (SurQL) into the internal model
3. Display Format — For displaying the internal model in the user’s language
4. Compute — Finally, the method to execute the code represented in the internal model

As you can see, these functions can be seen as translations between the different languages and between the different layers.

Now that we understand the high-level architecture, we can look at the code itself.

Conclusion

See, not so bad after all, if we just know how each piece fits into the larger architecture.

Now that we understand the high-level overview of the code, in the following posts we will dive into one subject at a time and look at the internal implementation.

Next up — understanding how Surreal parses the query language (we’ve seen a small example above).

See you there!