# Concepts ## Graph *Related:* [Why graph?](https://agdb.agnesoft.com/blog/why-graph) Graph is a set of nodes (also vertices, points) that are connected to each other through edges (also arcs, links). In `agdb` the data is plotted on directed graphs and there are no restrictions on their structure. They can be cyclic (forming a cycle), acyclic (being open-ended), sparse (having only some connections between nodes), disjointed (thus forming multiple graphs), having self-referential edges (nodes being connected to themselves), having multiple edges to the same node (even itself) and/or in the same direction. Nodes and edges are `graph elements` and each can have key-value pairs associated with them usually referred to as `values`. Each graph element has a signed integer `id` (`db id`) - nodes having positive values while edges negative values. Note that upon removal of a graph element its `id` is freed and can be reused by subsequent inserts of new graph elements. **Terminology:** - Graph (set of nodes and edges) - Node (point on a graph) - Edge (connection between two nodes) - Graph elements (nodes & edges) - `db id` (graph element `id`, positive for nodes, negative for edges) - Values (key-value pairs associated with a node or an edge) ## Query *Related:* [Why object queries?](https://agdb.agnesoft.com/blog/object-queries), [Queries](https://agdb.agnesoft.com/docs/references/queries) Query is a request to retrieve or manipulate data in a database (both the graph structure and `values` associated with the nodes and edges). In `agdb` queries are not texts (like in SQL) but rather objects that contain details about what is being requested. These objects are typically constructed via a query builder, but it is also possible to create them like any other object. The builder steps resemble, and often indeed are, direct translations of a well known SQL equivalents (e.g. `QueryBuilder::select() == SELECT`, `QueryBuilder::insert() == INSERT INTO`). Queries are executed by the database directly. The `agdb` distinguishes between `immutable` (retrieving data) and `mutable` (manipulating data) queries. Each query execution produces either a result or an error. In `agdb` there is a single `result` object containing a numerical result (i.e. number of affected elements or values) and a list of database elements. Each element in a result consists of a database `id` and a list of `values` (associated key-value pairs). In case of a failure the database execution yields an error detailing what went wrong instead of a result. See dedicated [queries](https://agdb.agnesoft.com/docs/references/queries) documentation for details. **Terminology:** - Query (request to retrieve or manipulate data) - Immutable query (request to retrieve data) - Mutable query (request to manipulate data) - Result (result of a query) ## Transaction *Related*: [Queries](https://agdb.agnesoft.com/docs/references/queries) Transactions are a way to provide atomicity, isolation and data consistency in a database (three of [ACID](https://en.wikipedia.org/wiki/ACID){rel=""nofollow""} properties). In `agdb` every query is a transaction, but it is also possible to execute multiple queries as a single transaction. Just like `queries` transactions are immutable or mutable. One important rule is borrowed directly from Rust and enforced on the type level: *"There can be either unlimited number of concurrent immutable transactions or exactly one mutable transaction"* In multithreaded environment you can easily synchronize the access to the database by using [`RwLock`](https://doc.rust-lang.org/std/sync/struct.RwLock.html){rel=""nofollow""}. Furthermore, unlike traditional transactions implemented in other database systems the `agdb` transactions are immediately executed requiring a closure containing (minimum) amount of code and queries required for the transaction to be performed. This forces the client to optimize their transactions and reduce the time the database is locked, which is particularly important for mutable transactions as they lock the entire database for their execution. **Terminology:** - Transaction (set of queries to be executed atomically against a database wrapped in a closure) - Mutable transaction (set of mutable & immutable queries wrapped in a closure) - Immutable transaction (set of immutable queries wrapped in a closure) ## Storage *Related*: [Why single file?](https://agdb.agnesoft.com/blog/single-file) Every persistent database eventually stores its data somewhere on disk in one or more files. The `agdb` stores its data in a single file (that is being shadowed by another temporary write ahead log file). Its internal structure is very similar to that of a memory which makes it very easy to map between the two. The file format is fully platform-agnostic, and the file can be safely transferred to another machine and loaded there. Similarly, the `agdb` is by default memory mapped database, but it could just as easily operate purely on the file itself at the cost of read performance (might be implemented as a feature in the future). The database durability is provided by the write-ahead-log (WAL) file which records reverse of every operation to be performed on the main file before it actually happens. In case of any catastrophic failure the main database file is repaired from the WAL on loading the database. By default the database relies on the operating system to flush dirty pages to disk in the background (`SyncMode::None`). This is fast and sufficient for local filesystems. On network filesystems with client-side write caching (e.g. WekaFS with `writecache`) the OS may reorder or delay flushes, which can corrupt the WAL protocol. For these environments set `SyncMode::Commit` which issues `fdatasync` at each transaction commit ensuring the correct ordering of writes: ```rs use agdb::{Db, SyncMode}; let mut db = Db::new("my.agdb")?; db.set_sync_mode(SyncMode::Commit); ``` This guarantees that committed data reaches durable storage before the WAL is cleared, at the cost of additional latency per transaction. Just like the memory the main database file will get fragmented over time. Sectors of the file used for the data that was later reallocated will remain unused (fragmented) until the database file is defragmented. That operation is performed automatically on database object instance drop. The storage taken by individual elements and properties is generally as follows: - node: 32 bytes - edge: 32 bytes - single key or value (<=15 bytes): 16 bytes - single key or value (>15 bytes): 32 bytes (+) - key-value pair: 32 bytes (+) The size of the graph elements (nodes & edges) is fixed. The size of the properties (key-value pairs) is at least 32 bytes (16 per key and 16 per value) but can be greater if the value itself is greater. This creates some inefficiency for small values (e.g. integers) but it also allows application of small value optimization where values up to 15 bytes in size (e.g. strings) do not allocate or take extra space. When a value is larger than 15 bytes it will be stored separately with another 16 bytes overhead making it at least `32 + value length` bytes. The reason for values taking 16 bytes at minimum instead of 8 is that the value needs to store a type information for which 1 byte is required. 9 bytes is an awkward and very inefficient (as measured where 16 byte values were much faster) size even if it could save some file space. The next alignment is therefore 16 bytes which also allows the aforementioned small value optimization. **Terminology:** - File storage (underlying single data file) - Write ahead log (WAL, shadowing file storage to provide durability) - Sync mode (`None` = rely on OS flushing, `Commit` = explicit fsync per transaction) ## Data types Supported types of both keys and values are: - `i64` - `u64` - `f64` - `String` - `Vec` - `Vec` - `Vec` - `Vec` - `Vec` It is an enum of limited number of supported types that are universal across all platforms and programming languages. They are serialized in file as follows: | Type | Layout | Size | | ------------- | ------------------------------------------------------------------------------- | -------- | | `i64` | little endian | 8 bytes | | `u64` | little endian | 8 bytes | | `f64` | little endian | 8 bytes | | `String` | size as `u64` little endian followed by UTF-8 encoded string as `u8` bytes | 8+ bytes | | `Vec` | size as `u64` little endian followed by individual `u8` bytes | 8+ bytes | | `Vec` | size as `u64` little endian followed by individual `i64` little endian elements | 8+ bytes | | `Vec` | size as `u64` little endian followed by individual `u64` little endian elements | 8+ bytes | | `Vec` | size as `u64` little endian followed by individual `f64` elements | 8+ bytes | | `Vec` | size as `u64` little endian followed by individual `String` elements | 8+ bytes | # Quickstart The following is the quickstart guide for the agdb embedded/application database. [Looking for server client guide instead?](https://agdb.agnesoft.com/api-docs/rust) ::steps ### Install Rust toolchain From the [official source](https://www.rust-lang.org/tools/install){rel=""nofollow""}. ### Create an application First we initialize an application called `agdb_app` with cargo: ```bash mkdir agdb_app cd agdb_app cargo init ``` ### Add dependencies ```bash cargo add agdb ``` ### Create the database In `main.rs` we create a memory mapped database: ```rs filename="main.rs" use agdb::DbError; fn main() -> Result<(), DbError> { let mut db = Db::new("agdb_app.agdb")?; Ok(()) } ``` :::note The namesake file "agdb\_app.agdb" will be created in your working directory. The .agdb extension is conventional. ::: ### Run queries We run our first query against the database inserting a node with alias "users": ```rs db.exec_mut(QueryBuilder::insert() .nodes() .aliases("users") .query())?; ``` We then insert additional nodes representing some users and connect them with the "users" node: ```rs // We derive from agdb::DbType // so we can use the type in the db. #[derive(Debug, DbType)] struct User { db_id: Option, // The db_id member is optional but // it allows insert your user type // directly into the database. username: String, age: u64, } let users = vec![User { db_id: None, username: "Alice".to_string(), age: 40 }, User { db_id: None, username: "Bob".to_string(), age: 30 }, User { db_id: None, username: "John".to_string(), age: 20 }]; let db_users = db.exec_mut(QueryBuilder::insert() .nodes() .values(&users) // We can pass users directly as // query parameter thanks to the // implementation of the agdb::DbType // trait via #[derive(DbType)]. .query())?; db.exec_mut( QueryBuilder::insert() .edges() .from("users") .to(&db_users) // We can feed result of a previous // query directly to the next one. .query(), )?; ``` :::tip We could also run all the queries as a single transaction with `transaction()` / `transaction_mut()` instead of exec variants that operate on a single query each time. ::: ### Find a user in the database matching some conditions ```rs // We combine search & select into a single query like so: let users: Vec = db .exec( QueryBuilder::select() .elements::() // Select only relevant properties for the User struct. .search() .from("users") // Start the search from the "users" node. .where_() .key("age") // Examine "age" property. .value(LessThan(40.into())) // Include it in the search result if the value // is less than 40. .query(), )? .try_into()?; // Convert the result into a list of User objects. println!("{:?}", users); // We can print the users thanks to #[derive(Debug)]. The result should be something like: // Vec [User { db_id: Some(DbId(3)), username: "John", age: 20 }, User { db_id: Some(DbId(4)), username: "Bob", age: 30 }] ``` ### Run the program ```bash cargo run ``` ### Full program {rel=""nofollow""} :: # Overview The `agdb_server` can be run in multiple ways. The following table directs you to the correct documentation based on your use case. It differentiates between running the server as a single server or as a cluster of multiple replicated nodes (using Raft consensus protocol). The documentation is provided for running the server on bare metal, using docker or Kubernetes (K8s). Please refer to the [server](https://agdb.agnesoft.com/docs/references/server) documentation for general information. | Type / Target | Bare Metal | Docker | K8s | | ------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------- | | Server | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-bare-metal) | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-docker) | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-k8s) | | Cluster | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-bare-metal) | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-docker) | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-k8s) | # Bare metal The `agdb_server` can be run on bare metal by building the binary and running it on the target machine: ::steps ### Install prerequisites - Git: [official source](https://git-scm.com/){rel=""nofollow""} (skip if you already have it) - Rust: [official source](https://www.rust-lang.org/tools/install){rel=""nofollow""} - [OPTIONAL, REQUIRED FOR STUDIO] Node.js: [official source](https://nodejs.org/en/download/){rel=""nofollow""} - [OPTIONAL, UNIX ONLY, REQUIRED FOR TLS] CMake: [official source](https://cmake.org/download/){rel=""nofollow""} - [OPTIONAL, WINDOWS ONLY, REQUIRED FOR TLS] Microsoft Build Tools (includes CMake): [official source](https://learn.microsoft.com/en-us/cpp/build/building-on-the-command-line){rel=""nofollow""} ### Install `agdb_server` ```shell cargo install agdb_server --features "tls studio" # The `tls` feature is optional and can be omitted if TLS support is not needed. Requires CMake and on Windows also MSVC. # The `studio` feature is optional can be omitted if you do not need or want server GUI. Requires `pnpm`. ``` You can also build the server manually. One advantage is that you can use a custom `pepper` value and bake it into the binary instead of using runtime configured value. The `pepper` file is located in sources as `agdb_server/pepper` and contains a random 16 character value that is used internally to additionally "season" the encrypted passwords. When building for production you should change this value to a different one and keep the pepper file as secret in case you needed to rebuild the server or build a new version. It can also be changed via configuration during runtime. The steps for a manual build (use `bash` on Unix or `git bash` on Windows): ```bash git clone https://github.com/agnesoft/agdb.git cd agdb/ git checkout $(git describe --tags) # checkout the latest released version echo "1234567891234567" > agdb_server/pepper #use a different value, this value will be a secret cargo build --release -p agdb_server --features "tls studio" # The `tls` feature is optional and can be omitted if TLS support is not needed. Requires CMake and on Windows also MSVC. # The `studio` feature is optional can be omitted if you do not need or want server GUI. Requires `pnpm`. mv target/release/agdb_server "" # Windows: target/release/agdb_server.exe ``` :::warning Server with a different pepper value (e.g. default non-prod version) won't be able to decode passwords in the internal database. If you lose the pepper value of your server and need to rebuild it you should generate a new pepper and then you will need to create a new admin account (by changing the config value to a non-existent user) and using that account you can reset passwords of all your users via `/api/v1/admin/user/{username}/change_password` API (including the old admin account). ::: Alternatively you can use the default pepper value but specify in configuration the "pepper\_path" from which the pepper would be loaded during runtime. This file and location should be treated as secret. All the caveats of manual build still apply including the recovery steps in case the pepper value is lost. ### Run the server ```bash agdb_server ``` The server upon starting will create few things in its working directory: - `agdb_server.yaml`: Configuration file. You can alter it as you wish or prepare one in advance. You would need to restart the server for the changes to take effect. - `agdb_data_dir/`: Folder for storing the user data. It can be changed in the configuration file (requires restart of the server and possibly moving the internal database and data, if any, to the new location). - `agdb_data_dir/agdb_server.agdb` (`agdb_data_dir/.agdb_server.agdb`): Internal database of the server (uses `agdb` itself) + it's write ahead file (the dotfile). and report where it listens at: ```bash 2024-01-26T17:47:30.956260Z INFO agdb_server: Listening at localhost:3000 ``` :::tip You can prepare the configuration file before starting the server. ::: [Please refer to the server reference for the configuration options.](https://agdb.agnesoft.com/docs/references/server) ### Test that the server is up with `curl` ```bash curl -v localhost:3000/api/v1/status # should return 200 OK ``` ### Create a database user It is recommended (but optional) to create a regular user rather than using the `admin` user (which is however still possible): ```bash # produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5" token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}') # using admin token to create a user curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/user/my_db_user/add -d '{"password":"password123"}' # login as the new user and producing their token token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"my_db_user","password":"password123"}') ``` ### Interact with the database server You can either continue using `curl`, interactive OpenAPI GUI from any browser `localhost:3000/api/v1` (provided by `rapidoc`) or choose one of the [available API clients](https://agdb.agnesoft.com/api-docs/openapi). The raw OpenAPI specification can be downloaded from the server at `localhost:3000/api/v1/openapi.json` as well. ### Shutdown the server The server can be shutdown with `CTRL+C` or programmatically posting to the shutdown endpoint as logged in server admin: ```bash # this will produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5" token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}') curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/shutdown ``` :: # Docker The `agdb_server` can be run in Docker using official Docker image. Optionally you can [build the image](https://github.com/agnesoft/agdb/blob/main/agdb_server/containerfile){rel=""nofollow""} yourself. ::steps ### Install Docker - Windows: {rel=""nofollow""} - Linux: {rel=""nofollow""} ### Pull or build the agdb\_server image The image is based on [Alpine Linux](https://alpinelinux.org/){rel=""nofollow""} using musl libc. The image is made available on Docker Hub or GitHub packages: | Vendor | Tag | Command | Description | | ---------- | ------ | ----------------------------------------- | ----------------------------------------------------------------------------------------- | | Docker Hub | latest | docker pull agnesoft/agdb\:latest | Equals latest released version | | Docker Hub | 0.x.x | docker pull agnesoft/agdb:0.x.x | Released version, e.g. 0.10.0 | | Docker Hub | dev | docker pull agnesoft/agdb\:dev | Equals latest development version on the main branch, refreshed with every commit to main | | GitHub | latest | docker pull ghcr.io/agnesoft/agdb\:latest | Equals latest released version | | GitHub | 0.x.x | docker pull ghcr.io/agnesoft/agdb:0.x.x | Released version, e.g. 0.10.0 | | GitHub | dev | docker pull ghcr.io/agnesoft/agdb\:dev | Equals latest development version on the main branch, refreshed with every commit to main | If you want to build the image yourself run the following in the root of the checked out `agdb` repository: ```bash docker build --pull -t agnesoft/agdb:dev -f agdb_server/containerfile . ``` ### Run the server ```bash docker run -v agdb_data:/agdb/agdb_data --name agdb -p 3000:3000 agnesoft/agdb:dev ``` This command runs the server using the default configuration (without TLS). It assigns a volume to the data directory for data persistence, gives the container a name (`agdb`) and publishes the container's exposed port (`3000`) to the host. You can publish to a different local port (e.g. `5000:3000` (host\:container)) where the container's port `3000` will be locally accessible on the port `5000`. ### Test that the server is up with `curl` ```bash curl -v localhost:3000/api/v1/status # should return 200 OK ``` ### Shutdown the server The server can be shutdown either by stopping the container or programmatically posting to the shutdown endpoint as logged in server admin: ```bash # this will produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5" token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}') curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/shutdown ``` ### TLS Support In order to enable TLS support you need to provide following config values as per the [server documentation](https://agdb.agnesoft.com/docs/references/server): ```yaml tls_certificate: /agdb/cert.pem tls_key: /agdb/cert.key ``` If you are using self-signed certificate you should provide also the root CA via: ```yaml tls_root: /agdb/root_ca.pem ``` You can then mount the custom config and the certificates into the container as volume(s). Assuming the `agdb_server.yaml` and the certificates are in `/local/path` on your host machine: ```bash docker run -v agdb_data:/agdb/agdb_data -v /local/path:/agdb --name agdb -p 3000:3000 agnesoft/agdb:dev ``` When you run the container this way it will load the configuration from `/agdb/agdb_server.yaml` and if it specifies the paths to the files they would be loaded as well. :: # Kubernetes The `agdb_server` can be run in Kubernetes using official Docker image. Optionally you can [build the image](https://github.com/agnesoft/agdb/blob/main/agdb_server/containerfile){rel=""nofollow""} yourself and host it at the repository of your choosing. Please refer to the [server-docker](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-docker) guide for available images. ::note This guide is for running `agdb_server` as a single instance. This is only recommended when resiliency is not required and/or if speed is of the essence. Consider running [cluster](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-k8s) instead. :: ### Requirements - Kubernetes cluster (you can enable local K8s cluster in Docker Desktop) - kubectl ### Stateful set yaml You can find an example Kubernetes deployment at {rel=""nofollow""} The example breakdown: ::steps ### Service First we deploy the K8s service of type `ClusterIP` that only allows communication inside the cluster. As we are running a database server it would typically serve other backends in the same cluster and not be accessible from the outside. If such access is needed consider using `LoadBalancer` type service or `Ingress` controller with the `ClusterIp` service. The port available in the cluster is `3000` under a name of `agdb`. Furthermore, we specify selector value `app: agdb` and also a label of the same value. ```yaml apiVersion: v1 kind: Service metadata: name: agdb labels: app: agdb spec: ports: - port: 3000 name: agdb clusterIP: None selector: app: agdb ``` ### Secrets Next document is the pepper secret `agdb-pepper`. ```yaml apiVersion: v1 kind: Secret metadata: name: agdb-pepper labels: app: agdb stringData: pepper: "1234567891234567" ``` Followed by the certificates. In production, you should create the certificates using kubectl command from secure files rather than embedding them into the manifest. However, for demonstration purposes the following example is provided. It uses self-signed certificate with the corresponding CA. It needs to be provided as `base64` value. You can use e.g. `cat cert.pem | base64` bash command to get the correct value (end of lines do not matter). :::warning The certificate must be issued (or used as an alternative name - SAN) for the DNS name of the server used in its configuration `address` field. E.g. `agdb.default.svc.cluster.local`. ::: ```yaml apiVersion: v1 kind: Secret metadata: name: agdb-certs labels: app: agdb data: cert.pem: ... key.pem: ... root_ca.pem: ... ``` ### ConfigMap The configuration named `agdb-config` via the `ConfigMap` is optional as the default configuration would work just as well. It might however be useful if you needed to change anything regarding the server later. For example disabling TLS if it is not needed etc. ```yaml --- apiVersion: v1 kind: ConfigMap metadata: name: agdb-config labels: app: agdb data: agdb_server.yaml: | bind: :::3000 address: http://agdb.default.svc.cluster.local:3000 basepath: "" static_roots: [] admin: admin log_level: INFO data_dir: /agdb/data pepper_path: /agdb/pepper/pepper tls_certificate: /agdb/certs/cert.pem tls_key: /agdb/certs/key.pem tls_root: /agdb/certs/root_ca.pem cluster_token: cluster cluster_heartbeat_timeout_ms: 1000 cluster_term_timeout_ms: 3000 cluster: [] ``` ### StatefulSet The main part of the deployment is the stateful set definition. While replica set could work to some extent the instances of `agdb_server` are not interchangeable and cannot be freely scaled horizontally. The stateful set type is therefore a better fit. It uses the selector and labels `app: agdb` in order to "link" the service and the underlying pod together. Kubernetes is using selectors rather than direct mapping when linking various things together such as services and pods. We specify 1 replica only (refer to the [agdb as K8s cluster](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-k8s) for an alternative). The container spec matches the port on the service by name `agdb` and exposes the container port `3000` (default). The security context specifies the user `1000` (default uid in the container) and disables root access as it is not needed and enhances security. Finally, we specify volumes and volume mounts to add the `secrets`, `configmap` and persistent volume claim (PVC) to the expected locations. The PVC is a way how data can survive restart or redeployment. By default, 1 GB of storage is specified which can be increased (but not decreased) in subsequent deployments. Certificates are provided for TLS support. ```yaml --- apiVersion: apps/v1 kind: StatefulSet metadata: name: agdb labels: app: agdb spec: serviceName: "agdb" replicas: 1 selector: matchLabels: app: agdb template: metadata: labels: app: agdb spec: containers: - name: agdb image: agnesoft/agdb:dev ports: - containerPort: 3000 name: agdb securityContext: runAsUser: 1000 runAsGroup: 1000 securityContext: allowPrivilegeEscalation: false volumeMounts: - name: agdb-data mountPath: /agdb/data - name: config mountPath: /agdb - name: pepper mountPath: /agdb/pepper - name: certs mountPath: /agdb/certs volumes: - name: config configMap: name: agdb-config defaultMode: 511 - name: pepper secret: secretName: agdb-pepper - name: certs secret: secretName: agdb-certs volumeClaimTemplates: - metadata: name: agdb-data labels: app: agdb spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi ``` ### Test that the server is up with `curl` The following command must be run from within the cluster unless the server was exposed via `LoadBalancer` or `Ingress`. The `.default.` bit is the name of the namespace where everything was deployed. ```bash curl -v https://agdb.default.svc.cluster.local:3000/api/v1/status # should return 200 OK ``` ### Additional considerations - You could use the `localhost:3000/api/v1/status` as a startup/readiness/health probe. - Standard shutdown procedure via the endpoint will not work as K8s will simply restart the server. - Consider running the [cluster](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-k8s) rather than just a single node. - You should be able to adapt the above to other platforms such as AWS EKS. - If you are using `agdb` only inside the K8s cluster with no visibility outside of it you may want to disable TLS. - If you are exposing `agdb` outside the K8s cluster use real production certificate. In that case leave the `tls_root` configuration option empty. :: # Bare metal (cluster) The `agdb_server` can be run as a cluster on bare metal. First you should build the server as described in the [server - bare metal](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-bare-metal) guide. In the following steps we will run a cluster of 3 nodes on a local machine however it should work essentially the same even when each server was run on its own machine: ::steps ### Prepare cluster configuration First we create the configuration files for our nodes. Notice that the only difference is the port in the address option. As described in the main [server](https://agdb.agnesoft.com/docs/references/server) documentation the address field determines the index of the local node (position in the cluster list). The list of nodes is then the same and having the same order for each instance as it provides the cluster "hash" for establishing trust in the cluster (in addition to the `cluster_token`): ```yaml #node0: ~/agdb_cluster/node0/agdb_server.yaml bind: :::3000 address: http://localhost:3000 basepath: "" static_roots: [] admin: admin log_level: INFO data_dir: agdb_server_data pepper_path: "" tls_certificate: "" tls_key: "" tls_root: "" cluster_token: cluster cluster_heartbeat_timeout_ms: 1000 cluster_term_timeout_ms: 3000 cluster: [http://localhost:3000, http://localhost:3001, http://localhost:3002] #node1: ~/agdb_cluster/node1/agdb_server.yaml bind: :::3000 address: http://localhost:3001 basepath: "" static_roots: [] admin: admin log_level: INFO data_dir: agdb_server_data pepper_path: "" tls_certificate: "" tls_key: "" tls_root: "" cluster_token: cluster cluster_heartbeat_timeout_ms: 1000 cluster_term_timeout_ms: 3000 cluster: [http://localhost:3000, http://localhost:3001, http://localhost:3002] #node2: ~/agdb_cluster/node2/agdb_server.yaml bind: :::3000 address: http://localhost:3002 basepath: "" static_roots: [] admin: admin log_level: INFO data_dir: agdb_server_data pepper_path: "" tls_certificate: "" tls_key: "" tls_root: "" cluster_token: cluster cluster_heartbeat_timeout_ms: 1000 cluster_term_timeout_ms: 3000 cluster: [http://localhost:3000, http://localhost:3001, http://localhost:3002] ``` ### Run the server Next we run all 3 nodes as background processes in their respective directories with the prepared config files. It is recommended to run each in its own shell so you can observe the logs otherwise they would be all writing to the same shell if run as background processes (i.e. `agdb_server &`). If you decide to run all of them in the same shell each log messages clearly indicates to which node it belongs using the node's index (e.g. `[0]`, `[1]` etc.) ```bash cd ~/agdb_cluster/node0/ #run each node in its respective directory agdb_server ``` ### Test that the cluster is up with `curl` The following commands will hit each node and return the list of nodes, their status and which one is the leader. If the servers are connected and operating normally the returned list should be the same from each node. ```bash curl -v localhost:3000/api/v1/cluster/status curl -v localhost:3001/api/v1/cluster/status curl -v localhost:3002/api/v1/cluster/status ``` ### Shutdown the servers The cluster must be shutdown one by one using the same mechanism as with single server including the `CTRL+C`. Using curl: ```bash # this will produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5" token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}') curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/shutdown token=$(curl -X POST -H 'Content-Type: application/json' localhost:3001/api/v1/user/login -d '{"username":"admin","password":"admin"}') curl -X POST -H "Authorization: Bearer ${token}" localhost:3001/api/v1/admin/shutdown token=$(curl -X POST -H 'Content-Type: application/json' localhost:3002/api/v1/user/login -d '{"username":"admin","password":"admin"}') curl -X POST -H "Authorization: Bearer ${token}" localhost:3002/api/v1/admin/shutdown ``` While it is technically possible to use cluster login and avoid logins to each node separately it might be fragile and not work well in situations where the cluster is in the bad shapes with nodes not being available etc. Local login and shutdown are guaranteed to work regardless of the overall cluster status. :: # Docker (cluster) The `agdb_server` can be run as a cluster in docker. Optionally you can [build the image](https://github.com/agnesoft/agdb/blob/main/agdb_server/containerfile){rel=""nofollow""} yourself. ::steps ### Install docker - Windows: {rel=""nofollow""} - Linux: {rel=""nofollow""} ### Pull or build the agdb\_server image The image is based on [Alpine Linux](https://alpinelinux.org/){rel=""nofollow""} using musl libc. The image is made available on Docker Hub or GitHub packages: | Vendor | Tag | Command | Description | | ---------- | ------ | ----------------------------------------- | ----------------------------------------------------------------------------------------- | | Docker Hub | latest | docker pull agnesoft/agdb\:latest | Equals latest released version | | Docker Hub | 0.x.x | docker pull agnesoft/agdb:0.x.x | Released version, e.g. 0.10.0 | | Docker Hub | dev | docker pull agnesoft/agdb\:dev | Equals latest development version on the main branch, refreshed with every commit to main | | GitHub | latest | docker pull ghcr.io/agnesoft/agdb\:latest | Equals latest released version | | GitHub | 0.x.x | docker pull ghcr.io/agnesoft/agdb:0.x.x | Released version, e.g. 0.10.0 | | GitHub | dev | docker pull ghcr.io/agnesoft/agdb\:dev | Equals latest development version on the main branch, refreshed with every commit to main | If you want to build the image yourself run the following in the root of the checked out `agdb` repository: ```bash docker build --pull -t agnesoft/agdb:dev -f agdb_server/containerfile . ``` ### Run the cluster You will need the `compose.yaml` file from the sources at: {rel=""nofollow""} ```bash # the -f path is where the file resides in the sources, you can change it to the actual location of the compose.yaml file docker compose -f agdb_server/compose.yaml up --wait ``` This command runs the 3 nodes as a docker cluster using docker compose that contains valid cluster configuration. The volumes are provided for each node so that the data is persisted. It exposes the nodes at the ports `3000`, `3001` and `3002`. By default, it is using TLS self-signed certificates. You can either remove the certificates and related configuration from the `compose.yaml` or provide your own certificates. Refer to the [server configuration](https://agdb.agnesoft.com/docs/references/server) for more details. ### Test that the cluster is up with `curl` The following commands will hit each node and return the list of nodes, their status and which one is the leader. If the servers are connected and operating normally the returned list should be the same from each node. ```bash curl -v localhost:3000/api/v1/cluster/status curl -v localhost:3001/api/v1/cluster/status curl -v localhost:3002/api/v1/cluster/status ``` ### Shutdown the cluster The cluster can be shutdown either by stopping the containers or programmatically posting to the shutdown endpoints as logged in server admin: ```bash # this will produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5" token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}') curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/shutdown token=$(curl -X POST -H 'Content-Type: application/json' localhost:3001/api/v1/user/login -d '{"username":"admin","password":"admin"}') curl -X POST -H "Authorization: Bearer ${token}" localhost:3001/api/v1/admin/shutdown token=$(curl -X POST -H 'Content-Type: application/json' localhost:3002/api/v1/user/login -d '{"username":"admin","password":"admin"}') curl -X POST -H "Authorization: Bearer ${token}" localhost:3002/api/v1/admin/shutdown ``` While it is technically possible to use cluster login and avoid logins to each node separately it might be fragile and not work well in situations where the cluster is in the bad shapes with nodes not being available etc. Local login and shutdown are guaranteed to work regardless of the overall cluster status. :: # Kubernetes (cluster) The `agdb_server` can be run in Kubernetes using official Docker image. Optionally you can [build the image](https://github.com/agnesoft/agdb/blob/main/agdb_server/containerfile){rel=""nofollow""} yourself and host it at the place of your choosing. Please refer to the [server-docker](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-docker) guide for available images. ### Requirements - Kubernetes cluster (you can enable local K8s cluster in Docker Desktop) - kubectl ### Stateful set yaml You can find an example Kubernetes deployment at {rel=""nofollow""} The example breakdown: ::steps ### Service First we deploy the K8s service of type `ClusterIP` that only allows communication inside the cluster. As we are running a database servers it would typically serve other backends in the same cluster and not be accessible from the outside. If such access is needed consider using `LoadBalancer` type service or `Ingress` controller with the `ClusterIp` service. The port available in the cluster is `3000` under a name of `agdb`. Furthermore we specify selector value `app: agdb` and also a label of the same value. ```yaml apiVersion: v1 kind: Service metadata: name: agdb labels: app: agdb spec: ports: - port: 3000 name: agdb clusterIP: None selector: app: agdb ``` :::warning When accessing the database cluster through the service name each request will be sent to random node (the sessions are not sticky) that may lead to inconsistent view into the cluster's data. Consider specifying a pod to connect to. Instead of `http://agdb.default.svc.cluster.local:3000` use for example `http://agdb-0.agdb.default.svc.cluster.local:3000`. ::: ### Secrets Next document is the pepper secret `agdb-pepper`. ```yaml apiVersion: v1 kind: Secret metadata: name: agdb-pepper labels: app: agdb stringData: pepper: "1234567891234567" ``` Followed by the certificates. In production, you should create the certificates using kubectl command from secure files rather than embedding them into the manifest. However, for demonstration purposes the following example is provided. It uses self-signed certificate with the corresponding CA. It needs to be provided as `base64` value. You can use e.g. `cat cert.pem | base64` bash command to get the correct value (end of lines do not matter). :::tip The certificate must be issued (or used as an alternative name - SAN) for the DNS names of the servers used in their configuration `address` field. E.g. `agdb-0.default.svc.cluster.local`, `agdb-1.default.svc.cluster.local`, `agdb-2.default.svc.cluster.local`. ::: ```yaml apiVersion: v1 kind: Secret metadata: name: agdb-certs labels: app: agdb data: cert.pem: ... key.pem: ... root_ca.pem: ... ``` ### ConfigMap The configuration named `agdb-config` via the `ConfigMap` is required as we need specific configuration for each node. Additionally, we specify a custom `start.sh` script that dynamically assigns the deployment index as a cluster index on startup. We are using the same certificate in all servers, and it must be valid for all the names it is used for, e.g. `agdb-0.default.svc.cluster.local`, `agdb-1.default.svc.cluster.local`, `agdb-2.default.svc.cluster.local`. ```yaml --- apiVersion: v1 kind: ConfigMap metadata: name: agdb-config labels: app: agdb data: start.sh: | cp /agdb/config/agdb_server.yaml /agdb/agdb_server.yaml sed -i "s/{id}/$AGDB_REPLICA_INDEX/g" /agdb/agdb_server.yaml /usr/local/bin/agdb_server agdb_server.yaml: | bind: :::3000 address: http://agdb-{id}.agdb.default.svc.cluster.local:3000 basepath: "" static_roots: [] admin: admin log_level: INFO data_dir: /agdb/data pepper_path: /agdb/pepper/pepper tls_certificate: /agdb/certs/cert.pem tls_key: /agdb/certs/key.pem tls_root: /agdb/certs/root_ca.pem cluster_token: cluster cluster_heartbeat_timeout_ms: 1000 cluster_term_timeout_ms: 3000 cluster: [http://agdb-0.agdb.default.svc.cluster.local:3000, http://agdb-1.agdb.default.svc.cluster.local:3000, http://agdb-2.agdb.default.svc.cluster.local:3000] ``` ### StatefulSet The main part of the deployment is the stateful set definition. It uses the selector and labels `app: agdb` in order to "link" the service and the underlying pods together. Kubernetes is using selectors rather than direct mapping when linking various things together such as services and pods. We specify 3 replicas as we want to run 3 node cluster. The container spec matches the port on the service by name `agdb` and exposes the container port `3000` (default). The security context specifies the user `1000` (default uid in the container) and disables root access as it is not needed and enhances security. Finally, we specify volumes and volume mounts to add the `secrets`, `configmap` and persistent volume claim (PVC) to the expected locations. The PVC is a way how data can survive restart or redeployment. By default, 1 GB of storage is specified which can be increased (but not decreased) in subsequent deployments. The custom command running the `start.sh` from the configmap and the environment variable `AGDB_REPLICA_INDEX` make sure the correct config is assigned to each node on startup. Certificates are provided for TLS support. ```yaml --- apiVersion: apps/v1 kind: StatefulSet metadata: name: agdb labels: app: agdb spec: serviceName: "agdb" replicas: 3 selector: matchLabels: app: agdb template: metadata: labels: app: agdb spec: containers: - name: agdb image: agnesoft/agdb:dev command: ["sh", "/agdb/config/start.sh"] ports: - containerPort: 3000 name: agdb securityContext: runAsUser: 1000 runAsGroup: 1000 securityContext: allowPrivilegeEscalation: false env: - name: AGDB_REPLICA_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['apps.kubernetes.io/pod-index'] volumeMounts: - name: agdb-data mountPath: /agdb/data - name: config mountPath: /agdb/config - name: pepper mountPath: /agdb/pepper - name: certs mountPath: /agdb/certs volumes: - name: config configMap: name: agdb-config defaultMode: 511 - name: pepper secret: secretName: agdb-pepper - name: certs secret: secretName: agdb-certs volumeClaimTemplates: - metadata: name: agdb-data labels: app: agdb spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi ``` ### Test that the server is up with `curl` The following command must be run from within the cluster unless the server was exposed via `LoadBalancer` or `Ingress`. The `.default.` bit is the name of the namespace where everything was deployed. ```bash curl -v https://agdb-0.agdb.default.svc.cluster.local:3000/api/v1/status # should return 200 OK curl -v https://agdb-1.agdb.default.svc.cluster.local:3000/api/v1/status # should return 200 OK curl -v https://agdb-2.agdb.default.svc.cluster.local:3000/api/v1/status # should return 200 OK ``` ### Additional considerations - You could use the `localhost:3000/api/v1/status` as a startup/readiness/health probe. - Standard shutdown procedure via the endpoint will not work as K8s will simply restart the servers. - You should be able to adapt the above to other platforms such as AWS EKS. - Always connect to a particular node (e.g. `agdb-0`) rather than the service address since the latter will route the request to a random node. - If you are using `agdb` only inside the K8s cluster with no visibility outside of it you may want to disable TLS. - If you are exposing `agdb` outside the K8s cluster use real production certificate. In that case leave the `tls_root` configuration option empty. :: # How to use studio? TBD # Troubleshooting ## Getting cross-origin (CORS) errors when connecting to the `agdb_server` CORS error can happen even locally when running the server and connecting to it using raw IP address (e.g. `127.0.0.1`). Try running the server and binding it to `localhost` (default) or another DNS name instead. # Guides The following guides are guided examples of usage of the agdb: - [Concepts](https://agdb.agnesoft.com/docs/guides/concepts) - [Quickstart](https://agdb.agnesoft.com/docs/guides/quickstart) - [How to run a server?](https://agdb.agnesoft.com/docs/guides/how-to-run-server) - [How to use the studio?](https://agdb.agnesoft.com/docs/guides/how-to-use-studio) - [Troubleshooting](https://agdb.agnesoft.com/docs/guides/troubleshooting) # Overview The following links lead you to the example code in the `agdb` repository. For the guided examples see [guides](https://agdb.agnesoft.com/docs/guides). - [app\_db](https://github.com/agnesoft/agdb/tree/main/examples/app_db){rel=""nofollow""} - [indexes](https://github.com/agnesoft/agdb/tree/main/examples/indexes){rel=""nofollow""} - [joins](https://github.com/agnesoft/agdb/tree/main/examples/joins){rel=""nofollow""} - [schema migration](https://github.com/agnesoft/agdb/tree/main/examples/schema_migration){rel=""nofollow""} - [server client - rust](https://github.com/agnesoft/agdb/tree/main/examples/server_client_rust){rel=""nofollow""} - [server client - typescript](https://github.com/agnesoft/agdb/tree/main/examples/server_client_typescript){rel=""nofollow""} - [strong types](https://github.com/agnesoft/agdb/tree/main/examples/user_types){rel=""nofollow""} # Overview In this section you will find complete references of all aspects of `agdb` including full reference of queries, and other aspects of `agdb`: - [Queries](https://agdb.agnesoft.com/references/queries) - [Server](https://agdb.agnesoft.com/references/server) - [Studio](https://agdb.agnesoft.com/references/studio) - [Cloud](https://agdb.agnesoft.com/references/cloud) - [Efficient agdb](https://agdb.agnesoft.com/references/efficient-agdb) - [Performance](https://agdb.agnesoft.com/references/performance) # Queries All interactions with the `agdb` are realized through queries. There are two kinds of queries: - Immutable queries - Mutable queries Immutable queries read the data from the database through `select` and `search` queries. Mutable queries write to or delete from the database through `insert` and `remove` queries. All queries follow the Rust rules about borrowing: ::note There can be unlimited number of immutable concurrent queries or exactly one mutable query running against the database. :: The queries are executed against the database by calling the corresponding method on the database object: ```rs impl Db { // immutable queries only pub fn exec(&self, query: &T) -> Result // mutable queries only pub fn exec_mut(&mut self, query: &T) -> Result } ``` Alternatively you can run a series of queries as a [transaction](https://agdb.agnesoft.com/#transactions). All queries return `Result`. The [`QueryResult`](https://agdb.agnesoft.com/#queryresult) is the universal data structure holding results of all queries in a uniform structure. The [`DbError`](https://agdb.agnesoft.com/#dberror) is the singular error type holding information of any failure or problem encountered when running the query. ## Types ### DbType The `DbType` trait is an interface that can be implemented for user defined types so that they can be seamlessly used with the database: ```rs pub trait DbType: Sized { fn db_id(&self) -> Option; fn db_keys() -> Vec; fn from_db_element(element: &DbElement) -> Result; fn to_db_values(&self) -> Vec; fn db_element_id() -> Option { None } } ``` Typically, you would derive this trait with `agdb::DbType` procedural macro that uses the field names as keys (of type `String`) and losslessly converts the values when reading/writing from/to the database from supported types (e.g. field type `i32` will become `i64` in the database). You can also use an alternative derive `agdb::DbElement` that acts like `agdb::DbType` but additionally implements the `db_element_id()` to return the type's name. It is then automatically used as additional property when using it with the `QueryBuilder`. When inserting type that derived `DbElement` an additional property `db_element_id` will be inserted for the given db element. Conversely, when select & searching elements the automatic condition matching the `db_element_id` with the type name is also inserted into the search. It is recommended but optional to have `db_id` field of type `DbId` or `Option>` (e.g. `DbId` or `Option` or `Option`) in your user defined types which will further allow you to directly update your values with query shorthands. However, it is optional, and all other features will still work including conversion from `QueryResult` or passing your types to `values()` in the builders or type matching when deriving from `DbElement`. The `agdb::DbType` macro also supports `Option`al types. When a value is `None` it will be omitted when saving the object to the database. Example: ```rs #[derive(DbType)] struct User { db_id: Option, name: String, } let user = User { db_id: None, name: "Bob".to_string() }; db.exec_mut(QueryBuilder::insert().nodes().values(user).query())?; let mut user: User = db.exec(QueryBuilder::select().values(User::db_keys()).ids(1).query())?.try_into()?; // User { db_id: Some(DbId(1)), name: "Bob" } user.name = "Alice".to_string(); db.exec_mut(QueryBuilder::insert().element(&user).query())?; //updates the user element with new name ``` You can optionally use `#[agdb(flatten)]` to flatten nested structs, `#[agdb(rename = "new_name")]` to disambiguate the duplicate keys (useful when flattening) or `#[agdb(skip)]` to omit a field (requires for the field's type to implement `Default`). In some cases you may want to implement the `DbType` trait yourself if you want to do some additional transformation besides the supported ones by the derive macro. Additionally, you can use these supporting derive macros: ```rs #[derive(DbTypeMarker)] // allows using vectorized custom types, e.g. Vec in fields of user defined types #[derive(DbValue)] // derives an implementation converting a user defined type // to DbValue for easy nesting of user defined types. // NOTE: it additionally requires agdb::AgdbSerialize trait to be implemented #[derive(DbSerialize)] // derives implementation of agdb::AgdbSerialize trait (includes both serialize and deserialize) for user defined types ``` Types not directly used in the database but for which the conversions are supported: - u32 <=> u64 - i32 <=> i64 - f32 <=> f64 - Vec :i32[ <=> Vec :i64] - Vec :u32[ <=> Vec :u64] - Vec :f32[ <=> Vec :f64] - \&str => String (only one way conversion to `String`) - Vec<\&str> => Vec :string[ (only one way conversion to `Vec`)] - bool (\*) \* The boolean type is not a native type in the `agdb`, but you can still use it in your types in any language. The `bool` type will be converted to `u64` (0 == false, 1 == true). The `Vec` type will be converted to `Vec` (bytes, 0 == false, 1 == true). The conversion back to `bool` is possible from wider range of values - the same rules apply for vectorized version which however cannot be converted to from single values: - u64 / i64: any non-zero value will be `true` - f64: any value except `0.0` will be `true` - string: only `"true"` or `"1"` will be `true` ### QueryResult The `QueryResult` is the universal result type for all successful queries. It can be converted to user defined types that implement [`DbType`](https://agdb.agnesoft.com/#dbtype) with `try_into()`. It looks like this: ```rs pub struct QueryResult { pub result: u64, pub elements: Vec, } ``` The `result` field holds numerical result of the query. It typically returns the number of database items affected. For example when selecting from the database it will hold the number of elements returned. When removing from the database it will hold the number of elements deleted from the database. The `from` and `to` fields will hold origin/destination `id` of an edge or first outgoing/incoming edge of a node (or 0 which is the invalid/empty element id). The `elements` field hold the [database elements](https://agdb.agnesoft.com/docs/guides/concepts#graph) returned. Each element looks like: ```rs pub struct DbElement { pub id: DbId, pub from: DbId, pub to: DbId, pub values: Vec, } ``` The `id` (i.e. `pub struct DbId(i64)`) is a numerical identifier of a database element. Positive number means the element is a `node` while negative number means the elements is an `edge`. The value `0` is a special value signifying no valid element and is used when certain queries return data not related to any particular element, e.g. aliases. The values are `key-value` pairs (properties) associated with the given element: ```rs pub struct DbKeyValue { pub key: DbValue, pub value: DbValue, } ``` Where `DbValue` is: ```rs pub enum DbValue { Bytes(Vec), I64(i64), U64(u64), F64(DbF64), String(String), VecI64(Vec), VecU64(Vec), VecF64(Vec), VecString(Vec), } ``` Note the `DbF64` type (i.e. `pub struct DbF64(f64)`) which is a convenient wrapper of `f64` to provide opinionated implementation of some of the operations that are not floating type friendly like comparisons. In `agdb` the float type is using [`total_cmp` standard library function](https://doc.rust-lang.org/std/primitive.f64.html#method.total_cmp){rel=""nofollow""}. Please see its documentation for important details about possible limits or issues on certain platforms. The enum variants can be conveniently accessed through methods named after each variant: ```rs fn bytes(&self) -> Result<&Vec, DbError>; fn to_bool(&self) -> Result; fn to_f64(&self) -> Result; fn to_i64(&self) -> Result; fn to_u64(&self) -> Result; fn to_string(&self) -> String; fn string(&self) -> Result<&String, DbError>; fn vec_f64(&self) -> Result<&Vec, DbError>; fn vec_i64(&self) -> Result<&Vec, DbError>; fn vec_u64(&self) -> Result<&Vec, DbError>; fn vec_string(&self) -> Result<&Vec, DbError>; fn vec_bool(&self) -> Result, DbError>; ``` The numerical variants (`I64`, `U64`, `DbF64`) will attempt loss-less conversions where possible. To avoid copies all other variants return `&` where conversions are not possible even if they could be done in theory. The special case is `to_string()` provided by the `Display` trait. It converts any values into string (it also copies the `String` variant) and performs possibly lossy conversion from `Bytes` to UTF-8 string. For `bool` conversion details refer to [DbType](https://agdb.agnesoft.com/#dbtype) section. ### DbError Failure when running a query is reported through a single `DbError` object which can optionally hold internal error (or chain of errors) that led to the failure. Most commonly it will represent **data error** or **logic error** in your query. Less commonly it may also report a failure to perform the requested operation due to underlying infrastructure issue (e.g. out of memory). It is up to the client code to handle the errors. ### QueryId, QueryIds Most queries operate over a set of database `ids`. The `QueryIds` type is actually an enum: ```rs pub enum QueryIds { Ids(Vec), Search(SearchQuery), } ``` It represents either a set of actual `ids` or a `search` query that will be executed as the larger query and its results fed as `ids` to the larger query. The `QueryId` is defined as another enum: ```rs pub enum QueryId { Id(DbId), Alias(String), } ``` This is because you can refer to the database elements via their numerical identifier or by the `string` alias (name). The `DbId` is then just a wrapper type: `pub struct DbId(pub i64)`. Both `QueryIds` and `QueryId` can be constructed from large number of different types like raw `i64`, `&str`, `String` or vectors of those etc. ### QueryValues The `QueryValues` is an enum type that makes a distinction between singular and multiple values like so: ```rs pub enum QueryValues { Single(Vec), Multi(Vec>), } ``` This is especially important because it can change the meaning of a query making use of this type. For example when inserting elements into the database and supplying `QueryValues::Single` all the elements will have the copy of the single set of properties associated with them. Conversely, `QueryValues::Multi` will initialize each element with a different provided set of properties but the number of inserted elements and the number of property sets must then match (it would be a query logic error if they did not match and the query would fail with such an error). ## Mutable queries Mutable queries are the way to modify the data in the database. Remember there can only be a mutable query running against the database at any one time preventing all other mutable or immutable queries running concurrently. There are two types of mutable queries: - insert - remove The `insert` queries are used for both insert and updating data while `remove` queries are used to delete data from the database. ## Immutable queries Immutable queries read the data from the database and there can be an unlimited number of concurrent queries running against the database at the same time. There are two types of immutable queries: - select - search The `select` queries are used to read the data from the database using known `id`s of elements. The `search` queries are used to find the `id`s and the result of search queries is thus often combined with the `select` queries. ## Transactions You can run a series of queries as a transaction invoking corresponding methods on the database object: ```rs impl Db { // immutable transaction pub fn transaction(&self, mut f: impl FnMut(&Transaction) -> Result) -> Result // mutable transaction pub fn transaction_mut>(&mut self, mut f: impl FnMut(&mut TransactionMut) -> Result) -> Result } ``` The transaction methods take a closure that itself takes a transaction object as an argument. This is to prevent long-lived transactions and force them to be as concise as possible. The transaction objects implement the same execution methods as the `Db` itself (`exec` / `exec_mut`). It is not possible to nest transactions, but you can run immutable queries within a mutable transaction `TransactionMut`. Note that you cannot manually abort, rollback or commit the transaction. These are handled by the database itself based on the result of the closure. If it's `Ok` the transaction will be committed (in case of the `mutable` queries as there is nothing to commit for `immutable` queries). If the result is `Err` the transaction will be rolled back. In both cases the result will be returned and the signature of the transaction methods allows for custom mapping of the default `Result` to an arbitrary `` result-error pair. Worth noting is that regular `exec / exec_mut` methods on the `Db` object are actually implemented as transactions. ## Insert There are 5 distinct insert queries: - insert aliases - insert edges - insert nodes - insert index - insert values ### Insert aliases | **Struct** | **Result** | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | ```rs pub struct InsertAliasesQuery { pub ids: QueryIds, pub aliases: Vec, } ``` | ```rs pub struct QueryResult { pub result: u64, // number of inserted/updated aliases pub elements: Vec, // empty } ``` | | **Builder** | | | ```rs QueryBuilder::insert().aliases("a").ids(1).query(); QueryBuilder::insert().aliases("a").ids("b").query(); // alias "b" is replaced with "a" QueryBuilder::insert().aliases(["a", "b"]).ids([1, 2]).query(); ``` | | Inserts or updates aliases of existing nodes (and only nodes, edges cannot have aliases) through this query. It takes `ids` [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) and list of `aliases` as arguments. The number of aliases must match the `ids` (even if they are a search query). Empty alias (`""`) are not allowed. Note that this query is also used for updating existing aliases. By inserting a different alias of an `id` that already has one that alias will be overwritten with the new one. ### Insert edges | **Struct** | **Result** | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ```rs pub struct InsertEdgesQuery { pub from: QueryIds, pub to: QueryIds, pub ids: QueryIds, pub values: QueryValues, pub each: bool, } ``` | ```rs pub struct QueryResult { pub result: u64, // number of inserted edges pub elements: Vec, // list of inserted edges (only ids) } ``` | | **Builder** | | | ```rs QueryBuilder::insert().edges().from(1).to(2).query(); QueryBuilder::insert().edges().from("a").to("b").query(); QueryBuilder::insert().edges().from("a").to([1, 2]).query(); QueryBuilder::insert().edges().from([1, 2]).to([2, 3]).query(); QueryBuilder::insert().edges().from([1, 2]).to([2, 3]).each().query(); QueryBuilder::insert().edges().from("a").to([1, 2]).values([[("k", 1).into()], [("k", 2).into()]]).query(); QueryBuilder::insert().edges().from("a").to([1, 2]).values_uniform([("k", "v").into(), (1, 10).into()]).query(); QueryBuilder::insert().edges().from(QueryBuilder::search().from("a").where_().node().query()).to(QueryBuilder::search().from("b").where_().node().query()).query(); QueryBuilder::insert().edges().from(QueryBuilder::search().from("a").where_().node().query()).to(QueryBuilder::search().from("b").where_().node().query()).values([[("k", 1).into()], [("k", 2).into()]]).query(); QueryBuilder::insert().edges().from(QueryBuilder::search().from("a").where_().node().query()).to(QueryBuilder::search().from("b").where_().node().query()).values_uniform([("k", "v").into(), (1, 10).into()]).query(); QueryBuilder::insert().edges().ids(-3).from(1).to(2).query(); QueryBuilder::insert().edges().ids([-3, -4]).from(1).to(2).query(); QueryBuilder::insert().edges().ids(QueryBuilder::search().from(1).where_().edge().query()).from(1).to(2).query(); ``` | | The `from` and `to` represents list of origins and destinations of the edges to be inserted. As per [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) it can be a list, single value, search query or even a result of another query (e.g. [insert nodes](https://agdb.agnesoft.com/#insert-nodes)) through the call of convenient `QueryResult::ids()` method. All `ids` must be `node`s and all must exist in the database otherwise data error will occur. If the `values` is [`QueryValues::Single`](https://agdb.agnesoft.com/#queryvalues) all edges will be associated with the copy of the same properties. If `values` is [`QueryValues::Multi`](https://agdb.agnesoft.com/#queryvalues) then the number of edges being inserted must match the provided values otherwise a logic error will occur. By default, the `from` and `to` are expected to be of equal length specifying at each index the pair of nodes to connect with an edge. If all-to-all is desired set the `each` flag to `true`. The rule about the `values` [`QueryValues::Multi`](https://agdb.agnesoft.com/#queryvalues) still applies though so there must be enough values for all nodes resulting from the combination. The values can be inferred from user defined types if they implement `DbType` trait (`#derive(agdb::DbType)`). Both singular and vectorized versions are supported. Optionally one can specify `ids` that facilitates insert-or-update semantics. The field can be a search sub-query. If the resulting list in `ids` is empty the query will insert edges as normal. If the list is not empty all `ids` must exist and refer to existing edges and the query will perform update of values instead. Note: the specified from/to (origin/destination) for the updated edges is not checked against those supplied via `ids`. ### Insert index | **Struct** | **Result** | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct InsertIndexQuery(pub DbValue); ``` | ```rs pub struct QueryResult { pub result: u64, // number of indexed values pub elements: Vec, // empty } ``` | | **Builder** | | | ```rs QueryBuilder::insert().index("key").query(); ``` | | Creates an index for a key. The index is valid for the entire database including any and all existing values in the database. The purpose of the index is to provide faster lookup for data that is not modelled on the graph itself. Example can be looking up users by their username or token. ### Insert nodes | **Struct** | **Result** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ```rs pub struct InsertNodesQuery { pub count: u64, pub values: QueryValues, pub aliases: Vec, pub ids: QueryIds, } ``` | ```rs pub struct QueryResult { pub result: u64, // number of inserted nodes pub elements: Vec, // list of inserted nodes (only ids) } ``` | | **Builder** | | | ```rs QueryBuilder::insert().nodes().count(2).query(); QueryBuilder::insert().nodes().count(2).values_uniform([("k", "v").into(), (1, 10).into()]).query(); QueryBuilder::insert().nodes().aliases(["a", "b"]).query(); QueryBuilder::insert().nodes().aliases(["a", "b"]).values([[("k", 1).into()], [("k", 2).into()]]).query(); QueryBuilder::insert().nodes().aliases(["a", "b"]).values_uniform([("k", "v").into(), (1, 10).into()]).query(); QueryBuilder::insert().nodes().values([[("k", 1).into()], [("k", 2).into()]]).query(); QueryBuilder::insert().nodes().ids(1).count(1).query(); QueryBuilder::insert().nodes().ids([1, 2]).count(1).query(); QueryBuilder::insert().nodes().ids("a").count(1).query(); QueryBuilder::insert().nodes().ids("a").aliases("a").query(), QueryBuilder::insert().nodes().ids(["a", "b"]).count(1).query(); QueryBuilder::insert().nodes().ids([1, 2]).values([[("k", "v").into()], [(1, 10).into()]]).query(), QueryBuilder::insert().nodes().ids([1, 2]).values_uniform([("k", "v").into(), (1, 10).into()]).query(), QueryBuilder::insert().nodes().ids(QueryBuilder::search().from(1).query()).count(1).query(); ``` | | The `count` is the number of nodes to be inserted into the database. It can be omitted (left `0`) if either `values` or `aliases` (or both) are provided. If the `values` is [`QueryValues::Single`](https://agdb.agnesoft.com/#queryvalues) you must provide either `count` or `aliases`. It is not an error if the count is set to `0`, but the query will be a no-op and return empty result. If both `values` [`QueryValues::Multi`](https://agdb.agnesoft.com/#queryvalues) and `aliases` are provided their lengths must be compatible (aliases <= values), otherwise it will result in a logic error. Empty aliases (`""`) are not allowed. The values can be inferred from user defined types if they implement `DbType` trait (`#derive(agdb::DbType)`). Both singular and vectorized versions are supported. Optionally one can specify `ids` that facilitates insert-or-update semantics. The field can be a search sub-query. If the resulting list in `ids` is empty the query will insert nodes as normal. If the list is not empty all `ids` must exist and must refer to nodes and the query will perform update instead - both aliases (replacing existing ones if applicable) and values. If an alias already exists in the database its values will be amended (inserted or replaced) with the provided values. ### Insert values | **Struct** | **Result** | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct InsertValuesQuery { pub ids: QueryIds, pub values: QueryValues, } ``` | ```rs pub struct QueryResult { pub result: u64, // number of inserted key-value pairs pub elements: Vec, // list of new elements } ``` | | **Builder** | | | ```rs QueryBuilder::insert().element(&T { ... }).query(); //Where T: DbType (i.e. #derive(agdb::DbType)) QueryBuilder::insert().elements(&vec![T {...}, T {...}]).query(); //Where T: DbType (i.e. #derive(agdb::DbType)) QueryBuilder::insert().values([vec![("k", "v").into(), (1, 10).into()], vec![("k", 2).into()]]).ids([1, 2]).query(); QueryBuilder::insert().values([vec![("k", "v").into(), (1, 10).into()], vec![("k", 2).into()]]).ids(QueryBuilder::search().from("a").query()).query(); QueryBuilder::insert().values([vec![("k", "v").into(), (1, 10).into()], vec![("k", 2).into()]]).search().from("a").query(); //Equivalent to the previous query QueryBuilder::insert().values_uniform([("k", "v").into(), (1, 10).into()]).ids([1, 2]).query(); QueryBuilder::insert().values_uniform([("k", "v").into(), (1, 10).into()]).ids(QueryBuilder::search().from("a").query()).query(); QueryBuilder::insert().values_uniform([("k", "v").into(), (1, 10).into()]).search().from("a").query(); //Equivalent to the previous query ``` | | Inserts or updates key-value pairs (properties) of existing elements or insert new elements (nodes). You need to specify the `ids` [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) and the list of `values`. The `values` can be either [`QueryValues::Single`](https://agdb.agnesoft.com/#queryvalues) that will insert the single set of properties to all elements identified by `ids` or [`QueryValues::Multi`](https://agdb.agnesoft.com/#queryvalues) that will insert to each `id` its own set of properties, but their number must match the number of `ids`. If the user defined type contains `db_id` field of type `Option>` you can use the shorthand `insert().element() / .insert().elements()` that will infer the values and `ids` from your types. The `values()` will be inferred from user defined types if they implement `DbType` trait (`#derive(agdb::DbType)`). Both singular and vectorized versions are supported. - If an `id` is non-0 or an existing alias that element will be updated in the database with provided values. - If an `id` is `0` or a non-existent alias new element (node) will be inserted into the database with that alias. Note: that this query is insert-or-update for both nodes and existing values. By inserting the same `key` its old value will be overwritten with the new one. ## Remove There are 4 distinct remove queries: - remove aliases - remove (elements) - remove index - remove values ### Remove aliases | **Struct** | **Result** | | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct RemoveAliasesQuery(pub Vec); ``` | ```rs pub struct QueryResult { pub result: u64, // number of removed aliases pub elements: Vec, // empty } ``` | | **Builder** | | | ```rs QueryBuilder::remove().aliases("a").query(); QueryBuilder::remove().aliases(["a", "b"]).query(); ``` | | The aliases listed will be removed from the database if they exist. It is NOT an error if the aliases do not exist in the database. ### Remove elements | **Struct** | **Result** | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct RemoveQuery(pub QueryIds); ``` | ```rs pub struct QueryResult { pub result: u64, // number of removed ids // (does not include removed edges // unless listed in query ids) pub elements: Vec, // empty } ``` | | **Builder** | | | ```rs QueryBuilder::remove().ids(1).query(); QueryBuilder::remove().ids("a").query(); QueryBuilder::remove().ids([1, 2]).query(); QueryBuilder::remove().ids(["a", "b"]).query(); QueryBuilder::remove().ids(QueryBuilder::search().from("a").query()).query(); QueryBuilder::remove().search().from("a").query(); //Equivalent to the previous query ``` | | The elements identified by [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) will be removed from the database if they exist. It is NOT an error if the elements to be removed do not exist in the database. All associated properties (key-value pairs) are also removed from all elements. Removing nodes will also remove all their edges (incoming and outgoing) and their properties. ### Remove index | **Struct** | **Result** | | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct RemoveIndexQuery(pub DbValue); ``` | ```rs pub struct QueryResult { pub result: u64, // number of values removed // from the index pub elements: Vec, // empty } ``` | | **Builder** | | | ```rs QueryBuilder::remove().index("key").query(); ``` | | Removes an index from the database. It is NOT an error if the index does not exist in the database. ### Remove values | **Struct** | **Result** | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct RemoveValuesQuery(pub SelectValuesQuery); ``` | ```rs pub struct QueryResult { pub result: u64, // number of actually removed // key-value pairs pub elements: Vec, // empty } ``` | | **Builder** | | | ```rs QueryBuilder::remove().values(["k1".into(), "k2".into()]).ids([1, 2]).query(); QueryBuilder::remove().values(["k1".into(), "k2".into()]).ids(QueryBuilder::search().from("a").query()).query(); QueryBuilder::remove().values(["k1".into(), "k2".into()]).search().from("a").query(); //Equivalent to the previous query ``` | | NOTE: See [`SelectValuesQuery`](https://agdb.agnesoft.com/#select-values) for more details. The properties (key-value pairs) identified by `keys` and associated with `ids` [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) will be removed from the database if they exist. It is an error if any of the `ids` do not exist in the database, but it is NOT an error if any of the keys does not exist or is not associated as property to any of the `ids`. ## Select There are following select queries: - select aliases - select all aliases - select edge count - select (elements) - select indexes - select keys - select key count - select values ### Select aliases | **Struct** | **Result** | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct SelectAliasesQuery(pub QueryIds); ``` | ```rs pub struct QueryResult { pub result: u64, // number of returned elements pub elements: Vec, // list of elements each with // a single property // (`String("alias")`: `String`) } ``` | | **Builder** | | | ```rs QueryBuilder::select().aliases().ids([1, 2]).query(); QueryBuilder::select().aliases().ids(QueryBuilder::search().from(1).query()).query(); QueryBuilder::select().aliases().search().from(1).query(); //Equivalent to the previous query ``` | | Selects aliases of the `ids` [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) or a search. If any of the `ids` does not have an alias running the query will return an error. ### Select all aliases | **Struct** | **Result** | | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct SelectAllAliasesQuery {} ``` | ```rs pub struct QueryResult { pub result: u64, // number of elements with aliases pub elements: Vec, // list of elements with an // alias each with a single // property (`String("alias"): String`) } ``` | | **Builder** | | | ```rs QueryBuilder::select().aliases().query(); ``` | | Selects all aliases in the database. ### Select edge count | **Struct** | **Result** | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct SelectEdgeCountQuery { pub ids: Ids, pub from: bool, pub to: bool } ``` | ```rs pub struct QueryResult { pub result: u64, // Sum of all edge_counts in all selected elements pub elements: Vec, // list of elements with an // alias each with a single // property (`String("edge_count"): String`) } ``` | | **Builder** | | | ```rs QueryBuilder::select().edge_count().ids([1, 2]).query(); QueryBuilder::select().edge_count_from().ids([1, 2]).query(); QueryBuilder::select().edge_count_to().ids([1, 2]).query(); QueryBuilder::select().edge_count().ids(QueryBuilder::search().from("a").query()).query(); QueryBuilder::select().edge_count().search().from("a").query(); // Equivalent to the previous query ``` | | Selects count of edges of nodes (`ids`). The `edge_count` variant counts all edges (outgoing & incoming). The `edge_count_from` counts only outgoing edges. The `edge_count_to` counts only incoming edges. NOTE: Self-referential edges (going from the same node to the same node) will be counted twice in the first variant (`edge_count`) as the query counts outgoing/incoming edges rather than unique database elements. As a result the `edge_count` result may be higher than the actual number of physical edges in such a case. ### Select indexes | **Struct** | **Result** | | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct SelectIndexesQuery {}; ``` | ```rs pub struct QueryResult { pub result: u64, // number of indexes in the database pub elements: Vec, // single element with id 0 and list of // properties representing each index // (`DbValue`: `u64`) where the key is // the indexed key and the value is number // of indexed values in the index. } ``` | | **Builder** | | | ```rs QueryBuilder::select().indexes().query(); ``` | | Selects all indexes in the database. ### Select keys | **Struct** | **Result** | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ```rs pub struct SelectKeysQuery(pub QueryIds); ``` | ```rs pub struct QueryResult { pub result: u64, // number of returned elements pub elements: Vec, // list of elements with only keys // defaulted values will be `I64(0)` } ``` | | **Builder** | | | ```rs QueryBuilder::select().keys().ids("a").query(); QueryBuilder::select().keys().ids([1, 2]).query(); QueryBuilder::select().keys().ids(QueryBuilder::search().from(1).query()).query(); QueryBuilder::select().keys().search().from(1).query(); // Equivalent to the previous query ``` | | Selects elements identified by `ids` [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) or search query with only keys returned. If any of the `ids` does not exist in the database running the query will return an error. This query is most commonly used for establishing what data is available in on the graph elements (e.g. when transforming the data into a table this query could be used to populate the column names). ### Select key count | **Struct** | **Result** | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct SelectKeyCountQuery(pub QueryIds); ``` | ```rs pub struct QueryResult { pub result: u64, // sum value of all key_counts from all selected elements pub elements: Vec, // list of elements each with a // single property // (`String("key_count")`: `u64`) } ``` | | **Builder** | | | ```rs QueryBuilder::select().key_count().ids("a").query(); QueryBuilder::select().key_count().ids([1, 2]).query(); QueryBuilder::select().key_count().ids(QueryBuilder::search().from(1).query()).query(); QueryBuilder::select().key_count().search().from(1).query(); // Equivalent to the previous query ``` | | Selects elements identified by `ids` [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) or search query with only key count returned. If any of the `ids` does not exist in the database running the query will return an error. This query is most commonly used for establishing how many properties there are associated with the graph elements. ### Select node count | **Struct** | **Result** | | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct SelectNodeCountQuery {} ``` | ```rs pub struct QueryResult { pub result: u64, // Count of nodes in the database pub elements: Vec, // empty list of elements } ``` | | **Builder** | | | ```rs QueryBuilder::select().node_count().query(); ``` | | Selects number (count) of nodes in the database. ### Select values | **Struct** | **Result** | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct SelectValuesQuery { pub keys: Vec, pub ids: QueryIds, } ``` | ```rs pub struct QueryResult { pub result: u64, // number of returned elements pub elements: Vec, // list of elements with only // selected properties } ``` | | **Builder** | | | ```rs QueryBuilder::select().ids("a").query(); QueryBuilder::select().ids([1, 2]).query(); QueryBuilder::select().ids(QueryBuilder::search().from(1).query()).query(); QueryBuilder::select().search().from(1).query(); // Equivalent to the previous query QueryBuilder::select().values(["k".into(), "k2".into()]).ids("a").query(); QueryBuilder::select().values(["k".into(), "k2".into()]).ids([1, 2]).query(); QueryBuilder::select().values(["k".into(), "k2".into()]).ids(QueryBuilder::search().from(1).query()).query(); QueryBuilder::select().values(["k".into(), "k2".into()]).search().from(1).query(); // Equivalent to the previous query QueryBuilder::select().element::(); //if followed by search() it will set limit to 1 QueryBuilder::select().elements::().ids(1).query(); QueryBuilder::select().elements::().ids(QueryBuilder::search().from("a").query()).query(); QueryBuilder::select().elements::().search().from("a").query(); // Equivalent to the previous query ``` | | Selects elements identified by `ids` [`QueryIds`](https://agdb.agnesoft.com/#queryids--queryid) or search query with only selected properties (identified by the list of keys). If any of the `ids` does not exist in the database or does not have all the keys associated with it then running the query will return an error. The search query is most commonly used to find, filter or otherwise limit what elements to select. You can limit what properties will be returned. If the list of properties to select is empty all properties will be returned. If you plan to convert the result into your user defined type(s) you should use either `elements::()` variant or supply the list of keys to `values()` with `T::db_keys()` provided through the `DbType` trait (`#[derive(agdb::DbType)]` or `#[derive(agdb::DbElement)]`) as argument to `values()`. If `T` derives from `DbElement` or otherwise implements `DbType::db_element_id()` the "select().search()" query will have automatic condition added, i.e. as if written by hand `key("db_element_id").value("T")` to simplify working with types, especially if they are overlapping. You can also often omit conditions altogether in some cases. ## Search | **Struct** | **Result** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ```rs pub struct SearchQuery { pub algorithm: SearchQueryAlgorithm, pub origin: QueryId, pub destination: QueryId, pub limit: u64, pub offset: u64, pub order_by: Vec, pub conditions: Vec, } ``` | ```rs pub struct QueryResult { pub result: u64, // number of elements found pub elements: Vec, // list of elements found (only ids) } ``` | | ```rs pub enum SearchQueryAlgorithm { BreadthFirst, DepthFirst, Index, Elements } pub enum DbKeyOrder { Asc(DbValue), Desc(DbValue), } ``` | | | **Builder** | | | ```rs QueryBuilder::search().from("a").query(); QueryBuilder::search().to(1).query(); //reverse search QueryBuilder::search().from("a").to("b").query(); //path search using A* algorithm QueryBuilder::search().breadth_first().from("a").query(); //breadth first is the default and can be omitted QueryBuilder::search().depth_first().from("a").query(); QueryBuilder::search().elements().query(); QueryBuilder::search().index("age").value(20).query(); //index search //limit, offset and order_by can be applied similarly to all the search variants except search index QueryBuilder::search().from(1).order_by([DbKeyOrder::Desc("age".into()), DbKeyOrder::Asc("name".into())]).query() QueryBuilder::search().from(1).offset(10).query(); QueryBuilder::search().from(1).limit(5).query(); QueryBuilder::search().from(1).order_by([DbKeyOrder::Desc("k".into())]).offset(10).query(); QueryBuilder::search().from(1).order_by([DbKeyOrder::Desc("k".into())]).limit(5).query(); QueryBuilder::search().from(1).order_by([DbKeyOrder::Desc("k".into())]).offset(10).limit(5).query(); QueryBuilder::search().from(1).offset(10).limit(5).query(); ``` | | There is only a single search query that provides the ability to search the graph or indexes. When searching the graph it examines connected elements and their properties. While it is possible to construct the search queries manually, specifying them that way can be excessively difficult and therefore **using the builder pattern is recommended**. The default search algorithm is `breadth first` however you can choose to use `depth first`. For path search the `A*` algorithm is used. For searching an index the algorithm is `index`. For searching disregarding the graph structure and indexes (full search) the algorithm is `elements`. Elements will never be examined twice during any search regardless of any cycles in the graph. Very often you would want the values / elements to be returned from the search query. To accomplish it you need to nest the search query in the select query with either `.search()` builder element or `ids()` step that takes a `SearchQuery` as argument. That fetches the data as the search query only traverses the graph. E.g. `QueryBuilder::select().search().from("alias").query()`. Refer to the [Select Values](https://agdb.agnesoft.com/#select-values) query for details. If the index search is done the graph traversal is skipped entirely as are most of the parameters including like limit, offset, ordering and conditions. The graph search query is made up of the `origin` and `destination` of the search and the algorithm. Specifying only `origin` (from) will result in a search along `from->to` edges. Specifying only `destination` (to) will result in the reverse search along the `to<-from` edges. When both `origin` and `destination` are specified the search algorithm becomes a path search and the algorithm used will be `A*`. Optionally you can specify a `limit` (0 = unlimited) and `offset` (0 = no offset) to the returned list of graph element `ids`. If specified (!= 0) the `origin` and the `destination` must exist in the database, otherwise an error will be returned. The elements can be optionally ordered with `order_by` list of keys allowing ascending/descending ordering based on multiple properties. When searching `elements` the database is being scanned in linearly one element (node & edge) at a time which can be very slow. Consider using `limit` in this case. However, this search can be useful in exploration, when the database structure is not known, when searching for abandoned/lost elements and other edge cases not covered by regular search algorithms. The default order of returned elements is from the lowest internal db `id` to the highest which does not necessarily indicate age of the elements as the `ids` can be reused when elements are deleted. Finally, the list of `conditions` that each examined graph element must satisfy to be included in the result (and subjected to the `limit` and `offset`). **NOTE:** When both `origin` and `destination` are specified, and the algorithm is switched to the `A*` the `limit` and `offset` are applied differently. In regular (open-ended) search the search will end when the `limit` is reached but with the path search (A\*) the `destination` must be reached first before they are applied. ### Breadth First The `breadth first` algorithm (the default one) examines every element on each level before moving to the next level. For instance starting at a node this algorithm will first examine all the edges in the selected direction (from/to) before examining the adjacent nodes reachable through those edges. The order of the elements is **from newest to oldest** where newest means most recently connected. Similarly, the next level is also examined in the same order. Example: Given a graph of 6 nodes connected together with 4 edges like so (NOTE: `ids` are for illustration only and does NOT indicate newer/older element): | Level 0 | Level 1 | Level 2 | Level 3 | Level 4 | | -------- | -------- | -------- | -------- | -------- | | Node (a) | Edge (b) | Node (d) | Edge (f) | Node (h) | | | Edge (c) | Node (e) | Edge (g) | Node (j) | The `breadth first` algorithm will first visit node (a) at level 0. Then it will visit all edges at level 1 starting with the newest edge (c) followed by the older edge (b). Then it will move on to the level 2 once more examining the newest node first (g) followed by (f). Lastly it will move on to level 4 examining nodes (j) and (h). The "newest" means most recently connected and does not necessarily mean it will have higher `id` because `ids` can be reused from deleted elements. ### Depth First The `depth first` algorithm follows every element to the next level first. When it cannot continue on to a next level it will step back to the previous level trying another direction if possible. When exhausted or not available it will backtrack again to the previous level and continue from there. The order of the elements is **from newest to oldest** where newest means most recently connected. Example: Given a graph of 6 nodes connected together with 4 edges like so (NOTE: `ids` are for illustration only and does NOT indicate newer/older element): | Level 0 | Level 1 | Level 2 | Level 3 | Level 4 | | -------- | -------- | -------- | -------- | -------- | | Node (a) | Edge (b) | Node (d) | | | | | Edge (c) | Node (e) | Edge (f) | Node (h) | | | | | Edge (g) | Node (j) | The `depth first` algorithm will first visit node (a) at level 0. Then it will visit the most recent edge (c) on level 1. Then it will follow it to its connected node (e) at level 2, then edge (g) at level 3 and finally node (j) at level 4. Since it cannot continue it will step back to level 3 and examine the edge (f) and follow it to node (h) at level 4. After that it will step back to level 3 and see nothing available so it will backtrack further to level 1 to examine edge (b) and follow it to node (d). That will conclude the search. NOTE: when a graph contains multiple edges leading to the same elements the extra edges will appear seemingly "out of order" in the search result (i.e. at the end). This is because no element can be visited twice yet the DFS algorithm will eventually backtrack and attempt to go in their direction possibly including them in the result. Typically, you might want to filter out all edges with `.where_().node()` condition. ### Paths Path search (`from().to()`) uses A\* algorithm. Every element (node or edge) has a cost of `1` by default. If it passes all the conditions (the `SearchControl` value `true`) the cost will remain `1` and would be included in the result (if the path it is on would be selected). If it fails any of the conditions (the `SearchControl` value `false`) its cost will be `2`. This means that the algorithm will prefer paths where elements match the conditions rather than the absolutely shortest path (that can be achieved with no conditions). If the search is not to continue beyond certain element (through `beyond()`, `not_beyond()` or `distance()` conditions) its cost will be `0` and the paths it is on will no longer be considered for that search. ### Conditions | **Struct** | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ```rs pub struct QueryCondition { pub logic: QueryConditionLogic, pub modifier: QueryConditionModifier, pub data: QueryConditionData, } pub enum QueryConditionLogic { And, Or, } pub enum QueryConditionModifier { None, Beyond, Not, NotBeyond, } pub enum QueryConditionData { Distance(CountComparison), Edge, EdgeCount(CountComparison), EdgeCountFrom(CountComparison), EdgeCountTo(CountComparison), Ids(Vec), KeyValue { key: DbValue, value: Comparison }, Keys(Vec), Node, Where(Vec), } pub enum CountComparison { Equal(u64), GreaterThan(u64), GreaterThanOrEqual(u64), LessThan(u64), LessThanOrEqual(u64), NotEqual(u64), } pub enum Comparison { Equal(DbValue), GreaterThan(DbValue), GreaterThanOrEqual(DbValue), LessThan(DbValue), LessThanOrEqual(DbValue), NotEqual(DbValue), Contains(DbValue), StartsWith(DbValue), EndsWith(DbValue), } ``` | | **Builder** | | ```rs //the where_() can be applied to any of the basic search queries after order_by/offset/limit //not() and not_beyond() can be applied to all conditions including nested where_() QueryBuilder::search().from(1).where_().distance(CountComparison::LessThan(3)).query(); QueryBuilder::search().from(1).where_().edge().query(); QueryBuilder::search().from(1).where_().edge_count(CountComparison::GreaterThan(2)).query(); QueryBuilder::search().from(1).where_().edge_count_from(1).query(); QueryBuilder::search().from(1).where_().edge_count_to(CountComparison::NotEqual(1)).query(); QueryBuilder::search().from(1).where_().node().query(); QueryBuilder::search().from(1).where_().key("k").value(1).query(); QueryBuilder::search().from(1).where_().keys(vec!["k1".into(), "k2".into()]).query(); QueryBuilder::search().from(1).where_().not().keys(vec!["k1".into(), "k2".into()]).query(); QueryBuilder::search().from(1).where_().ids([1, 2]).query(); QueryBuilder::search().from(1).where_().beyond().keys(vec!["k".into()]).query(); QueryBuilder::search().from(1).where_().not().ids([1, 2]).query(); QueryBuilder::search().from(1).where_().not_beyond().ids("a").query(); QueryBuilder::search().from(1).where_().node().or().edge().query(); QueryBuilder::search().from(1).where_().node().and().distance(CountComparison::GreaterThanOrEqual(3)).query(); QueryBuilder::search().from(1).where_().node().or().where_().edge().and().key("k").value(1).end_where().query(); QueryBuilder::search().from(1).where_().node().or().where_().edge().and().key("k").value(Comparison::Contains(1.into())).end_where().query(); QueryBuilder::search().from(1).where_().node().or().where_().edge().and().key("k").value(Comparison::Contains(vec![1, 2].into())).end_where().query(); ``` | The currently supported conditions are: - Where (opens nested list of conditions) - Edge (if the element is an `edge`) - Node (if the element is a `node`) - Distance (if the current distance of the search satisfies the numerical comparison, each graph element away from the start increases the distance, including edges, i.e. second node from start is at distance `2`) - EdgeCount (if the element is a node and total number of edges (in and out) satisfies the numerical comparison - self-referential edges are counted twice) - EdgeCountFrom (if the element is a node and total number of outgoing edges satisfies the numerical comparison) - EdgeCountTo (if the element is a node and total number of incoming edges satisfies the numerical comparison) - Ids (if the element `id` is in the list) - KeyValue (if the element's property has the `key` and its value satisfies `value` comparison) - Keys (if the element has all the `keys` regardless of their values) - EndWhere (closes nested list of conditions) All conditions can be further modified as follows: - Beyond (controls traversal only: continues the search only beyond this element if the condition passes, does not affect element selection) - Not (reverses the condition selection result) - NotBeyond (controls traversal only: stops the search beyond this element if the condition passes, does not affect element selection) The conditions can be changed with logic operators: - And (logical `and`) - Or (logical `or`) NOTE: The use of `where_` with an underscore as the method name is necessary to avoid conflict with the Rust keyword. The conditions are applied one at a time to each visited element and chained using logic operators `AND` and `OR`. They can be nested using `where_` and `end_where` (in place of brackets). The condition evaluator supports short-circuiting not evaluating conditions further if the logical outcome cannot change. The condition comparators are type strict meaning that they do not perform type conversions nor coercion (e.g. `Comparison::Equal(1_i64).compare(1_u64)` will evaluate to `false`). Slight exception to this rule is the `Comparison::Contains` as it allows vectorized version of the base type (e.g. `Comparison::Contains(vec!["bc", "ef"]).compare("abcdefg")` will evaluate to `true`). Similarly, `Comparison::StartsWith` and `Comparison::EndsWith` are provided with the same semantics as `Comparison::Contains` matching only from the beginning or end respectively (both single value and vectorized and vice versa). The condition `Distance` and the condition modifiers `Beyond` and `NotBeyond` are particularly important because they can directly influence the search. The former (`Distance`) can limit the depth of the search and can help with constructing more elaborate queries (or sequence thereof) extracting only fine-grained elements (e.g. nodes whose edges have particular properties or are connected to other nodes with some properties). The latter (`Beyond` and `NotBeyond`) can limit search to only certain areas of an otherwise larger graph, but they only control traversal (whether the search continues or stops at an element) and do not select or reject elements on their own. All visited elements are selected by default with `(not)_beyond` conditions. To control selection of elements, combine them with additional conditions. Their most basic usage would be with condition `ids` to flat out stop the search at certain elements or continue only beyond certain elements. ### Truth tables The following information should help with reasoning about the query conditions. Most of it should be intuitive, but there are some aspects that might not be obvious especially when combining logic operators and condition modifiers. The search is using the following `enum` when evaluating conditions: ```rs pub enum SearchControl { Continue(bool), Finish(bool), Stop(bool), } ``` The type controls the search and the boolean value controls if the given element should be included in the search result. The `Stop` will prevent the search expanding beyond current element (stopping the search in that direction). `Finish` will immediately exit the search returning accumulated elements (`ids`) and is only used internally with `offset` and `limit` (NOTE: path search and `order_by` still require complete search regardless of `limit`). Each condition contributes to the final control result as follows with the starting/default value being always `Continue(true)`: #### And | Left | Right | Result | | -------------- | --------------- | ----------------------- | | Continue(left) | Continue(right) | Continue(left && right) | | Continue(left) | Stop(right) | Stop(left && right) | | Continue(left) | Finish(right) | Finish(left && right) | | Stop(left) | Stop(right) | Stop(left && right) | | Stop(left) | Finish(right) | Finish(left && right) | | Finish(left) | Finish(right) | Finish(left && right) | #### Or | Left | Right | Result | | -------------- | --------------- | ------------------------- | | Continue(left) | Continue(right) | Continue(left \|\| right) | | Continue(left) | Stop(right) | Continue(left \|\| right) | | Continue(left) | Finish(right) | Continue(left \|\| right) | | Stop(left) | Stop(right) | Stop(left \|\| right) | | Stop(left) | Finish(right) | Stop(left \|\| right) | | Finish(left) | Finish(right) | Finish(left \|\| right) | #### Modifiers Modifiers will change the result of a condition based on the control value (the boolean) as follows: | Modifier | TRUE | FALSE | | --------- | ------------------- | ---------------- | | None | - | - | | Beyond | `&& Continue(true)` | `Stop(true)` | | Not | `!` | `!` | | NotBeyond | `&& Stop(true)` | `Continue(true)` | #### Results Most conditions result in `Continue(bool)` except for `distance()` and nested `where()` which can also result in `Stop(bool)`: | Condition | Continue | Stop | | ----------- | -------- | ---- | | Where | YES | YES | | Edge | YES | NO | | Node | YES | NO | | Distance | YES | YES | | EdgeCount\* | YES | NO | | Ids | YES | NO | | Key(Value) | YES | NO | | Keys | YES | NO | --- For further examples and use cases see the [efficient agdb](https://agdb.agnesoft.com/docs/references/efficient-agdb). # Server The `agdb_server` is the OpenAPI REST server that provides remote `agdb` database management. Running the server is trivial as there are no dependencies, no complicated configuration etc. It can be run on any platform supported by Rust. Please follow the guide: [How to run the server?](https://agdb.agnesoft.com/docs/guides/how-to-run-server) The server is based on [`axum`](https://github.com/tokio-rs/axum){rel=""nofollow""} and uses OpenAPI to specify its API (via [`utoipa`](https://github.com/juhaku/utoipa){rel=""nofollow""}) and [`rapidoc`](https://rapidocweb.com/){rel=""nofollow""} for the OpenAPI GUI. To interact with the server you can use the rapidoc GUI, `curl` or any of the [available API clients](https://agdb.agnesoft.com/api-docs/openapi). Internally it uses the `agdb` database: GUI accessible at (run in a browser when the server is running): ```bash http://localhost:3000/api/v1 ``` ## Configuration The server will create default configuration when run and always reads it from the working directory if it exists: ```yaml # agdb_server.yaml bind: ":::3000" # address to listen at (bind to) address: "http://localhost:3000" # address the incoming connections will come from basepath: "" # base path to append to the address in case the server is to be run behind a reverse proxy static_roots: [] # list of static folders to serve in format : (e.g. /static:/home/user/www) admin: admin # the admin user that will be created automatically for the server, the password will be the same as name (admin by default, recommended to change after startup) token_expiry_seconds: 3600 # for how long are login tokens valid for, min: 60 = 1 minute, max: 86400 = 1 day, default: 3600 = 1 hour data_dir: agdb_server_data # directory to store user data log_level: INFO # Options are: OFF, ERROR, WARN, INFO, DEBUG, TRACE log_body_limit: 10240 # maximum length of the body of the request that will be logged in bytes, default is 10KB request_body_limit: 10485760 # maximum length of the body of the request that will be accepted in bytes, default is 10MB pepper_path: "" # Optional path to a runtime secret file containing 16 bytes "pepper" value for additionally "seasoning" (hashing) passwords. If empty a built-in pepper value is used - see "How to run the server?" guide for details tls_certificate: "" # path to the TLS certificate file tls_key: "" # path to the TLS key file tls_root: "" # path to the TLS root CA file cluster_token: cluster # token used between members of the cluster for authentication, treat this value as secret cluster_heartbeat_timeout_ms: 1000 # number of milliseconds since last message sent to a node in the cluster before the leader sends a heartbeat message cluster_term_timeout_ms: 3000 # number of milliseconds without receiving a message from the leader after which the nodes will consider leader to be off and begin a new term cluster_election_factor_ms: 1000 # number of milliseconds delay for starting election in each node, this time will be multiplied by the index of the node (e.g. node 0 * cluster_election_factor_ms == starts election immediately) cluster: [] # list of "address" fields of all nodes in the cluster including local node - the order and values must be the same in all members of the cluster sync_mode: none # controls fsync behavior: "none" (default) or "commit" ``` You can prepare it in advance in a file `agdb_server.yaml`. After the server database is created changes to the `admin` field will have no effect, but the other settings can be changed later. All config changes require server restart to take effect. ### Sync Mode The `sync_mode` setting controls whether the server issues explicit `fdatasync` calls at transaction commit boundaries. It applies uniformly to the server database, cluster log, and all user databases. | Value | Behavior | | -------- | ---------------------------------------------------------------------- | | `none` | Rely on the operating system to flush dirty pages in the background. | | `commit` | Issue `fdatasync` on every transaction commit ensuring write ordering. | **Default: `none`** — on local filesystems (ext4, xfs, ZFS, etc.) the OS page cache provides sufficient write ordering guarantees for the WAL protocol. The kernel flushes dirty pages in program order and the filesystem journal protects metadata. The probability of a reorder causing data loss on local disk is negligible in practice. **When to enable `commit`:** - **Network filesystems with client-side write caching** (e.g. WekaFS mounted with `writecache`, CephFS with client buffering). These filesystems may reorder or delay flushes across files. Without explicit fsync the WAL can be cleared on disk before the main data file is flushed, which would cause an already-committed transaction to roll back on restart. - **Any storage layer where write ordering between two open file descriptors is not guaranteed** by the OS (FUSE-based mounts, certain cloud block storage with write-back caching). **When `none` is safe:** - Local physical disks with a POSIX filesystem — the OS provides ordering. - Cluster deployments on local disk — Raft replication provides node-level durability through quorum acknowledgement. Even in the theoretical local-disk reorder scenario, other nodes hold the committed entry. - Any environment where a small window of potential data loss on power failure is acceptable (similar to SQLite's `PRAGMA synchronous=NORMAL`). The `commit` mode adds latency proportional to the storage backend's fsync speed (typically sub-millisecond on local NVMe, 1-5ms on spinning disk, 5-50ms on network filesystems). Enable it only when the underlying storage requires it. The server is built with a default `pepper` (can be changed as [part of the build](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-bare-metal) ) that is used if `pepper_path` is not specified. If unique it makes sure the two different `agdb_server` instances do not produce the same password hashes and enhances the security. The `pepper` value is to be considered a secret. The TLS is turned on by specifying the `tls_certificate` and `tls_key` and internally uses (`rustls`) [{rel=""nofollow""}]. The `tls_root` is optional (can be empty) and needs to be specified only if you are using self-signed certificates. The certificate use in `tls_certificate` must be issued for the name (or one of alternative names) used in the `address` and `cluster` fields. For self-signed certificate and root CA (usable in docker compose or K8s deployments) you can use the following: ```bash cargo install rustls-cert-gen rustls-cert-gen \ --common-name=agdb \ --ca-file-name=root_ca \ --cert-file-name=cert \ --country-name=CZ \ --organization-name=Agnesoft \ --san=localhost \ --san=agdb0 \ --san=agdb1 \ --san=agdb2 \ --output=. ``` ## Users The server has a single admin account (`admin` by default, configurable with password being the name) that can perform any regular user action + all admin actions such as creating users. You can use this account for using the database locally, but it would be advisable to use it only for maintaining the server and to create a regular user for use with the databases: ```bash # produce an admin API token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5" token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}') # using admin token to create a user curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/user/my_db_user/add -d '{"password":"password123"}' # login as the new user and producing their token token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"my_db_user","password":"password123"}') ``` Users are allowed to change their password and to create and manipulate databases. Login tokens are now session based. Each successful login creates a new API token bound to a specific session rather than reusing one shared token per user. The current active sessions can be inspected via `/api/v1/user/status`. Available user APIs: | Action | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | /api/v1/user/change\_password | changes the current user's password | | /api/v1/user/login | logs in the user returning a new API token for a new session | | /api/v1/user/logout | logs out the current session by default, all sessions with `?session=all`, all other sessions with `?session=others`, or a specific session by id | | /api/v1/user/status | returns current user's username, whether it is a server admin, and the list of current login sessions | | /api/v1/cluster/user/login | logs in the user cluster wide and creates the same new session token on all nodes in the cluster | | /api/v1/cluster/user/logout | same logout options as `/api/v1/user/logout`, but applied cluster wide | Examples: ```bash # logout only the current session curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/user/logout # logout all sessions of the current user curl -X POST -H "Authorization: Bearer ${token}" "localhost:3000/api/v1/user/logout?session=all" # logout all other sessions but keep the current one curl -X POST -H "Authorization: Bearer ${token}" "localhost:3000/api/v1/user/logout?session=others" ``` ## Databases Any user can create, remove and manipulate their own databases. To create a database: ```bash curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/db/my_db_owner/my_db/add?db_type=mapped ``` Note that a user can only create databases under their own name. The `db_type` can be one of: ```yaml memory # memory only database, basically a cache mapped # memory mapped database, using memory for reading but persisting changes to the disk file # file based database only, no memory caching, reading/writing from/to disk ``` It is possible to add an existing database to the server. Move the db file to the server data folder and run `/api/v1/db/{owner}/{db}/add` API as if you were creating a new database with the db's name. If the file exists it will be added rather than created. Similarly, you can remove database (instead of deleting it) from the server with `/api/v1/db/{owner}/{db}/remove` API that will disassociate the db from the server which you can then move and use elsewhere. ### Database Users Each database is scoped to one user (owner) who can exercise full control over it. The owner can add more users (they must exist on the server) to the database including admin level users with one of three roles: ```yaml read # can only run immutable exec queries write # can run mutable and immutable exec queries admin # same as write but can also admin the database ``` The admin users can do some (but not all) actions that the owner can: ### Database Actions | Action | Permission | Description | | ----------------------------------- | ---------- | ------------------------------------------------------------------------------------------------ | | /api/v1/db/{owner}/{db}/add | owner | adds (from existing files) or creates a database (memory, memory mapped, file only) | | /api/v1/db/{owner}/{db}/audit | read | returns the log of all mutable queries that ran against the database (with user who ran them) | | /api/v1/db/{owner}/{db}/backup | admin | creates an automatic backup snapshot of the database (see backup docs below) | | /api/v1/db/{owner}/{db}/clear | admin | clears the content of the database (either all, db only, audit only, backup only) | | /api/v1/db/{owner}/{db}/convert | admin | converts db between memory/mapped/file | | /api/v1/db/{owner}/{db}/copy | read | creates a copy of the database under the current user | | /api/v1/db/{owner}/{db}/delete | owner | deletes the database including files on disk | | /api/v1/db/{owner}/{db}/exec | read | executes queries against the database (does not allow mutable queries) | | /api/v1/db/{owner}/{db}/exec\_mut | write | executes queries against the database (allows both mutable and immutable queries) | | /api/v1/db/{owner}/{db}/list | read | lists the databases with role of the current user (owned and others') | | /api/v1/db/{owner}/{db}/optimize | write | optimizes the underlying file storage packing the data reclaiming unused regions (defragmenting) | | /api/v1/db/{owner}/{db}/remove | owner | removes the database from the server but keeps the files on disk (main, WAL, backup, audit) | | /api/v1/db/{owner}/{db}/rename | owner | changes the name of the database (this API can be used to transfer db ownership) | | /api/v1/db/{owner}/{db}/restore | admin | restores the database from the automatic backup while keeping the backup unchanged | | /api/v1/db/{owner}/{db}/rollback | admin | swaps the current database with the backup, effectively rolling back to the backup state | | /api/v1/db/{owner}/{db}/user/add | admin | adds a user to the database | | /api/v1/db/{owner}/{db}/user/list | read | list users of the database with their roles | | /api/v1/db/{owner}/{db}/user/remove | admin | removes a user from the database | ### Backups Each database can be backed up. The backup API `/api/v1/db/{owner}/{db}/backup` has no parameters and will always back up the database under the same name to the "backups" subfolder in the owner's data. The database can be restored with `/api/v1/db/{owner}/{db}/restore`. This replaces the current database contents with the backup while leaving the backup intact, so repeating `restore` will reapply the same snapshot. If you want the old swap-style behavior, use `/api/v1/db/{owner}/{db}/rollback`. This swaps the current database with the backup, so running `rollback` twice toggles between the two states. The backup and restore flow also keeps the audit log snapshot aligned with the database backup. If you need more granular backup or multiple backups you can devise your own scheme using the `/api/v1/db/{owner}/{db}/copy`, `/api/v1/db/{owner}/{db}/rename` and possibly `/api/v1/db/{owner}/{db}/remove` or `/api/v1/db/{owner}/{db}/delete` APIs. ### Queries All queries are executed using the single `/api/v1/db/{owner}/{db}/exec` (read only queries) and `/api/v1/db/{owner}/{db}/exec_mut` (for queries that also write to the database) endpoint and are exactly the same as in the embedded/application database (see [Queries documentation](https://agdb.agnesoft.com/docs/references/queries)). However, depending on the user's role the server may reject executing the queries (i.e. mutable queries executed by the user with `read` role in the database). The endpoints accept a list of queries and the entire list is run as a transaction meaning either all queries succeed or none of them do. The endpoint will return list of results, one per executed query. It is possible to reference queries from each other in the list and the server will inject results of the referenced queries to the next one. This is slight extension to the vanilla `agdb` queries. It is best illustrated by an example: ```rs let queries = &vec![ QueryBuilder::insert().nodes().count(1).query().into(), // :0 QueryBuilder::insert().nodes().count(1).query().into(), // :1 QueryBuilder::insert().edges().from(":0").to(":1").query().into(), // :2 QueryBuilder::search().from(":0").to(":1").query().into(), //:3 ]; ``` In places where the `alias` can be used (an `ids` identifier) you can use an index prefixed by `:` to inject result of the previous query in the list. In the example we are inserting two separate nodes and then creating an edge between them and finally searching from one node to the other. An index to the results can be used in most places including conditions. What is currently not possible is to inject data (i.e. key-value properties) from results to subsequent queries. **Transactions** While the server serves each request asynchronously and can serve any number of clients at the same time the queries and individual databases must still follow the basic principles of the `agdb` that are the same as in the embedded variant (and derived from Rust itself): There can be either: - unlimited amount of immutable transactions - exactly one mutable transaction However, the `agdb` is written in such a way that it performs excellently even under heavily contested read/write load. See `agdb_benchmark` and [performance documentation](https://agdb.agnesoft.com/docs/references/performance). ## Admin Each `agdb_server` has exactly one admin account (`admin` by default) that acts as a regular user but additionally is allowed to execute APIs under `/admin/`. These mostly copies the APIs for regular users but some of the restrictions are not enforced (i.e. ownership or db role). Furthermore, the admin has access to the following exclusive APIs: | Action | Description | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | /api/v1/admin/db/\* | provides same endpoints as for regular users but without owner/role restrictions | | /api/v1/admin/shutdown | gracefully shuts down the server | | /api/v1/admin/status | lists extended statistics of the server - uptime, # dbs, # users, # logged users, server data size, log level | | /api/v1/admin/set\_log\_level | sets log level, one of `trace`, `debug`, `info`, `warn`, `error`, `off` | | /api/v1/admin/user/{username}/add | adds new user to the server | | /api/v1/admin/user/{username}/change\_password | changes password of a user | | /api/v1/admin/user/{username}/logout | force logout of all sessions of a user by default, or a specific session with `?session=` | | /api/v1/admin/user/logout\_all | force logout of all non-admin users | | /api/v1/admin/user/{username}/delete | deletes user and all their data (databases) from the server | | /api/v1/admin/user/list | lists the all users on the server | | /api/v1/cluster/admin/user/{username}/logout | same admin logout options as above, but applied on all nodes in the cluster | | /api/v1/cluster/admin/user/logout\_all | force logout of all non-admin users from all nodes in the cluster | ## Shutdown The server can be gracefully shutdown with `CTRL+C` or programmatically by using the `/api/v1/admin/shutdown` endpoint which requires admin token, e.g. ```bash token=$(curl -X POST -H 'Content-Type: application/json' localhost:3000/api/v1/user/login -d '{"username":"admin","password":"admin"}') #will produce a token, e.g. "bb2fc207-90d1-45dd-8110-3247c4753cd5" curl -X POST -H "Authorization: Bearer ${token}" localhost:3000/api/v1/admin/shutdown ``` ## Cluster You can an in most cases should run the server in a cluster for resiliency and durability of the data. The exception would be if you require speed at the cost of resiliency and durability and the `agdb` is used more as a cache rather than main database. The `agdb_server` is using custom implementation of a [Raft consensus algorithm](https://en.wikipedia.org/wiki/Raft_\(algorithm\)){rel=""nofollow""}. The benefits of the algorithm are durability, data consistency and speed as the actions do not need to be persisted to all nodes (only majority) before acknowledging the client. The custom implementation in `agdb` additionally offers the following features and guarantees atop of Raft: - Write action acknowledgement happens only when the respective action was executed fully (not just that the action was received by the majority as is the standard implementation). - Clients can request write operations through any member of the cluster (not only the leader) and the node will forward it to the current leader and act as a proxy. - When forwarding the action the node will only acknowledge the client when both of the following becomes true: - The leader committed the operation meaning the majority of the nodes persisted the action. - The node through which the action was performed executed the action itself and in full. - Even when there is no leader elected such as when the cluster is being (re)deployed the read operations are always available This means that you can freely choose a node and perform any action through it and observe consistent results at the minor performance penalty due to the forwarding (unless you picked the leader node). It might be useful in situations where reads are more frequent as they would be more spread out across the cluster. This however does NOT prevent inconsistencies in situations where every request is sent to a different node (e.g. when accessing the cluster through single service in Kubernetes). The `agdb` does not offer consistency across all nodes at all times - only leader + local node (if different) + number of other (unspecified) nodes required to reach majority. ::note The pepper value (used for additionally hashing the passwords, see the [configuration](https://agdb.agnesoft.com/#configuration) section) also needs to be the same across the cluster as when sending actions to do with passwords only the hashed values are sent between nodes rather than raw passwords for security reasons. :: The nodes use `cluster_token` from the config (see above) and calculated cluster `hash` to authenticate to each other. The hash is the calculated value based on the `cluster` array in the config listing all the nodes (`address` values) in the cluster. It is required for the `cluster` array to be the same in the config files of all nodes. Additionally, when the `cluster` is not empty the local `address` value must match one of the addresses. The index of such address is the index of the node in the cluster. Example: ```yaml #node0 agdb_server.yaml address: http://localhost:3000 cluster: ["http://localhost:3000", "http://localhost:3001", "http://localhost:3002"] #node1 abdb_server.yaml address: http://localhost:3001 cluster: ["http://localhost:3000", "http://localhost:3001", "http://localhost:3002"] #node2 agdb_server.yaml address: http://localhost:3002 cluster: ["http://localhost:3000", "http://localhost:3001", "http://localhost:3002"] ``` ::note You should run a cluster of at least 2 nodes. Recommended number is 3 or any odd number. The even number of nodes will still work but the benefits of the consensus algorithm will be somewhat diminished due to the majority rule. Majority of 2 is 2 nodes, of 3 nodes is 2 nodes, of 4 nodes is 3 nodes, of 5 nodes is also 3 nodes etc. :: The node0 will have the index `0`, the node1 will have the index `1` and node2 will have the index `2` as their respective addresses (`address` field) match these indexes in the `cluster` array which itself is the same in all configs and thus produces the same cluster hash. This mechanism helps protect the cluster during topology change. However, if you need to change number of nodes in the cluster, you risk the [split brain issue](https://en.wikipedia.org/wiki/Split-brain_\(computing\)){rel=""nofollow""}. For instance if you changed the number of nodes from 3 to 5 on node0 then the node1 and node2 would still see the old topology of 3 nodes, elect either as leader and continued as normal. Whereas the node0 with the 2 new nodes could do the same (as majority in cluster of 5 is 3 nodes). If you must for any reason change the topology: - Perform backup of all nodes - Prevent new nodes from starting (do NOT start new nodes under any circumstances yet) - Update the configuration of the OLD (existing) nodes with the new nodes one by one - Wait for the cluster leader to be established (this is only possible if the majority rules allow it: going from 3 to 5 is fine, going from 3 to 7 is not) - Bring the new nodes up and let them synchronize with the leader ::warning Make topology changes only when absolutely necessary and excercise great care when doing it. :: ## Misc Following are the special or miscellaneous endpoints: | Endpoint | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------- | | /api/v1 | serves rapidoc OpenAPI GUI (use this in the browser) | | /api/v1/openapi.json | returns the server's OpenAPI specification as json | | /api/v1/status | returns 200 OK if the server is ready (up) | | /api/v1/cluster/status | returns the list of cluster nodes indicating which nodes are reachable from the current node and which node is the leader | # Studio TBD # Cloud TBD # Efficient agdb In this document we will explore more realistic use of the `agdb`. It should help you understand how to make the best use of the `graph` data schema and how to build complex queries. The premise that we will be working on is building a database for a social network. The users of the network can create posts and share them with other users to comment and like. You can see the complete code under [tests/efficient\_agdb.rs](https://github.com/agnesoft/agdb/blob/main/agdb/tests/efficient_agdb/mod.rs){rel=""nofollow""}. ## The setup ```rs fn create_db() -> Result>, DbError> { let db = Arc::new(RwLock::new(Db::new("social.agdb")?)); db.write()?.transaction_mut(|t| -> Result<(), DbError> { t.exec_mut( QueryBuilder::insert() .nodes() .aliases(["root", "users", "posts"]) .query(), )?; t.exec_mut( QueryBuilder::insert() .edges() .from("root") .to(["users", "posts"]) .query(), )?; Ok(()) })?; Ok(db) } ``` We are setting up the database for the multithreaded use with the `Arc` and the `RwLock` in order to leverage unlimited read parallelism. We create nodes for `users` and one for `posts` and create a `root` node and connect the other two to it. The `agdb` does allow disjointed graphs, but it is not easy to navigate an unknown database (e.g. when opening the database file in a data editor/explorer without knowing its content). A useful convention is thus to specify a root node. If it is a first node (that would always end up with the `id` == `1`) or has an alias (i.e. `root`) the entry-point is known. We can then connect other nodes to it or insert their aliases (or `ids`) as properties to the `root` node. There is no preferred or hard-coded method to do this which is intentional. You may also choose not to do it at all if you do not have a need for data discovery. ### Users The users of our social network will be nodes connected to the `users` node. The information we want to store about our users are: - username - e-mail - password Lets firs define the `User` struct to hold this information: ```rs #[derive(DbType)] struct User { username: String, email: String, password: String, } ``` We derive from `agdb::DbType` so we can use the `User` type directly in our queries. A query creating the user would therefore look like this: ```rs fn register_user(db: &mut Db, user: &User) -> Result { db.transaction_mut(|t| -> Result { if t.exec( QueryBuilder::search() .from("users") .where_() .key("username") .value(&user.username) .query(), )? .result != 0 { return Err(DbError::from(format!( "User {} already exists.", user.username ))); } let user = t .exec_mut( QueryBuilder::insert() .element(user) .query(), )? .elements[0] .id; t.exec_mut( QueryBuilder::insert() .edges() .from("users") .to(user) .query(), )?; Ok(user) }) } ``` First we check if the user exists and return error if the username is taken. We then use a transaction to create a user node and edge from `users` node. The reason why this is done in two steps (queries) is to keep the queries simpler and because we want to get back the result of the node insertion - the user `id`. If we fed the insert nodes query to the insert edges query the `id` would be lost. ### Posts The users should be able to create posts. The data we want to store about the posts are: - title - body - author Once again let's define the `Post` type. The specially treated `db_id` field will become useful later on: ```rs #[derive(DbType)] struct Post { db_id: Option, title: String, body: String, } ``` The first two will become properties while the `author` will be represented as an edge. To create a post: ```rs fn create_post(db: &mut Db, user: DbId, post: &Post) -> Result { db.transaction_mut(|t| -> Result { let post = t .exec_mut( QueryBuilder::insert() .element(post) .query(), )? .elements[0] .id; t.exec_mut( QueryBuilder::insert() .edges() .from([QueryId::from("posts"), user.into()]) .to(post) .values([vec![], vec![("authored", 1_u64).into()]]) .query(), )?; Ok(post) }) } ``` Beside connecting the node to two others we are also adding a property `authored` to the edge coming from the `user`. This is to distinguish it from other possible edges coming from the user - comments and likes. ### Comments The comments are created by the users and are either top level comments on a post or replies to other comments. Information we want to store about the comments are: - body - author - parent (post OR comment) We define the comment type: ```rs #[derive(DbType)] struct Comment { body: String, } ``` To create a comment: ```rs fn create_comment( db: &mut Db, user: DbId, parent: DbId, comment: &Comment, ) -> Result { db.transaction_mut(|t| -> Result { let comment = t .exec_mut( QueryBuilder::insert() .element(comment) .query(), )? .elements[0] .id; t.exec_mut( QueryBuilder::insert() .edges() .from([parent, user]) .to(comment) .values([vec![], vec![("commented", 1_u64).into()]]) .query(), )?; Ok(comment) }) } ``` The `parent` parameter can be either a post `id` or a comment `id`. The edges from the user have now a property `commented` to distinguish them from `authored` edges. ### Likes Likes can be best modelled as connections from users to posts and comments: ```rs fn like(db: &mut Db, user: DbId, id: DbId) -> Result<(), DbError> { db.exec_mut( QueryBuilder::insert() .edges() .from(user) .to(id) .values_uniform([("liked", 1).into()]) .query(), )?; Ok(()) } ``` The query is fairly self-explanatory. The edge has the `liked` property that distinguishes it from the other edges from a user (i.e. `authored` and `commented`). Since users can decide that they no longer like a post or comment we need to have the ability to remove it: ```rs fn remove_like(db: &mut Db, user: DbId, id: DbId) -> Result<(), DbError> { db.transaction_mut(|t| -> Result<(), DbError> { t.exec_mut( QueryBuilder::remove() .search() .from(user) .to(id) .where_() .keys("liked") .query(), )?; Ok(()) }) } ``` This query removes elements returned by the search. The search is the "path search" starting (`from`) the user and looking for the `id` (`to`). It selects only the element with the `liked` property which would be the edge we are looking for. The query is simple because it takes advantage of several facts: - if the `id` exists the path to it will contain 3 elements: starting node, an edge and the `id` node - elements not selected for the result by the condition are penalized in the path search eliminating the candidate path through the `authored` node - `limit(1)` is not useful here because path search applies the limit after it found the best path which would be as described - containing just one suitable element anyway Still if we were unsure the `id` exists or if we wanted to limit the search area as much as possible we could create a chain of conditions to only restrict the search to a particular distance and prevent the other edges to be followed: ```rs .where_() .keys("liked") .and() .beyond() .keys("liked") ``` This condition ensures that traversal only follows edges that have the `liked` property. The search stops at the first node reached after traversing the last such `liked` edge, which may be the destination node. Additionally, it will not continue from a node unless there is an adjacent edge with the `liked` property. ## Selects & Searches Now that we have the data in our database and means to add (or remove) more it is time to create the select and search queries. Recall that the search queries find the `ids` of the database (graph) elements. To read properties the properties you would need to combine it with a `select` query. ### Login First the user login which means searching the database for a particular username and matching its password: ```rs fn login(db: &Db, username: &str, password: &str) -> Result { let result = db .exec( QueryBuilder::select() .values("password".into()) .search() .depth_first() .from("users") .limit(1) .where_() .neighbor() .and() .key("username") .value(username) .query(), )? .elements; let user = result .first() .ok_or(DbError::from(format!("Username '{username}' not found")))?; let pswd = user.values[0].value.to_string(); if password != pswd { return Err(DbError::from("Password is incorrect")); } Ok(user.id) } ``` First we retrieve the password if the user exists: 1. Start the search at the `users` node using depth first algorithm. The depth first is better here because it allows us to examine users in sequence rather than first examining all the edges from the `users` node and only then all the users. 2. Limit the search to just a single element (`limit(1)`) as we want just one user, and we want to stop once it is found. 3. Limit the distance of the search to elements at distance 2 (distance 0 = starting node, distance 1 == edges from users, distance 2 == user nodes). 4. Check `username` property for a match against the passed in username. Upon success, we attempt to get the first element in the result returning "user not found" if it is not there. Finally, we get the value of the password (we have selected single property so we know it is there) from the result and check if the password matches. You may be wondering why we do not check the password in the query directly. The reason is that we have no way of stopping the further search if only the `username` matched but not the `password`. The search would then needlessly continue over all users. Therefore, we only retrieve the password and match it in the code. Since it would be salted and hashed anyway it would not be possible to do it with a database query in the first place. ### User content Showing users their content or content they liked can be done with a following query first retrieving the `ids` of the posts: ```rs fn user_posts_ids(db: &Db, user: DbId) -> Result, DbError> { Ok(db .exec( QueryBuilder::search() .from(user) .where_() .neighbor() .and() .beyond() .keys("authored") .query(), )? .ids()) } ``` This time we search from the logged in `user` node identified by its `id` (i.e. returned from the `login()` function). We know the posts are at distance 2 from there (beyond just a single edge) so the first condition is `distance(Equal(2))` (neighbor) to select the elements we want. We narrow down the search further with `beyond()` and a condition that limits where we want the search to go. That is the purpose of the `keys("authored")` so only elements with `authored` key are followed. Note that `beyond()` only controls traversal and does not select elements on its own - the `neighbor()` condition handles the selection. The same outcome can be reached with number of other conditions as well. For example using `keys("title")` condition. However, it would also examine all the comments and likes (you may remember similar discussion in the method to [remove likes](https://agdb.agnesoft.com/#likes)). Another option could be using `not_beyond()` with `where_().keys("commented").or().keys("liked")` - explicitly stopping at edges with those properties (`commented` and `liked`). The `keys()` condition is "all or nothing" so it needs to be `or`ed and specified twice in this case. Similarly to `user_posts` we can fetch the user comments and liked posts with slight modification of the condition: - user comments: `.keys("commented")` - liked posts: `keys("title")` and `.keys("liked")` Notice as well that the function returns the `ids` of the elements we were interested in which gives us flexibility in what we want to retrieve about the posts. In order to retrieve say titles of the posts we would need to feed it to a select query: ```rs fn post_titles(db: &Db, ids: Vec) -> Result, DbError> { Ok(db .exec( QueryBuilder::select() .values("title") .ids(ids) .query(), )? .elements .into_iter() .map(|post| post.values[0].value.to_string()) .collect()) } ``` Here we take advantage of the fact that we have selected a single property so that every element in the result is guaranteed to have it. ### Posts Selecting all posts is a fairly straightforward query, but we would rarely need all of them at once. A common need for large collections of data is "paging". That means returning only a chunk of data at a time. Similarly to SQL we can use both `offset` and `limit` to achieve this: ```rs fn posts(db: &Db, offset: u64, limit: u64) -> Result, DbError> { db .exec( QueryBuilder::select() .elements::() .search() .from("posts") .offset(offset) .limit(limit) .where_() .neighbor() .query(), )? .try_into() } ``` By running the function repeatedly and incrementing the `offset` by the `limit` we would iterate over all posts in `limit` steps (usually called "pages"). Notice the `distance` condition which is all we need to limit the search to just posts. There is something missing though as we would want to also order the posts by the number of likes they have. That would be possible with the current schema but not very easily. We will revisit this a later when we will discuss the schema updates. ### Comments Now that we have the posts we will want to fetch the comments on them. Our schema says that the only outgoing edges from posts are the comments so getting the comments can be done like this: ```rs fn comments(db: &Db, id: DbId) -> Result, DbError> { db .exec( QueryBuilder::select() .elements::() .search() .depth_first() .from(id) .where_() .node() .and() .distance(CountComparison::GreaterThan(1)) .query(), )? .try_into() } ``` Using the `depth_first` algorithm will help in organizing the comments in their natural tree structure in the result. The comments are nodes so `.node()` is the first condition. We are starting at the post, but we are not interested in selecting that hence the condition `.distance(CountComparison::GreaterThan(1))`. Since we are selecting the `body` property we can assume it when extracting it to a vector of comments. There is another flaw here however, do you see it? We currently do not have a way to tell which comment is a top level comment to correctly present the comments to the user other than in a flat list. This is another case for a schema update to satisfy this requirement. ## Schema updates Possibly the most common problem with any database is that it contains the information we want in some form, but it does not allow us using it in the way we would like. Perhaps we want to join the information together or get it in a different format than in which it is stored. Or the information is not there, and we need to start capturing it. These issues are not unique to `agdb` or to databases in general for that matter. They are ubiquitous in all software as requirements and our understanding of the problem domains change over time. The ability to change is what matters the most. Let's see how `agdb` tackles it. In our case we have already identified two such issues with our database so far: - ordering posts based on likes - determining level of comments We can perhaps already come up with more such as getting the authors of posts or comments, missing timestamp information etc. There are certainly more but for now let's focus on the two highlighted ones: ### Likes Let's start with the likes. The query to make use of the `liked` edges would not be terribly difficult (counting the `liked` edges incoming to a post or comment) but it certainly does not seem that easy or fast. Especially as we would be doing it over and over. Instead, we could simply introduce a counter property called `likes` and essentially cache the information on the posts (or comments) themselves. That would simplify and speed up things: ```rs fn add_likes_to_posts(db: &mut Db) -> Result<(), DbError> { db.transaction_mut(|t| -> Result<(), DbError> { let posts = t.exec( QueryBuilder::search() .from("posts") .where_() .neighbor() .query(), )?; let mut likes = Vec::>::new(); for post in posts.ids() { let post_likes = t .exec( QueryBuilder::search() .to(post) .where_() .distance(1) .and() .keys("liked") .query(), )? .result; likes.push(vec![("likes", post_likes).into()]); } t.exec_mut(QueryBuilder::insert().values(likes).ids(posts).query())?; Ok(()) }) } ``` We are doing a mutable transaction to prevent any new posts, likes or other modifications to interfere while we do this. First we get the `ids` of all the posts. Then we count the `liked` edges of each post (exactly what we would be doing if we did not want to change the schema) and finally we insert a new `likes` property with that count back to the posts. Furthermore, we should update our definition of `Post`: ```rs #[derive(DbType)] struct PostLiked { db_id: Option, title: String, body: String, likes: i64, } ``` This allows us to select and order posts based on likes: ```rs fn liked_posts(db: &Db, offset: u64, limit: u64) -> Result, DbError> { db .exec( QueryBuilder::select() .elements::() .search() .from("posts") .order_by([DbKeyOrder::Desc("likes".into())]) .offset(offset) .limit(limit) .where_() .neighbor() .query(), )? .try_into() } ``` However, this change is not "free" in that caching any information means it now exists in two places and those places must be synchronized (the famous cache invalidation problem). Fortunately this instance is not as hard. We simply make sure that whenever we add or remove `likes` we also update the counter on the post or comment. Since when we do those operations we also have the post/comment `id` doing that would be trivial. ### Comments Another issue we found was that comments do not track their level, and we cannot present them hierarchically. To make things simpler let's add only a simple distinction between top level comments and replies disregarding any further nesting. To do that we would simply mark the top level comments with a new property: ```rs fn mark_top_level_comments(db: &mut Db) -> Result<(), DbError> { db.exec_mut( QueryBuilder::insert() .values_uniform([("level", 1).into()]) .search() .from("posts") .where_() .distance(4) .query(), )?; Ok(()) } ``` Although this task might have seemed daunting it could be done with a simple query that once more takes advantage of the graph schema. We know that the only outgoing edges from the posts are the comments and that they are hierarchical (replies are attached to the comments they reply to). Therefore, when searching from the `posts` node at distance `4` we will find only the top level comments. We then uniformly apply a property `level=1` to them. Such a property could then be used by the client code to determine how the data is displayed to the users. ## Summary In this guide we have gone through a realistic example of an inception of a database setting it up from scratch, designing search queries and leveraging the graph schema. We have used the ability to limit the search area based on our data and the graph schema multiple times. Instead of searching possibly millions of records and filtering them out to get what we want we could search & select just the relevant fraction of the data set. That is the main advantage of the graph databases. If a user authored just 3 posts the query would do exactly the same work if there were 30 posts total in the database as if there were 3 billion. We have also discovered issues with the schema and were able to seamlessly fix them. It demonstrated yet another important aspect of graph databases which is fearless schema updates. Modelling data on a graph feels natural and changing it to fit new or changing requirements is just as natural. Lastly we have seen that the queries can be simple, readable, statically checked and completely native while still providing complex functionality such as filtering through conditions, paging, ordering etc. Moreover, while the features of object queries won't make them always logically correct they eliminate entire categories of issues like syntax errors, type errors, security issues like SQL injection, and even certain logic errors etc. For the comprehensive overview of all queries see the [query reference](https://agdb.agnesoft.com/docs/references/queries). For the code used in this document see [the code in the tests](https://github.com/agnesoft/agdb/blob/main/agdb/tests/efficient_agdb/mod.rs){rel=""nofollow""}. # Performance Database performance is one of the key metrics when judging the suitability of the solution for a given use case. Individual metrics such as how many inserts or selects can a database handle in a tight loop are not very interesting or indicative of real performance. In this document we will therefore examine the performance of `agdb` in more realistic use cases via `agdb_benchmark` that simulate real world usage. The `agdb` is designed with the following principles: - ACID database - O(1) complexity for direct access - O(n) complexity for search - Unlimited read concurrency - Exclusive writes The database is ACID compliant, operations must be transactional = `atomic` (A) meaning they are "all or nothing" operations, `consistent` (C) so that the queries will only produce valid state of the data, `isolated` (I) meaning the transactions do not affect each other when in flight and `durable` (D) meaning the database is resistant to system failure and will preserve integrity of the data. Direct access read/write operations have constant complexity of O(1) while search operations are O(n) but the `n` can be limited to a subgraph greatly reducing the time the operation takes. Let's see if the `agdb` lives up to these principles. ## The benchmark The `agdb_benchmark` project is building upon the [Efficient agdb](https://agdb.agnesoft.com/docs/references/efficient-agdb) simulating the traffic in a "social network" database. It simulates concurrent read & write operations on the same database: - Posters: Writes social media posts - Commenters: Writes comments to the existing posts - Post readers: Reads existing posts - Comment readers: Read existing comments It is highly configurable through the `agdb_benchmark.yaml` (or custom file passed as an argument, produced on first run) with the following settings: - Running using embedded database or against a server - How many of each category of users (post writers, comment writers, post readers, comment readers) - How many operations should each user perform - How large each operation should be [readers only] (e.g. how many posts to read) - Contents of each operation [writes only] (e.g. post title, post body) - Delay between each operation For writers the configured content is additionally augmented by the user `id` to produce unique content. The delays are further shifted by the user `id` to prevent unrealistic resource contention by everyone in a single millisecond. The read operations are repeated if no result is yielded effectively "waiting" for the readers to input data first. The benchmark uses tokio tasks spawning everything together. It measures each database operation (transaction as some operations are multiple queries) for minimum, average, maximum and total elapsed time. Additionally, it shows total database size after all operations finished and furthermore after running the optimization algorithm compacting (defragmenting) the data. ### Default settings - Insert user nodes (for post & comment writers) - 10 post writers (100 posts each, 100ms delay, non-small title (>15 bytes) & body (>15 bytes)) - 10 comment writers (100 comments each, 100ms delay, non-small body (>15 bytes)) - 100 post readers (100 reads each, 10 posts per read, 100ms delay) - 100 comment readers (100 reads each, 10 comments per read, 100ms delay). ### Measured operations - Insert user nodes: a node aliased `"users"` with individual users connected to it with a blank edge. Each user has properties `"name"` and `"email"` (values small values `<15 bytes`). - Write posts: a post node connected with a blank edge to the single node aliased `"posts"` and with an edge (property `"authored": 1`) to the respective user node. The properties are `"title"` and `"body"` from config (values are large `>15 bytes`). - Write comments: a comment node connected with a blank edge to the latest post (found via search from `"posts"` node) and with an edge (property `"commented": 1`) to the respective user node. The properties are only `"body"` from config (value is large `>15 bytes`). - Read posts: reads configured amount (e.g. 10 by default) of recent posts on each iteration (found via search from `"posts"` node). - Read comments: reads configured amount (e.g. 10 by default) of recent comments on the latest post (found via search from `"posts"` node). - Database size: after all operations finished & after optimization algorithm is run. ### Run command ```bash cargo run --release -p agdb_benchmark ``` ## Results The following benchmarks were run on: - CPU: Intel Core i7-7700 4 cores (8 logical cores) @ 3,6 GHz - RAM: Crucial Ballistix Sport LT 16GB (2x8GB) DDR4 @ 2400 MHz - DISK: HyperX Savage - 240GB (KINGSTON SHSS37A240G, 4 cores, 8 channels Phison S10, 560 MB/s read, 530 MB/s write, SATA III (6 Gb/s)) - OS: Windows 10 22H2 (19045.3448), Debian: Version 12 (bookworm) [running in Hyper-V/WSL2] When running on a different machine your results will vary, but the relative comparisons should still hold. ### Memory mapped (default) The benchmark run with [default settings](https://agdb.agnesoft.com/#default-settings) using memory mapped file persistent storage (database size is limited to available RAM): **Windows** | Description | Threads | Iters | Per iter | Count | Min | Avg | Max | Total | | -------------- | -------- | ------ | -------- | ------ | ----- | ------ | ------ | ----- | | Creating users | 1 | 1 | 20 | 20 | - | 1 ms | - | 30 ms | | Write posts | 10 | 100 | 1 | 1 000 | 1 ms | 25 ms | 3 s | 8 s | | Write comments | 10 | 100 | 1 | 1 000 | 1 ms | 29 ms | 3 s | 8 s | | Read posts | 100 | 100 | 10 | 10 000 | 14 μs | 387 μs | 282 ms | 9 s | | Read comments | 100 | 100 | 10 | 10 000 | 9 μs | 295 μs | 21 ms | 9 s | | Database size | 1 627 kB | 785 kB | | | | | | | **Debian (Hyper-V/WSL2)** | Description | Threads | Iters | Per iter | Count | Min | Avg | Max | Total | | -------------- | -------- | ------ | -------- | ------ | ------ | ------ | ------ | ----- | | Creating users | 1 | 1 | 20 | 20 | - | 316 μs | - | 6 ms | | Write posts | 10 | 100 | 1 | 1 000 | 390 μs | 1 ms | 141 ms | 2 s | | Write comments | 10 | 100 | 1 | 1 000 | 397 μs | 1 ms | 286 ms | 1 s | | Read posts | 100 | 100 | 10 | 10 000 | 12 μs | 716 μs | 287 ms | 5 s | | Read comments | 100 | 100 | 10 | 10 000 | 6 μs | 445 μs | 286 ms | 5 s | | Database size | 1 627 kB | 785 kB | | | | | | | --- The data shows that the average write operation without contention is very fast (Creating users). Concurrent writes that also contest the database with read operations increase the latency by an order of magnitude. The read operations that can be as fast as <10μs can slow down with contention up to two orders of magnitude particularly due to frequent reads. ### File only The benchmark run with [default settings](https://agdb.agnesoft.com/#default-settings) using file persistent storage only (no memory use but unlimited database size): **Windows** | Description | Threads | Iters | Per iter | Count | Min | Avg | Max | Total | | -------------- | -------- | ------ | -------- | ------ | ------ | ------ | ------ | ----- | | Creating users | 1 | 1 | 20 | 20 | - | 1 ms | - | 38 ms | | Write posts | 10 | 100 | 1 | 1 000 | 1 ms | 650 ms | 96 s | 306 s | | Write comments | 10 | 100 | 1 | 1 000 | 1 ms | 1 s | 149 s | 306 s | | Read posts | 100 | 100 | 10 | 10 000 | 604 μs | 23 ms | 758 ms | 305 s | | Read comments | 100 | 100 | 10 | 10 000 | 390 μs | 28 ms | 775 ms | 304 s | | Database size | 1 627 kB | 785 kB | | | | | | | **Debian (Hyper-V/WSL2)** | Description | Threads | Iters | Per iter | Count | Min | Avg | Max | Total | | -------------- | -------- | ------ | -------- | ------ | ------ | ------ | ------ | ----- | | Creating users | 1 | 1 | 20 | 20 | - | 340 μs | - | 6 ms | | Write posts | 10 | 100 | 1 | 1 000 | 459 μs | 9 ms | 304 ms | 67 s | | Write comments | 10 | 100 | 1 | 1 000 | 431 μs | 15 ms | 99 ms | 65 s | | Read posts | 100 | 100 | 10 | 10 000 | 384 μs | 23 ms | 351 ms | 68 s | | Read comments | 100 | 100 | 10 | 10 000 | 61 μs | 24 ms | 213 ms | 68 s | | Database size | 1 627 kB | 785 kB | | | | | | | Running purely off a file significantly decreases performance. While the minimum write times remain expectedly the same as with memory mapped option (that uses the same underlying persistent file storage for writes) the average and particularly maximum times increased dramatically. This indicates that for data sets too large to fit to RAM running purely off a file is not a viable option either due to prohibitively bad performance. Therefore, a different strategy would be required (in-memory caching, splitting the data set over multiple databases etc.). The file based database might be suitable for write heavy use cases with huge amounts of data such as log store where operations can be serialized to limit the contention and reads/searches are relatively infrequent and do not collide with writes often. ### In memory (cache only) The benchmark run with [default settings](https://agdb.agnesoft.com/#default-settings) using in-memory cache only (no persistence): **Windows** | Description | Threads | Iters | Per iter | Count | Min | Avg | Max | Total | | -------------- | -------- | ------ | -------- | ------ | ----- | ------ | ------ | ------ | | Creating users | 1 | 1 | 20 | 20 | - | 9 μs | - | 189 μs | | Write posts | 10 | 100 | 1 | 1 000 | 11 μs | 5 ms | 442 ms | 3 s | | Write comments | 10 | 100 | 1 | 1 000 | 10 μs | 10 ms | 440 ms | 3 s | | Read posts | 100 | 100 | 10 | 10 000 | 14 μs | 300 μs | 7 ms | 6 s | | Read comments | 100 | 100 | 10 | 10 000 | 13 μs | 319 μs | 7 ms | 6 s | | Database size | 1 627 kB | 785 kB | | | | | | | **Debian (Hyper-V/WSL2)** | Description | Threads | Iters | Per iter | Count | Min | Avg | Max | Total | | -------------- | -------- | ------ | -------- | ------ | ---- | ------ | ---- | ------ | | Creating users | 1 | 1 | 20 | 20 | - | 6 μs | - | 125 μs | | Write posts | 10 | 100 | 1 | 1 000 | 6 μs | 503 μs | 7 ms | 2 s | | Write comments | 10 | 100 | 1 | 1 000 | 4 μs | 374 μs | 1 ms | 1 s | | Read posts | 100 | 100 | 10 | 10 000 | 9 μs | 729 μs | 8 ms | 5 s | | Read comments | 100 | 100 | 10 | 10 000 | 6 μs | 741 μs | 5 ms | 5 s | | Database size | 1 627 kB | 785 kB | | | | | | | Unsurprisingly by far the fastest option. Operating purely off RAM offers unmatched performance but without any persistence such a database can be of limited use. Still for caching purposes this solution is very viable offering sub-millisecond performance for all operations (read & write) with minimal impact of contention even in highly contested cases such as the one being benchmarked. ### In Memory (10x) Increasing the number of writers and readers 10x: | Description | Threads | Iters | Per iter | Count | Min | Avg | Max | Total | | -------------- | ------- | -------- | -------- | ------- | ----- | ------ | ------ | ----- | | Creating users | 1 | 1 | 200 | 200 | - | 9 μs | - | 1 ms | | Write posts | 100 | 100 | 1 | 10 000 | 10 μs | 152 ms | 34 s | 497 s | | Write comments | 100 | 100 | 1 | 10 000 | 9 μs | 154 ms | 64 s | 496 s | | Read posts | 1 000 | 100 | 10 | 100 000 | 21 μs | 3 ms | 235 ms | 497 s | | Read comments | 1 000 | 100 | 10 | 100 000 | 12 μs | 4 ms | 344 ms | 497 s | | Database size | 12 MB | 6 528 kB | | | | | | | ## Flamegraph The following is the ["flamegraph"](https://github.com/flamegraph-rs/flamegraph){rel=""nofollow""} illustrating what the benchmark is spending most time on. The obvious answer (as predicted) is the async orchestration through tokio followed by the database running queries. Digging down the callgraph there is no immediate performance bottleneck (such as memory allocation) that could be significantly optimized. The database functionality seems to be evenly distributed matching the expectations given what is being run: ![Flamegraph](https://agdb.agnesoft.com/images/flamegraph.svg) ## Conclusion The used benchmark simulates highly contested database environment where dozens of writers and hundreds of readers are using the database at the same time. Tweaking the values (e.g. increasing/decreasing) the writers/readers had no significant effect on overall results meaning that the database can scale, and the principles hold under all circumstances. Running the benchmark in 2 OSs with 3 different storage backends showed that results will vary depending primarily on use (or not) of RAM for caching and on the level of data contention. The contention slowdown however is by no means linear is largely down to the scheduling of tasks (tasks not actually being executed and waiting their turn) - more powerful hardware would improve the results significantly (vertical scalability). As demonstrated `agdb` can handle even an extreme load such as the one in the benchmark. The flamegraph has also shown that the database itself is well optimized and there are no obvious/easy wins with most of the time being taken by orchestration (Tokio runtime) as expected. When using `agdb` your bottlenecks will likely lay elsewhere and not in the database itself. Some advice: - Always measure your use case but do not rely on micro-benchmarks, use realistic workloads. See `Creating users` line in each table which is equivalent to an isolated micro-benchmark and compare it with the rest of the table that demonstrates realistic load with contention. - Correct storage backend matters. While the default is usually the best choice offering persistence and speed it comes with certain caveats: - Do not use memory mapped database if you store terabytes of data or your data set is likely to exceed your available RAM size. - Do not use memory mapped database if your use case is write-heavy with infrequent reads. The memory mapping aids only in reading and slows down the writes a little bit. - Do not use in-memory cache if you need persistence even though it is the fastest. # OpenAPI The [agdb server](https://agdb.agnesoft.com/docs/references/server) can be accessed using OpenAPI (REST) via any HTTP client. In addition to the API specification `agdb` offers wide range of clients for many languages that uses the same API but provides convenience and ease-of-use: :language-icons :open-api-code-block # Rust The rust agdb API client is **async only** and can be used with any HTTP client that would implement the `agdb_api::HttpClient` trait. The default implementation uses [reqwest](https://crates.io/crates/reqwest){rel=""nofollow""}. The following is the quickstart guide for the agdb client in Rust (connecting to the server). It assumes an `agdb_server` is running locally. Please refer to the [server guide](https://agdb.agnesoft.com/docs/guides/how-to-run-server) to learn how to run the server. Looking for... [how to run a server?](https://agdb.agnesoft.com/docs/guides/how-to-run-server) | [another language?](https://agdb.agnesoft.com/api-docs/openapi) | [embedded db guide?](https://agdb.agnesoft.com/docs/guides/quickstart) ## Usage ::steps ### Install Rust Please install the Rust toolchain from the [official source](https://www.rust-lang.org/tools/install){rel=""nofollow""}. ### Create an application First we initialize an application called `agdb_client` with cargo: ```bash cargo init agdb_client ``` ### Add dependencies ```bash cargo add agdb --features serde,openapi cargo add agdb_api cargo add tokio --features full cargo add anyhow ``` ### Create the client The client should point to a running `agdb_server`: ```rs use agdb_api::AgdbApi; use agdb_api::ReqwestClient; #[tokio:main] async fn main() -> anyhow::Result<()> { let mut client = AgdbApi::new(ReqwestClient::new(), "localhost:3000"); Ok(()) } ``` ### Create a database user First we need to log in as default admin user and create our database user and then login as them: ```rs client.user_login("admin", "admin").await?; // The authentication login is stored in // the client for subsequent API calls. // Default admin credentials are "admin/admin". client.admin_user_add("my_user", "password123").await?; client.user_login("my_user", "password123").await?; // Login as our newly created user. ``` ### Create a database ```rs use agdb_api::DbType; client.db_add("my_user", "my_db", DbType::Mapped).await?; // Memory mapped database called "my_db" // will be created under our "my_user". ``` ### Run our first queries To run queries against the database we call `db_exec` (for read only queries) and `db_exec_mut` (for queries that also write to the database). In this case we will insert node "users" and 3 user nodes and connect them together. :::tip Note that we can feed the result of a previous query directly to the next one referencing it with an index into the (previous) results starting with semicolon followed by the index, e.g. `:0`, `:1`. ::: ```rs // We derive from agdb::DbType // so we can use the type in the db. #[derive(Debug, DbType)] struct User { db_id: Option, // The db_id member is optional but // it allows querying your user type // from the database. username: String age: u64, } let users = vec![User { db_id: None, username: "Alice".to_string(), age: 40 }, User { db_id: None, username: "Bob".to_string(), age: 30 }, User { db_id: None, username: "John".to_string(), age: 20 }]; // We can pass users directly as // query parameter thanks to the // implementation of the agdb::DbType // trait via #[derive(DbType)]. let queries: Vec = vec![QueryBuilder::insert().nodes().aliases("users").query().into(), QueryBuilder::insert().nodes().values(&users).query().into(), QueryBuilder::insert().edges().from("users").to(":1").query().into(), ]; client.db_exec_mut("my_user", "my_db", &queries).await?; ``` ## Run more queries Run another query searching & selecting the users and converting them back to the native local object and printing the result: ```rs let queries = vec![QueryBuilder::select() .values(User::db_keys()) // Select only relevant properties for the User struct. .ids( QueryBuilder::search() .from("users") // Start the search from the "users" node. .where_() .key("age") // Examine "age" property. .value(LessThan(40.into())) // Include it in the search result if the value // is less than 40. .query(), ) .query()]; // Runs the query against the db, grabs the first result and converts it to the collection of users. let users: Vec = client.db_exec("my_user", "my_db", &queries).await?[0].try_into()?; println!("{:?}", users); ``` ### Run the program ```bash cargo run ``` ### Full program {rel=""nofollow""} :: # PHP The PHP agdb API client is generated with [openapi-generator](https://github.com/OpenAPITools/openapi-generator/blob/master/docs/generators/php.md){rel=""nofollow""}. The following is the quickstart guide for the agdb client in PHP (connecting to the server). It assumes an `agdb_server` is running locally. Please refer to the [server guide](https://agdb.agnesoft.com/docs/guides/how-to-run-server) to learn how to run the server. Looking for... [how to run a server?](https://agdb.agnesoft.com/docs/guides/how-to-run-server) | [another language?](https://agdb.agnesoft.com/api-docs/openapi) | [embedded db guide?](https://agdb.agnesoft.com/docs/guides/quickstart) ## Usage The following is the from-scratch guide to use `agdb-api` PHP package. ::steps ### Install PHP {rel=""nofollow""} ### Install Composer {rel=""nofollow""} ### Create your project Create your project's folder (e.g. `my_agdb`) and initialize the package: ```bash mkdir my_agdb cd my_agdb composer init # follow the steps & prompts ``` ### Add `agnesoft/agdb_api` as a dependency ```bash composer install agnesoft/agdb_api ``` :::tip Consider using other dev packages such as `phpunit/phpunit` and `phpstan/phpstan`. ::: ### Create your main script E.g. `src/index.php` and create a client to connect to the server: ```php status(false); ``` ### Create a database user To create a database user we use the default admin user: ```php // Login as server admin $token = self::$client->userLogin( new UserLogin(["username" => "admin", "password" => "admin"]) ); $client->getConfig()->setAccessToken($token); // Creat user "php_user1" $client->adminUserAdd( "php_user1", new UserCredentials(["password" => "php_user1"]) ); // Login as "php_user1" $token = self::$client->userLogin( new UserLogin([ "username" => "php_user1", "password" => "php_user1", ]) ); $client->getConfig()->setAccessToken($token); ``` ### Create a database To create a database we associate it with a user and give it a name and type (one of `MAPPED`, `MEMORY`, `FILE`): ```php // Creates memory mapped database "db1" for user "php_user1" $client->dbAdd("php_user1", "db1", DbType::MAPPED); ``` ### Execute queries To execute queries against the database we call `dbExec` (read only queries) and `dbExecMut` (for queries that also write to the database) with the user and their database. :::note Notice we are feeding results of the previous query to the next one with special alias `":0"` and `":1"` referencing first and second result respectively. ::: ```php // Prepare the queries to be executed on the remote database. $queries = [ // :0: Inserts a root node aliased "users". QueryBuilder::insert() ->nodes() ->aliases(["users"]) ->query(), // :1: Inserts more nodes with some data. QueryBuilder::insert() ->nodes() ->values([ [ "username" => "user1", "password" => "password123", ], [ "username" => "user2", "password" => "password456", ], ]) ->query(), // :2: Connect the root to the inserted nodes with edges referencing both from previous queries. QueryBuilder::insert()->edges()->from(":0")->to(":1")->query(), // :3: Find a node starting at the "users" node (could also be ":0" in this instance) with specific username. QueryBuilder::select() ->search() ->from("users") ->where() ->key("username") ->value(ComparisonBuilder::Equal("user1")) ->query() ]; // Execute queries. Since it includes mutable queries (inserts) // we need to call dbExecMut() rather than read-only dbExec() $result = $client->dbExecMut("php_user1", "db1", $queries); ``` ### Print the the result We print the result of the of the final query to the console: ```php // Print the result of the last query printf($result[3]); // { // "elements": [ // { // "from": null, // "id": 3, // "to": null, // "values": [ // { // "key": { // "String": "username" // }, // "value": { // "String": "user1" // } // }, // { // "key": { // "String": "password" // }, // "value": { // "String": "password456" // } // } // ] // }, // ], // "result": 1 // } ``` ### Run the program :::warning Make sure the agdb\_server is running at `localhost:3000`. ::: :::tip If you are running this from the examples you may need to call `composer install` first. ::: ```bash php src/index.php ``` ### Full program {rel=""nofollow""} :: # Typescript / Javascript The typescript agdb API client is based on [openapi-client-axios](https://www.npmjs.com/package/openapi-client-axios){rel=""nofollow""}. The following is the quickstart guide for the agdb client in Javascript/Typescript (connecting to the server). It assumes an `agdb_server` is running locally. Please refer to the [server guide](https://agdb.agnesoft.com/docs/guides/how-to-run-server) to learn how to run the server. Looking for... [how to run a server?](https://agdb.agnesoft.com/docs/guides/how-to-run-server) | [another language?](https://agdb.agnesoft.com/api-docs/openapi) | [embedded db guide?](https://agdb.agnesoft.com/docs/guides/quickstart) ## Usage The following is the from-scratch guide to use `agdb-api` typescript/javascript package. ::steps ### Install NodeJS {rel=""nofollow""} ### Create your project Let's create a directory (e.g. `my_agdb`) and initialize the package: ```bash mkdir my_agdb cd my_agdb npm init # follow the steps & prompts ``` ### Add `typescript` ```bash npm install typescript --save-dev ``` and create its configuration file `tsconfig.json`: ```json { "compilerOptions": { "module": "ESNext", "sourceMap": true, "lib": ["ES2015", "DOM"], "moduleResolution": "node", "allowJs": true, "esModuleInterop": true } } ``` :::tip Consider using other dev packages such as `prettier` and `eslint` (and `@typescript-eslint/parser`) ::: ### Add `@agnesoft/agdb_api` as a dependency. ```bash npm install @agnesoft/agdb_api ``` ### Create a client In your main script (`index.ts` or `main.ts` depending on your `package.json`'s `"main"` field) create a client connecting to the server: ```ts import { QueryBuilder, Comparison, AgdbApi } from "@agnesoft/agdb_api"; async function main() { // Creates a client connecting to the remote server. let client = await AgdbApi.client("http://localhost:3000"); } ``` ### Create a database user To create a database user we use the default admin user: ```ts await client.login("admin", "admin"); await client.admin_user_add("user1", { password: "password123" }); await client.login("user1", "password123"); ``` ### Create a database ```ts await client.db_add({ owner: "user1", db: "db1", db_type: "mapped", //memory mapped type, other options are "memory" and "file" }); ``` ### Execute queries To execute queries against the database we call `db_exec` (read only queries) and `db_exec_mut` (for queries that also write to the database) with the user and their database. :::note Notice we are feeding results of the previous query to the next one with special alias `":0"` and `":1"` referencing first and second result respectively. ::: ```ts // Prepare the queries to be executed on the remote database. let queries = [ // :0: Inserts a root node aliased "users" QueryBuilder.insert().nodes().aliases(["users"]).query(), // :1: Inserts more nodes with some data QueryBuilder.insert() .nodes() .values([ [ ["username", "user1"], ["password", "password123"], ], [ ["username", "user1"], ["password", "password456"], ], ]) .query(), // :2: Connect the root to the inserted nodes with edges referencing both from previous queries QueryBuilder.insert().edges().from(":0").to(":1").query(), // :3: Find a node starting at the "users" node (could also be ":0" in this instance) with specific username QueryBuilder.select() .search() .from("users") .where() .key("username") .value(Comparison.Equal("user1")) .query(), ]; // Execute queries. let results = (await client.db_exec_mut({ owner: "user1", db: "db1" }, queries)) .data; ``` ### Print the result of the final query to the console: ```ts console.log(`User (id: ${results[3].elements[0].id})`); for (let { key, value } of results[3].elements[0].values) { console.log(`${key["String"]}: ${value["String"]}`); } ``` ### Run the program :::warning Make sure the agdb\_server is running at `localhost:3000`. ::: :::tip If you are running this from the examples you may need to call `npm install` first. ::: ```bash npm run ``` ### Full program {rel=""nofollow""} :: # Distance The `distance` is a unique property of graphs and perhaps their most important feature. In this post we will examine what distance actually is and how it is useful when working and thinking about data on a graph. First let's consider the following simplest graph: ![distance - simple](https://agdb.agnesoft.com/images/distance_simple.png) We have two nodes - "projects" and "project A" - connected with an edge. We search from "projects" and therefore the "projects" node is at distance 0. The adjacent edge is at distance 1 and finally the "project A" is at distance 2. The distance is also relative to the direction of the search. Consider a reverse search of the above graph: ![distance - simple - reversed](https://agdb.agnesoft.com/images/distance_simple_reversed.png) Here the origin node is "project A" and therefore it is at distance 0. The edge is once again at distance 1 and the "projects" node is at distance 2. ::tip The distance is only relevant when we are searching the graph and is always measured from the point of origin of our search regardless of the search direction. :: ## Distance of 2 When searching graphs perhaps the most significant distance is distance of 2. Why? Because at distance 2 there will always be all the neighboring nodes. Typically you would have a `root` node (such as "projects") being connected to individual nodes having relation to the root (individual projects). That we be akin to a "table" in a relational database. For instance: ![distance of 2](https://agdb.agnesoft.com/images/distance_2.png) In `agdb` the query to find all projects would look like this (using `Rust` as the query language): ```rust QueryBuilder::search() .from("projects") .where_() .neighbor() .query(); ``` Searching from the node "projects" and only returning elements at distance 2. The distance here serves two purposes. One is that it tells the algorithm what to return (elements at distance 2) and even more importantly it tells it when to stop searching. Since we are asking only for elements at the distance 2 it knows that it does not need to consider anything beyond that distance and can safely ignore it! ::tip At distance 2 there will always be all the adjacent (neighboring) nodes to the origin of search. :: ## Limit the search In the previous section we have covered the basics of distance and hinted to its power, so let's explore it properly. Expanding our graph with outputs of our projects: ![distance - outputs](https://agdb.agnesoft.com/images/distance_outputs.png) The original query searching for elements at distance 2 remains unaffected. It would still stop searching at distance 2 regardless if there are now further elements beyond that distance. It therefore does not matter how large and complicated the graph is because with `distance` we can ignore it all and focus the search to only the relevant `subgraph` that we are interested in. Even if the graph had a billion nodes, the above query confined to this subgraph would have to do the same number of operations and would be as performant as if there were only 3 nodes on the graph. Another way the `distance` limits the required work when searching the graph is that even if the element is reached via the distance constrained, e.g. node "projects" at distance 0 and all the connected edges, because it cannot be included in the final result (we asked for distance 2) the algorithm will not be examining its properties at all regardless of additional constraints. For instance searching for a particular project: ```rust QueryBuilder::search() .from("projects") .where_() .neighbor() .and() .key("name") .value("B") .query(); ``` The search algorithm will only look at the "name" properties of the elements at distance 2 rather than all elements in encountered. The query would have the same result without the `distance` condition but would be less efficient as the algorithm would need to do more work (examine properties of all elements and not stopping at distance 2). It is generally a good idea to lead the conditions with the distance constraints if possible for this very reason. Even more efficient version would be to switch to depth-first-search algorithm and limit the search to a single returned element: ```rust QueryBuilder::search() .depth_first() // use depth-first algorithm which is more efficient when searching for a single element .from("projects") .limit(1) // return maximum 1 element .where_() .neighbor() .and() .key("name") .value("B") .query(); ``` Now the algorithm will do least amount of work necessary to find what we are looking for. ## No more joins One of the most ubiquitous actions done in relational databases is joining the data from multiple tables. If we were to model the graph in a relational database we would likely end up with two tables - one for "projects" and one for "outcomes". We would additionally need a link (relation) between them that would most likely be accomplished with a foreign key column in "projects" referencing rows in the "outputs" table. In the query we would then use a join to extract data from both tables. On a graph this is much simpler because the relations are directly represented and joins are not necessary. Once more we can utilize the distance to get the outputs: ```rust QueryBuilder::search() .from("projects") .where_() .distance(4) .query(); ``` Since we know the structure of our graph we also know that the outputs are at distance 4 and can leverage that information to easily extract all outputs. But what if we wanted only outputs of a particular project? Then we would additionally use the `beyond` condition like so: ```rust QueryBuilder::search() .from("projects") .where_() .distance(4) .and() .beyond() .where_() .distance(CountComparison::NotEqual(2)) .or() .key("name") .value("B") .query(); ``` That would result in a following search: ![distance - outputs](https://agdb.agnesoft.com/images/distance_outputs_search.png) Yellow elements would be visited, red is where the algorithm would stop and not continue further and green ones are the elements found and to be included in the final result of the query. Let's break it down: - search starts at the node "projects" (distance 0) - we want only elements at distance 4 (the outputs): `.distance(4)` - additionally we want to limit the search to continue only `beyond` certain elements: `.and().beyond().where_()` (NOTE: the nested `.where_()` is like opening a bracket so we specify multiple conditions for our beyond segment) - the beyond conditions then are: - continue only if the distance is NOT 2 (`.distance(CountComparison::NotEqual(2))`) - OR - if the element has property "name" with the value "B" The `beyond` condition might seem alien but when you look at the picture of the search it should become clear. We are simply telling the algorithm to search normally and not consider anything at distance 2 or beyond unless the `or` condition is satisfied, in this case property "name" equals value "B". Because only one element in our graph satisfies such condition ("project B") the algorithm will continue beyond it (and stop at distance 2 otherwise) and eventually reach the distance 4 and return all elements it finds there. This potentially greatly limits the scope of work needed. The detailed steps taken by the algorithm and how it evaluates each element: ::steps ### Distance 0: ```text - Distance is less than 4 meaning the first condition evaluates `false` (nothing is selected but search will continue). - Second condition has modifier "beyond" meaning it only controls traversal (whether the search continues or stops), not element selection. - Distance is not 2 so the beyond condition evaluates to `true` and due to the short circuit of the `or` condition we will not reach the `key` condition at all. The beyond condition lets the search continue. ``` ### Distance 1: ```text - Same evaluation as at distance 0. ``` ### Distance 2: ```text - Distance is still less than 4 and the first condition evaluates `false` (nothing is selected). - The beyond qualified condition for distance now evaluates to `false` as we are at distance 2. - The `or` however will examine key `name` of all elements at this distance and thus will evaluate to `true` only if the value is `B` and continue the search only beyond that element stopping at all others. ``` ### Distance 3: ```text - Same evaluation as 0 and 1. - This level is however reached only via elements that passed the previous beyond condition. ``` ### Distance 4: ```text - Distance is now 4 and the first condition evaluates `true` and elements will be selected for the result. - Algorithm is now stopped as nothing further away can satisfy the first condition and evaluation of the beyond conditions is therefore not relevant. ``` :: For general reference of the conditions and their evaluation see [truth tables in queries documentation](https://agdb.agnesoft.com/docs/references/queries#truth-tables). ::tip The algorithms will never visit same element twice and are immune to cycles. Similarly each element will appear in the result only once at the position of the first encounter. :: ## Conclusion As we have seen the `distance` is the powerful property of graphs and is one of the key advantages of modelling data on graphs as opposed to tables. They help us limit the scope of our search and can easily confine the search to small subgraph(s) even if the entire graph consists of billions of elements. Furthermore, you can easily extract related data without the need for joins and the algorithm will only examine the properties of the elements that are relevant to the search. # Why graph? The database area is dominated by relational database systems (tables) and text queries since the 1970s. However, the issues with the relational database systems are numerous, and they even gave rise the regular SW profession - database engineer. This is because contrary to their name they are very awkward at representing actual relations between data which is always demanded by the real world applications. They typically use foreign keys and/or proxy tables to represent them. Additionally, the tables naturally enforce fixed immutable data schema upon the data they store. To change the schema one needs to create a new database with the changed schema and copy the data over (this is called database migration). Such operation is very costly and most database systems fair poorly when there are foreign keys involved (requiring them to be disabled for the migration to happen). As it turns out nowadays no database schema is truly immutable. New and changed requirements happen so often that the database schemas usually need updating (migrating) nearly every time there is an update to the systems using it. There is no solution to this "schema" issue because it is the inherent feature of representing data in tabular form. It can be only mitigated to some degree but your mileage will vary greatly when using these techniques many of which are considered antipatterns. Things like indexes, indirection (storing data with varied length), storing blob data, data with internal format unknown to the database itself (e.g. JSON) are all the ways to prevent the need for database migration at the cost of efficiency. While there are good reasons for representing data in tabular form (lookup speed and space efficiency) the costs of very often far exceed the benefits. Plus as it turns out it is not even that efficient! The tables are represented as fixed size records (rows) one after another (this is what makes the schema immutable). This representation is the most efficient when we are reading entire rows at the time (all columns) which is very rarely the case. Most often we want only some of the columns which means we are discarding some (or most) of the row when reading it. This is the same problem the CPU itself has when using memory. It reads is using cache lines. If we happen to read only some of the line the rest is wasted and another line needs to be fetched for the next item(s) (this is called a `cache miss`). This is why contiguous collections (like a `vector`) are almost always the most efficient because they minimize the cache misses. Chandler Carruth had numerous talks at CPPCon on this subject demonstrating that by far the biggest performance impact on software are the cache misses (over 50 % and up to 80 % !!!) with everything else being dwarfed in comparison. Beside trying to optimize the tables the most prominent "solution" are the NoSQL databases. They typically use a different way to store data, often in a "schema-less" to cater to the above use cases - easing database migrations (or eliminating them) and providing more efficient data lookup. They typically choose some combination of key-value representation, document representation or a graph representation to scaffold the data instead of tables. They often trade in ACID properties, use write only - never delete "big tables" and other techniques. Of NoSQL databases the graph databases stand out in particular because by definition they actually store the relations between the data in the database. How the values are then "attached" to the graph vary but the graph itself serves as an "index" as well as a "map" to be efficiently searched and reason about. The sparse graph (not all nodes are connected to all others) representation is then actually the most flexible and accurate way to store and represent the sparse data (as mentioned, nearly all real world data is sparse). There are two key properties of representing data as a graph that directly relates to the aforementioned issues of schema and data searching. Firstly the graph itself is the schema that can change freely as needed at any time without any restrictions eliminating the schema issue entirely. You do not need to be clairvoyant and agonize over the right database schema. You can do what works now and change your mind later without any issues. Secondly the graph allows accurately representing any actual relations between the data allowing the most efficient native traversal and lookup of data (vaguely resembling traditional indexing) making the lookup constantly efficient regardless of the data set size. Where table performance will deteriorate as it grows the graph will stay constantly efficient if you can traverse only the subset of the nodes via their relations even if the graph itself contained billions of nodes. This is in a nutshell why the graph database is the best choice for most problem domains and data sets out there and why `agdb` is the graph database. ## Costs Everything has the cost and graph databases are no exception. Some operations and some data representations may be costlier in them as opposed to table based databases. For example if you had immutable schema that never updates then table based database might a better fit as the representation in form of tables is more storage efficient. Or if you always read the whole table or whole rows then once again the table based databases might be more performant. Typically, though these are uncommon edge cases unlikely to be found in the real world applications. The data is almost always sparse and diverse in nature, the schema is never truly stable etc. On the other hand most use cases benefit greatly from graph based representation and thus such a database is well worth it despite some (often more theoretical) costs. ## Why not an existing graph database? The following is the list of requirements for an ideal graph database: - Free license - Faster than table based databases in most common use cases - No new language for querying - No text based queries - Rust and/or C++ driver - Resource efficient (storage & memory) Surprisingly there is no database that would fit the bill. Even the most popular graph databases such as `Neo4J` or `OrientDB` fall short on several of these requirements. They do have their own text based language (e.g. Cypher for Neo4J). They lack the drivers for C++/Rust. They are not particularly efficient (being mostly written in Java). Even the recent addition built in Rust - `SurrealDb` - is using text based SQL queries. Quite incomprehensibly its driver support for Rust itself is not very mature so far and was added only later despite the system being written in Rust. Something which is oddly common in the database world, e.g. `RethinkDb`, itself a document database, written mostly in C++, has no C++ support but does officially support for example Ruby. Atop of these issues they often do not leverage the graph structure very well (except for Neo4J which does great job at this) still leaning heavily towards tables. # Object queries The most ubiquitous database query language is SQL which is text based language created in the 1970s. Its biggest advantage is that being text based it can be used from any language to communicate with the database. However, just like relational (table) bases databases from the same era it has some major flaws: - It needs to be parsed and interpreted by the database during runtime leading to common syntax errors that are hard or impossible to statically check. - Being a separate programming language from the client coding language increases cognitive load on the programmer. - It opens up the database to attacks from SQL-injection where the attacker is trying to force the interpreter to treat the user input (e.g. table or column names) as SQL code itself issuing malicious commands such as stealing or damaging the data. - Being "Turing-complete" and complex language on itself means it can lead (and often leads) to incredibly complex and unmaintainable queries. The last point is particularly troublesome because it partially stems from the `schema` issue discussed in the previous points. One common way to avoid changing the schema is to transform the data via queries. This is not only less efficient than representing the data in the correct form directly but also increases the complexity of queries significantly. The solutions include heavily sanitizing the user inputs in an attempt to prevent SQL injection attacks, wrapping the constructing of SQL in a builder-pattern to prevent syntax errors and easing the cognitive load by letting programmers create their queries in their main coding language. The complexity is often being reduced by the use of stored SQL procedures (pre-created queries). However, all of these options can only mitigate the issues SQL has. Using native objects representing the queries eliminate all of SQL issues sacrificing the portability between languages. However, that can be relatively easily be made up via already very mature (de)serialization of native objects available in most languages. Using builder pattern to construct these objects further improve their correctness and readability. Native objects carry no additional cognitive load on the programmer and can be easily used just like any other code. # Single file All operating systems have fairly low limit on number of open file descriptors for a program and for all programs in total making this system resource one of the rarest. Furthermore, operating over multiple files does not seem to bring in any substantial benefits for the database while it complicates its implementation significantly. The graph database typically needs to have access to the full graph at all times unlike say key-value stores or document databases. Splitting the data into multiple files would therefore be actually detrimental. Lastly overall storage taken by the multiple files would not actually change as the amount of data would be the same. Conversely, using just a single file (with a second temporary write ahead log file) makes everything simpler and easier. You can for example easily transfer the data to a different machine - it is just one file. The database can also operate on the file directly if memory mapping was turned off to save RAM at the cost of performance. The program would not need to juggle multiple files consuming valuable system resources. The one file is the database and the data. # Sharding, replication and performance at scale Most databases tackle the issue of (poor) performance at scale by scaling up using replication/sharding strategies. While these techniques are definitely useful, and they are planned for `agdb` they should be avoided as much as possible. The increase in complexity when using replication and/or sharding is dramatic, and it has adverse performance impact meaning it is only worth it if there is no other choice. The `agdb` is designed so that it performs well regardless of the data set size. Direct access operations are O(1) and there is no limit on concurrency. Write operations are O(1) amortized however they are exclusive - there can be only one write operation running on the database at any given time preventing any other read or write operations at the same time. You will still get O(n) complexity when searching the (sub)graph as reading 1000 connected nodes will take 1000 O(1) operations = O(n) same as reading 1000 rows in a table. However, if the data does not indiscriminately connect everything to everything one can have as large data set as the hardware can fit without performance issues. The key is querying only subset of the graph (subgraph) since your query will have performance based on that subgraph and not all the data stored in the database. The point here is that scaling has significant cost regardless of technology or clever tricks. Only when the database starts exceeding limits of a single machine they shall be considered because adding data replication/backup will mean huge performance hit. To mitigate it to some extent caching can be used, but it can never be as performant as local database. The features "at scale" are definitely coming you should avoid using them as much as possible even if available. [For real world performance see dedicated documentation.](https://agdb.agnesoft.com/docs/references/performance) # Why not SQL? SQL, or Structured Query Language, is a programming language that has dominated the database space for decades. Initially created 50 years ago in 1974 just two years after C. Despite its undeniable success its age means it has many less-than-ideal properties particularly unsuitable for a modern database(s) like the `agdb`. ## Yet another programming language SQL is a Turing complete programming language which has grown to be vast, complex and complicated. When a programmer needs to access or manipulate a database using SQL they need to learn entire additional language in addition to the one in which they are writing their program. This cognitive (over)load is very often being solved by abstracting the "raw" SQL behind an ORM (Object-relational mapping) to seemingly use the database directly from the main programming language. The keyword here is seemingly because under the hood any ORM will simply translate to a "raw" SQL that quite often might not be as performant as when written by hand. ::tip ORM often violates the famous zero-cost abstraction rule favorite in languages like C++ or Rust that states "the abstraction is as performant as if written by hand". :: The textual nature of SQL means that it can be used from any programming language with ease, but that is of little importance for an individual programmer writing it in their program. While they enjoy the benefits of their IDE and language server for their primary languages, there is not much that can be done about the embedded SQL statements in their code. Until they are run against a database it cannot even be known if even the basic syntax is correct (certain plugins do offer limited SQL support for syntax etc. but the ultimate truth is the particular implementation of the given database). This unpleasant experience is worsened further with advent of new programming tooling like Copilot where the main language support is better than ever while SQL support lags behind as it is nearly impossible to improve the experience in that area. ## Interpreting an injection SQL injection is one of the most prevalent attacks using SQL programming languages. This is made possible by two major properties: 1. SQL is a textual language. 2. SQL is a runtime-interpreted language. There is a long Wikipedia page on [this attack](https://en.wikipedia.org/wiki/SQL_injection){rel=""nofollow""} citing serious recent cases of data breaches due to it. This problem cannot be definitively solved because it stems from the fundamental properties of the language. As long as it must take user input and is effectively interpreted only during runtime this type of attack can and will happen. Various mitigation strategies while effective (e.g. stored procedures, database permissions) also increases complexity and cost of using the language. Probably the closest thing to eliminate SQL injection attacks entirely is use of server-side stored procedures. Essentially a type of remotely executed pre-defined function. However, the procedure itself must take precautions and to sanitize the inputs. Making changes to remote procedures is also much more difficult than simply changing the SQL statement in your code. Not to mention that even when the client code is perfectly fine and sanitized the SQL interpreted on the database side does not necessarily have to be (although it typically will be). ## Complexity and performance As mentioned the SQL is a regular programming language and as such it allows writing literally anything in it. Back in the 70s and decades after this property was very valuable because client machines (unlike database servers) were not particularly performant. Letting the server do the heavy lifting and massaging the data into the correct output made sense and often was the only viable approach. Unfortunately it can also lead to quite complex queries. Have you ever seen 10 000 line long SQL query? Yes, singular. And yes, 4 zeroes. Making changes to such a query will take many days spent on understanding it and figuring out how to change it. Nowadays, client machines are so powerful that doing any extra work on any server is generally undesirable. The situation is pretty much completely reversed. Particularly embedding application/business logic in database queries can lead to hard to change and hard to maintain code because SQL is not the primary language of any application and is often not even very suitable for it in the first place (it is a domain specific language for interfacing with the databases). Performance of SQL can never match compiled or even JIT compiled languages. Being interpreted means it must be parsed (in itself very non-trivial task that can lead to SQL injections mentioned above) and then translated into internal database structures and commands to be executed. And once more mitigation steps exist (like stored procedures again) but they are simply a patch over a fundamental property of the language. ## An alternative Despite all of its shortcomings the SQL is still the dominant database language and the `agdb` will have to support it at some point to certain degree as well. Somewhat frustratingly there is no good off the shelf alternative. Many no-SQL databases went on and created their own textual interpreted language immediately suffering from the very same issues described above but without any of the 50 years of hardening SQL has had. And the only benefit of being usable from "any" programming language is usually immediately thrown away by the creation of an ORM for each language. So what is the alternative? [Object based queries](https://agdb.agnesoft.com/blog/object-queries). Having a binary compiled format eliminates most of the above problems. A binary query cannot morph into another because of some user input. The code, including the queries, is written entirely and only in the language of choice. The performance is unmatched on both sides as the database does not need to interpret the text into its own structures and everything is as expected (or results in a query error if not). Language server and IDE can easily help like with any other code. There can never be a syntax error. It does not lend itself very easily to a thousand lines long data queries and basically forces the application to process the data in itself rather than on the database server (beyond basic operations like sorting or filtering). The cost of course is that the query system needs to be either written or generated for each language (initial cost) and that such a query system is not Turing complete programming language (which is perhaps more of a benefit). And that seems like a small price for everything else. Nearly everything has changed and after 50 years it is time for something better in the database space and to start treating SQL like we do the languages from the same era - replace them with better and safer alternatives. # Blog Articles written about the `agdb` and related topics adding insights into technologies and thinking behind the database. ## Articles - [Distance](https://agdb.agnesoft.com/blog/distance) - [Why graph?](https://agdb.agnesoft.com/blog/why-graph) - [Object queries](https://agdb.agnesoft.com/blog/object-queries) - [Single file](https://agdb.agnesoft.com/blog/single-file) - [Replication, sharding and performance](https://agdb.agnesoft.com/blog/replication-sharding-performance) - [Why not SQL?](https://agdb.agnesoft.com/blog/why-not-sql) # Technical Consultation We offer technical consultation for your project. Our experts will help you answer any of your questions, and we offer a free, 30 minute consultation to get you started. Drop us an e-mail to book a slot at: ## Enterprise & Commercial Support If you are interested in using `agdb` in your business please visit the dedicated [enterprise](https://agdb.agnesoft.com/enterprise) section for commercial options. ## General Questions - Search the [documentation](https://agdb.agnesoft.com/docs). - Send an e-mail with your questions to - Post a question in [r/agdb](https://www.reddit.com/r/agdb){rel=""nofollow""} subreddit. - Open a [discussion](https://github.com/agnesoft/agdb/discussions){rel=""nofollow""}. - Open an [issue](https://github.com/agnesoft/agdb/issues){rel=""nofollow""}. # Cloud The cloud solution is currently not publically available. Please contact us at . # Self-hosted If you are interested in self-hosting `agdb` we offer basic guides with server or cluster variants for the publically available target platforms: | Type / Target | Bare Metal | Docker | K8s | | ------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------- | | Server | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-bare-metal) | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-docker) | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/server-k8s) | | Cluster | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-bare-metal) | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-docker) | [LINK](https://agdb.agnesoft.com/docs/guides/how-to-run-server/cluster-k8s) | If you require support setting up `agdb` in your environment please contact us at: # Support We offer following support channels: - [Technical Consultation](https://agdb.agnesoft.com/#technical-consultation) - Post a question in [r/agdb](https://www.reddit.com/r/agdb/){rel=""nofollow""} subreddit. - Open a [discussion](https://github.com/agnesoft/agdb/discussions){rel=""nofollow""}. - Open an [issue](https://github.com/agnesoft/agdb/issues){rel=""nofollow""}. For additional or dedicated support please contact us at: # Sponsors The sponsorship is available on GitHub at: {rel=""nofollow""} If you would like to become an official sponsor, get recognition on the homepage as well as in the repository please reach out to: # Enterprise The `agdb` is open-source software provided as-is under [Apache 2.0 license](https://agdb.agnesoft.com/license). You can obtain a commercial license that would grant you access to various services like a cloud hosted database, support for self-hosted solutions and direct general support. You can also consider [sponsoring](https://agdb.agnesoft.com/enterprise/sponsors) the development of `agdb` and get mentioned on the homepage and in the repository! - [Technical Consultation](https://agdb.agnesoft.com/enterprise/consultation) - [Self-hosted](https://agdb.agnesoft.com/enterprise/self-hosted) - [Cloud](https://agdb.agnesoft.com/enterprise/cloud) - [Support](https://agdb.agnesoft.com/enterprise/support) - [Sponsors](https://agdb.agnesoft.com/enterprise/sponsors)