Custom Storages

Graphersal's reference storage is TraversalGraph, an in-memory graph. A host (a server, an application) can bring its own in-memory storage instead: any type that implements the GraphStorage trait. The engine, the Rhai DSL, saved queries, the session of the playground and the dev server (graphersal-session) and the persist Store then work over it.

EXPERIMENTAL in 0.1.x. GraphStorage, PersistentStorage, AnyGraph and the conformance crate graphersal-storage-tests may change in any 0.1.x release. Pin the exact versions (graphersal = "=0.1.0", graphersal-storage-tests = "=0.1.0"). graphersal-session is internal and has no stability guarantees at all.

How it fits together

The engine (GraphTraversalSource<'g, S>) is generic over the storage and statically dispatched: a traversal over a host storage is compiled for that storage, with no virtual call per element. Everything above the engine holds one object-safe type per layer instead, and makes one virtual call per operation (a terminal, a schema method, a catalog call, a view):

Graph<S> (in graphersal::storage, also in the prelude) is the shared form of a storage: S behind a read-write lock, with read()/write() guards that dereference to S. Graph without a parameter means Graph<TraversalGraph>.

What a host implements

1. GraphStorage (required). Only the read methods, the element-creating and element-removing methods and the property writes are required; everything else has a default that is correct for a simple, non-transactional storage. The trait's rustdoc ("A guide for storage implementers") is the reference: handles that never alias, the write checks and their public helpers, units of atomicity, capabilities, change capture, statistics, schema and the catalog.
2. Default (required by the DSL and the session). A whole script runs as one unit by moving the storage out of the shared lock while the write lock is held (std::mem::take), which leaves an empty instance behind. The session also builds every new graph (a sample, a file load) by reading it into a TraversalGraph and copying it into an S::default() (through the load sink of a PersistentStorage, through the write methods otherwise).
3. Send + Sync + 'static, so the graph can be shared between threads.
4. Change capture (optional; needed for commit hooks, dry runs and the Store). A storage that claims change_capture (and therefore atomic) in capabilities() builds the ChangeSet of every committed unit itself with the public constructors (ChangeSet::builder, one Mutation constructor per kind). There is no reusable undo log or capture component: the storage derives what a unit changed from its own bookkeeping. It embeds a CommitHooks (graphersal::storage) and calls it at its commit point, so the order of the hooks (user hooks in registration order, then the journal) is implemented once; the rustdoc of CommitHooks says when to call each event.
5. PersistentStorage (optional, feature persist; needed for snapshots and the Store): the lineage id, the commit position, a load sink outside units, clear, the journal slot and replay. See Your Own Storage.

Units and transactions come for free: the Transactional extension trait (transaction, transaction_with, dry_run) is implemented for every GraphStorage on top of the trait's unit methods. dry_run needs change_capture.

Plugging it in

A Store requires the capabilities atomic and change_capture and fails with StoreFailure::MissingCapability without them. The files a Store<S> writes do not depend on S: a store written by a host storage opens as Store<TraversalGraph> (in graphersal store, the dev server, Python) and the other way round.

A complete host

The example below is a test of graphersal-session (crates/graphersal-session/tests/custom_storage_example.rs), compiled and run by cargo test -p graphersal-session. The Forwarding storage of graphersal-storage-tests stands in for the host's storage (see below).

use std::sync::Arc;

use graphersal::TraversalGraph;
use graphersal::persist::{MemDir, Store, StoreDir, StoreOptions};
use graphersal::prelude::{ElementProperty, GraphTraversalSource};
use graphersal::script::eval_value;
use graphersal::storage::{Graph, GraphStorage, Transactional};
use graphersal_session::{RunOptions, Session, StorageKind};
use graphersal_storage_tests::forwarding::Forwarding;

/// The host's storage. A real host writes its own `GraphStorage` (+ `Default`, and
/// `PersistentStorage` to be stored); `Forwarding` stands in for it.
type MyStorage = Forwarding;

fn modern() -> MyStorage {
    Forwarding::new(TraversalGraph::tinkerpop_modern())
}

#[test]
fn a_host_plugs_its_own_storage_into_every_front_end() -> Result<(), Box<dyn std::error::Error>> {
    // 1. The engine is generic: a traversal over `&S` (reads) or `&mut S` (writes).
    let mut storage = modern();
    let people = GraphTraversalSource::new(&storage)
        .v(None)
        .has_label("person")
        .count()
        .next()?;
    assert_eq!(people, Some(ElementProperty::from(4)));
    storage.transaction(|g| {
        GraphTraversalSource::new(&mut *g).add_v("robot").next()?;
        Ok::<_, Box<dyn std::error::Error>>(())
    })?;
    assert_eq!(storage.vertex_count(), 7);

    // 2. The DSL and saved queries take any `Arc<Graph<S>>` (`impl IntoAnyGraph`).
    //    `new_persistent` also gives the DSL's `g.export_snapshot(path)`.
    let graph = Arc::new(Graph::new_persistent(modern()));
    let _definition = eval_value(
        graph.clone(),
        r#"g.define_query(#{name: "older_than", params: #{age: "integer"},
               body: "fn older_than(age) { g.V().has(\"age\", P.gt(age)).values(\"name\").order() }"})"#,
    )?;
    let names = eval_value(
        graph.clone(),
        r#"g.query("older_than", #{age: 30}).toList()"#,
    )?;
    assert_eq!(names.to_string(), r#"["josh", "peter"]"#);

    // 3. A Store of the storage: `create_in` / `open_in` (it must claim `atomic` and
    //    `change_capture`). Every commit goes to the write-ahead log.
    let dir: Arc<dyn StoreDir> = Arc::new(MemDir::named("custom-storage-example"));
    let store = Store::create_in(&dir, modern(), StoreOptions::new())?;
    store.graph().write().transaction(|g| {
        GraphTraversalSource::new(&mut *g)
            .add_v("person")
            .property("name", "ada")
            .next()?;
        Ok::<_, Box<dyn std::error::Error>>(())
    })?;
    store.close()?;
    // `open_in` takes how to build the empty instance the snapshot is loaded into.
    let store: Store<MyStorage> = Store::open_in(&dir, StoreOptions::new(), MyStorage::default)?;
    assert_eq!(store.graph().read().vertex_count(), 7);

    // 4. A session over the storage: every graph it builds (samples, loads, stores in memory,
    //    forks) is a `MyStorage`; here it serves the Store opened above.
    let mut session = Session::with_storage(Arc::new(StorageKind::<MyStorage>::persistent()));
    session.attach_store(Arc::new(store), "my store");
    let answer = session.execute(
        r#"g.V().has("name", "ada").count().next()"#,
        &RunOptions::default(),
    );
    assert!(answer.contains(r#""ok":true"#), "{answer}");
    assert_eq!(
        session.graph().storage_type_name(),
        std::any::type_name::<MyStorage>()
    );
    let mark = session.store_mark("after-ada")?;
    assert_eq!(mark["mark"]["name"], "after-ada");
    Ok(())
}

What stays TraversalGraph-only

• Scratch graphs. Graphs a script creates itself are always TraversalGraphs, whatever the host's storage: GraphSource::*, __, _g, the results of subgraph()/cap()/toGraph() and schema.toGraph(). A host-chosen scratch storage is not planned.
• withSideEffect(name, graph) into a host graph. A subgraph() writes only into a TraversalGraph; a foreign target fails with GraphError::Unsupported. Let the subgraph land in a scratch graph and copy what is needed.
• Compressed properties. A foreign storage keeps compression rules in its catalog like any definition (defined, listed, persisted, dropped), but nothing applies them: recompress fails with GraphError::Unsupported and the compression statistics are absent. The session's compression manager lists and edits the rules and says "not supported by this storage" for the rest.
• The undo log and its change-capture derivation. A foreign storage brings its own (see "Change capture" above).
• The Store's internal working graphs. The merge checkpoint's delta graph, salvage (maintenance mode) and repair work on internal TraversalGraphs and write files; they never touch the host's graph.
• The shipped front ends. The CLI (REPL, one-shot runner), the dev server and its MCP endpoint, the Python binding and the WebAssembly playground serve TraversalGraph. A host that serves its own storage builds on graphersal-session instead.
• Sample graphs and TraversalGraph::with_execution_policy. The sample constructors (TraversalGraph::tinkerpop_modern(), ...) build TraversalGraphs (the session copies them into the host storage); the trait method execution_policy() gives a host storage the same policy hook.

Operations a storage does not implement fail with GraphError::Unsupported, which names the operation; its help points to the storage's documentation and capabilities.

Testing a storage

The conformance suite

Add graphersal-storage-tests as a dev-dependency (with its persist feature to test the PersistentStorage side) and invoke the macro in one test of the implementing crate:

graphersal_storage_tests::conformance!(my_storage, base = MyStorage::default() => base);

Each contract item becomes its own test; the capability-conditional items (rollback, savepoints, hooks, change sets, dry runs) run for what capabilities() claims. See Storage Conformance.

The Forwarding pattern

graphersal_storage_tests::forwarding::Forwarding forwards every GraphStorage (and, with persist, PersistentStorage) method to a wrapped TraversalGraph, but has its own element handle types. Graphersal uses it to prove that the front ends are storage-independent:

• crates/graphersal/tests/all/foreign_storage_tests.rs runs every documented DSL example and a curated set of mutating, whole-script, dry-run and saved-query scripts on both storages and requires the same results;
• crates/graphersal/tests/all/foreign_store_tests.rs runs the Store on Forwarding over every backend (commits, reopen, checkpoints, rollback, backups, ZIP, marks, damage) and opens the files with the other storage type;
• crates/graphersal-session/src/storage/tests.rs requires the same JSON from a session over either storage;
• GRAPHERSAL_TEST_STORAGE=forwarding runs the whole TinkerPop suite on it, gated against the same baseline;
• the CI job "foreign storage proof" runs all four.

A host can follow the same pattern for its own storage: run the same scripts on a Graph<TraversalGraph> and on a Graph<MyStorage> loaded with the same data and compare the rendered results. Differences are either bugs of the storage or behaviour the host chose (and documents).

Cost

A storage behind Graph<S> runs within 1-4 % of TraversalGraph itself on the measured traversals (measured with the Forwarding test storage), because the engine is monomorphised for it. The price is build time: one engine instantiation per storage type a host uses.