TinkerPop Compliance

Graphersal measures its Gremlin compliance against the reference semantics that Apache TinkerPop ships as executable Gherkin scenarios. Each scenario names a graph, a traversal and the exact expected result, so the suite works as a ready-made differential test against real Gremlin. It serves three purposes:

1. Compliance measure: what passes, what does not, and why, grouped by root cause.
2. Regression gate: a scenario that passed must keep passing; cargo test fails otherwise.
3. Backlog generator: the failure report ranks the fix tasks.

The suite

• Pinned version: Apache TinkerPop 3.8.2 (commit 952d8d8e4c3cd37ed8e192ecb97348e6e82594a9), vendored under crates/graphersal/tests/tinkerpop/features/. Only the scenarios in the compatibility scope count (see Out of scope and Deliberate incompatibilities); their number is in the generated headline numbers. The vendored directory's README.md records the upgrade procedure.
• Runner: the integration test target crates/graphersal/tests/tinkerpop_features.rs, with its harness in crates/graphersal/tests/tinkerpop/harness/. Every scenario is translated to the Rhai DSL and runs through the script engine, the same path users take, so the suite also tests DSL parity.
• Gate: crates/graphersal/tests/tinkerpop/baseline.toml lists every scenario that does not pass yet, with its class and a one-line reason.

Headline numbers

The numbers below are the suite's result on the current code, with the strategy scenarios and the deliberate incompatibilities excluded from the compatibility scope. All totals and percentages count in-scope scenarios only. The scope, the tables and the counts between the tinkerpop:begin/tinkerpop:end markers on this page are generated by the suite and checked by every test run (see Running and regenerating); never edit them by hand.

Scope of this run:

1675 in-scope scenarios; 215 excluded (strategies), 182 excluded (deliberate incompatibilities).

Unsupported = translate 51, missing 20. Wrong behaviour = runtime 0, wrong_result 0, wrong_error 0, timeout 0, panic 0.

Every remaining non-pass is a missing step, overload or token, or a literal type the translator cannot express (see Classes); the scenarios that ran but differed on purpose are the deliberate incompatibilities. The per-class counts are in the line below the table.

API inventory (TinkerPop 3.8.2): 22 of 151 methods, 6 of 21 token classes, 39 of 127 members of registered token classes not registered

The largest remaining blockers are the date/time steps and literals (asDate, datetime, planned for 0.2.0), the set literals and GType.SET/TREE/VPROPERTY (no such value types), ranked with their scenario counts in the report and in the API inventory section of it. The full ranking is in the report. The open gaps are listed in TinkerPop Deviations.

Out of scope

Graphersal will provide subgraph views, partitions, read-only mode and row-level security through its own, more flexible access-control mechanism, and execution settings through its own unified, hierarchical execution-metadata mechanism (replacing with()/withStrategies()/parameters) — not through TinkerPop TraversalStrategys. GraphComputer/OLAP strategies do not apply to an in-memory OLTP engine. Therefore every scenario whose traversal, graph initializer or graph-count traversal calls withStrategies(...) or withoutStrategies(...) is excluded from the compatibility scope: it is not run, gets the class excluded, has no baseline entry, and is left out of every total and percentage. Scenarios that only use withSideEffect, withSack or with(key, value) stay in scope. The rule matches the strategy call wherever the scenario lives, not the feature file name; the table is crates/graphersal/tests/tinkerpop/harness/scope.rs.

The counts per reason and strategy (generated):

Excluded from compatibility scope: 215 scenarios [views/partitions/read-only/row-level security are provided by Graphersal's own access-control mechanism, not by TinkerPop strategies: 93 (SubgraphStrategy 62, PartitionStrategy 24, ReadOnlyStrategy 7); GraphComputer/OLAP — not applicable to Graphersal: 16 (VertexProgramStrategy 3, VertexProgramRestrictionStrategy 2, ComputerVerificationStrategy 2, ComputerFinalizationStrategy 2, HaltedTraverserStrategy 3, MessagePassingReductionStrategy 2, GraphFilterStrategy 2); TinkerPop strategy configuration is not supported — Graphersal will provide its own unified, hierarchical execution-metadata mechanism (replacing with()/withStrategies()/parameters): 106 (AdjacentToIncidentStrategy 4, ByModulatorOptimizationStrategy 2, ConnectiveStrategy 2, CountStrategy 2, EarlyLimitStrategy 3, EdgeLabelVerificationStrategy 3, ElementIdStrategy 2, FilterRankingStrategy 2, IdentityRemovalStrategy 2, IncidentToAdjacentStrategy 2, InlineFilterStrategy 2, LambdaRestrictionStrategy 2, LazyBarrierStrategy 2, MatchAlgorithmStrategy 3, MatchPredicateStrategy 2, OptionsStrategy 3, OrderLimitStrategy 2, PathProcessorStrategy 2, PathRetractionStrategy 2, ProductiveByStrategy 29, ProfileStrategy 2, ReferenceElementStrategy 2, RepeatUnrollStrategy 18, ReservedKeysVerificationStrategy 3, SeedStrategy 6, StandardVerificationStrategy 2)]

Most of them are in integrated/ (all its strategy feature files); the rest live in a few other files (map/Max, Mean, Min, Sum, filter/Sample, sideEffect/Aggregate and single scenarios elsewhere). A feature file whose scenarios are all excluded is not listed in the report's per-file table. The report's "Excluded from compatibility scope" section lists the counts per reason and strategy and every excluded id.

Deliberate incompatibilities

Some scenarios cannot pass because Graphersal chose a different behaviour, or does not plan the feature at all. Each reason is a design decision, is documented on the page linked in the table and as a row of TinkerPop Deviations, and is listed explicitly by scenario id in INCOMPATIBILITIES in crates/graphersal/tests/tinkerpop/harness/scope.rs (ids, never tags, so a scenario a suite upgrade adds is never excluded silently). These scenarios are not run, have no baseline entry and are in no total or percentage; a guard test fails when one of them starts to pass, and another when an id does not exist. To move one back into scope, remove its id from the table and run the suite.

The report's "Deliberate incompatibilities" section has the same table, with the full id list per code.

Running and regenerating

# Run the suite (also part of the plain `cargo test --package graphersal --features script,io,display,serde,persist`)
cargo test --package graphersal --features script,io,display,serde,persist --test tinkerpop_features

# Rewrite the baseline AND the generated blocks of this page after intended changes; review the diff like code
TINKERPOP_UPDATE_BASELINE=1 cargo test --package graphersal --features script,io,display,serde,persist --test tinkerpop_features

Every run prints the summary to stderr and writes the full report to target/tinkerpop-report.md: headline, per-category and per-file tables, failures grouped by class (within missing ranked by the missing name, within runtime/wrong_result by the exercised step), baseline class changes, order-relaxed passes and the API inventory. TINKERPOP_DETAILS=1 prints one line per non-passing scenario; TINKERPOP_LANES=<n> sets the number of parallel worker lanes (default: one per core).

The test fails when:

• a scenario that is not in the baseline fails (a regression);

• a scenario in the baseline passes now, no longer exists or is excluded from the scope (the entry is stale; remove it in the same change);

• an id of the deliberate-incompatibility table does not exist, is also excluded by a strategy, has no existing doc page, or names a scenario that passes now (deliberate_table_is_consistent, deliberate_scenarios_do_not_pass_silently).

• a generated block of this page (between <!-- tinkerpop:begin NAME --> and <!-- tinkerpop:end NAME -->: scope, headline, categories, inventory, excluded, incompat) differs from what the run computes. The message shows the differing lines and the command above. Only trailing whitespace is ignored. A missing, duplicated or unclosed marker fails with a clear error too.

A normal run never writes a file (apart from the report under target/). With TINKERPOP_UPDATE_BASELINE=1 the run rewrites baseline.toml and the text between the markers of this page, nothing else: the prose and the step tables are hand-written. The blocks render the same Markdown as target/tinkerpop-report.md, so the page and the report cannot disagree. The page path resolves through CARGO_MANIFEST_DIR, so the suite works from the repository root and from the crate directory. Commit the regenerated page together with the change that moved the numbers.

A baseline scenario that keeps failing under a different class is reported, not failed.

GRAPHERSAL_TEST_STORAGE=forwarding runs the whole suite on a storage that is not TraversalGraph (the Forwarding wrapper of graphersal-storage-tests, with its own element handle types) to prove the front ends are storage-independent: it is gated against the same baseline and generated blocks, additionally fails when a scenario changes class, never rewrites either (TINKERPOP_UPDATE_BASELINE is refused with it) and writes its report to target/tinkerpop-report-forwarding.md.

Classes

The headline splits the non-passing, non-skipped scenarios into two groups, because they mean different things. Graphersal does not plan to support 100 % of Gremlin's steps, but every step it does support must behave as the specification says:

• Unsupported (classes missing and translate): the step, overload, token or literal type does not exist in Graphersal. An open feature gap, not a defect.
• Wrong behaviour of supported steps (classes runtime, wrong_result, wrong_error, timeout, panic): the step exists and runs, but differs from the specification or fails. This is the number to keep at 0, and any entry in it is a bug.

The report, the console summary and the per-category and per-file tables use the same two groups; the per-class counts stay in a line below the headline table and in the "Failures by class" section.

Documented normalizations

The runner applies a small, fixed set of normalizations. None of them hides a semantic gap.

• Tolerant id comparison. Graphersal's element ids are strings, TinkerPop's stock datasets use small integers with the same values. An expected d[N].i/d[N].l that stands for an id is stringified and compared with the string id. This applies to ids only.
• Number-class comparison. Graphersal has int64 and float64 only, so numbers compare by class (integer vs. floating point), ignoring the declared width: d[1].i and d[1].l both match an int64, d[1].f and d[1].d both match a float64.
• 1 ulp float tolerance. Two float64 result values compare equal when they are equal or differ by at most 1 ulp (NaN equals NaN, infinities exact, +0 equals -0, other signs must agree). This absorbs last-digit differences between Rust's libm and Java (sin(4.0)), not an arithmetic deviation of Graphersal (see Deviations). It applies to float vs. float result values only, never to integers, ids, orderings or error cases.
• Order relaxation. When a scenario expects an ordered result, the sequence differs but the multiset matches, and the translated traversal has no explicit ordering step (order()/by()), the scenario passes as an "order-relaxed pass", listed separately in the report. Graphersal does not guarantee TinkerGraph's insertion order across GraphStorage implementations. With an explicit ordering step the exact sequence is required.
• Tree assertions. the result should be a tree with a structure of compares the single result as a map of key to nested map (a leaf is an empty map). Keys are compared by value (vertices by id, numbers by class, strings) and the siblings of a node match as a set, because TinkerPop's Tree is an unordered map.
• Subgraph assertions. the result should be a subgraph with the following compares the graph value's {"vertices": [..], "edges": [..]} listing with the edge and the vertex table, each as a multiset (vertices and edges by id).
• Lexical-only translation. The translator from canonical Gremlin to the Rhai DSL rewrites only lexical differences between Groovy and Rhai: numeric suffixes (1L, 1d), list and map literals, string quoting, parameter and side-effect bindings, and Groovy's static imports (a bare out() becomes __.out(), a bare desc becomes Order.desc). Token map keys (T.label:, (T.id):, Direction.OUT:, and the t[..]/D[..] keys of parameter maps) become the DSL's reserved string keys "T.label", "T.id", "Direction.OUT", "Direction.IN". A missing step, overload or token is never emulated: it reaches the engine and fails there, counted against that step.