1. Kenny Helsens, who presented a wrap-up of the zimmo.be project
2. Gabriel Reid, who gave an introduction to Apache Crunch, of which is is a committer
3. Myself, giving a presentation on Datomic and FluxGraph

The presentation highlights various aspects that, to my opinion, set Datomic apart from other types of noSQL databases. At the end of the presentation, I introduced FluxGraph, a distributed, temporal graph database implemented on top of Datomic. FluxGraph implements the Tinkerpop Blueprints API and allows users to perform queries against a specific version of the graph in the past.

Already looking forward to the next BigData.be MeetUp!

1. mongoDB Graph

The mongoDB Blueprints implementation provides users with a scalable, distributed Graph Database implementation. mongoDB graph does not require any information on the underlying physical mongoDB setup, so read/write scalability can easily be achieved through mongoDB’s natively supported replication and sharding functionalities. Some initial benchmarks (writing/reading 100.000 vertices where each 2 vertices are connected through an edge) on a single mongoDB node shows the following performance:

Not too shabby for running it on a single mongoDB instance. Currently, each write is executed as a singular commit to the mongoDB store. Although this process could be improved by performing writes through optimized batches, we need to ensure that the transactional semantics of the Blueprints stack are still respected. Further performance optimizations will be released soon, so keep an eye on the blueprints-mongodb-graph github project.

2. Datomic Graph

Datomic is a novel distributed database system designed to enable scalable, flexible and intelligent applications, running on next-generation cloud architectures. Datomic employs a powerful data model (based upon the concept of Datoms) and an expressive query language (based upon the concept of Datalog. Additionally, it introduces an explicit notion of time, which allows for the execution of queries against both the previous and future states of the database. The RDF and SPARQL feel of the Datomic data model and query approach makes it an ideal target for implementing a property graph. Hence, the blueprints-datomic-graph Blueprints implementation.

Clever use the time-aware nature of the Datomic datastore, makes Datomic Graph the very first distributed, temporal graph database: users can perform queries against a specific version of the graph in the past. The code sample below illustrates how a time-aware social graph can be created, stored and queried through the Datomic Graph implementation.

As one would expect, this outputs the following information:

Datomic Graph does not only support versioning of the vertices and edges, but also on the properties of individual vertices/edges. Pretty slick, not? We are currently enhancing the implementation with additional time-based operations, including the easy comparison of subgraphs over time. So stay tuned!

1. The Datomic data model

Datomic stores facts (i.e. your data points) as datoms. A datom represents the addition (or retraction) of a relation between an entity, an attribute, a value, and a transaction. The datom concept is closely related to the concept of a RDF triple, where each triple is a statement about a particular resource in the form of a subject-predicate-object expression. Datomic adds the notion of time by explicitly tagging a datom with a transaction identifier (i.e. the exact time-point at which the fact was persisted into the Datomic database). This allows Datomic to promote data immutability: updates are not changing your existing facts; they are merely creating new datoms that are tagged with a more recent transaction. Hence, the system keeps track of all the facts, forever.

Datomic does not enforce an explicit entity schema; it’s up to the user to decide what type of attributes he/she want to store for a particular entity. Attributes are part of the Datomic meta model, which specifies the characteristics (i.e. attributes) of the attributes themselves. Our genealogical example data model stores information about persons and their ancestors. For this, we will require two attributes: name and parent. An attribute is basically an entity, expressed in terms of the built-in system attributes such as cardinality, value type and attribute description.

All entities in a Datomic database need to have an internal key, called the entity id. In our case, we generate a temporary id through the tempid utility method. All entities are stored within a specific database partition that groups together logically related entities. Attribute definitions need to reside in the :db.part/db partition, a dedicated system partition employed exclusively for storing system entities and schema definitions. :person/name is a single-valued attribute of value type string. :person/parent is a multi-valued attribute of value type ref. The value of a reference attribute points to (the id) of another entity stored within the Datomic database. Once our attribute schema is persisted, we can start populating our database with concrete person entities.

We will create three concrete persons: myself, my dad Gilbert Suvee and my grandfather Edmond Suvee. Similarly to the definition of attributes, we again employ the tempid utility method to retrieve temporary ids for our newly created entities. This time however, we store our persons within the :db.part/user database partition, which is the default partition for storing application entities. Each person is given a name (via the :person/name attribute) and parent (via the :person/parent attribute). When calling the transact method, each entity is translated into a set of individual datoms that together describe the entity. Once persisted, Datomic ensures that temporary ids are replaced with their final counterparts.

2. The Datomic query language

Datomic’s query model is an extended form of which will feel quite familiar to people who have experience with SPARQL and/or Prolog. The declarative query language makes use of a pattern matching mechanism to find all combinations of values (i.e. facts) that satisfy a particular set of conditions expressed as clauses. Let’s have a look at a few example queries:

We consider entities to be of type person if they own a :person/name attribute. The :where-part of the first query, which aims at finding all persons stored in the Datomic database, specifies the following “conditional” clause: [?person :person/name ?name]. ?person and ?name are variables which act as placeholders. The Datalog query engine retrieves all facts (i.e. datoms) that match this clause. The :find-part of the query specifies the “values” that should be returned as the result of the query.

The second and the third query aim at retrieving the parents and grandparents of all persons stored in the Datomic database. These queries specify multiple clauses that are solved through the use of unification: when a variable name is used more than once, it must represent the same value in every clause in order to satisfy the total set of clauses. As expected, only Davy Suvee has been identified as having a grandparent, as the necessary facts to satisfy this query are not available for neither Gilbert Suvee and Edmond Suvee.

If several queries require this “grandparent” notion, one can define a reusable rule that encapsulates the required clauses. Rules can be flexibly combined with clauses (and other rules) in the :where-part of a query. Our third query can be rewritten using the following rules and clauses:

Rules can also be used to write recursive queries. Imagine the ancestor-relationship. It’s impossible to predict the number of parent-levels one needs to go up in order to retrieve the ancestors of a person. As Datomic rules supports the notion of recursion, a rule can call itself within its definition. Similar to recursion in other languages, recursive rules are build up out of a simple base case and a set of clauses which reduce all other cases toward this base case.

3. Back To The Future I

As already mentioned in section 1, Datomic does not perform in-place updates. Instead, all facts are stored and tagged with a transaction such that the most up-to-date value of a particular entity attribute can be retrieved. By doing so, Datomic allows you to travel back into time and perform queries against previous states of the database. Using the asOf method, one can retrieve a version of the database that only contains facts that were part of the database at that particular moment in time. The use of a checkpoint that predates the storage of my own person entity will result in parent-query results that do not longer contain results related to myself.

4. Back To The Future II

Datomic also allows to predict the future. Well, sort of … Similar to the asOf method, one can use the with method to retrieve a version of the database that gets extended with a list of not-yet transacted datoms. This allows to run queries against future states of the database and to observe the implications if these new facts were to be added.

5. Conclusion

The use of Datoms and Datalog allows you to express simple, yet powerful queries. This article introduces only a fraction of the features offered by Datomic. To get myself better acquainted with the various Datomic gotchas, I implemented the Tinkerpop Blueprints API on top of Datomic. By doing so, you basically get a distributed, temporal graph database, which is, as far as I know, unique within the Graph database ecosystem. The source code of this Blueprints implementation can currently be found on the Datablend public GitHub repository and will soon be merged within the Tinkerpop project..