1. mongoDB Graph

The mongoDB Blueprints implementation provides users with a scalable, distributed Graph Database implementation. mongoDB graph does not require any information on the underlying physical mongoDB setup, so read/write scalability can easily be achieved through mongoDB’s natively supported replication and sharding functionalities. Some initial benchmarks (writing/reading 100.000 vertices where each 2 vertices are connected through an edge) on a single mongoDB node shows the following performance:

Not too shabby for running it on a single mongoDB instance. Currently, each write is executed as a singular commit to the mongoDB store. Although this process could be improved by performing writes through optimized batches, we need to ensure that the transactional semantics of the Blueprints stack are still respected. Further performance optimizations will be released soon, so keep an eye on the blueprints-mongodb-graph github project.

2. Datomic Graph

Datomic is a novel distributed database system designed to enable scalable, flexible and intelligent applications, running on next-generation cloud architectures. Datomic employs a powerful data model (based upon the concept of Datoms) and an expressive query language (based upon the concept of Datalog. Additionally, it introduces an explicit notion of time, which allows for the execution of queries against both the previous and future states of the database. The RDF and SPARQL feel of the Datomic data model and query approach makes it an ideal target for implementing a property graph. Hence, the blueprints-datomic-graph Blueprints implementation.

Clever use the time-aware nature of the Datomic datastore, makes Datomic Graph the very first distributed, temporal graph database: users can perform queries against a specific version of the graph in the past. The code sample below illustrates how a time-aware social graph can be created, stored and queried through the Datomic Graph implementation.

As one would expect, this outputs the following information:

Datomic Graph does not only support versioning of the vertices and edges, but also on the properties of individual vertices/edges. Pretty slick, not? We are currently enhancing the implementation with additional time-based operations, including the easy comparison of subgraphs over time. So stay tuned!

As mentioned in my previous blog post, I recently got asked to implement a storage and querying platform for biological RDF (Resource Description Framework) data. Traditional RDF stores are not really an option as my solution should also provide the ability to calculate shortest paths between random subjects. Calculating shortest path is however one of the strong selling points of Graph Databases and more specifically Neo4J. Unfortunately, the neo-rdf-sail component, which suits my requirements perfectly, is no longer under active development. Tinkerpop’s Sail implementation however, fills the void with an even better alternative!

1. What is Tinkerpop?

Tinkerpop is an open source project that provides an entire stack of technologies within the Graph Database space. At the core of this stack is the Blueprints framework. Blueprints can be considered as the JDBC of Graph Databases. By providing a collection of generic interfaces, it allows to develop graph-based applications, without introducing explicit dependencies on concrete Graph Database implementations. Additionally, Blueprints provides concrete bindings for the Neo4J, OrientDB and Dex Graph Databases. On top of Blueprints, the Tinkerpop team developed an entire range of graph technologies, including Gremlin, a powerful, domain-specific language designed for traversing graphs. Hence, once a Blueprints binding is available for a particular Graph Database, an entire range of technologies can be leveraged.

2. Tinkerpop and Sail

Last time, I talked about exposing a Neo4J Graph Database (containing RDF triples) through the Sail interface, which is part of the the Sail interface. Once you have your Sail available, storing and querying RDF is analogous to the piece of code shown in my previous blog article.

The two first lines of code require some more clarification. A TransactionalGraph can be run in MANUAL or AUTOMATIC transaction mode. In AUTOMATIC mode, transactions are basically ignored, in the sense that each item that gets created is immediately persisted in the underlying Graph Database. Although this fits my needs, AUTOMATIC mode is extremely slow in case of Neo4J because of the continuous IO access. MANUAL mode on the other hand is very fast; a new transaction is created at the moment the import of the RDF data file starts and is only committed to the Neo4J data store once all RDF triples are parsed and created. Unfortunately, MANUAL mode does not scale either in my specific situation; as some of my RDF data files contain over 50 million RDF triples, they can not fit into memory (i.e. Java heap space error). Requiring fast imports, I extended the default Neo4J Blueprints binding to support intermediate commits. I based my implementation on Neo4J’s best practices for big transactions. The idea is rather simple: you specify the maximum number of items that can be kept in memory, before they should be committed to the Neo4J data store. Once this number is reached, the current transaction is committed and a new one is automatically started. Simple, but very effective!

Based upon the “MyNeo4jGraph”-idea described above, the BluePrints team extended their API to support transaction buffers. More information on its use can be found here and here.

3. Shortest path calculation

Although Blueprints allows you to abstract away the Neo4J implementation details, it still provides you with access to the raw Neo4J data store if needed. Hence, one can still use the graph algorithms provided in the neo4j-graph-algo component to calculate shortest paths between random subjects. The complete source code can be found on the Datablend public GitHub repository.