In one of my previous articles, I explained how Neo4J and the Tinkerpop framework can be used to load and query RDF triples. The newly released Neo4J plugin now allows to visually browse these RDF triples and perform some more fancy operations such as finding patterns and executing social network analysis algorithms from within Gephi itself. Tinkerpop’s Sail Ouplementation also supports the notion of RDF Schema inferencing. Inferencing is the process where new (RDF) data is automatically deducted from existing (RDF) data through reasoning. Unfortunately, the Sail reasoner cannot easily be integrated within Gephi, as the Gephi plugin grabs a lock on the Neo4J store and no RDF data can be added, except through the plugin itself.

Being able to visualize the RDF Schema reasoning process and graphically indicate which RDF triples were added manually and which RDF data was automatically inferred would be a nice to have. To implement this feature, we should be able to push graph changes from Tinkerpop and Neo4J to Gephi. Luckily, the Gephi graph streaming plugin allows us to do just that. In the rest of this article, I will detail how to setup the required Gephi environment and how we can stream (inferred) RDF data from Neo4J to Gephi.

1. Adding the (inferred) RDF data

Let’s start by setting up the required Neo4J/Tinkerpop/Sail environment that we will use to store and infer RDF triples. The setup is similar to the one explained in my previous Tinkerpop article. However, instead of wrapping our GraphSail as a SailRepository, we will wrap it as a ForwardChainingRDFSInferencer. This inferencer will listen for RDF triples that are added and/or removed and will automatically execute RDF Schema inferencing, applying the rules as defined by the RDF Semantics Recommendation.

We are now ready to add RDF triples. Let’s create a simple loop that allows us to read-in RDF triples and add them to the Sail store.

The inference method itself is rather simple. We first start by parsing the RDF subject, predicate and object. Next, we start a new transaction, add the statement and commit the transaction. This will not only add the RDF triple to our Neo4J store but will additionally run the RDF Schema inferencing process and automatically add the inferred RDF triples. Pretty easy!

But how do we retrieve the inferred RDF triples that were added through the inference process? Although the ForwardChainingRDFSInferencer allows us to register a listener that is able to detect changes to the graph, it does not provide the required API to distinct between the manually added or inferred RDF triples. Luckily, we can still access the underlying Neo4J store and capture these graph changes by implementing the Neo4J TransactionEventHandler interface. After a transaction is committed, we can fetch the newly created relationships (i.e. RDF triples). For each of these relationships, the start node (i.e. RDF subject), end node (i.e. RFD object) and relationship type (i.e. RDF predicate) can be retrieved. In case a RDF triple was added through inference, the value of the boolean property “inferred” is “true”. We filter the relationships to the ones that are defined within our domain (as otherwise the full RDFS meta model will be visualized as well). Finally we push the relevant nodes and edges.

2. Pushing the (inferred) RDF data

The streaming plugin for Gephi allows reading and visualizing data that is send to its master server. This master server is a REST interface that is able to receive graph data through a JSON interface. The PushUtility used in the PushTransactionEventHandler is responsible for generating the required JSON edge and node data format and pushing it to the Gephi master.

3. Visualizing the (inferred) RDF data

Start the Gephi Streaming Master server. This will allow Gephi to receive the (inferred) RDF triples that we send it through its REST interface. Let’s run our Java application and add the following RDF triples:

The first two RDF triples above state that a teacher teaches a student. The last RDF triple states that Davy teaches Bob. As a result, the RDF Schema inferencer deducts that Davy must be a teacher and that Bob must be a student. Let’s have a look at what Gephi visualized for us.

Mmm … That doesn’t really look impressive . Let’s use some formatting. First apply Force Atlas lay-outing. Afterwards, scale the edges and enable the labels on both the edges and the nodes. Finally, apply partitioning on the edges by coloring the arrows using the inferred property on the edges. We can now clearly identify the inferred RDF statements (i.e. Davy being a teacher and Bob being a student).

Let’s add some additional RDF triples.

Basically, these RDF triples state that both teacher and student are subclasses of person. As a result, the RDFS inferencer is able to deduct that both Davy and Bob must be persons. The Gephi visualization is updated accordingly.

4. Conclusion

With just a few lines of code we are able to stream (inferred) RDF triples to Gephi and make use of its powerful visualization and analysis tools to explore and inspect our datasets. As always, the complete source code can be found on the Datablend public GitHub repository. Make sure to surf the internet to find some other nice Gephi streaming examples, the coolest one probably being the visualization of the Egyptian revolution on Twitter.

As mentioned in my previous blog post, I recently got asked to implement a storage and querying platform for biological RDF (Resource Description Framework) data. Traditional RDF stores are not really an option as my solution should also provide the ability to calculate shortest paths between random subjects. Calculating shortest path is however one of the strong selling points of Graph Databases and more specifically Neo4J. Unfortunately, the neo-rdf-sail component, which suits my requirements perfectly, is no longer under active development. Tinkerpop’s Sail implementation however, fills the void with an even better alternative!

1. What is Tinkerpop?

Tinkerpop is an open source project that provides an entire stack of technologies within the Graph Database space. At the core of this stack is the Blueprints framework. Blueprints can be considered as the JDBC of Graph Databases. By providing a collection of generic interfaces, it allows to develop graph-based applications, without introducing explicit dependencies on concrete Graph Database implementations. Additionally, Blueprints provides concrete bindings for the Neo4J, OrientDB and Dex Graph Databases. On top of Blueprints, the Tinkerpop team developed an entire range of graph technologies, including Gremlin, a powerful, domain-specific language designed for traversing graphs. Hence, once a Blueprints binding is available for a particular Graph Database, an entire range of technologies can be leveraged.

2. Tinkerpop and Sail

Last time, I talked about exposing a Neo4J Graph Database (containing RDF triples) through the Sail interface, which is part of the the Sail interface. Once you have your Sail available, storing and querying RDF is analogous to the piece of code shown in my previous blog article.

The two first lines of code require some more clarification. A TransactionalGraph can be run in MANUAL or AUTOMATIC transaction mode. In AUTOMATIC mode, transactions are basically ignored, in the sense that each item that gets created is immediately persisted in the underlying Graph Database. Although this fits my needs, AUTOMATIC mode is extremely slow in case of Neo4J because of the continuous IO access. MANUAL mode on the other hand is very fast; a new transaction is created at the moment the import of the RDF data file starts and is only committed to the Neo4J data store once all RDF triples are parsed and created. Unfortunately, MANUAL mode does not scale either in my specific situation; as some of my RDF data files contain over 50 million RDF triples, they can not fit into memory (i.e. Java heap space error). Requiring fast imports, I extended the default Neo4J Blueprints binding to support intermediate commits. I based my implementation on Neo4J’s best practices for big transactions. The idea is rather simple: you specify the maximum number of items that can be kept in memory, before they should be committed to the Neo4J data store. Once this number is reached, the current transaction is committed and a new one is automatically started. Simple, but very effective!

Based upon the “MyNeo4jGraph”-idea described above, the BluePrints team extended their API to support transaction buffers. More information on its use can be found here and here.

3. Shortest path calculation

Although Blueprints allows you to abstract away the Neo4J implementation details, it still provides you with access to the raw Neo4J data store if needed. Hence, one can still use the graph algorithms provided in the neo4j-graph-algo component to calculate shortest paths between random subjects. The complete source code can be found on the Datablend public GitHub repository.

1. Introduction

Recently, I got asked to implement a storage and querying platform for biological RDF (Resource Description Framework) data. RDF data is a set of statements about resources in the form of subject-predicate-object expressions (also referred to as triples). Let’s have a look at some simple RDF triples that define ‘me’, Davy Suvee:

Each subject is identified through an URI (Uniform Resource Identifier). For instance, I identify myself as being http://www.example.org/person/Davy_Suvee. A predicate, also identified through an URI, either points to a literal value or to a concrete object (which is again identified through an URI). In the example above, the first_name, last_name and age predicates all point to a literal value, while the company predicate points to http://www.example.org/company/DataBlend, the company I work for. The DataBlend subject also exhibits a number of properties, including name and VAT-number. Today’s triplestores allow you to save billions of these triples and information is retrieved through so-called SPARQL-queries. For instance, to retrieve my first name and age, I can use the following SPARQL-query:

2. Neo4J as a RDF data store

Similar to SQL, SPARQL provides a set of powerful querying constructs that allow you to declaratively specify your needs. Calculating shortest paths between random subjects on the contrary, can not easily be accomplished through SPARQL (unless one encodes the specific path structure, which kind of defeats the point). Being able to quickly calculate shortest paths, which is a requirement for the project I’m implementing, is one of the main selling points of Graph Databases. As RDF data can be thought of as a graph, it comes as no surprise that many Graph Databases, including Neo4J, provide native support for storing and querying RDF data. In case of Neo4J, this is achieved through the use of the neo4j-rdf, neo4j-rdf-sparql and neo-rdf-sail components. Unfortunately, I couldn’t find a recent piece of code that details the various steps for automatically importing RDF triple files within Neo4J. Hence, this article. The complete source code can be found on the Datablend public GitHub repository.

Start by setting up the Neo4J database connection:

An embedded Neo4J graph database (EmbeddedGraphDatabase) is used for importing 5MB of RDF tuples containing airline flight information. (This example data set was found at rdfdata.org, a great resource for some open RDF data sets). In order to easily find back flight information, we fully text-index our RDF triples (through Lucene). Next, we wrap the embedded Neo4J graph database as a VerboseQuadStore (one of internal triples store implementations provided by Neo4J). Finally, we expose our triple store through the Sail interface, which is part of the openrdf.org project. By doing so, we can use an entire range of RDF utilities (parsers and query evaluators) that are part of the openrdf.org project. Once we have a sail connection available, we can import the required RDF triples through the add-method.

That’s it! Once the import is finished, you can query your RDF triplets by executing a SPARQL-query. The query below for instance, will retrieve the flight number, departure and destination city of all flights that have a duration of 1 hour and 35 minutes.

3. Shortest path calculation

Through the SimpleFulltextIndex we can easily find back the Neo4J node equivalent of a particular RDF subject. Once we got hold of the required nodes, we can use the graph algorithms provided in the neo4j-graph-algo component to calculate (shortest) paths. Very cool!