<p>Disclaimer: After having adopted Neo4j as our database solution for a long time, we have joined the Startup Program offering Sourcer free access to the Enterprise version of their products.</p>
<h2>What is Neo4j?</h2>
<p>Any software company that deals with lots of data needs a good way to store said data. And depending on the type of data, the amount of it, and how you would like it to be structured, there is a specific solution to most efficiently handle it. In the case of Sourcer, we need to store large amounts of news articles, insights, and much more.</p>
<p>One good solution came forth for how we wanted to structure this. Neo4j is a company that provides database solutions and is the leader in Graph Technology. One of their products, the Neo4j Graph Database, is what Sourcer uses to store and access thousands and millions of articles.</p>
<p>Now, what is a graph database, of which Neo4j is one? Unlike more conventional databases, which consist of tables (imagine big spreadsheets of data), graph databases have the unique property of structuring information in terms of nodes and relations.</p>
<p>Think of your social network as an excellent way to imagine what nodes and relations can be. You and your friends are all nodes. You can have different nodes for friends, family, or coworkers. The relations with these people, or nodes, are the relations that can be found in the database too. Relations might be 'friends with' or 'daughter of', connecting two nodes.</p>
<p>Besides different types of nodes and relations, they can also contain additional information like a person's name and address. These are called properties, and properties can be numbers, text, a yes or no, but even dates and lists of other values.</p>
<p>Nodes in Neo4j are often depicted as circles, and relations are often depicted as arrows pointing from one node to another. You can find an example of such a social network in the image below, which is created using the Neo4j Browser platform.</p>
<p><img src="/assets/blog/neo4j/neo4j_example.png" alt=""></p>
<h2>How do we use Neo4j?</h2>
<p>As we mentioned a bit earlier, picking the right database solution depends on what type of data you are planning to store and how you plan to interact with it. We needed a solution that allowed us to store not only news articles that we analyzed and their metadata but also authors, publishers, keywords, stakeholders, and other information that is useful to the user. We also needed a way to connect an author with an article and other related articles.</p>
<p>Storing this data in a table, like a standard database, did not make sense to us and would result in us having to combine many different data tables to find any meaningful insights. The advantage of Neo4j is that traversing all these connections is made very intuitive and also happens very efficiently.</p>
<h3>Storing media content with nodes and relations</h3>
<p>We mentioned all the items we have to store, namely articles, authors, publishers, keywords, stakeholders, and many more, and we store these as nodes. Each node also contains metadata, like the name of the author or the logo of a publisher.</p>
<p>To make meaningful connections between all these data items, it made perfect sense to connect an author to an article they have written with a simple relation, like 'has written'. Since an author can write multiple news articles, adding more relations was all the easier.</p>
<p>Now that we have created many different nodes and relations to describe media content, we can run our algorithms that take advantage of the unique attribute of graph databases. For instance, we use relations between articles and keywords to help us determine related articles. We will also be able to predict trends, observe author track records, and much more using the relations we create. Building algorithms to gain insight from media content has been made much easier using graphs, saving us time developing and providing results for you faster.</p>
<p><img src="/assets/blog/neo4j/sourcer_db_example.png" alt=""></p>
<h3>Our proprietary ORM</h3>
<p>To interface with any database, a querying language is required. Cypher is the query language used by Neo4j to retrieve data from the graph database. The nice thing about Cypher, which makes it unique, is that it is a visual way of matching patterns and relationships [3].</p>
<p>However, each query is like writing a perfect search term to get the information you need from Google. If we want to get an article from the database, writing a Cypher query, or using a function that executes the Cypher query, is not the most elegant. That's why we use an ORM, or Object-Relational Mapping, a helpful technique that converts the graph data into useful self-contained objects. The advantage is that the code becomes cleaner, there is more reusability and maintainability, and it abstracts the complicated database to a few simple objects and functions.</p>
<p>Instead of using third-party software, we created our own ORM. Our ORM contains typing and creating of objects, simple reusable queries to the database, and the option for more complex custom queries to be created as well. The nice thing is that creating a new object corresponding to a type of node can be done easily by our developers with our ORM handling the complexity and ensuring a safe interaction with the database.</p>
<p>ORMs can be slower than directly interacting with the database, so as we mentioned, our system can be very easily adapted to allow more complicated queries as well.</p>
<h3>The vision of our information database</h3>
<p>Now that we have outlined our database, how it works, what it stores, and how we interact with it, it is good to look at what we are trying to accomplish by maintaining such a database of media content. As you have seen, we are not only storing the metadata of articles but also of authors, publishers, and many more data points.</p>
<p>The purpose of storing this information is not to create an extensive backup of others content, as we do not store the article itself but only its metadata. Instead, with this database, we aim to learn more from the information put out by the media, from all media. By indexing information from many different sources and running our algorithms to gain insights, we can deliver you what you need to make a well-informed decision and learn from the news you are reading. Every node containing metadata, and every relation comparing two articles, will get you closer to understanding more.</p>
<p>We hope to continue delivering on our vision, so our solution is built to be scalable. Thanks to Neo4j for enabling us to store thousands and millions of articles. They help us be future-proof, so we can prove the future of finding credible online information.</p>
<h2>What do we do with your data?</h2>
<p>Finally, to close on the topic of storing data, it is all to common for software companies to acquire as much information from their users as possible. As we hope to deliver additional valuable insights,  about information popularity and trending news, some usage data is collected, accompanied by anonymized account data to allow for authentication to premium features. Luckily for you, we are actually not interested in you, especially not you in particular. We pride our solution on being completely anonymous, tracking usage data, and allowing users to authenticate themselves for our premium features that will come out soon.</p>
<h3>Anonymized user data</h3>
<p>What does it mean for user data to be anonymous? Unlike confidential data containing a link that is used to retrieve the user's identity, user data being anonymous means it can never be linked back to the user who supplied the information. Any user currently using Sourcer cannot be traced back to them. The vital difference here is that even though we store user data, it cannot be considered personal data, as it is anonymous and not identifiable.</p>
<h3>Why we store user data at all</h3>
<p>What user data do we store, if not personal data? There is much more helpful information that can be gained from users using a product besides the information that can be linked back to them. As long as the data can be anonymized, it can be used to analyze and learn from it.</p>
<p>For instance, usage data can be precious. When many users use a product or service, it is good to know how it is used. Since we are interested in anonymous data, we are not asking the question 'How are users using the product?' but 'How is the product used?'—the change in questioning shifts the conversation from focusing on people to features and insights of the product. The usage data we track is, for instance, what anonymous users have read an article and at what time. But also, what related articles are clicked on by users from the extension, and what articles are not. These can be very useful to deliver insights back to the user, like the view count of an article and its trendiness, and for us to improve our product for you.</p>
<p>The point of storing user data is to improve the product and deliver value to you. We will always ensure that our business model is not driven by collecting personal data but only by useful user data. Many more features, yet to be released, will improve your security and privacy, with more blogs about these following too.</p>
<h3>Why we do not store personal data</h3>
<p>There are a few good reasons why we do not store personal data. All users are saved on the database using a long and random ID. This ID is all we know about the user and says nothing about them. For instance, when installing the extension, we never ask for your email or name. There are also no cookies; the only consent is to allow the extension to run on any news article. If we had to collect personal data, it would make your experience much more troublesome, yet we wanted to ensure that using the extension is accessible to you. The ID is taken care of in the background, it cannot be linked back to you as it only contains a series of random letters and numbers, and it is all we need to gain a few insights on how our product is used.</p>
<p>As of the writing of this blog post, personal data is not needed, not necessary, and not interesting to us. Soon though, we will release a set of productivity tools and even better insights for a small subscription. To authenticate your access to these exciting features, all we will need is your email address, which will not even be stored on our incredible graph database. So no password is necessary either!</p>
<h2>Sources</h2>
<p><a href="https://neo4j.com/product/neo4j-graph-database/">Neo4j Graph Database - Neo4j</a></p>
<p><a href="https://neo4j.com/developer/graph-database/">What is a Graph Database? - Neo4j</a></p>
<p><a href="https://neo4j.com/developer/cypher/">Cypher Query Language - Neo4j</a></p>
<p><a href="https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping">Object-Relational Mapping</a></p>
<h2>Disclaimer</h2>
<p>Sourcer assumes no responsibility or liability for any errors or omissions in the content of this article. The information contained in this article is provided on an "as is" basis with no complete guarantees of completeness, accuracy, usefulness or timeliness.</p>


Product

Company

Connect