
In this article, we'll discuss an open source, creative commons set of UI designs for a web-based personal data store.

- Click through [the demo](https://www.figma.com/proto/wkc2XEA6Lddai8n4PUU83C/Solidstarter-Prototype?page-id=327%3A0&node-id=335%3A32985&viewport=289%2C241%2C0.04754795879125595&scaling=scale-down) on figma
- View the [style book](https://zeroheight.com/756e7c07f/p/919ed8-solid-ui-kit) on ZeroHeight
- Check out the [Github Repository](https://github.com/ontola/solid-ui-designs), containing the source files

![solid-1](/images/solid-ui/solid-1.jpg)

## Why we need Personal Online Datastores

30 Years ago, the first website was created by Tim Berners-Lee.
The web was designed to be decentralized, where people would have their own servers to share their thoughts on an open internet.
But today, our web has become more centralized than every before.
A few large tech companies control most of our personal data.
Using cloud applications enabled us to access our data from every machine, but it took away a great deal of control.
Moving data from one app to the other is often difficult (or impossible), as many cloud apps have a tight integration between client and server.
For example, we can only view our facebook posts using official Facebook apps.
On the other hand, standards like e-mail, RSS, and HTML allow users to choose how they view and edit their data - we can pick our own e-mail apps, RSS readers and our own web browsers.
Wouldn't it be awesome if we could do the same thing with our social media posts, our chat messages, our todo apps and more?

One approach to solve this problem, is having a _personal online data store_, a Pod.
This is a computer that is always online, and contains your personal data.
It is similar to a service such as Dropbox or Google Drive, but it does a bit more.
Instead of only dealing with files, it deals with _structured standardized data_.
This enables a higher degree of interoperability, which means it becomes easier for app developers to re-use data.
It also hosts your online identity, stores notifications and more.

One notable project that aims to provide such a standard, is Solid.
This project was started by Tim Berners-Lee himself.
At this time, the specification is being formalized and various implementations are being worked on (such as our own [DexPod Server](https://gitlab.com/ontola/dexpod), or the [Solid-Community-Server](https://github.com/solid/community-server)).
The majority of the Solid community is focused on making all of this work, which mostly means getting the technology and the specification in good shape.
Another Pod project is [Atomic Data](https://atomicdata.dev), which is what I've been working on for the past year.
Both of these projects share the same goal: giving people more control over their data.
Because

## Designing a user interface for Pods

How would a Pod look like? How would you interact with it? What kind of UI challenges and opportunities does a Pod present?
We've partnered up with studio [Jager & Prooi](https://jagerenprooi.nl/) to answer these questions and design a user interface for Pods.
Thanks to the [SIDN Fonds](https://www.sidnfonds.nl/nieuws/techneuten-en-ontwerpers-bundelen-krachten-8-projecten-van-start) for funding this project.

We started our project off by interviewing two groups: Solid developers, and potential early adapters.
That way, we'd have a good grasp on technical opportunities and limitations, and what aspects of the UI would be most important for making the first group of users happy.

We realized that designing a Pod would pose an interesting challenge, even for experienced designers.
A Pod is not just one of many apps - it is a place where many apps could run.
It is similar to an Operating System, such as Windows or iOS: where you launch and manage apps, manage your data and manage your system.
This means it needs:

- **a homescreen** from which you can open apps and view activities
- a way to **browse through your files**, like a file manager where you can edit, organize and delete things
- a tool to **search** through your data
- a **settings app** to manage the hardware and UI
- a **notification tray** or a different way to view incoming changes and alerts
- **persistent UI elements** to quickly open search / notifications / settings
- a **way to open files with various different apps** (e.g. open a PDF in Adobe or a Browser).

These are the things that we've learned to expect from operating systems - these are also present in Windows, iOS and others.
But Pods are different for two reasons:

1. Pods are _always_ connected to the internet, which means that they can be places that others can find information
1. Pods use _structured data_ instead of _binary data_ (regular files on your computer), which means that you get new features to manage information

These differences result in new things that you might want a Pod operating system to achieve:

- **Manage external access to data**. In a Pod, you can allow other to access files without actually sending it to them. Your Pod is a server, so it's connected to the internet, which means that (if you want) others can collaborate on or view your data, too. This needs to be managed, which is something that is not actually common in other operating systems.
- **Version control / history control**. Storing structured data makes it easier to actually save various version, which means that user can undo their mistakes and see who changed what over time.
- **Structured Search**. In a regular operating system, we could search for filenames, but in a Pod, we can search through the data itself. This means that we can use the global search feature to find Todo's, appointments, playlists and more.
- **Manage personal information**. Since Pods are inherently _personal_, they are great for storing data about yourself. This includes data that you'd share publicly on your linkedin profile, or medical data that you'd want to keep private, but do want to have managed somewhere.

To keep the scope of this project manageable, we decided to focus on Home, Personal Information Management, Sharing, File Exploring and Search.

![personal data management](/images/solid-ui/solid-2.jpg)

**[Click through the demo](https://www.figma.com/proto/wkc2XEA6Lddai8n4PUU83C/Solidstarter-Prototype?page-id=327%3A0&node-id=335%3A32985&viewport=289%2C241%2C0.04754795879125595&scaling=scale-down)** to see how it turned out!
And feel free to use these designs in your own apps - we're sharing the [source files on Github](https://github.com/ontola/solid-ui-designs) under a CC-BY license for specifically that reason.


In this article, we'll discuss an open source, creative commons set of UI designs for a web-based personal data store.

Designing a UI for a Personal Online Datastore


In this article, we'll discuss how linked data can make surveys more easy to manage, increase response rates using rewards, create new benefits for users and more.

- Why it's important to improve survey interoperability
- How we could increase interoperability using linked data
- How we can use rewards to increase respondent rates for surveys

## Why we need Survey interoperability

Surveys are a pretty interesting and complex type of data model.
You can think of a Survey as describing three concepts at once:

1. a data schema / model (datatypes, required fields, etc.)
1. a UI (the questions show to the user)
1. and a customer journey (the sequence of questions, sometimes with conditional paths).

Due to this complexity, it should not be a big surprise that there are many approaches to creating a Survey, and this has contributed to the low degree of interoperability of surveys.

That's a shame, because having interoperable Surveys would be _really cool_!

- It would be easier for researchers to **standardize how questions are asked**. For example, many medical surveys are highly standardized, up to how specific words can be used. You can probably imagine how a small change in phrasing a question or answer could have a serious impact on average results. If we standardize surveys and their questions on a data level, researchers could simply import an existing survey, instead of going through the time consuming process of manually creating every single field in their survey software.
- Respondents could have more **freedom in chooing a UI**. Not all user interfaces are easily accessible, pretty, or fun to use. People can have vastly different preferences regarding how a survey should be filled in. If we standardize how Survey Data is structured, we can let respondents pick their own front-end, similar to how you can open a webpage in any webbrowser that you like.
- We could **auto-fill** data from a personal source, which makes entering data less tedious. Browsers already do this for addresses and credit cards, but we can extend this _way_ further by using links! We'll get to that later
- We can **improve insights in ourselves** by storing and then analyzing data that we filled in in a survey. For example, filling in a survey about how your shortness of breath linked to air pollution has been today could be used in a different app to make a graph that visualizes how your shortness of breath has progressed over the months for personal insight.

Achieving these last two steps requires a high degree of standardization in both the surveys and the responses. The survey and its questions should provide information about:

- The **question** itself. This is required in all survey questions, of course.
- The **required datatype** of the response, such as 'string', or 'datetime' or some 'enumeration'.
- A (link to a) **semantic definition** of the property being described. This is a bit more obscure: all pieces of linked data use links, instead of keys, to describe the relation between some resource and its property. For example, a normal resource might have a 'birthdate', while in linked data, we'd use 'https://schema.org/birthDate'. This semantic definition makes things easier to share, because it prevents misinterpretation. Links remove ambiguity.
- A **query description**. This is even more obscure, but perhaps the most interesting. A query description means describing how a piece of information can be retrieved. Perhaps a question in a survey will want to know what your payment pointer is. If a piece of software wants to auto-fill this field, it needs to know where it can find your payment pointer.

## Making surveys interoperable

A good place to start standardizing, is to use _links_ everywhere. Use links for Properties, use links as identifiers for Surveys, for Responses, for Questions...
Links are great, because they can be _opened_.
Read my article on [Linked Data](https://ontola.io/blog/what-is-linked-data) if this is new to you.

Okay, so we standardize the Questions, Surveys and Responses using links. But which links? And how?
At Ontola, we've been building dynamic RDF based forms for our [Argu.co](https://argu.co) e-democracy application.
We started out by using the [W3C SHACL specification](https://www.w3.org/TR/shacl/), which is a linked data standard for describing data models.
It turned out to be a great first step, as it ticked the first box of concepts in surveys: it describes the _schema_.
But we still have two concepts left to define: the _UI_ and the _Journey_.
For example, the SHACL spec does not provide UI concepts for describing whether to use a _radio select_ or a _dropdown_, or how to render a _helper text_.
And the _Journey_ should include conditional paths, which is also not possible.
These conditions are also important when the _rights_ of a user determines whether that user can edit a certain field.
We could either calculate all the form fields dynamically, or we could let the front-end use conditional fields to determine which fields are shown.
We opted for the latter, as this resulted in better performance.

So we're currently working on a Form ontology which adds things like `Pages`, `Groups`, `helper text` and `Conditions`.
As this is still prone to changes and under-specified, we'd rather not share it in its current form, but we do aim to do this in the near future.
An open-source implementation of our Survey software can be found on [Gitlab](https://gitlab.com/ontola/linked-surveys).

## Rewarding responses

Filling in a survey isn't the most fun thing to do, so getting people to fill in a survey can be a challenge.
That's why it's pretty common these days to see surveys that promise a chance to win rewards like nice gadgets (iPads, mostly) or coupons.
The first problem with this approach is that the one filling in the survey doesn't have a clear estimate of what their chances of winning are, which makes the deal a bit less attractive.
The second problem is that it requires the respondent to provide privacy sensitive details, such as address.
A third problem is that not everybody wants an iPad - money is often a bit more attractive.
Ideally, we'd provide a clear and transparent reward, and do so in an anonymous fashion.

A couple of months ago, we discovered the Grant for the Web, a grant that aims to change how we do payments on the internet.
We submitted a proposal to implement a solution for the problems stated above: we wanted to build a service that sends a payment to people who completed a survey.
Because we wanted to provide a high degree of privacy, we wanted our own services to be completely unaware of the payment address of the receiver.
We also wanted to make this project re-usable for other usages (other survey tools, or totally different things).
We therefore up building an open source service, called [**cashlink**](https://gitlab.com/ontola/cashlink).
It works like this:

- Create unique links (hence _cashlink_) which encode the value of the link. These can be used only once.
- Share this unique URL in some application, such as at the end of a survey.
- When the user opens this URL, they can submit their own _Payment Pointer_, which is an open specification for payment addresses built for the web.
- After submitting this, the payment is sent by the cashlink service.

## In conclusion

- Having a higher degree of standardization can offer various benefits: more standardized research, auto fill forms, UI freedom, and get more insights in our own personal data
- Linked Data can be used to facilitate this standardization process, but we need more (well-documented) standards to do this.
- Payments can be an effective tool for getting more people to fill in surveys.


In this article, we'll discuss how linked data can make surveys more easy to manage, increase response rates using rewards, create new benefits for users and more.

On linked data surveys, rewards and data ownership


EDIT: This tutorial is outdated. We no longer support the `link-lib` library. We recommend checking out [Atomic Data](/cases/atomic-data) and [@tomic/react](https://codesandbox.io/s/atomic-data-react-template-4y9qu) instead.

This tutorial will help you build a linked data front-end application, compatible with Tim Berners-Lee's Solid initiative.
Knowledge of React and Typescript / Javascript is assumed, as is a little knowledge of RDF.
For an introduction to RDF, read [this](https://ontola.io/blog/what-is-linked-data/).

Working with RDF (linked data) can be a bit different from using more conventional JSON REST APIs.
We need tooling to deal with some of the complexities that arise when working with linked data:

- The client (the browser app) does not know in advance what kind of data it will get back from a server. Link will simply fetch a URL using content negotiation, and get some [RDF serialized](https://ontola.io/blog/rdf-serialization-formats/) response.
- Deciding which view to render is often done by checking routes, but this is not possible in a linked data app. Instead, we can fetch the URL and check which `rdf:type` is being set (e.g. `Person` or `Blogpost`).
- Many strings are URLs, which can be a bother to type and might cause typo-related-bugs. To deal with this, we [set URLs programmatically using @ontologies/core](https://ontola.io/blog/ontologies-in-js/).
- [Ordered data in RDF](https://ontola.io/blog/ordered-data-in-rdf/) is a bit more complicated than JSON Arrays, so we'll use tooling to abstract these to arrays.

In this tutorial we're going to use `react` and the [`link-lib`](https://github.com/fletcher91/link-lib/) + [`link-redux`](https://github.com/fletcher91/link-redux) (in short: Link) libraries to create a simple front-end app to render data from a Solid Pod.
You can see the [final result on CodeSandbox](https://codesandbox.io/s/github/ontola/link-solid-boilerplate/tree/master/?file=/src/app.tsx)

## Understanding Link: Store, Views and Topologies

The Link libraries have an interesting, but unconventional way of rendering data.
The first concept to understand, is the LinkedRenderStore (the Store).
The **Store** is responsible for most of the heavy lifting: it contains the stored data, keeps rendered data in sync, parses changes, and stores registered Views.
**Views** are (in this case react) components that render one or more `rdf:Class`es in one or more Topologies.
**Topologies** are contexts in the DOM, such as `Page` or `Card` or `Detail`.

Let's say you want to view your public Solid Pod folder, such as `https://joep.inrupt.net/public/`, as a full page.
First, you pass the subject (the URL of the resource) to a Link `Resource` component.
This will let the LRS check if that resource is already present in the Store.
If not, it will be fetched.
Once it has been fetched and parsed, the resource will be added to the Store.
Then, the Resource component will be notified of new data, and then the Store will try to find the _most suitable View_.
This is done by checking the `rdf:type` of the Resource and the current Topology.
In this case, that type is an `ldp:Container`, and the topology is `fullPage`.
If there is a `fullPage` View registered in the Store for `ldp:Container`, it will render that View.
If there is not such a View, it might try a more generic view.

## Setting up the basics

First, we'll need to set up a react project.
You can use our boilerplate, or you can set it up manually.

### Easy mode: Using the link-solid-boilerplate

You can get started right away using the [`link-solid-boilerplate` on CodeSandbox](https://codesandbox.io/s/github/ontola/link-solid-boilerplate/tree/master/?file=/src/app.tsx).
Alternatively, you can use the [Template feature on Github to fork it](https://github.com/ontola/link-solid-boilerplate/generate), and follow the readme to get it running locally.
Got it running?
Skip the next section and go to [Loading Data](#loading-data).

### Hard mode: Without using the boilerplate

_(skip this if you're using the boilerplate)_

1. Make sure `node` and `npm` are installed.
1. Start a new react project: `npx create-react-app my-app --template typescript`.
1. Add the type definitions: `yarn add @types/node @types/react @types/react-dom @types/jest `.
1. Add `link-lib` and `link-redux` as dependencies: `yarn add link-lib link-redux @rdfdev/delta @rdfdev/actions @rdfdev/iri @rdfdev/collections`
1. Add their `peerDepdencies`: `yarn add @ontologies/as @ontologies/core @ontologies/schema @ontologies/shacl @ontologies/xsd http-status-codes n-quads-parser`
1. [Set-up the Linked Render Store](https://github.com/fletcher91/link-redux#1-set-up-the-store).
1. Got stuck? Check the [boilerplate](https://github.com/ontola/link-solid-boilerplate) for inspiration and use the [link-redux wiki](https://github.com/fletcher91/link-redux/wiki).
1. Continue with the rest of this tutorial.

## Loading data

As a data source, we could use multiple back-ends. Here's a list of tested and open source options:

- A Solid pod hosted on [Inrupt.net](https://inrupt.net/)
- A [DexPod](https://gitlab.com/ontola/dexpod/), hosted on [dexpods.eu](https://dexpods.eu).
- [Atomic-Server](https://github.com/joepio/atomic), which also serializes data to RDF formats.
- Any other public RDF server that supports `n-triples` and has permissive `cors` headers.

But for this tutorial, we'll stick with the Inrupt.net pod.
We'll try to render `https://joep.inrupt.net/public/`, which is an `ldp:Container`.
That Container, as its name suggests, contains a bunch of resources, similar to a folder.

## Registering a View

Create the following file `views/Container.tsx` with a functional React component.
All we need is a single property

```tsx
import { Resource, useProperty } from "link-redux";
import React from 'react';
import ldp from '../../ontology/ldp'

const Container = () => {
  // This hook allows you to get any property / value combination from the resource.
  // The ldp.contains essentially contains a URL for the predicate.
  const contains = useProperty(ldp.contains);

  return (
    <>
      {contains.map(member => (
        // This renders a nested Resource.
        // Each subject will be fetcher by Link
        <Resource
          key={member.value}
          subject={member}
        />
      ))}
    </>
  );
};

// The Type attribute dictates for which RDF Classes
Container.type = ldp.Container;

export default Container;
```

Next, go to `views/index.ts` and add the newly created Container component to the `getViews` list:

```ts
const getViews = (): RegistrableComponent<any>[] => [
  // here!
  Container,
  ErrorResource,
  LoadingResource,
  // ....
]
```

This will make sure that when the app initializes, the `Container` component will be registered to the store.
That's it! The component should now render.
However, you won't see much.
We also want to render the `members` from above.
But these should not be rendered on a full page.
They need a different _topology_.

So let's create one for the current context.
We'll call this one `topologies/BrowserList.ts`, as it's a list in the file browser.

```ts
import { TopologyProvider } from 'link-redux'

import { appNS } from '../helpers/app'

// This is the same as `new NamedNode("myappurl/browserlist")`.
// Export this, you'll use it later!
export const browserListTopology = appNS('browserList');

// The link-redux TopologyProvider deals with most of the complexity
class BrowserList extends TopologyProvider {
  constructor(props: any) {
    super(props);

    // Create a NamedNode for your Topology, and bind it to this.topology
    this.topology = browserListTopology;
    // This determines the HTML element type (optional)
    this.elementType = 'ul';
    // The CSS Classname (optional)
    this.className = 'BrowserList';
  }
}

export default BrowserList;
```

Great! Now we can use this BrowserList topology in our Container component.

```tsx
import { Resource, useProperty } from "link-redux";
import React from 'react';

import ldp from '../../ontology/ldp'
import BrowserList from '../../topologies/BrowserList'

const Container = () => {
  const contains = useProperty(ldp.contains);

  return (
    // We've wrapped our new Resources in our BrowserList topology
    <BrowserList>
      {contains.map(member => (
        <Resource
          key={member.value}
          subject={member}
        />
      ))}
    </BrowserList>
  );
};

// The Type attribute dictates for which RDF Classes
Container.type = ldp.Container;

export default Container;
```

Now we need to register a new View for our `members` and our `BrowserList` toplogy.
We'll create a highly _generic_ view, for all sorts of things.
Let's call it `ThingBrowserList`.
It's good practice to use the `ConceptTopology` naming convention.

```tsx
import dcterms from '@ontologies/dcterms'
import { Literal, NamedNode } from "@ontologies/core";
import foaf from '@ontologies/foaf'
import rdfs from '@ontologies/rdfs'
import schema from '@ontologies/schema'
import { FC, Property, useProperty } from "link-redux";
import React from 'react';

import BrowserListItem from '../../components/BrowserListItem'
import { browserListTopology } from '../../topologies/BrowserList'
import { filename } from '@rdfdev/iri';

export interface Props {
  name: Literal;
}

// We can use multiple predicates.
// Link will select the first one that matches.
// This is especially useful in generic kinds of views, such as this one.
const namePredicates = [
  schema.name,
  rdfs.label,
  dcterms.title,
  foaf.name,
]

const ThingBrowserList: FC<Props> = ({
  subject,
}) => {
  const [name] = useProperty(namePredicates)
  const [modified] = useProperty(dcterms.modified);
  // When using a value from usePropety, you'll often want to use the internal raw string value using `.value`
  const displayName = name?.value || filename(subject as NamedNode);

  return (
    <BrowserListItem
      name={displayName}
      title={subject.value}
      to={subject}
    >
      {modified.value}
    </BrowserListItem>
  );
};

ThingBrowserList.type = rdfs.Resource;
// This is where we let this View know that it should be rendered in the BrowserList topology!
ThingBrowserList.topology = browserListTopology;

export default ThingBrowserList;
```

And for the final step, we'll register this View in `views/index.ts` like before:

```ts
const getViews = (): RegistrableComponent<any>[] => [
  Container,
  ThingBrowserList,
  // ....
]
```

Now, open the app and enter your public Solid Pod URL (e.g. `https://joep.inrupt.net/public/`).

## Got stuck?

- Check the [**result on codesandbox**](https://codesandbox.io/s/github/ontola/link-solid-boilerplate/tree/master/?file=/src/app.tsx)
- If you need dev tools In your browser console, use the [link dev tools](https://github.com/ontola/link-devtools): `dev.help`.
- If you encounter a problem, please [create an issue in the Issue tracker on github](https://github.com/ontola/link-solid-boilerplate/issues).
- Check out the [link-redux wiki](https://github.com/fletcher91/link-redux/wiki) for many useful hooks and other features for rendering RDF!


You can use Link to build a React application that consumes linked data. We'll help you step-by-step.

Tutorial: Building a React front-end app with RDF Data, powered by Link and Solid


How might a web application work if it exclusively uses [Linked Data](https://ontola.io/blog/what-is-linked-data/) (RDF) to communicate between server and client?
In this article, I'll tell you about our API journey, why we chose linked data (RDF), the challenges that we faced, and some of the solutions that we came up with.

## About the project

In 2016, we started [Argu.co](https://argu.co): an e-democracy platform.
We wanted more people to become politically engaged, and make sure that their opinions and ideas were heard by decision makers.
We needed a web app where people would discuss issues, vote, share ideas and make decisions together.
We started with Ruby on Rails, and built a traditional server-side monolithic HTML serving app.
This proved to be a quick and effective way to start, but as time passed, we noticed more and more **limitations of this server-side approach**:

- The HTML pages relied on the internal ORM, so we didn't need an API for internal use. Having **no API** limits what customers can do with their data, and it also means that it's gonna be hard (or impossible) to build a native mobile App.
- Having **interactivity** (menu's that pop up, rich text editors, notifications, live editing, etc.) was difficult to achieve with server-side rendering. We liked react, but using that in a server-rendered Rails app was complex and error-prone.
- **Performance** was not as good as we wanted it to be, as most clicks required some server processing and an HTTP roundtrip.

It was clear that we needed a dedicated client-side single page application and an API.

## Designing a client-server contract (API)

Re-inventing the wheel seems like a bad idea, so we searched for best practices and existing standards in API design.
We initially went with [JSON-API](https://jsonapi.org/) (not just 'a' JSON API, but a formal specification that describes things like pagination).
Since we were already using React, and saw its popularity climbing, we went with that for the front-end.
Now we had to decide how the internal state of the client should look like.
We picked Redux, which uses a single (immutable) JS object which contains all application state (including data).

As we worked on this, our front-end started looking more and more like our back-end.
It now had explicit knowledge of:

- Many of the models from the back-end (Comment / Question / Vote / User)
- The routing logic
- The accessible parameters in endpoints
- Various actions (e.g. notifications / alerts)
- Forms and their validations.

This seemed _wrong_.
We were taught to write DRY code: _don't repeat yourself_, but it seemed like we were repeating ourselves all the time.
This meant that when we wanted some feature changed, we had to adjust _both_ the front and the back-end.
Something seemed off.

## Why we opted for Linked Data

Our CTO Thom van Kalkeren thoroughly studied the principles behind REST, HATEOS and Hypermedia, and concluded that we need to give _everything_ URLs.
Not just the _pages_ that we present to our users, but _every single thing_ that can be used by a client - including menu items, buttons, actions and form fields.
Luckily for us, the good folks at W3C had written quite a bit of specs about how data should look when you use URLs: the RDF specification (and friends).

Linked Data (or RDF) has some unique qualities:

- **Browsable**. Using links for everything in data, means that data can be surfed like how web pages work: just follow the URLs. That's actually really helpful in a web application, since this means that the client no longer needs to be aware of your routing logic.
- **Flexibility**. Everything can be serialized to various formats (JSON, XML, Turtle, N-Triples, and [more](https://ontola.io/blog/rdf-serialization-formats/)), which gives some really nice and flexible export functionality.
- **Self-describing APIs**. Simply use content-type negotiation to fetch a resource as HTML or some RDF format, and browse the data like you would browse a website: by following the links. API docs become kind of unnecessary, since navigating the website shows you all the endpoints.
- **Re-use other linked data**. Because links can point to _anywhere_ (not just your server), you can use all the publicly available linked data! This, for me, has always been the number one reason to believe in Linked Data.
- **Enables true data ownership**. Because of this, it enables _decentralized networks_ where people _own their data_, which can help to combat the existing web oligopoly. That's what the [Solid project](/solid) is all about.

So, we went with Linked Data.

## The cost of being an early adopter

Although the first [RDF spec](https://www.w3.org/TR/rdf11-concepts/) was already 10 years old, there were not many existing libraries (let alone tutorials) for us to get started.
Practically all Linked Data projects functioned as viewers that directly show individual RDF Statements in a table.
We wanted something else, we wanted a user-friendly web app that resembled social media platforms.
We needed interactivity, notifications, forms...
And doing things like that with Linked Data, turned out to be pretty hard.
Luckily for you, we've shared quite a bit of tools, libraries and ideas that could make your life a bit easier if you choose to go Linked Data.

## Getting used to RDF

RDF is a weird data model.
It was primarily designed as a language for the _semantic web_, which aims to (formally, logically) describe our world using _semantic triples_.
The design decisions made to achieve that, however, make things harder for you as a developer:

- RDF uses a lot of very long strings (URLs) instead of keys, which makes selecting values difficult.
- RDF does not have key uniqueness (subject-predicate uniqueness), which means that you could get many values if you want to know a specific property.
- RDF doesn't support native arrays, but has [a bunch of (confusing) ways of representing ordered data](https://ontola.io/blog/ordered-data-in-rdf/).
- RDF knows [many serialization formats](https://ontola.io/blog/rdf-serialization-formats/), which is a blessing and a curse. The most mentioned format in the docs (the earliest one, RDF/XML), is also the most confusing.
- Properties are not linked to specific datatypes, so you have to check datatypes for every single statement.

## Making Ruby on Rails work with RDF

Ruby on Rails is an opinionated MVC framework.
By default, it works great for generating HTML pages (or JSON objects, using Rails-API), but RDF is a whole different beast.
We created our own serializers ([rdf-serializers](https://github.com/ontola/rdf-serializers)), but more importantly, Thom and Arthur Dingemans created [linked-rails](https://github.com/ontola/linked_rails/wiki).
This gem provides some abstractions for working with RDF, including:

- Serialization (including errors)
- Policies (what users can or can't do)
- Collections (sort / filter / pagination)
- Forms (we'll get to that later)
- Menus for actions and navigation
- Actions handler (we'll get to that later, too)

Check out the [Wiki](https://github.com/ontola/linked_rails/wiki) if you want to learn more!

## Forms and form validation

Argu was always about getting opinions from people, so it needs a lot of forms.
And these can differ for many reasons:

- The model (of course)
- Specific settings for a tenant (customer)
- User and tenant-specific rights

So, how should the server communicate these forms to the client?

We stumbled upon [SHACL](https://www.w3.org/TR/shacl/), a schema language for RDF (written in RDF), that provides a way to validate RDF graphs.
This seemed like the way to go!
However, it became obvious that forms cannot _entirely_ depend on shapes.
Should you render a radio select, or a dropdown?
How do you describe a multi-page form?
How do you describe interrelated form fields (if x is a, show form option y).
And to make things even more challenging: how do you make these forms _cacheable_, given some fields might render differently for different users?

Again, we had to create our own form abstraction.
We used the SHACL spec where possible, but had to create some new Classes and Properties to solve the challenges that we faced.
The Forms DSL in [linked-rails](https://github.com/ontola/linked_rails/wiki#forms) provides a simple interface to create these forms.

## Turning RDF into HTML

Ultimately, the user needs to show a pretty web app, which means the RDF from the server has to be transformed into HTML.
Step one is fetching the data from the server, and storing it in the client.

We started using [RDFlib.js](https://github.com/linkeddata/rdflib.js), written partially by the inventor of Linked Data (and HTTP, HTML, the WWW...) Tim Berners-Lee himself.
It provided a lot of useful features for dealing with RDF: parse, fetch, serialize, mutate, search...
However, our app had kind of a unique set of requirements (mostly related to performance when dealing with many, dynamic triples), so we started working on a different RDF store.

My colleague Thom wrote two libraries: [link-lib](https://github.com/fletcher91/link-lib) and [link-redux](https://github.com/fletcher91/link-redux).
Link-lib does most of the heavy lifting: it provides a store, deals with parsing, serializing, fetching, sending RDF data, handling the actions, registering the views (more an that in a minute).
Link-redux (should be called link-react, but that name was taken) adds some useful functions for using it in React.

In practice, it works like this:

- When the user requests any URL as HTML (default browser request), they receive a mostly empty HTML page with some cached RDF data (that matches the Subject of the fetched URL) that helps to build the first view as quick as possible.
- The JS app (using Link-lib, link-redux and react) is loaded, and all the existing Views are registered. These Views are React components that will render for one or more RDF Classes, in specific Topologies (e.g. fullpage, in a column, in a menu item).
- The RDF data is added to the store, and the corresponding view is rendered.
- Note that there is no routing going on in the front-end - the rendered component is based on the Class of the resource, not the URL route.
- Depending on the rendered view, new resources might be needed (e.g. a rendered Article might show some Comments). These will be fetched by the front-end, and loading indicators will be shown in the places where data is missing.
- When the user interacts with the content (e.g. click on a link, open a menu item, press a button), they will dispatch an action and modify the internal state - all using RDF. These actions _might_ make changes on the server, and these changes _might_ have an impact on the user after that. The changes are communicated using [linked-deltas](https://github.com/ontola/linked-delta), which describes how single triples have been either changed, removed or added. When a delta is processed, components might re-render, because their properties have changed. This is a flux-like / redux-like pattern.
- Because we a _lot_ of URLs, we need some way to prevent typos. That's where the [@ontologies](https://github.com/ontola/ontologies) library comes in handy!

## Serialization & parsing performance

RDF can be serialized in many ways ([at least 10](https://ontola.io/blog/rdf-serialization-formats/)).
Some formats look like the ones you know (JSON-LD, RDF/XML), some reflect the internal model really well (N-Triples), and some have interesting and complex semantic features (Notation3).
We didn't really care about any of that - we just wanted to make things fast.
Parsing JSON-LD was _extremely_ hard and slow, Parsing Turtle was better (but still slow), so for a while we opted for the with `N-Triples`.
But ultimately, Thom came up with a new format: [HexTuples-NDJSON](https://github.com/ontola/hextuples), which was actually very simple to implement both in the Ruby back-end and in the JS front-end.
It made our _entire_ server app about 2x faster (it was working a _lot_ on serialization), and in the front-end we noticed a similar speed increase.

## Bulk-API, caching and more performance improvements

Still, our app wasn't quick enough.
The core problem: every single resource on screen with has a URL, and needs to be fetched independently.
In other words, opening a single page could require over a 100 round-trips.
Thom came up with a solution: the bulk-API, which would accept a body containing all the URLs that needed to be fetched.
This improved many things, but our back-end was still running each resource request internally with all the existing overhead (checking the user, session, etc.).
Even when that is fixed, we're still performing way too much hard work for every request - we need some form of caching.

That's why were working on a new RDF Triple Strore, written in Rust, and it is _fast_.
It offers three ways of querying:

- Single resource (HTTP GET to the subject URL)
- Bulk-API (list of subjects)
- Triple Pattern Fragments

It's not rolled out yet in production, but we'll open source it soon enough.

## FAQ

### Which triple store do you use?

Although our back-end serializes linked data, we don't use a triple store.
The Rails back-end has a mostly strict schema in SQL tables, but we do have a table in our Postgres schema that stores single triples.
Keep in mind that the RDF representation of data can be created during serialization - so your app can create linked data without using a triple store!
This is often a preferable approach in apps that have business logic or any type constraints.
But as I've mentioned earlier, we're working on a new triple store (written in Rust) that we're planning on open sourcing soon, which we use for caching to keep performance optimal.

### Why not use SPARQL?

SPARQL is the de facto query language for RDF data, so it seems logical to use it somewhere in our stack.
However, we don't.
Getting SPARQL performant is actually pretty difficult, and we don't need the powerful query options that it provides.
Most of the requests from the front-end just ask for all triples about one or multiple subjects, and these kind of queries don't require SPARQL.
SPARQL is useful for more complex graph property traversal queries, but is not necessarily the best approach for simpler queries.

### RDF tools that we've built

- [link-lib](https://github.com/fletcher91/link-lib) JS lib for managing an RDF store, registering views and handling actions
- [link-redux](https://github.com/fletcher91/link-redux) JS lib for React components, works with link-lib
- [linked-rails](https://github.com/ontola/linked_rails) Ruby gem that does a whole lot of linked data goodness
- [rdf-serializers](https://github.com/ontola/rdf-serializers) Ruby gem that for serializing RDF
- [linked-delta](https://github.com/ontola/linked-delta) for communicating state changes (between server / client, or persisting as event log)
- [@ontologies](https://github.com/ontola/ontologies) - make it easier to use ontologies (schema, owl, dcterms..) in JS projects
- [js.rdf.dev](https://js.rdf.dev/) A whole lot of RDF tools for JS!
- Check out the rest at [Github](https://github.com/ontola/).

If you need any help, [get in touch](mailto:joep@ontola.io)!


We went all-in on RDF, and build an application that's linked data through-and-through. It was a challenging ride, and it led us to some interesting solutions.

Full-stack linked data: lessons from building an RDF web app


Sooner or later when working with RDF, you'll need to work with ordered data / n-ary relations.
The `subject predicate object` model does not support Arrays, and no - you can't use the order in which triples appear.

However, RDF _does_ support _Collections_ (`rdf:List`) and _Containers_ (`rdf:Bag`, `rdf:Seq`, `rdf:Alt`).
And the open nature of RDF allows for even more alternatives, such as the [Ordered List Ontology](http://smiy.sourceforge.net/olo/spec/orderedlistontology.html).
All these concepts are (subtly) different, and can be quite confusing.
In this article, I'll explain the models behind these concepts, show how they are serialized, and give some insights into when you should use which.

Skip to the bottom for a TL;DR!

## RDF Collections

Maybe you've seen something like this in JSON-LD:

```json
{
  "@list": [ "Arnold", "Bob", "Catherine" ]
}
````

Or something like this in a Turtle document:

```
someList :b ( "Arnold" "Bob" "Catherine")
```

These are `rdf:Collection`s.
These serialization formats (turtle and JSON-LD) have syntactic sugar for Collections, so they appear as regular arrays.
Under the hood, however, they have a very different data model.
Collections are [linked lists](https://en.wikipedia.org/wiki/Linked_list) and its chains consist of `rdf:List` nodes, connected by `rdf:rest` relations:

![RDF:List](/images/posts/ordered/rdflist_basic.png)

Every `rdf:List` has a `rdf:first` object, which refers to the actual content of the list item.
Intuitively, it might be a bit weird to refer to Bob as a `first` in a List.
However, since Lists are recursive and often contain other Lists, it actually makes sense to refer to the content as `first`.
Collections always end with an `rdf:nil` namednode, which means they have a _formally known ending_.
If we'd express the same information in N-Triples, we'd get something like this:

```
_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "Arnold" .
_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> _:b1 .
_:b1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "Bob" .
_:b1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> _:b2 .
_:b2 <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "Catherine" .
_:b2 <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> <http://www.w3.org/1999/02/22-rdf-syntax-ns#nil> .
```

This explicitly linked nature means that it's possible to start a list in location A, and have it continue in a completely different domain:

![RDF:List](/images/posts/ordered/rdflist_external.png)

It also means that inserting items is quite easy, as only the node before it has to be adjusted:

![RDF:List](/images/posts/ordered/rdflist_insert.png)

However... This structure can be kind of hard to deal with.
Parsing is non-trivial, and might require a lot of lookups, which is very costly.
Also, appending a single node to a List is hard, because it requires that the link of the item before it is adjusted.

## RDF Containers

There are three types of RDF Containers `rdf:Seq`, `rdf:Bag`, `rdf:Alt`

- An `rdf:Seq` is an _ordered_ container
- An `rdf:Bag` is an _unordered_ container
- An `rdf:Alt` is an _unordered_ set of alternatives, in which the first one is the default option

RDF Containers use a numbered predicate (e.g. `rdf:_1`) to indicate that something is a child / member of the Container:

![RDF:Seq](/images/posts/ordered/rdfseq.png)

This model is often easier to parse and serialize than an `rdf:List`.

```
_:someSeq <http://www.w3.org/1999/02/22-rdf-syntax-ns#_1> "Arnold" .
_:someSeq <http://www.w3.org/1999/02/22-rdf-syntax-ns#_2> "Bob" .
_:someSeq <http://www.w3.org/1999/02/22-rdf-syntax-ns#_3> "Catherine" .
```

Appending items is easy as well, simply add one new statement and increment the predicate.
However, inserting items requires you to rewrite many triples.
If you use a reasoner that has implemented the basic RDFS spec, you can use [`rdfs:member`](https://www.w3.org/TR/rdf-schema/#ch_member) (the superclass of every `rdf:_n` property) to get all members of an RDF Container.

## Pagination with ActivityStreams and Hydra Collections

When your arrays will be either too long to serialize, or too computationally heavy to generate at run-time, you'll need some form of pagination.
It's recommended to use an existing ontology for this.
The W3C Activity Streams 2.0 ontology defines the [`as:Collection`](https://www.w3.org/TR/activitystreams-core/#collection).
It provides keys for things like `next`, `totalItems` and `orderedItems`.
This spec seems to be designed with `JSON-LD` serialization in mind, so it relies on JSON arrays, which represent `rdf:List` Collections.

Similar to this is the [`hydra:Collection`](https://www.hydra-cg.com/spec/latest/core/#collections).
The Hyrda spec still is in draft status, but it has been updated in october last year.

## Converting to Arrays

If you're building an RDF application, and you want to access your RDF Lists / Collections / Sequences as an array, make sure to find an RDF library that has support for ordered data.
We've open-sourced a JS library that might be useful for this: [`@rdfdev/collections`](https://js.rdf.dev/modules/_rdfdev_collections).
[Get in touch](mailto:joep@ontola.io) if we can help!

## TL;DR

There are no arrays in RDF, and don't use the order in which serialized triples appear.

**RDF Containers:**
- Come in three forms: `rdf:Seq` (ordered), `rdf:Bag` (unordered), `rdf:Alt` (alternatives with default)
- You can add new items by simply adding RDF triples
- Inserting items is hard: requires rewriting *many* statements
- Must be stored in a single graph / machine / server (centralized)
- Have a formally unknown ending (open world assumption)

**RDF Collections:**
- An ordered chain of `rdf:List` resources
- You have to edit / remove statements before you can add new items
- Inserting items is easy: requires changing *just a few* statements
- Can span many graphs / machines / servers (decentralized)
- Have a known ending (the `rdf:nil`)

**Pagination:**
- Use [ActivityStreams collections](https://www.w3.org/TR/activitystreams-core/#collection)

**Converting to arrays:**
- Use a library, such as [`@rdfdev/collections`](https://js.rdf.dev/modules/_rdfdev_collections)


In this article, we'll discuss one of the most confusing but important topics of RDF: the various ways you can create ordered data.

Ordered data in RDF: About Arrays, Lists, Collections, Sequences and Pagination


[Open Decision Making](https://openbesluitvorming.nl) (Open Besluitvorming in Dutch) lets you search through what has been discussed in Dutch governments.
In collaboration with local governments, the VNG and the Open State Foundation, we've built software that standardizes and aggregates the meeting data of more than 120 governments.
All this open information makes it easier for journalists, policymakers and well-informed citizens to get involved in.
With the principles of [linked data](/blog/what-is-linked-data/) we ensured that the data is open, uniform and reusable for anyone that likes to build upon it.

## How it started

In 2014 we started our e-democracy platform [Argu.co](https://argu.co), which aims to engage citizens in decision-making processes.
We worked with many governments and hosted online discussions where people shared ideas, opinions and voted on issues.
Although we've seen that e-democracy and civic participation can be very successful, one key element is often missing: the actual decision-making process.
This has to do with the fact that e-democracy discussions tend to happen parallel to the actual governmental meetings.
We wanted to bring these two worlds together.
To achieve that goal, we decided to work on gathering and standardizing the data that these decision-makers create: meetings, votes, agenda-items and many other documents.

In 2016 we started working with the [Open State Foundation](https://openstate.eu/), who initiated Open Raadsinformatie (Open Municipality Data).
VNG (the Dutch Municipality Union) provided funding, we joined forces, got involved with the technology, and helped to scale the project up from 7 to over 120 municipalities and 6 provinces.

## Going from PDF to enriched text

Dutch municipalities produce a *lot* of information in their meetings.
They use meeting software to organize and distribute their agenda items, documents, motions, and reports online.
However, all these applications tend to work quite differently, all with their own information schemes.
Also, many of the proceedings and documents are saved in PDF format and are often not publicly accessible to download.
With our access to these systems and a lot of time to understand and implement the various APIs, we were able to bring everything together and create a standardized, searchable base of information that is always publicly accessible.

Most of the interesting information hides in PDF files.
We wrote ETL (extract, load, transform) software using Python, Celery and Elasticsearch that gets all the source files, standardizes it to a single model and loads the text into various query systems.
We store and cache all these files in Google Cloud Storage, along with the referencing sources and metadata.
This enables us to serve files that are not accessible through the default API of the meeting software sources.
This also makes it possible to see what has changed between versions.

Additionally, two organizations contributed two enrichers that add derived information to the resources:

- Theme classifier: a [machine-learning classifier](https://github.com/openstate/ori-theme-classifier) written by the Open State Foundation.
Scores document relevance to a set of specified themes, such as Economy, Safety and Education.

- Location classifier: based on the project [LocLinkVis](https://bitbucket.org/aolieman/loclinkvis/) made by Alex Olieman and the University of Amsterdam.
It finds streets and locations in document texts and translates it into geographical coordinates of city neighborhoods and districts.

These enrichers allow all users of Open Decision Making to filter documents by theme tags and use geographical information to pinpoint which locations are mentioned.

## Storing RDF

We love [linked data](what-is-linked-data/), and its merits are exceptionally valuable in a project such as this one.
Linked data is highly re-usable in other applications without needing duplication, and the [diverse serialization formats](/blog/rdf-serialization-formats/) of RDF will provide something for any use-case.

However, working with linked data poses new challenges, such as choosing the right data store for the statements.
You might think that an existing triple store is the most logical option since we're working with RDF, but you don't need triple stores to do serve linked data.
Triple stores make a lot of sense when you have schema-less RDF data and require SPARQL as a query language.
We didn't need those, as our schema was constrained by our internal models, and SPARQL didn't suit our query requirements.
The primary goal was providing full-text search and RESTful RDF resources.

Before we joined the project, Elasticsearch was used as the only data store.
It ticked the text search box, as Elasticsearch has a powerful text search query-language, but it is not designed for relational database queries.
Also, it's not a database as it's not ACID compliant (i.e., highly resistant to corruption or data loss).

So, looking for a suitable store, we started with ArangoDB, which is a NoSQL document-based graph store.
It seemed appropriate at the time because we were working with lots of documents, all connected in a graph.
However, we found that the graph queries were taking too long to be suitable for this project.

Next we tried Neo4j, which is the market leader in Graph databases.
It offers a powerful query language called Cypher, but we found that memory use was growing too early too fast.
Another problem related to storing RDF in Neo4j - it should be possible, but there are some differences between the graph models of RDF and Neo4j.
We had some trouble with using URLs as keys (a requirement for RDF), which had to be escaped, making the queries long and dreadful.
Also, you have to think about how to cope with external resources.
Whereas in an RDF store one can add a triple referencing an external resource, in Neo4j one should create a node to mimic the same behavior.

In the end, we abandoned graph databases altogether for this project and stuck with Postgres.
We didn't need graph-like queries or schemaless storage of triples.
I am not saying that those graph databases are not good enough, merely not suitable for our purpose and budget.

Since the project depends heavily on fast text search queries, we found that Elasticsearch is still essential.
So we kept it on for full-text searching.
But since its Query DSL is changing quite a lot over time, it is not suitable as a stable API.
We had to come up with an answer to this and created the ORI API service which can be found [here](https://id.openraadsinformatie.nl/).
The idea here was that every resource should have its own resolvable identifier that would stay the same.

## Open Data

Several applications now actively use the freely available data from Open Decision Making:

### OpenBesluitvorming.nl

We have built the [primary open-source search application (Dutch)](https://openbesluitvorming.nl) for users to discover and explore what is being said in their local government.
The beauty of open data is that everyone can build their own applications based on the same API.

![openbesluitvorming.nl](/images/posts/ori/zoek-openraadsinformatie.jpg "openbesluitvorming.nl")

&nbsp;

### Raadstalk.nl

VNG is the Dutch umbrella organization for all municipalities and has been the driving force in this project.
They've asked us to create a [widget](https://raadstalk.nl) to see what words are trending for all municipalities in each month.

![raadstalk.nl](/images/posts/ori/raadstalk.jpg "Raadstalk.nl")

&nbsp;

### Waaroverheid.nl

[WaarOverheid](https://waaroverheid.nl) shows all documents that mention a location in your neighborhood and district.
It provides a map interface to navigate from neighborhood to neighborhood to find documents in streets located there.

![waaroverheid.nl](/images/posts/ori/waaroverheid.jpg "Waaroverheid.nl")

&nbsp;

### 1848.nl

[1848](https://1848.nl) gathers political information from different sources to search and receive alerts to be able to stay on top of things.

![1848.nl](/images/posts/ori/1848.jpg "1848.nl")

&nbsp;

## Going forward

It all started with municipality data, but the scope has now broadened to provincial data and will soon include data from the national level as well.
We are working with the Dutch parliament on integrating their brand new API.

We believe that this is just a preview of what is possible with open government data.
If we can convince governments to adopt a more data-focused information publishing strategy (using machine-readable formats, such as RDF, instead of PDF), we can create even more engaging web applications: imagine clicking on a representative to see which motions they submitted, or compare voting behavior between political parties.

Working on this project has been challenging, we've overcome quite some hurdles that come with managing such amounts of data and systems.
Along the way, we have learned a great deal about ETL processes, Elasticsearch, Kafka and Kubernetes, to name a few.

[Contact us](/contact) to know how we can help you achieving your goals in modeling and transforming information to linked data.


We've aggregated meeting data from over 250+ Dutch governmental organisations and are now publishing it as linked data.

Publishing Dutch governmental meetings as linked data


Ontologies are resources that describe concepts and how several classes and properties in a certain domain relate to each other.
For example, if you're creating a social network app, you might want to use the Friend of a Friend ([FOAF](http://xmlns.com/foaf/spec/)) ontology, which describes things like ["Person"](http://xmlns.com/foaf/spec/#term_Person) and ["knows"](http://xmlns.com/foaf/spec/#term_knows).
When creating linked data applications, you're likely to work a lot with ontologies.
Since ontologies, their classes and their properties all use URLs as identifiers, that means that you might write a lot of long strings.
This is not only bothersome, but also a source of unnecessary typos and a possible performance bottleneck.

That's why we created the [@ontologies](https://github.com/ontola/ontologies) project.
This is an open source repository that converts RDF ontologies into JS exports, pubslihed in NPM for easy re-use.
If your IDE is smart enough, it will auto-import the ontologies and provide and useful tooltips with ontological descriptions for each item.

## How to use it

```sh
## Add it to your project
yarn add @ontologies/core @ontologies/schema
```

```js
// Initialize it once (./init.js)
import { setup } from "@ontologies/core";
setup();
```

```js
// Use it
import "./init";
import { name } from "@ontologies/schema";

document.getElementById("app").innerHTML = `
<h1>Schema URL: ${name.value}</h1>
`;
```

Check out a [running example on CodeSandbox](https://codesandbox.io/s/ontologies-41zgg).

We've currently added ActivityStreams, DCElements, DCTerms, DCMIType, FOAF, OWL, PROV-O, RDF, RDFS, Schema, SHACL, SKOS and XSD.

PRs for new ontologies are welcome!


Our new @ontologies NPM package helps developers write RDF applications in Javascript.

Using ontologies in Javascript apps


Contrary to some other data models, RDF is not bound by a single serialization format.
Triple statements (the data atoms of RDF) can be serialized in many ways, which leaves developers with a possibly tough decision: how should I serialize my [linked data](/blog/what-is-linked-data)?

## To answer the title's question: _it depends_

Skip to the [TL;DR](#tldr) if you're feeling hasty.
The order in which they appear is **chronological / historical and does not reflect preference**.

## RDF/XML

The first and perhaps most well-known RDF serialization format is [RDF/XML](https://www.w3.org/TR/rdf-syntax-grammar/).
It's also the most [despised](https://github.com/mhausenblas/rdfxml.info/blob/master/input/RDF-XML%20sucks%20-%20praise%20and%20damnation.txt).
Many systems were able to parse, store and serialize XML when RDF was invented almost 20 years ago, so RDF/XML seemed like a logical default.

Unfortunately, RDF/XML is a weird mixture of two fundamentally different concepts: a tree-like document, and a triple-based graph.
This makes RDF/XML conceptually difficult and quite verbose, compared to other standards.
For XML developers, it might look familiar, but since it does not clearly reflect the triple model, it will probably cause confusion.

Use it only if you need to work with XML.

```xml
<?xml version="1.0"?>

<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:schema="http://schema.org/">

<rdf:Description rdf:about="https://www.w3.org/People/Berners-Lee/">
  <schema:birthDate>1966-06-08</schema:birthDate>
  <schema:birthPlace rdf:resource="http://dbpedia.org/resource/London"/>
</rdf:Description>

</rdf:RDF>
```

## RDFa

[RDFa](https://www.w3.org/TR/rdfa-core/) is RDF inside HTML.
By adding attributes to HTML elements, you can give semantic context to the content inside your webpages.
Google parses it (amongst [JSON-LD](#JSON-LD), [Microdata](https://w3c.github.io/microdata/#microdata-and-rdf), Microformats and Pagemaps) to [enhance their search previews](https://developers.google.com/custom-search/docs/structured_data), although they [recommend using JSON-LD](https://www.searchenginejournal.com/google-structured-data-preference/297479/).
W3C's own Respec documentation tool [dropped support for RDFa](https://github.com/w3c/respec/issues/1503),
mainly because the adoption of RDFa was too low, the required code was messy and even Google didn't parse it correctly.

RDFa is fundamentally different from the other mentioned formats: it combines RDF with view data (HTML).
This means that it makes your HTML documents a bit larger and more complicated, and parsing it for triples will be more costly than parsing an RDF only format like N-Triples.
This makes it less useful if your application relies on a lot of RDF data.

Use RDFa if you want to make your existing website/ blog / HTML based application more semantic.

```html
<div about="https://www.w3.org/People/Berners-Lee/">
  <p>
    Tim is born on
    <span property="http://schema.org/birthDate">1955-06-07</span>
    in
    <a
      property="http://schema.org/birthPlace"
      href="http://dbpedia.org/resource/London"
    >
      London
    </a>
  </p>
</div>
```

## Notation3 (.n3)

Tim-Berners Lee wanted something better than RDF/XML, and came up with [N3](https://www.w3.org/TeamSubmission/n3/).
Contrary to RDF/XML, N3 closely resembles the RDF Subject / Predicate / Object model.
This makes N3 very easy on the eyes and helps to understand how RDF works.
By using `@prefix`es, N3 can be quite compact.

However, N3 is relatively costly to serialize, which could hinder performance.
It's also quite feature-heavy since it supports [RDF rules](http://www.ninebynine.org/RDFNotes/RDFFactsAndRules.html), which makes it harder to parse.

Unless you need the reasoning/rules features of N3, use its more popular (and very similar) successor Turtle.

```
@prefix tim: <https://www.w3.org/People/Berners-Lee/>.
@prefix schema: <http://schema.org/>.
@prefix dbpedia: <http://dbpedia.org/resource/>.

<tim> schema:birthDate "1955-06-08"^^<http://www.w3.org/2001/XMLSchema#date>.
<tim> schema:birthPlace <dbpedia:London>.
<tim> schema:birthPlace <dbpedia:London>.
```

## Turtle (.ttl)

[Turtle](https://www.w3.org/TR/turtle/), the _Terse RDF Triple Language_, is a subset of N3.

It strips some of the syntactic sugar en features of N3, which makes parsing Turtle a bit simpler.
This, in turn, made Turtle more popular, which means that it's easier to find libraries for it.

Unfortunately, it's still quite costly to parse compared to [N-Triples](#N-Triples).

Turtle is highly human-readable and is, therefore, a good candidate if you need to edit RDF by hand.

```
@prefix tim: <https://www.w3.org/People/Berners-Lee/>.
@prefix schema: <http://schema.org/>.
@prefix dbpedia: <http://dbpedia.org/resource/>.

<tim> schema:birthDate "1955-06-08"^^<http://www.w3.org/2001/XMLSchema#date>.
<tim> schema:birthPlace <dbpedia:London>.
```

## N-Triples (.nt) and N-Quads (.nq)

[N-Triples](https://www.w3.org/TR/n-triples/) is a very simple subset of Turtle, which in turn is a simple subset of N3.
N-Triples does not support `@prefixes` or any fancy features.
This makes N-Triples trivial to parse / serialize.
Therefore, many libraries for N-Triples are available and you can easily write one yourself.
It also makes serialization and parsing highly performant.

```
<https://www.w3.org/People/Berners-Lee/> <http://schema.org/birthDate> "1955-06-08"^^<http://www.w3.org/2001/XMLSchema#date>.
<https://www.w3.org/People/Berners-Lee/> <http://schema.org/birthPlace> <http://dbpedia.org/resource/London>.
```

However, the lack of prefixes and shorthands makes the format lengthy and a bit tough to read.
The lengthy URLs also mean that you'll need some form of compression (e.g.g-zip) if you don't want to waste precious bandwidth or storage capacity, so make sure to enable that in your server.

Since writing a parser / serializer for N-Triples is so simple, it's a good idea to support this pretty much always.
And since N-Triples is a subset of Turtle and N3, it means that Turtle / N3 parsers know how to deal with N-Triples, too.

[N-Quads](https://www.w3.org/TR/n-quads/) are like N-Triples, but they have an optional fourth column, which can be used to denote a graph label.
The graph label often refers to the source of the data, e.g. the URL of the HTML document or some external RDF resource.

## JSON-LD (.jsonld)

JSON is, without a doubt, the most popular way to serialize data in web applications.
[JSON-LD](https://json-ld.org/spec/latest/json-ld/) is an extension of JSON and is valid JSON as well.
You can turn your regular plain old JSON into RDF by adding [`@context`](https://json-ld.org/spec/latest/json-ld/#the-context).
This object mainly serves as a mapping, so your plain keys get turned into fancy links to RDF Classes and Properties.
You can add context either by adding an `@content` header in your HTTP response, by including the link in your JSON body, or by adding the entire `@context` object to you JSON.
This means that if you want to upgrade your JSON API to JSON-LD, you get to keep your serializers.

```json
{
  "@context": {
    "dbpedia": "http://dbpedia.org/resource/",
    "schema": "http://schema.org/"
  },
  "@id": "https://www.w3.org/People/Berners-Lee/",
  "schema:birthDate": "1955-06-08",
  "schema:birthPlace": {
    "@id": "dbpedia:London"
  }
}
```

JSON-LD is **easy to read**, and will feel familiar even to those new to RDF and linked data.
Because it's still valid JSON, it's easily usable for those who don't want to deal with URLs.
JSON arrays are converted to [RDF Lists](/blog/ordered-data-in-rdf/) (but can also be converted to [other sequence types]()).
I recommend spending some time in the [JSON-LD playground](https://json-ld.org/playground/) to get familiar with how it works.

Unfortunately, JSON-LD [difficult and costly to parse](http://www.dr-chuck.com/csev-blog/2016/04/json-ld-performance-sucks-for-api-specs/) if you need the RDF data instead of the JSON object.
Parsing JSON-LD often involves requesting data from the internet, and needs clever caching to be performant.
This complexity in parsing limits how many (bug-free) JSON-LD parsers are available, makes libraries heavy, and it also means that parsing JSON-LD takes relatively long.

JSON-LD is a compromise.
It supports RDF, it supports JSON, and it does both _okay_.
Use JSON-LD if you already have a RESTful JSON API, and if performant RDF parsing is not crucial.

## JSON-AD (Atomic Data)

_Disclaimer: I'm the creator of JSON-AD_.

[JSON-AD](https://docs.atomicdata.dev/core/json-ad.html) (JSON Atomic Data) only supports a strict subset of RDF, so it is an odd one in this list.
In other words: it cannot be used to serialize all existing RDF data.
However, because it allows only a strict subset, it also has a couple of advantages over other formats:

- Very **low parsing cost / high performance**. It can be parsed as plain JSON, which means that many efficient and performant parsers are available.
- **Easy to understand**. Since it supports only a subset of RDF, it doesn't suffer from some of the [quirks of RDF](https://docs.atomicdata.dev/interoperability/rdf.html) such as predicate-non-uniqueness
- It supports **native JSON arrays** as collections, which prevents the complexities of dealing with [ordered data in RDF](https://ontola.io/blog/ordered-data-in-rdf/).
- **Type safety**. Each key in a JSON-AD object should resolve to a [Property](https://atomicdata.dev/classes/Property), which describes [datatype](https://atomicdata.dev/properties/datatype) and [description](https://atomicdata.dev/properties/description).

Here's the `description` Property from above, serialized as JSON-AD:

```json
{
  "@id": "https://atomicdata.dev/properties/description",
  "https://atomicdata.dev/properties/datatype": "https://atomicdata.dev/datatypes/markdown",
  "https://atomicdata.dev/properties/description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.",
  "https://atomicdata.dev/properties/isA": [
    "https://atomicdata.dev/classes/Property"
  ],
  "https://atomicdata.dev/properties/shortname": "description"
}
```

Because these properties also have [`shortnames`](https://atomicdata.dev/properties/shortname), we can easily [serialize Atomic Data to plain JSON](https://docs.atomicdata.dev/interoperability/json.html), without the long URLS:

```json
{
  "@id": "https://atomicdata.dev/properties/description",
  "datatype": "https://atomicdata.dev/datatypes/markdown",
  "description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.",
  "is-a": [
    "https://atomicdata.dev/classes/Property"
  ],
  "shortname": "description"
}
```

## HexTuples

[HexTuples](https://github.com/ontola/hextuples) is an [NDJSON](http://ndjson.org/) (Newline Delimited JSON) based RDF serialization format.
It is designed to achieve the best possible performance in a JS context (i.e. the browser).
It uses plain JSON arrays, in which the position of the items denote `subject`, `predicate`, `object`, `datatype`, `lang` and `graph`.

```json
["https://www.w3.org/People/Berners-Lee/", "http://schema.org/birthDate", "1955-06-08", "http://www.w3.org/2001/XMLSchema#date", "", ""]
["https://www.w3.org/People/Berners-Lee/", "http://schema.org/birthPlace", "http://dbpedia.org/resource/London", "http://www.w3.org/1999/02/22-rdf-syntax-ns#namedNode", "", ""]
```

HexTuples is designed by Thom van Kalkeren (a colleague of mine) because he noticed that parsing / serialization was unnecessarily costly in our stack, even when using the relatively performant `n-quads` format.
Since HexTuples is serialized in NDJSON, it benefits from the [highly optimized JSON parsers in browsers](https://v8.dev/blog/cost-of-javascript-2019#json).
It uses NDJSON instead of regular JSON because it makes it easier to parse concatenated responses (multiple root objects in one document).
As an added plus, this enables streaming parsing as well, which gives it another performance boost.
Our JS RDF libraries ([link-lib](https://github.com/fletcher91/link-lib/), [link-redux](https://github.com/fletcher91/link-redux/)) have an internal RDF graph model which uses these arrays as well, which means that there is minimal mapping cost when parsing Hex-Tuple statements.
This format is especially suitable for real front-end applications that use dynamic RDF data.

## HDT

[HDT](http://www.rdfhdt.org/what-is-hdt/) (Header, Dictionary, Triples) is a compact data structure and binary serialization format for RDF, so it's more than just a way to serialize RDF.
Its data structure saves space and bandwidth (it's half the size of gzipped N-Triples).
Its design has indexing built-in, which means it can be searched or browsed efficiently.
Check out the impressive [technical specification](http://www.rdfhdt.org/technical-specification/) if you want to learn more about how it works.
HDT compression is a costly process, so it's not that attractive for highly dynamic data / data that changes over time.
Although some really useful libraries for HDT exist, be sure to check if there exists libraries that work with your stack.

## RDF Binary Thrift

[RDF Binary thrift](https://afs.github.io/rdf-thrift/rdf-binary-thrift.html) is an encoding of RDF data that uses Apache Thrift.
It is used in the Apache Jena RDF store.
It's a binary format, and therefore it's cheap to parse and serialize.
I've never used it, but it sounds interesting, although you'll need Thrift tooling for decoding it.

## Let your users choose a format

Choosing an RDF serialization format for your application or service might be a false dilemma.
Since you control your application and probably have an internal model, you can offer multiple serialization options.
Therefore, you can implement a serialization library (e.g. our [rdf-serializers](https://github.com/ontola/rdf-serializers) gem for Rails) and use [HTTP Content negotiation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation), so your project can handle all kinds of formats.

## TL;DR

- Use Hex-Tuples if you want high performance in JS with dynamic data.
- Use JSON-AD if you don't have to support existing RDF data, but do value JSON compatibility and type safety.
- Use HDT if you have big, static datasets and want the best performance and compression.
- Use N-Triples / N-Quads if you want decent performance and high compatibility.
- Use JSON-LD if you want to improve your existing JSON API, and don't need performant RDF parsing.
- Use Turtle if you want to manually read & edit your RDF.
- Use Notation3 if you need RDF rules.
- Use RDFa to extend your existing HTML pages.
- Use RDF/XML if you need to use XML.
- If you can, support all of them and use content negotiation.


Since RDF has a wide variety of serialization formats, it's hard to pick the best one. This post will help you to find out.

What's the best RDF serialization format?


[JSON-HAL](http://stateless.co/hal_specification.html) and [JSON-LD](https://json-ld.org/) are pretty popular formats for desiging APIs. They both have [their merits](https://sookocheff.com/post/api/on-choosing-a-hypermedia-format/), but they aren't compatible with each other. However, with some additions, we can turn JSON-HAL into valid JSON-LD, without changing any of the original keys. This way, you can keep compatibility with HAL clients and benefit from the [advantages of linked data](/blog/what-is-linked-data), such as conversion to other RDF formats.

Let's start with a JSON-HAL body object from the [docs](https://tools.ietf.org/html/draft-kelly-json-hal-08).

```json
{
  "_links": {
    "self": { "href": "/orders/523" },
    "warehouse": { "href": "/warehouse/56" },
    "invoice": { "href": "/invoices/873" }
  },
  "currency": "USD",
  "status": "shipped",
  "total": 10.20
}
```

To identify the main resource, we add an `@id` value:

```json
{
  "@id": "https://example.com/orders/123",
  "_links": {
    "self": { "href": "/orders/523" },
    "warehouse": { "href": "/warehouse/56" },
    "invoice": { "href": "/invoices/873" }
  },
  "currency": "USD",
  "status": "shipped",
  "total": 10.20
}
```

Let's add an `@context` object to map our keys to URIs:

```json
{
  "@id": "https://example.com/orders/123",
  "@context": {
    "currency": "https://example.com/ns/shipping#currency",
    "status": "https://example.com/ns/shipping#status",
    "total": "https://example.com/ns/shipping#total",
    "warehouse": "https://example.com/ns/shipping#warehouse",
    "invoice": "https://example.com/ns/shipping#invoice"
  },
  "_links": {
    "self": { "href": "/orders/523" },
    "warehouse": { "href": "/warehouse/56" },
    "invoice": { "href": "/invoices/873" }
  },
  "currency": "USD",
  "status": "shipped",
  "total": 10.20
}
```

If we try to [parse this as JSON-LD](https://json-ld.org/playground/#startTab=tab-nquads&json-ld=%7B%22%40id%22%3A%22https%3A%2F%2Fexample.com%2Forders%2F123%22%2C%22%40context%22%3A%7B%22currency%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23currency%22%2C%22status%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23status%22%2C%22total%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23total%22%2C%22warehouse%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23warehouse%22%2C%22invoice%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23invoice%22%7D%2C%22_links%22%3A%7B%22self%22%3A%7B%22href%22%3A%22%2Forders%2F523%22%7D%2C%22warehouse%22%3A%7B%22href%22%3A%22%2Fwarehouse%2F56%22%7D%2C%22invoice%22%3A%7B%22href%22%3A%22%2Finvoices%2F873%22%7D%7D%2C%22currency%22%3A%22USD%22%2C%22status%22%3A%22shipped%22%2C%22total%22%3A10.2%7D), we only get three valid triples. That's because the items in `_links` are not yet parsed. To do this, we need to let the JSON-LD parser know three things:

- The object in `_links` describes the same resource as the root object. We do this by adding an `@id` value in the `_links` object with a value identical to the `@id` of the root object.
- The `href` values in the objects in `_links` are actually the URI values of these properties. We do this by adding an2 `@context` object to the `_links` body that binds `href` to `@id`.
- The `_links` object contains useful information, don't skip it, parser! For the JSON-LD playground parser, we can do this by mapping `_links` to some arbitrary URL. Since documenting such a hacky solution might be useful, let's link to this article. This is the most hacky part of my solution, so let me know if you find something better!

```json
{
  "@id": "https://example.com/orders/123",
  "@context": {
    "currency": "https://example.com/ns/shipping#currency",
    "status": "https://example.com/ns/shipping#status",
    "total": "https://example.com/ns/shipping#total",
    "warehouse": "https://example.com/ns/shipping#warehouse",
    "invoice": "https://example.com/ns/shipping#invoice",
    "_links": "https://ontola.io/blog/json-hal-ld"
  },
  "_links": {
    "@id": "https://example.com/orders/123",
    "@context": { "href": "@id" },
    "self": { "href": "/orders/523" },
    "warehouse": { "href": "/warehouse/56" },
    "invoice": { "href": "/invoices/873" }
  },
  "currency": "USD",
  "status": "shipped",
  "total": 10.20
}
```

If we [parse this as JSON-LD](https://json-ld.org/playground-dev/#startTab=tab-nquads&json-ld=%7B%22%40id%22%3A%22https%3A%2F%2Fexample.com%2Forders%2F123%22%2C%22%40context%22%3A%7B%22currency%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23currency%22%2C%22status%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23status%22%2C%22total%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23total%22%2C%22warehouse%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23warehouse%22%2C%22invoice%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23invoice%22%2C%22_links%22%3A%22https%3A%2F%2Fontola.io%2Fblog%2Fjson-hal-ld%22%7D%2C%22_links%22%3A%7B%22%40id%22%3A%22https%3A%2F%2Fexample.com%2Forders%2F123%22%2C%22%40context%22%3A%7B%22href%22%3A%22%40id%22%7D%2C%22self%22%3A%7B%22href%22%3A%22%2Forders%2F523%22%7D%2C%22warehouse%22%3A%7B%22href%22%3A%22%2Fwarehouse%2F56%22%7D%2C%22invoice%22%3A%7B%22href%22%3A%22%2Finvoices%2F873%22%7D%7D%2C%22currency%22%3A%22USD%22%2C%22status%22%3A%22shipped%22%2C%22total%22%3A10.2%7D), we get all six triples:

```
<https://example.com/orders/123> <https://example.com/ns/shipping#currency> "USD" .
<https://example.com/orders/123> <https://example.com/ns/shipping#invoice> <https://json-ld.org/invoices/873> .
<https://example.com/orders/123> <https://example.com/ns/shipping#status> "shipped" .
<https://example.com/orders/123> <https://example.com/ns/shipping#total> "1.02E1"^^<http://www.w3.org/2001/XMLSchema#double> .
<https://example.com/orders/123> <https://example.com/ns/shipping#warehouse> <https://json-ld.org/warehouse/56> .
<https://example.com/orders/123> <https://ontola.io/blog/json-hal-ld> <https://example.com/orders/123> .
```

Awesome! Just by adding a few keys, we converted JSON-HAL into JSON-LD, without breaking anything. However, the resulting object is not that pretty and quite verbose. Only use this is you really _have_ to support both JSON-HAL and JSON-LD. If you haven't invested in JSON-HAL yet, but do need a linked data API, I recommend you go with JSON-LD plus [Hydra](https://www.hydra-cg.com/spec/latest/core/), which also offers standardization for things like actions, collections and pagination.

## TL;DR

- Add an `@id` to both the root object and the `_links` object with the main resouce URL.
- In the root `@context`, add your usual JSON-LD mapping.
- In the root `@context`, map`_links` to some URL.
- In the `_links` `@context`, map `href` to `@id`.


JSON-HAL and JSON-LD can be made to be fully compatible with each other. We'll discuss how in this article.

Upgrade JSON-HAL to JSON-LD


The internet started off as a place for linked documents.
However, where humans are perfectly capable of understanding HTML documents through a browser, this does not apply for machines that need a specific piece of information.
That’s why every decent web service has an API, and probably uses it for their website, app, integrations and external services.
Unfortunately, too many APIs are unnecessarily hard to use and unintuitive.
In this article, I'll give some practical advice on designing a RESTful, hypermedia API that follows web conventions.

## Use URLs as IDs
Every thing in your API, every individual resource, should have its own URL. The URL should serve as both an _identifier_ as well as a _locator_: it is the identity of the thing and it provides a way to fetch information about that thing. This makes URLs really useful.

Firstly, URLs make your responses _far easier to navigate_. Imagine a JSON object representing a social media post that references some author with an identity of `18EA91FB19`. From this information, you wouldn't know where to find that author. You need to read the API docs, discover the endpoint for authors and compose your request. If the ID was a URL, you would instantly know where to send that request to. This is not just great for humans, but also for machines, since they can't read your API docs - but they can navigate URLs.

Secondly, URLs are always unique. URLs are not just unique identifiers in a single system, but also unique across different systems. The domain name takes care of that. This means that you can use your data across multiple systems without worrying about name collisions in identifiers. This is one of the properties [that makes linked data awesome](https://ontola.io/blog/what-is-linked-data).

Make sure that your URLs are stable. [Cool URIs don’t change](https://www.w3.org/Provider/Style/URI). If they really have to change, make sure the old URLs redirect to the new ones. Nobody likes broken links.

## Your API endpoint is your website
You don't need a subdomain for your API, like `api.example.com` or a sub-path, like `example.com/api`. Your endpoint should be the root of your webpage: `example.com`.

This is useful, because as discussed above the URL should be both the identifier as the locator of a single resource. Whether someone is looking for a HTML version or for example a JSON representation of a resource, he should be able to use the same URL. This makes your API easier to use, because someone who navigates your website can know at any time how to access the same resource in some other format.

But if the URL does not change across format, how do you request the right one? This is where [HTTP content negotiation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept) comes in handy. A client can send preferences about what kind of content it wants to receive in the `Accept` HTTP header. The default header for web browsers is `text/HTML`, but for most APIs, a machine readable setting such as `application/json` is more suitable.

But what about API versioning? We want our URLs _not to change_, so we should not use different URLs for different API versions. The solution, again, is to use a HTTP header. Use an `api-version` header or a [specific Mime type](https://developer.github.com/v3/media/) in your requests.

## Use sensible hierarchy in URL paths
Having a URL hierarchy that makes sense is not just important for your website, but also for your API - especially if your API structure resembles your website structure. Try to come up with a sensible URL strategy, discuss it with your colleages and do all of this early in the development process.

A few things to consdier:
* Move from large to small, from generic to specific.
* The user should be able to remove the last part of the URL and arrive at a parent resource.
* Let the hierarchy reflect the UX of navigating the website.
* Try to keep URLs as short as possible.
* Human readable URLs are easier to understand and share, and they are great for SEO.
* [Cool URIs don't change](https://www.w3.org/Provider/Style/URI). Leave out anything that might change, such as author, file name extensions, status or subject.

## Use query parameters correctly
The [URI spec](https://tools.ietf.org/html/rfc3986#section-3.4) tells us to use query parameters only for _non-hierarchical data_.

Don't use query parameters to identify a resource; use a path.

* **Bad**: `example.com/posts?id=123`
* **Good**: `example.com/posts/123`

Use query parameters for _optional_ things like limiting, sorting, filtering and other modifiers:

* **Good**: `example.com/posts?limit=30`
* **Good**: `example.com/posts/123?show_hidden=true`

## Use HTTP methods
Instead of having a bunch of endpoints for various types of actions, use _a single URL for every single resource_ in your application. Distinguish between actions using [HTTP methods](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods).

* **Bad**: `GET example.com/showPost/123`
* **Bad**: `GET example.com/removePost/123`
* **Good**: `GET example.com/posts/123`
* **Good**: `DELETE example.com/posts/123`

There is a big difference between requests that aim to read content, create content or edit content. Make sure to use the GET, POST, PUT and PATCH HTTP methods correctly. The GET and PUT operations are _idempotent_, which means that a request can be repeated multiple times without side effects. This distinction is important, because it tells the client whether it can try again if an error occurs. It also helps with caching, since only GET request should be cacheable.

If you want to offer a form to delete or edit a resource, _that form will be a different resource from the original item_, so it will need a seperate URL. A nice convention is to nest that form resource below the original item. This way, the user just adds `/edit` to the a URL if he wants to edit that resource.

* **Good**: `GET example.com/posts/123/remove`
* **Good**: `GET example.com/posts/123/edit`

## Use HTTP status codes
Pretty much all types of error messages can be categorized in the existing [HTTP status codes](https://developer.mozilla.org/nl/docs/Web/HTTP/Status). These are not just useful to humans, but especially to machines. Status codes can be parsed far more quickly than a body text. Another advantage is that they are standardized, so the client library is likely to know what the status code represents. You don’t have to support every single one, but at the very least make sure that you use the five categories:
* **1xx: informational** - just letting you know
* **2xx: successful** - everything’s OK
* **3xx: redirection** - your content is somewhere else
* **4xx: client error** - you’re doing something wrong
* **5xx: server error** - we’re doing something wrong

## Add context to your JSON
Assuming you use JSON as a serialization format, you can use [@context](https://json-ld.org/spec/FCGS/json-ld/20180607/#the-context). The @context object is a nifty little idea to make your API more self-descriptive. It describes what the various keys in your JSON actually represent. It provides links to where the definition can be found.

Make sure all your IDs are actually links, and your context is included. Now all your JSON has become JSON-LD, which is [linked data](https://ontola.io/blog/what-is-linked-data). That means that your JSON data is now convertible to other RDF formats (Turtle, N3, N-triples, etc.), which means it becomes far more reusable.

Keep in mind that the links that you use should preferably resolve to some document that explains what your concept represents. A good starting point to find relevant concepts is [schema.org](https://schema.org).

## Offer various serialization options
Be as flexible as possible in your serialization options. Someone who uses your API might be good at dealing with JSON, but may not know how to work with XML. Serializing an in-memory object to various serialization options is often not that complicated. If you use Ruby on Rails, check out our [rdf-serializers library](https://github.com/argu-co/rdf-serializers) which supports JSON-LD, RDF/XML, N3, N-triples and Turtle. Use the [aforementioned](#your-api-endpoint-is-your-website) HTTP accept header to handle content negotiation.

## Standardize index pages and pagination
You’re probably going to need index pages with pagination. How to deal with that? Pagination is not a trivial problem, but luckily for you, you’re not the first to encounter it. Don’t try to reinvent the wheel; use something that already exists, such as [W3C activity stream collections](https://www.w3.org/TR/activitystreams-core/#collections) or [Hydra collections](http://www.hydra-cg.com/spec/latest/core/#collections).

## Don’t require an API key
Your default API (the HTML one) doesn’t need one, so your JSON API [shouldn’t need one](https://ruben.verborgh.org/blog/2013/11/29/the-lie-of-the-api/#api-keys-are-a-lie) as well. Use rate limiting to make sure your servers don’t fry. You can still use API keys or authentication to give access to special parts of your API, of course.

## Use a `doc.` subdomain for API docs
Here's [a clever little idea](https://medium.com/@fletcher91/semantic-documentation-1177d563783c): make your API documentation available at `doc.example.com`. If a user wants to know how your api works for a certain page, he just adds `doc.` in front of his current URL. Show the user a page that tells something useful about how to use the API at that route.

## Use your own API
Finally, and perhaps most importantly: eat your own dog food. Make your API a first-class citizen by using it as the only way to access information from that system. API-driven development forces you to make your API actually work. It helps you document your API properly, since your colleagues need to use it as well. Besides, you'll make your application more modular and gradually realize a microservice architecture, which has its own set of benefits.

[Let me know](mailto:joep@argu.co) if I'm missing something, or if you want help with API design!


How do you design great RESTful APIs? This artricle provides do's and dont's for designing an API.

Best practices for RESTful API design

What is linked data, and what makes it useful? In this article, we'll take a piece of information and upgrade its data quality step by step, until it's linked data.

A brief introduction to linked data


## Our democracy and the Internet

Two of our favorite inventions.
Too bad they don't work so well together yet.
There are two problems we need to solve:

### 1. The internet can play a better role in our democracy

There are millions of us living in a country, but all the big decisions are made by small groups in old wooden rooms.
Countless valuable insights and ideas are skipped because of this.
With the Internet, people could share their ideas, think along and give their opinions from all sorts of places.

### 2. Social media are not designed for solving problems

Social media are mostly designed for people to look at ads, not to solve problems together, or to get a nuanced view of something.
Online discussions are often cluttered, terse, and thus not very useful.
Understandably, then, many organizations are afraid to use the Internet.

<BigImage image="cases/argu_abstract_en.png" />

## Our solution

So we set to work to build an online democracy / community platform.
Designed to make it as easy as possible for citizens to participate, and for civil servants to ask questions in the best possible way.
We launched Argu in 2015, and have now completed over 150 participation projects.
We have built in numerous [functionalities](https://argu.co/info/functionaliteiten), such as digital voting, responding, dynamic forms, questionnaires, map views, notifications, rights management, groups, custom domain names, export capabilities, custom theming, and much more.

<BigImage image="cases/argu_body.png" />

## Technology

Argu is by far our biggest project, and a major source of our revenue.
It is a SAAS solution that we have been developing and keeping live for several clients for years.
Developing Argu has taught us to create web applications that are maintainable, flexible and scalable.

Argu's stack consists of several services, two of which are particularly relevant:

### Back-end: Apex

The foundation of our back-end of Argu is built based on [Ruby on Rails](/en/tech/ruby-on-rails).
Given that we had the desire to make the data available as [Linked Data](/enl/blog/what-is-linked-data), we wrote an Open Source library for this: [Linked_Rails](https://github.com/ontola/linked_rails).
This gives Apex a number of unique qualities:

- **Everything is linked data**. Even the buttons, forms, and actions are described as data. This gives us great flexibility, so we can use the same source code for very different use cases.
- **The API is the same as the website**. If you can view a page, you can retrieve the data from that page using the same URL. Actually incredibly logical (see our [article on API design](/blog/api-design)), but oddly enough quite rare.

### Front-end: Libro

Libro is a web application written in [React](/en/tech/react) + [Typescript](/en/tech/typescript).
Libro stands for _linked data browser_.
This is what makes Libro unique: at its core it is not a regular app, but a _data browser_.
Under the hood, libro works with the [Link libraries](https://github.com/rescribet/link-redux), developed by our CTO Thom van Kalkeren.
Libro communicates exclusively with Apex by exchanging linked data (RDF).
We are one of the first SAAS companies to put linked data so far into its stack.
That has given us [some unique challenges](/blog/full-stack-linked-data/), such as a performance bottleneck for which we developed a [new RDF serialization format](https://github.com/ontola/hextuples).

## Created with

<TechWrapperSmall>
  <TechPill technology={'react'} />
  <TechPill technology={'typescript'} />
</TechWrapperSmall>

## Source code available

The Argu project is entirely Open Source and free to use.
Check out our [Gitlab repository](https://gitlab.com/ontola/argu/) for instructions to host an Argu server.


An open source e-democracy and community platform, to help organizations reach their constituencies.

Argu


## Decentralizing the Internet Again

In 1989, Tim Berners-Lee published the first website.
In doing so, he had a vision of a decentralized internet, a web, where people would host their own websites on their own servers.
Now we're more than 30 years on, and the web is more centralized than ever.
A handful of tech giants manage our data, determine what our digital interactions look like, and in what order we read our news.
We need to be able to do that differently.

Tech giants are keeping us on board by connecting our own data.
For example, your Google Calendar works perfectly with your GMail, and you can quickly attach your Google Drive files.
As a tech giant, you have the data to link all of this together, making users dependent on your services.
Fortunately, there is another way to solve this: **standardize**.
With good standards, we can move our data from App to App, so you don't get stuck in the ecosystems of the tech giants.
In addition, by standardizing a lot more, you can do things you can't do now, like search through all your data, see who changed what and when, open the same data in different apps, and more...
But how do we get there?
What do standards need to meet before we can live in this tech utopia?

## From Linked Data to Atomic Data

Tim Berners-Lee gained an important insight 10 years after the first website: the same links we use for websites, we can also use in _data_.
Thus _Linked Data_ was born.
Linked Data makes it possible to use the entire web as a large, connected database.
With that, you could create apps that are completely decentralized, that work much better together, where people themselves have full control over their data.

At least, that's the theory.
In practice, it is still very difficult to build apps with Linked Data.
We can know, because as Ontola we went [all-in](/blog/full-stack-linked-data) on this technology a few years ago.
We rewrote our e-democracy platform [Argu](https://argu.co) and used Linked Data for literally every button and menu.
This was a big job, and we had to create numerous tools to achieve this goal.
From this year-long development process, we learned how to make Linked Data even better - and most importantly, much _easier_.

Our CEO Joep Meindertsma started working on a new form of Linked Data in 2020 in his spare time.
The goal: to find an easier, more powerful abstraction that would allow developers to more quickly create apps that work well together.
Grabbing the best stuff from Linked Data, but improving where possible.
After a few months, there was a [comprehensive documentation](https://docs.atomicdata.dev/) of Atomic Data online, and not much later the first server, libraries and browser.
As a result, Atomic Data quickly became a pretty mature system, with solid implementations and various tools.

## Why Atomic Data

- **Faster Standardization**. Tuning how to shape a particular model is too often a time-consuming task. As a result, there is _too little standardization_, and that limits how well applications can work together. Atomic Data is designed to share data models quickly and easily.
- **Faster development**. Existing tools (such as Atomic-Server) allow you to get started quickly, and you don't have to reinvent the wheel. Storage, searching, versioning, rights management.... It's all already taken care of.
- **Convertible to virtually anything**. You can export Atomic Data to virtually any format: JSON, XML, JSON-LD, RDF, Turtle, N-Triples...
- **Easy to share**. All objects have links, and these links can be looked up - just like websites. The data itself describes where you can find more information. This makes Atomic Data very sharable.
- **Verifiable and correctable**. Atomic Data introduces [Commits](https://docs.atomicdata.dev/commits/intro.html), which stores every change. This creates a history of each object, so you can always determine who did what and when.
- **Authentication and authorization**. Control who has access to which piece of data.

## The best of linked data, JSON and type safety

- **The simplicity of JSON**. Developers know it, and it works everywhere.
- **Type safety**. You can be sure your data has a certain type.
- **Linked data**. Link your data together - even if it's on different machines.

<img src="/images/cases/atomic_venn.svg" />

## What we've built

- [**Atomic-Server**](https://crates.io/crates/atomic-server): a very fast database + server, written in Rust.
- [**Atomic-Data-Browser**](https://github.com/joepio/atomic-data-browser): front-end application for management and display of Atomic Data.
- [**Atomic-CLI**](https://crates.io/crates/atomic-cli): convenient command-line interface tool
- [**Atomic-lib**](https://crates.io/crates/atomic_lib): library for Rust
- [**@tomic/react**](https://www.npmjs.com/package/@tomic/react): library for React, see [template on codesandbox](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx)
- [**@tomic/lib**](https://www.npmjs.com/package/@tomic/lib): library for javascript / typescript
- [**Documentation**](https://docs.atomicdata.dev/): over 80 pages of technical specifications and examples.

## Atomic Server

Atomic Data is a modular specification, which means you can grab bits from it that you find useful, but you don't have to use everything.
If you want to get started quickly, we recommend using our Atomic-Server, which has all the specs implemented.

Atomic-Server is a powerful, extremely fast database + server with an incredible amount of features.
Completely free and open source.
Written in the Rust programming language.
The front-end is written in [Typescript](/en/tech/typescript) with [React](/en/tech/react).

Atomic-Server is designed to be the personal server of the future: a lightweight application that you can easily run at home on a cheap computer, like a Raspberri Pi or an old smartphone.

[Read how to install Atomic-Server here](https://github.com/joepio/atomic-data-rust/).

<video controls src="https://user-images.githubusercontent.com/2183313/139728539-d69b899f-6f9b-44cb-a1b7-bbab68beac0c.mp4"/>


## Created with

<TechWrapperSmall>
  <TechPill technology={'react'} />
  <TechPill technology={'rust'} />
  <TechPill technology={'typescript'} />
</TechWrapperSmall>


An ambitious, fully open source initiative to increase data ownership and make apps work better together.

Atomic Data


## The question

As a political party, your members are your most important tool for growth - they are your campaginers.
An important part of their activities, is going door-to-door.
They need a route, where are they going to ring the bell?
Which areas require most attention, which have the most potential?
And above all - what do they do with the results?
GLAPP was developed for this purpose.

## Our solution

We developed the GLAPP, the Green Left App.
This web application makes it easy for volunteers to join the campaign and for campaign managers to make sure everything goes smooth.

## Processing results

Results of the conversations volunteers have while going door-to-door are entered into the GLAPP.
This updates the potential in that neighborhood, so other volunteers know in which areas they have most impact.
The data is always converted to an average on street level, so no personal data is stored.

## Privacy first

For our client, privacy is a major concern.
That's why we kept privacy in our mind during every step of the development.
Filtering our logs, limiting access to data from certain areas, two factor authentication, aggregating results instead of storing it per address: everything to make sure no personal data is ever stored or shown to and unauthorized user.

## Multilevel dashboards

We created multiple dashboard so different people with different roles can access the data they need.
The data can be filtered and exported to fit all uses.

## Gamification

Doing repetitive tasks can be exhausting.
That's why we made the app not only as easy to use as possible, but also fun to use! Both campaign managers and volunteers are encouraged to continue their efforts by showing cheerful texts and granting badges for their hard work.

<BigImage image="cases/glapp_body.png" />

## Gemaakt met

<TechWrapperSmall>
  <TechPill technology={'react'} />
  <TechPill technology={'typescript'} />
</TechWrapperSmall>


A web application for Groenlinks, modernizing its online and offline campaigns.

The GLAPP


## Addition to online democracy

In 2015, we started [Argu](/en/cases/argu) to give people more influence in our democracy.
With this e-democracy platform, we had developed a new way to have a say, making it easier than ever before to participate in decisions made in your municipality from behind your computer.
But we noticed a problem: the online participation in Argu ran _parralel_ to the formal decision-making within municipalities.
As a result, there was not enough feedback to residents about what happened to their ideas.

We therefore wanted to use data from formal decision-making within municipalities to improve participation in Argu.
We envisioned how we could link council meeting agendas with questions to residents, and how we could share decisions and votes during meetings with residents to keep them optimally informed.

## From Open Council Information to Open Decision Making

We contacted the [Open State Foundation](https://openstate.eu/nl/) (OSF), an organization dedicated to open data in the Netherlands.
Together we lobbied the Lower House to make their data open, and we were the first in the Netherlands to be allowed to work with their data.
Not entirely coincidentally, OSF was then also working with the VNG to make the data of city councils open, in the Open Council Information project.
After OSF made the first links to meeting systems, we were commissioned by the VNG to improve the search engine and scale up the project.
In one year we went from 7 to almost 100 councils connected.
This made Open Council Information one of the largest open data projects in the Netherlands.
Next, OSF was also working with a number of provinces. This project was called Open States Information.
Because we wanted to show both data sources in one interface, in 2019 we chose to keep the whole thing under the name OpenDecisions.
In subsequent years, we made sure that new municipalities were added, and maintained the interfaces with the meeting systems.

## From OpenDecision Making to PLOOI

In 2022, the Open Government Act took effect, which requires municipalities to open up council information, among other things.
The government organization KOOP was commissioned to create software for this, called PLOOI.
It is planned that [PLOOI](https://www.koopoverheid.nl/voor-overheden/rijksoverheid/plooi-platform-open-overheidsinformatie) will soon replace Open Decision Making.

## Technology

OpenDecision Making consists of the following components:

- Extract, Load & Transform (ETL) pipeline, written in Pyhton. This systeem fetches the data from various sources (Notubiz, Parleaus, iBabs and more) and converts it to a standaradised format.
- Search back-end, based on Elasticsearch. This enables full-text search.
- Front-end, gebaseerd op [React](/nl/tech/react). This is the responsive web interface that users see.

Read more about the technology in [our blogpost](/blog/publishing-dutch-governmental-meetings-as-linked-data/).

The code is [open source on Github](https://github.com/ontola/openbesluitvorming).

<video controls src="https://user-images.githubusercontent.com/2183313/208661302-385fb00d-5be2-4c50-8159-591c7805e51c.mov"/>

## Built with

<TechWrapperSmall>
  <TechPill technology={'react'} />
  <TechPill technology={'typescript'} />
  <TechPill technology={'kubernetes'} />
</TechWrapperSmall>


One of the largest open data projects in the Netherlands. Standardized meeting data and API from more than 270 municipalities and provinces.

Software Development

Full-service

Cases