
In this article, we'll discuss an open source, creative commons set of UI designs for a web-based personal data store.

- Click through [the demo](https://www.figma.com/proto/wkc2XEA6Lddai8n4PUU83C/Solidstarter-Prototype?page-id=327%3A0&node-id=335%3A32985&viewport=289%2C241%2C0.04754795879125595&scaling=scale-down) on figma
- View the [style book](https://zeroheight.com/756e7c07f/p/919ed8-solid-ui-kit) on ZeroHeight
- Check out the [Github Repository](https://github.com/ontola/solid-ui-designs), containing the source files

![solid-1](/images/solid-ui/solid-1.jpg)

## Why we need Personal Online Datastores

30 Years ago, the first website was created by Tim Berners-Lee.
The web was designed to be decentralized, where people would have their own servers to share their thoughts on an open internet.
But today, our web has become more centralized than every before.
A few large tech companies control most of our personal data.
Using cloud applications enabled us to access our data from every machine, but it took away a great deal of control.
Moving data from one app to the other is often difficult (or impossible), as many cloud apps have a tight integration between client and server.
For example, we can only view our facebook posts using official Facebook apps.
On the other hand, standards like e-mail, RSS, and HTML allow users to choose how they view and edit their data - we can pick our own e-mail apps, RSS readers and our own web browsers.
Wouldn't it be awesome if we could do the same thing with our social media posts, our chat messages, our todo apps and more?

One approach to solve this problem, is having a _personal online data store_, a Pod.
This is a computer that is always online, and contains your personal data.
It is similar to a service such as Dropbox or Google Drive, but it does a bit more.
Instead of only dealing with files, it deals with _structured standardized data_.
This enables a higher degree of interoperability, which means it becomes easier for app developers to re-use data.
It also hosts your online identity, stores notifications and more.

One notable project that aims to provide such a standard, is Solid.
This project was started by Tim Berners-Lee himself.
At this time, the specification is being formalized and various implementations are being worked on (such as our own [DexPod Server](https://gitlab.com/ontola/dexpod), or the [Solid-Community-Server](https://github.com/solid/community-server)).
The majority of the Solid community is focused on making all of this work, which mostly means getting the technology and the specification in good shape.
Another Pod project is [Atomic Data](https://atomicdata.dev), which is what I've been working on for the past year.
Both of these projects share the same goal: giving people more control over their data.
Because

## Designing a user interface for Pods

How would a Pod look like? How would you interact with it? What kind of UI challenges and opportunities does a Pod present?
We've partnered up with studio [Jager & Prooi](https://jagerenprooi.nl/) to answer these questions and design a user interface for Pods.
Thanks to the [SIDN Fonds](https://www.sidnfonds.nl/nieuws/techneuten-en-ontwerpers-bundelen-krachten-8-projecten-van-start) for funding this project.

We started our project off by interviewing two groups: Solid developers, and potential early adapters.
That way, we'd have a good grasp on technical opportunities and limitations, and what aspects of the UI would be most important for making the first group of users happy.

We realized that designing a Pod would pose an interesting challenge, even for experienced designers.
A Pod is not just one of many apps - it is a place where many apps could run.
It is similar to an Operating System, such as Windows or iOS: where you launch and manage apps, manage your data and manage your system.
This means it needs:

- **a homescreen** from which you can open apps and view activities
- a way to **browse through your files**, like a file manager where you can edit, organize and delete things
- a tool to **search** through your data
- a **settings app** to manage the hardware and UI
- a **notification tray** or a different way to view incoming changes and alerts
- **persistent UI elements** to quickly open search / notifications / settings
- a **way to open files with various different apps** (e.g. open a PDF in Adobe or a Browser).

These are the things that we've learned to expect from operating systems - these are also present in Windows, iOS and others.
But Pods are different for two reasons:

1. Pods are _always_ connected to the internet, which means that they can be places that others can find information
1. Pods use _structured data_ instead of _binary data_ (regular files on your computer), which means that you get new features to manage information

These differences result in new things that you might want a Pod operating system to achieve:

- **Manage external access to data**. In a Pod, you can allow other to access files without actually sending it to them. Your Pod is a server, so it's connected to the internet, which means that (if you want) others can collaborate on or view your data, too. This needs to be managed, which is something that is not actually common in other operating systems.
- **Version control / history control**. Storing structured data makes it easier to actually save various version, which means that user can undo their mistakes and see who changed what over time.
- **Structured Search**. In a regular operating system, we could search for filenames, but in a Pod, we can search through the data itself. This means that we can use the global search feature to find Todo's, appointments, playlists and more.
- **Manage personal information**. Since Pods are inherently _personal_, they are great for storing data about yourself. This includes data that you'd share publicly on your linkedin profile, or medical data that you'd want to keep private, but do want to have managed somewhere.

To keep the scope of this project manageable, we decided to focus on Home, Personal Information Management, Sharing, File Exploring and Search.

![personal data management](/images/solid-ui/solid-2.jpg)

**[Click through the demo](https://www.figma.com/proto/wkc2XEA6Lddai8n4PUU83C/Solidstarter-Prototype?page-id=327%3A0&node-id=335%3A32985&viewport=289%2C241%2C0.04754795879125595&scaling=scale-down)** to see how it turned out!
And feel free to use these designs in your own apps - we're sharing the [source files on Github](https://github.com/ontola/solid-ui-designs) under a CC-BY license for specifically that reason.


In this article, we'll discuss an open source, creative commons set of UI designs for a web-based personal data store.

Designing a UI for a Personal Online Datastore


In this article, we'll discuss how linked data can make surveys more easy to manage, increase response rates using rewards, create new benefits for users and more.

- Why it's important to improve survey interoperability
- How we could increase interoperability using linked data
- How we can use rewards to increase respondent rates for surveys

## Why we need Survey interoperability

Surveys are a pretty interesting and complex type of data model.
You can think of a Survey as describing three concepts at once:

1. a data schema / model (datatypes, required fields, etc.)
1. a UI (the questions show to the user)
1. and a customer journey (the sequence of questions, sometimes with conditional paths).

Due to this complexity, it should not be a big surprise that there are many approaches to creating a Survey, and this has contributed to the low degree of interoperability of surveys.

That's a shame, because having interoperable Surveys would be _really cool_!

- It would be easier for researchers to **standardize how questions are asked**. For example, many medical surveys are highly standardized, up to how specific words can be used. You can probably imagine how a small change in phrasing a question or answer could have a serious impact on average results. If we standardize surveys and their questions on a data level, researchers could simply import an existing survey, instead of going through the time consuming process of manually creating every single field in their survey software.
- Respondents could have more **freedom in chooing a UI**. Not all user interfaces are easily accessible, pretty, or fun to use. People can have vastly different preferences regarding how a survey should be filled in. If we standardize how Survey Data is structured, we can let respondents pick their own front-end, similar to how you can open a webpage in any webbrowser that you like.
- We could **auto-fill** data from a personal source, which makes entering data less tedious. Browsers already do this for addresses and credit cards, but we can extend this _way_ further by using links! We'll get to that later
- We can **improve insights in ourselves** by storing and then analyzing data that we filled in in a survey. For example, filling in a survey about how your shortness of breath linked to air pollution has been today could be used in a different app to make a graph that visualizes how your shortness of breath has progressed over the months for personal insight.

Achieving these last two steps requires a high degree of standardization in both the surveys and the responses. The survey and its questions should provide information about:

- The **question** itself. This is required in all survey questions, of course.
- The **required datatype** of the response, such as 'string', or 'datetime' or some 'enumeration'.
- A (link to a) **semantic definition** of the property being described. This is a bit more obscure: all pieces of linked data use links, instead of keys, to describe the relation between some resource and its property. For example, a normal resource might have a 'birthdate', while in linked data, we'd use 'https://schema.org/birthDate'. This semantic definition makes things easier to share, because it prevents misinterpretation. Links remove ambiguity.
- A **query description**. This is even more obscure, but perhaps the most interesting. A query description means describing how a piece of information can be retrieved. Perhaps a question in a survey will want to know what your payment pointer is. If a piece of software wants to auto-fill this field, it needs to know where it can find your payment pointer.

## Making surveys interoperable

A good place to start standardizing, is to use _links_ everywhere. Use links for Properties, use links as identifiers for Surveys, for Responses, for Questions...
Links are great, because they can be _opened_.
Read my article on [Linked Data](https://ontola.io/blog/what-is-linked-data) if this is new to you.

Okay, so we standardize the Questions, Surveys and Responses using links. But which links? And how?
At Ontola, we've been building dynamic RDF based forms for our [Argu.co](https://argu.co) e-democracy application.
We started out by using the [W3C SHACL specification](https://www.w3.org/TR/shacl/), which is a linked data standard for describing data models.
It turned out to be a great first step, as it ticked the first box of concepts in surveys: it describes the _schema_.
But we still have two concepts left to define: the _UI_ and the _Journey_.
For example, the SHACL spec does not provide UI concepts for describing whether to use a _radio select_ or a _dropdown_, or how to render a _helper text_.
And the _Journey_ should include conditional paths, which is also not possible.
These conditions are also important when the _rights_ of a user determines whether that user can edit a certain field.
We could either calculate all the form fields dynamically, or we could let the front-end use conditional fields to determine which fields are shown.
We opted for the latter, as this resulted in better performance.

So we're currently working on a Form ontology which adds things like `Pages`, `Groups`, `helper text` and `Conditions`.
As this is still prone to changes and under-specified, we'd rather not share it in its current form, but we do aim to do this in the near future.
An open-source implementation of our Survey software can be found on [Gitlab](https://gitlab.com/ontola/linked-surveys).

## Rewarding responses

Filling in a survey isn't the most fun thing to do, so getting people to fill in a survey can be a challenge.
That's why it's pretty common these days to see surveys that promise a chance to win rewards like nice gadgets (iPads, mostly) or coupons.
The first problem with this approach is that the one filling in the survey doesn't have a clear estimate of what their chances of winning are, which makes the deal a bit less attractive.
The second problem is that it requires the respondent to provide privacy sensitive details, such as address.
A third problem is that not everybody wants an iPad - money is often a bit more attractive.
Ideally, we'd provide a clear and transparent reward, and do so in an anonymous fashion.

A couple of months ago, we discovered the Grant for the Web, a grant that aims to change how we do payments on the internet.
We submitted a proposal to implement a solution for the problems stated above: we wanted to build a service that sends a payment to people who completed a survey.
Because we wanted to provide a high degree of privacy, we wanted our own services to be completely unaware of the payment address of the receiver.
We also wanted to make this project re-usable for other usages (other survey tools, or totally different things).
We therefore up building an open source service, called [**cashlink**](https://gitlab.com/ontola/cashlink).
It works like this:

- Create unique links (hence _cashlink_) which encode the value of the link. These can be used only once.
- Share this unique URL in some application, such as at the end of a survey.
- When the user opens this URL, they can submit their own _Payment Pointer_, which is an open specification for payment addresses built for the web.
- After submitting this, the payment is sent by the cashlink service.

## In conclusion

- Having a higher degree of standardization can offer various benefits: more standardized research, auto fill forms, UI freedom, and get more insights in our own personal data
- Linked Data can be used to facilitate this standardization process, but we need more (well-documented) standards to do this.
- Payments can be an effective tool for getting more people to fill in surveys.


In this article, we'll discuss how linked data can make surveys more easy to manage, increase response rates using rewards, create new benefits for users and more.

On linked data surveys, rewards and data ownership


EDIT: This tutorial is outdated. We no longer support the `link-lib` library. We recommend checking out [Atomic Data](/cases/atomic-data) and [@tomic/react](https://codesandbox.io/s/atomic-data-react-template-4y9qu) instead.

This tutorial will help you build a linked data front-end application, compatible with Tim Berners-Lee's Solid initiative.
Knowledge of React and Typescript / Javascript is assumed, as is a little knowledge of RDF.
For an introduction to RDF, read [this](https://ontola.io/blog/what-is-linked-data/).

Working with RDF (linked data) can be a bit different from using more conventional JSON REST APIs.
We need tooling to deal with some of the complexities that arise when working with linked data:

- The client (the browser app) does not know in advance what kind of data it will get back from a server. Link will simply fetch a URL using content negotiation, and get some [RDF serialized](https://ontola.io/blog/rdf-serialization-formats/) response.
- Deciding which view to render is often done by checking routes, but this is not possible in a linked data app. Instead, we can fetch the URL and check which `rdf:type` is being set (e.g. `Person` or `Blogpost`).
- Many strings are URLs, which can be a bother to type and might cause typo-related-bugs. To deal with this, we [set URLs programmatically using @ontologies/core](https://ontola.io/blog/ontologies-in-js/).
- [Ordered data in RDF](https://ontola.io/blog/ordered-data-in-rdf/) is a bit more complicated than JSON Arrays, so we'll use tooling to abstract these to arrays.

In this tutorial we're going to use `react` and the [`link-lib`](https://github.com/fletcher91/link-lib/) + [`link-redux`](https://github.com/fletcher91/link-redux) (in short: Link) libraries to create a simple front-end app to render data from a Solid Pod.
You can see the [final result on CodeSandbox](https://codesandbox.io/s/github/ontola/link-solid-boilerplate/tree/master/?file=/src/app.tsx)

## Understanding Link: Store, Views and Topologies

The Link libraries have an interesting, but unconventional way of rendering data.
The first concept to understand, is the LinkedRenderStore (the Store).
The **Store** is responsible for most of the heavy lifting: it contains the stored data, keeps rendered data in sync, parses changes, and stores registered Views.
**Views** are (in this case react) components that render one or more `rdf:Class`es in one or more Topologies.
**Topologies** are contexts in the DOM, such as `Page` or `Card` or `Detail`.

Let's say you want to view your public Solid Pod folder, such as `https://joep.inrupt.net/public/`, as a full page.
First, you pass the subject (the URL of the resource) to a Link `Resource` component.
This will let the LRS check if that resource is already present in the Store.
If not, it will be fetched.
Once it has been fetched and parsed, the resource will be added to the Store.
Then, the Resource component will be notified of new data, and then the Store will try to find the _most suitable View_.
This is done by checking the `rdf:type` of the Resource and the current Topology.
In this case, that type is an `ldp:Container`, and the topology is `fullPage`.
If there is a `fullPage` View registered in the Store for `ldp:Container`, it will render that View.
If there is not such a View, it might try a more generic view.

## Setting up the basics

First, we'll need to set up a react project.
You can use our boilerplate, or you can set it up manually.

### Easy mode: Using the link-solid-boilerplate

You can get started right away using the [`link-solid-boilerplate` on CodeSandbox](https://codesandbox.io/s/github/ontola/link-solid-boilerplate/tree/master/?file=/src/app.tsx).
Alternatively, you can use the [Template feature on Github to fork it](https://github.com/ontola/link-solid-boilerplate/generate), and follow the readme to get it running locally.
Got it running?
Skip the next section and go to [Loading Data](#loading-data).

### Hard mode: Without using the boilerplate

_(skip this if you're using the boilerplate)_

1. Make sure `node` and `npm` are installed.
1. Start a new react project: `npx create-react-app my-app --template typescript`.
1. Add the type definitions: `yarn add @types/node @types/react @types/react-dom @types/jest `.
1. Add `link-lib` and `link-redux` as dependencies: `yarn add link-lib link-redux @rdfdev/delta @rdfdev/actions @rdfdev/iri @rdfdev/collections`
1. Add their `peerDepdencies`: `yarn add @ontologies/as @ontologies/core @ontologies/schema @ontologies/shacl @ontologies/xsd http-status-codes n-quads-parser`
1. [Set-up the Linked Render Store](https://github.com/fletcher91/link-redux#1-set-up-the-store).
1. Got stuck? Check the [boilerplate](https://github.com/ontola/link-solid-boilerplate) for inspiration and use the [link-redux wiki](https://github.com/fletcher91/link-redux/wiki).
1. Continue with the rest of this tutorial.

## Loading data

As a data source, we could use multiple back-ends. Here's a list of tested and open source options:

- A Solid pod hosted on [Inrupt.net](https://inrupt.net/)
- A [DexPod](https://gitlab.com/ontola/dexpod/), hosted on [dexpods.eu](https://dexpods.eu).
- [Atomic-Server](https://github.com/joepio/atomic), which also serializes data to RDF formats.
- Any other public RDF server that supports `n-triples` and has permissive `cors` headers.

But for this tutorial, we'll stick with the Inrupt.net pod.
We'll try to render `https://joep.inrupt.net/public/`, which is an `ldp:Container`.
That Container, as its name suggests, contains a bunch of resources, similar to a folder.

## Registering a View

Create the following file `views/Container.tsx` with a functional React component.
All we need is a single property

```tsx
import { Resource, useProperty } from "link-redux";
import React from 'react';
import ldp from '../../ontology/ldp'

const Container = () => {
  // This hook allows you to get any property / value combination from the resource.
  // The ldp.contains essentially contains a URL for the predicate.
  const contains = useProperty(ldp.contains);

  return (
    <>
      {contains.map(member => (
        // This renders a nested Resource.
        // Each subject will be fetcher by Link
        <Resource
          key={member.value}
          subject={member}
        />
      ))}
    </>
  );
};

// The Type attribute dictates for which RDF Classes
Container.type = ldp.Container;

export default Container;
```

Next, go to `views/index.ts` and add the newly created Container component to the `getViews` list:

```ts
const getViews = (): RegistrableComponent<any>[] => [
  // here!
  Container,
  ErrorResource,
  LoadingResource,
  // ....
]
```

This will make sure that when the app initializes, the `Container` component will be registered to the store.
That's it! The component should now render.
However, you won't see much.
We also want to render the `members` from above.
But these should not be rendered on a full page.
They need a different _topology_.

So let's create one for the current context.
We'll call this one `topologies/BrowserList.ts`, as it's a list in the file browser.

```ts
import { TopologyProvider } from 'link-redux'

import { appNS } from '../helpers/app'

// This is the same as `new NamedNode("myappurl/browserlist")`.
// Export this, you'll use it later!
export const browserListTopology = appNS('browserList');

// The link-redux TopologyProvider deals with most of the complexity
class BrowserList extends TopologyProvider {
  constructor(props: any) {
    super(props);

    // Create a NamedNode for your Topology, and bind it to this.topology
    this.topology = browserListTopology;
    // This determines the HTML element type (optional)
    this.elementType = 'ul';
    // The CSS Classname (optional)
    this.className = 'BrowserList';
  }
}

export default BrowserList;
```

Great! Now we can use this BrowserList topology in our Container component.

```tsx
import { Resource, useProperty } from "link-redux";
import React from 'react';

import ldp from '../../ontology/ldp'
import BrowserList from '../../topologies/BrowserList'

const Container = () => {
  const contains = useProperty(ldp.contains);

  return (
    // We've wrapped our new Resources in our BrowserList topology
    <BrowserList>
      {contains.map(member => (
        <Resource
          key={member.value}
          subject={member}
        />
      ))}
    </BrowserList>
  );
};

// The Type attribute dictates for which RDF Classes
Container.type = ldp.Container;

export default Container;
```

Now we need to register a new View for our `members` and our `BrowserList` toplogy.
We'll create a highly _generic_ view, for all sorts of things.
Let's call it `ThingBrowserList`.
It's good practice to use the `ConceptTopology` naming convention.

```tsx
import dcterms from '@ontologies/dcterms'
import { Literal, NamedNode } from "@ontologies/core";
import foaf from '@ontologies/foaf'
import rdfs from '@ontologies/rdfs'
import schema from '@ontologies/schema'
import { FC, Property, useProperty } from "link-redux";
import React from 'react';

import BrowserListItem from '../../components/BrowserListItem'
import { browserListTopology } from '../../topologies/BrowserList'
import { filename } from '@rdfdev/iri';

export interface Props {
  name: Literal;
}

// We can use multiple predicates.
// Link will select the first one that matches.
// This is especially useful in generic kinds of views, such as this one.
const namePredicates = [
  schema.name,
  rdfs.label,
  dcterms.title,
  foaf.name,
]

const ThingBrowserList: FC<Props> = ({
  subject,
}) => {
  const [name] = useProperty(namePredicates)
  const [modified] = useProperty(dcterms.modified);
  // When using a value from usePropety, you'll often want to use the internal raw string value using `.value`
  const displayName = name?.value || filename(subject as NamedNode);

  return (
    <BrowserListItem
      name={displayName}
      title={subject.value}
      to={subject}
    >
      {modified.value}
    </BrowserListItem>
  );
};

ThingBrowserList.type = rdfs.Resource;
// This is where we let this View know that it should be rendered in the BrowserList topology!
ThingBrowserList.topology = browserListTopology;

export default ThingBrowserList;
```

And for the final step, we'll register this View in `views/index.ts` like before:

```ts
const getViews = (): RegistrableComponent<any>[] => [
  Container,
  ThingBrowserList,
  // ....
]
```

Now, open the app and enter your public Solid Pod URL (e.g. `https://joep.inrupt.net/public/`).

## Got stuck?

- Check the [**result on codesandbox**](https://codesandbox.io/s/github/ontola/link-solid-boilerplate/tree/master/?file=/src/app.tsx)
- If you need dev tools In your browser console, use the [link dev tools](https://github.com/ontola/link-devtools): `dev.help`.
- If you encounter a problem, please [create an issue in the Issue tracker on github](https://github.com/ontola/link-solid-boilerplate/issues).
- Check out the [link-redux wiki](https://github.com/fletcher91/link-redux/wiki) for many useful hooks and other features for rendering RDF!


You can use Link to build a React application that consumes linked data. We'll help you step-by-step.

Tutorial: Building a React front-end app with RDF Data, powered by Link and Solid


How might a web application work if it exclusively uses [Linked Data](https://ontola.io/blog/what-is-linked-data/) (RDF) to communicate between server and client?
In this article, I'll tell you about our API journey, why we chose linked data (RDF), the challenges that we faced, and some of the solutions that we came up with.

## About the project

In 2016, we started [Argu.co](https://argu.co): an e-democracy platform.
We wanted more people to become politically engaged, and make sure that their opinions and ideas were heard by decision makers.
We needed a web app where people would discuss issues, vote, share ideas and make decisions together.
We started with Ruby on Rails, and built a traditional server-side monolithic HTML serving app.
This proved to be a quick and effective way to start, but as time passed, we noticed more and more **limitations of this server-side approach**:

- The HTML pages relied on the internal ORM, so we didn't need an API for internal use. Having **no API** limits what customers can do with their data, and it also means that it's gonna be hard (or impossible) to build a native mobile App.
- Having **interactivity** (menu's that pop up, rich text editors, notifications, live editing, etc.) was difficult to achieve with server-side rendering. We liked react, but using that in a server-rendered Rails app was complex and error-prone.
- **Performance** was not as good as we wanted it to be, as most clicks required some server processing and an HTTP roundtrip.

It was clear that we needed a dedicated client-side single page application and an API.

## Designing a client-server contract (API)

Re-inventing the wheel seems like a bad idea, so we searched for best practices and existing standards in API design.
We initially went with [JSON-API](https://jsonapi.org/) (not just 'a' JSON API, but a formal specification that describes things like pagination).
Since we were already using React, and saw its popularity climbing, we went with that for the front-end.
Now we had to decide how the internal state of the client should look like.
We picked Redux, which uses a single (immutable) JS object which contains all application state (including data).

As we worked on this, our front-end started looking more and more like our back-end.
It now had explicit knowledge of:

- Many of the models from the back-end (Comment / Question / Vote / User)
- The routing logic
- The accessible parameters in endpoints
- Various actions (e.g. notifications / alerts)
- Forms and their validations.

This seemed _wrong_.
We were taught to write DRY code: _don't repeat yourself_, but it seemed like we were repeating ourselves all the time.
This meant that when we wanted some feature changed, we had to adjust _both_ the front and the back-end.
Something seemed off.

## Why we opted for Linked Data

Our CTO Thom van Kalkeren thoroughly studied the principles behind REST, HATEOS and Hypermedia, and concluded that we need to give _everything_ URLs.
Not just the _pages_ that we present to our users, but _every single thing_ that can be used by a client - including menu items, buttons, actions and form fields.
Luckily for us, the good folks at W3C had written quite a bit of specs about how data should look when you use URLs: the RDF specification (and friends).

Linked Data (or RDF) has some unique qualities:

- **Browsable**. Using links for everything in data, means that data can be surfed like how web pages work: just follow the URLs. That's actually really helpful in a web application, since this means that the client no longer needs to be aware of your routing logic.
- **Flexibility**. Everything can be serialized to various formats (JSON, XML, Turtle, N-Triples, and [more](https://ontola.io/blog/rdf-serialization-formats/)), which gives some really nice and flexible export functionality.
- **Self-describing APIs**. Simply use content-type negotiation to fetch a resource as HTML or some RDF format, and browse the data like you would browse a website: by following the links. API docs become kind of unnecessary, since navigating the website shows you all the endpoints.
- **Re-use other linked data**. Because links can point to _anywhere_ (not just your server), you can use all the publicly available linked data! This, for me, has always been the number one reason to believe in Linked Data.
- **Enables true data ownership**. Because of this, it enables _decentralized networks_ where people _own their data_, which can help to combat the existing web oligopoly. That's what the [Solid project](/solid) is all about.

So, we went with Linked Data.

## The cost of being an early adopter

Although the first [RDF spec](https://www.w3.org/TR/rdf11-concepts/) was already 10 years old, there were not many existing libraries (let alone tutorials) for us to get started.
Practically all Linked Data projects functioned as viewers that directly show individual RDF Statements in a table.
We wanted something else, we wanted a user-friendly web app that resembled social media platforms.
We needed interactivity, notifications, forms...
And doing things like that with Linked Data, turned out to be pretty hard.
Luckily for you, we've shared quite a bit of tools, libraries and ideas that could make your life a bit easier if you choose to go Linked Data.

## Getting used to RDF

RDF is a weird data model.
It was primarily designed as a language for the _semantic web_, which aims to (formally, logically) describe our world using _semantic triples_.
The design decisions made to achieve that, however, make things harder for you as a developer:

- RDF uses a lot of very long strings (URLs) instead of keys, which makes selecting values difficult.
- RDF does not have key uniqueness (subject-predicate uniqueness), which means that you could get many values if you want to know a specific property.
- RDF doesn't support native arrays, but has [a bunch of (confusing) ways of representing ordered data](https://ontola.io/blog/ordered-data-in-rdf/).
- RDF knows [many serialization formats](https://ontola.io/blog/rdf-serialization-formats/), which is a blessing and a curse. The most mentioned format in the docs (the earliest one, RDF/XML), is also the most confusing.
- Properties are not linked to specific datatypes, so you have to check datatypes for every single statement.

## Making Ruby on Rails work with RDF

Ruby on Rails is an opinionated MVC framework.
By default, it works great for generating HTML pages (or JSON objects, using Rails-API), but RDF is a whole different beast.
We created our own serializers ([rdf-serializers](https://github.com/ontola/rdf-serializers)), but more importantly, Thom and Arthur Dingemans created [linked-rails](https://github.com/ontola/linked_rails/wiki).
This gem provides some abstractions for working with RDF, including:

- Serialization (including errors)
- Policies (what users can or can't do)
- Collections (sort / filter / pagination)
- Forms (we'll get to that later)
- Menus for actions and navigation
- Actions handler (we'll get to that later, too)

Check out the [Wiki](https://github.com/ontola/linked_rails/wiki) if you want to learn more!

## Forms and form validation

Argu was always about getting opinions from people, so it needs a lot of forms.
And these can differ for many reasons:

- The model (of course)
- Specific settings for a tenant (customer)
- User and tenant-specific rights

So, how should the server communicate these forms to the client?

We stumbled upon [SHACL](https://www.w3.org/TR/shacl/), a schema language for RDF (written in RDF), that provides a way to validate RDF graphs.
This seemed like the way to go!
However, it became obvious that forms cannot _entirely_ depend on shapes.
Should you render a radio select, or a dropdown?
How do you describe a multi-page form?
How do you describe interrelated form fields (if x is a, show form option y).
And to make things even more challenging: how do you make these forms _cacheable_, given some fields might render differently for different users?

Again, we had to create our own form abstraction.
We used the SHACL spec where possible, but had to create some new Classes and Properties to solve the challenges that we faced.
The Forms DSL in [linked-rails](https://github.com/ontola/linked_rails/wiki#forms) provides a simple interface to create these forms.

## Turning RDF into HTML

Ultimately, the user needs to show a pretty web app, which means the RDF from the server has to be transformed into HTML.
Step one is fetching the data from the server, and storing it in the client.

We started using [RDFlib.js](https://github.com/linkeddata/rdflib.js), written partially by the inventor of Linked Data (and HTTP, HTML, the WWW...) Tim Berners-Lee himself.
It provided a lot of useful features for dealing with RDF: parse, fetch, serialize, mutate, search...
However, our app had kind of a unique set of requirements (mostly related to performance when dealing with many, dynamic triples), so we started working on a different RDF store.

My colleague Thom wrote two libraries: [link-lib](https://github.com/fletcher91/link-lib) and [link-redux](https://github.com/fletcher91/link-redux).
Link-lib does most of the heavy lifting: it provides a store, deals with parsing, serializing, fetching, sending RDF data, handling the actions, registering the views (more an that in a minute).
Link-redux (should be called link-react, but that name was taken) adds some useful functions for using it in React.

In practice, it works like this:

- When the user requests any URL as HTML (default browser request), they receive a mostly empty HTML page with some cached RDF data (that matches the Subject of the fetched URL) that helps to build the first view as quick as possible.
- The JS app (using Link-lib, link-redux and react) is loaded, and all the existing Views are registered. These Views are React components that will render for one or more RDF Classes, in specific Topologies (e.g. fullpage, in a column, in a menu item).
- The RDF data is added to the store, and the corresponding view is rendered.
- Note that there is no routing going on in the front-end - the rendered component is based on the Class of the resource, not the URL route.
- Depending on the rendered view, new resources might be needed (e.g. a rendered Article might show some Comments). These will be fetched by the front-end, and loading indicators will be shown in the places where data is missing.
- When the user interacts with the content (e.g. click on a link, open a menu item, press a button), they will dispatch an action and modify the internal state - all using RDF. These actions _might_ make changes on the server, and these changes _might_ have an impact on the user after that. The changes are communicated using [linked-deltas](https://github.com/ontola/linked-delta), which describes how single triples have been either changed, removed or added. When a delta is processed, components might re-render, because their properties have changed. This is a flux-like / redux-like pattern.
- Because we a _lot_ of URLs, we need some way to prevent typos. That's where the [@ontologies](https://github.com/ontola/ontologies) library comes in handy!

## Serialization & parsing performance

RDF can be serialized in many ways ([at least 10](https://ontola.io/blog/rdf-serialization-formats/)).
Some formats look like the ones you know (JSON-LD, RDF/XML), some reflect the internal model really well (N-Triples), and some have interesting and complex semantic features (Notation3).
We didn't really care about any of that - we just wanted to make things fast.
Parsing JSON-LD was _extremely_ hard and slow, Parsing Turtle was better (but still slow), so for a while we opted for the with `N-Triples`.
But ultimately, Thom came up with a new format: [HexTuples-NDJSON](https://github.com/ontola/hextuples), which was actually very simple to implement both in the Ruby back-end and in the JS front-end.
It made our _entire_ server app about 2x faster (it was working a _lot_ on serialization), and in the front-end we noticed a similar speed increase.

## Bulk-API, caching and more performance improvements

Still, our app wasn't quick enough.
The core problem: every single resource on screen with has a URL, and needs to be fetched independently.
In other words, opening a single page could require over a 100 round-trips.
Thom came up with a solution: the bulk-API, which would accept a body containing all the URLs that needed to be fetched.
This improved many things, but our back-end was still running each resource request internally with all the existing overhead (checking the user, session, etc.).
Even when that is fixed, we're still performing way too much hard work for every request - we need some form of caching.

That's why were working on a new RDF Triple Strore, written in Rust, and it is _fast_.
It offers three ways of querying:

- Single resource (HTTP GET to the subject URL)
- Bulk-API (list of subjects)
- Triple Pattern Fragments

It's not rolled out yet in production, but we'll open source it soon enough.

## FAQ

### Which triple store do you use?

Although our back-end serializes linked data, we don't use a triple store.
The Rails back-end has a mostly strict schema in SQL tables, but we do have a table in our Postgres schema that stores single triples.
Keep in mind that the RDF representation of data can be created during serialization - so your app can create linked data without using a triple store!
This is often a preferable approach in apps that have business logic or any type constraints.
But as I've mentioned earlier, we're working on a new triple store (written in Rust) that we're planning on open sourcing soon, which we use for caching to keep performance optimal.

### Why not use SPARQL?

SPARQL is the de facto query language for RDF data, so it seems logical to use it somewhere in our stack.
However, we don't.
Getting SPARQL performant is actually pretty difficult, and we don't need the powerful query options that it provides.
Most of the requests from the front-end just ask for all triples about one or multiple subjects, and these kind of queries don't require SPARQL.
SPARQL is useful for more complex graph property traversal queries, but is not necessarily the best approach for simpler queries.

### RDF tools that we've built

- [link-lib](https://github.com/fletcher91/link-lib) JS lib for managing an RDF store, registering views and handling actions
- [link-redux](https://github.com/fletcher91/link-redux) JS lib for React components, works with link-lib
- [linked-rails](https://github.com/ontola/linked_rails) Ruby gem that does a whole lot of linked data goodness
- [rdf-serializers](https://github.com/ontola/rdf-serializers) Ruby gem that for serializing RDF
- [linked-delta](https://github.com/ontola/linked-delta) for communicating state changes (between server / client, or persisting as event log)
- [@ontologies](https://github.com/ontola/ontologies) - make it easier to use ontologies (schema, owl, dcterms..) in JS projects
- [js.rdf.dev](https://js.rdf.dev/) A whole lot of RDF tools for JS!
- Check out the rest at [Github](https://github.com/ontola/).

If you need any help, [get in touch](mailto:joep@ontola.io)!


We went all-in on RDF, and build an application that's linked data through-and-through. It was a challenging ride, and it led us to some interesting solutions.

Full-stack linked data: lessons from building an RDF web app


Sooner or later when working with RDF, you'll need to work with ordered data / n-ary relations.
The `subject predicate object` model does not support Arrays, and no - you can't use the order in which triples appear.

However, RDF _does_ support _Collections_ (`rdf:List`) and _Containers_ (`rdf:Bag`, `rdf:Seq`, `rdf:Alt`).
And the open nature of RDF allows for even more alternatives, such as the [Ordered List Ontology](http://smiy.sourceforge.net/olo/spec/orderedlistontology.html).
All these concepts are (subtly) different, and can be quite confusing.
In this article, I'll explain the models behind these concepts, show how they are serialized, and give some insights into when you should use which.

Skip to the bottom for a TL;DR!

## RDF Collections

Maybe you've seen something like this in JSON-LD:

```json
{
  "@list": [ "Arnold", "Bob", "Catherine" ]
}
````

Or something like this in a Turtle document:

```
someList :b ( "Arnold" "Bob" "Catherine")
```

These are `rdf:Collection`s.
These serialization formats (turtle and JSON-LD) have syntactic sugar for Collections, so they appear as regular arrays.
Under the hood, however, they have a very different data model.
Collections are [linked lists](https://en.wikipedia.org/wiki/Linked_list) and its chains consist of `rdf:List` nodes, connected by `rdf:rest` relations:

![RDF:List](/images/posts/ordered/rdflist_basic.png)

Every `rdf:List` has a `rdf:first` object, which refers to the actual content of the list item.
Intuitively, it might be a bit weird to refer to Bob as a `first` in a List.
However, since Lists are recursive and often contain other Lists, it actually makes sense to refer to the content as `first`.
Collections always end with an `rdf:nil` namednode, which means they have a _formally known ending_.
If we'd express the same information in N-Triples, we'd get something like this:

```
_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "Arnold" .
_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> _:b1 .
_:b1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "Bob" .
_:b1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> _:b2 .
_:b2 <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "Catherine" .
_:b2 <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> <http://www.w3.org/1999/02/22-rdf-syntax-ns#nil> .
```

This explicitly linked nature means that it's possible to start a list in location A, and have it continue in a completely different domain:

![RDF:List](/images/posts/ordered/rdflist_external.png)

It also means that inserting items is quite easy, as only the node before it has to be adjusted:

![RDF:List](/images/posts/ordered/rdflist_insert.png)

However... This structure can be kind of hard to deal with.
Parsing is non-trivial, and might require a lot of lookups, which is very costly.
Also, appending a single node to a List is hard, because it requires that the link of the item before it is adjusted.

## RDF Containers

There are three types of RDF Containers `rdf:Seq`, `rdf:Bag`, `rdf:Alt`

- An `rdf:Seq` is an _ordered_ container
- An `rdf:Bag` is an _unordered_ container
- An `rdf:Alt` is an _unordered_ set of alternatives, in which the first one is the default option

RDF Containers use a numbered predicate (e.g. `rdf:_1`) to indicate that something is a child / member of the Container:

![RDF:Seq](/images/posts/ordered/rdfseq.png)

This model is often easier to parse and serialize than an `rdf:List`.

```
_:someSeq <http://www.w3.org/1999/02/22-rdf-syntax-ns#_1> "Arnold" .
_:someSeq <http://www.w3.org/1999/02/22-rdf-syntax-ns#_2> "Bob" .
_:someSeq <http://www.w3.org/1999/02/22-rdf-syntax-ns#_3> "Catherine" .
```

Appending items is easy as well, simply add one new statement and increment the predicate.
However, inserting items requires you to rewrite many triples.
If you use a reasoner that has implemented the basic RDFS spec, you can use [`rdfs:member`](https://www.w3.org/TR/rdf-schema/#ch_member) (the superclass of every `rdf:_n` property) to get all members of an RDF Container.

## Pagination with ActivityStreams and Hydra Collections

When your arrays will be either too long to serialize, or too computationally heavy to generate at run-time, you'll need some form of pagination.
It's recommended to use an existing ontology for this.
The W3C Activity Streams 2.0 ontology defines the [`as:Collection`](https://www.w3.org/TR/activitystreams-core/#collection).
It provides keys for things like `next`, `totalItems` and `orderedItems`.
This spec seems to be designed with `JSON-LD` serialization in mind, so it relies on JSON arrays, which represent `rdf:List` Collections.

Similar to this is the [`hydra:Collection`](https://www.hydra-cg.com/spec/latest/core/#collections).
The Hyrda spec still is in draft status, but it has been updated in october last year.

## Converting to Arrays

If you're building an RDF application, and you want to access your RDF Lists / Collections / Sequences as an array, make sure to find an RDF library that has support for ordered data.
We've open-sourced a JS library that might be useful for this: [`@rdfdev/collections`](https://js.rdf.dev/modules/_rdfdev_collections).
[Get in touch](mailto:joep@ontola.io) if we can help!

## TL;DR

There are no arrays in RDF, and don't use the order in which serialized triples appear.

**RDF Containers:**
- Come in three forms: `rdf:Seq` (ordered), `rdf:Bag` (unordered), `rdf:Alt` (alternatives with default)
- You can add new items by simply adding RDF triples
- Inserting items is hard: requires rewriting *many* statements
- Must be stored in a single graph / machine / server (centralized)
- Have a formally unknown ending (open world assumption)

**RDF Collections:**
- An ordered chain of `rdf:List` resources
- You have to edit / remove statements before you can add new items
- Inserting items is easy: requires changing *just a few* statements
- Can span many graphs / machines / servers (decentralized)
- Have a known ending (the `rdf:nil`)

**Pagination:**
- Use [ActivityStreams collections](https://www.w3.org/TR/activitystreams-core/#collection)

**Converting to arrays:**
- Use a library, such as [`@rdfdev/collections`](https://js.rdf.dev/modules/_rdfdev_collections)


In this article, we'll discuss one of the most confusing but important topics of RDF: the various ways you can create ordered data.

Ordered data in RDF: About Arrays, Lists, Collections, Sequences and Pagination


[Open Decision Making](https://openbesluitvorming.nl) (Open Besluitvorming in Dutch) lets you search through what has been discussed in Dutch governments.
In collaboration with local governments, the VNG and the Open State Foundation, we've built software that standardizes and aggregates the meeting data of more than 120 governments.
All this open information makes it easier for journalists, policymakers and well-informed citizens to get involved in.
With the principles of [linked data](/blog/what-is-linked-data/) we ensured that the data is open, uniform and reusable for anyone that likes to build upon it.

## How it started

In 2014 we started our e-democracy platform [Argu.co](https://argu.co), which aims to engage citizens in decision-making processes.
We worked with many governments and hosted online discussions where people shared ideas, opinions and voted on issues.
Although we've seen that e-democracy and civic participation can be very successful, one key element is often missing: the actual decision-making process.
This has to do with the fact that e-democracy discussions tend to happen parallel to the actual governmental meetings.
We wanted to bring these two worlds together.
To achieve that goal, we decided to work on gathering and standardizing the data that these decision-makers create: meetings, votes, agenda-items and many other documents.

In 2016 we started working with the [Open State Foundation](https://openstate.eu/), who initiated Open Raadsinformatie (Open Municipality Data).
VNG (the Dutch Municipality Union) provided funding, we joined forces, got involved with the technology, and helped to scale the project up from 7 to over 120 municipalities and 6 provinces.

## Going from PDF to enriched text

Dutch municipalities produce a *lot* of information in their meetings.
They use meeting software to organize and distribute their agenda items, documents, motions, and reports online.
However, all these applications tend to work quite differently, all with their own information schemes.
Also, many of the proceedings and documents are saved in PDF format and are often not publicly accessible to download.
With our access to these systems and a lot of time to understand and implement the various APIs, we were able to bring everything together and create a standardized, searchable base of information that is always publicly accessible.

Most of the interesting information hides in PDF files.
We wrote ETL (extract, load, transform) software using Python, Celery and Elasticsearch that gets all the source files, standardizes it to a single model and loads the text into various query systems.
We store and cache all these files in Google Cloud Storage, along with the referencing sources and metadata.
This enables us to serve files that are not accessible through the default API of the meeting software sources.
This also makes it possible to see what has changed between versions.

Additionally, two organizations contributed two enrichers that add derived information to the resources:

- Theme classifier: a [machine-learning classifier](https://github.com/openstate/ori-theme-classifier) written by the Open State Foundation.
Scores document relevance to a set of specified themes, such as Economy, Safety and Education.

- Location classifier: based on the project [LocLinkVis](https://bitbucket.org/aolieman/loclinkvis/) made by Alex Olieman and the University of Amsterdam.
It finds streets and locations in document texts and translates it into geographical coordinates of city neighborhoods and districts.

These enrichers allow all users of Open Decision Making to filter documents by theme tags and use geographical information to pinpoint which locations are mentioned.

## Storing RDF

We love [linked data](what-is-linked-data/), and its merits are exceptionally valuable in a project such as this one.
Linked data is highly re-usable in other applications without needing duplication, and the [diverse serialization formats](/blog/rdf-serialization-formats/) of RDF will provide something for any use-case.

However, working with linked data poses new challenges, such as choosing the right data store for the statements.
You might think that an existing triple store is the most logical option since we're working with RDF, but you don't need triple stores to do serve linked data.
Triple stores make a lot of sense when you have schema-less RDF data and require SPARQL as a query language.
We didn't need those, as our schema was constrained by our internal models, and SPARQL didn't suit our query requirements.
The primary goal was providing full-text search and RESTful RDF resources.

Before we joined the project, Elasticsearch was used as the only data store.
It ticked the text search box, as Elasticsearch has a powerful text search query-language, but it is not designed for relational database queries.
Also, it's not a database as it's not ACID compliant (i.e., highly resistant to corruption or data loss).

So, looking for a suitable store, we started with ArangoDB, which is a NoSQL document-based graph store.
It seemed appropriate at the time because we were working with lots of documents, all connected in a graph.
However, we found that the graph queries were taking too long to be suitable for this project.

Next we tried Neo4j, which is the market leader in Graph databases.
It offers a powerful query language called Cypher, but we found that memory use was growing too early too fast.
Another problem related to storing RDF in Neo4j - it should be possible, but there are some differences between the graph models of RDF and Neo4j.
We had some trouble with using URLs as keys (a requirement for RDF), which had to be escaped, making the queries long and dreadful.
Also, you have to think about how to cope with external resources.
Whereas in an RDF store one can add a triple referencing an external resource, in Neo4j one should create a node to mimic the same behavior.

In the end, we abandoned graph databases altogether for this project and stuck with Postgres.
We didn't need graph-like queries or schemaless storage of triples.
I am not saying that those graph databases are not good enough, merely not suitable for our purpose and budget.

Since the project depends heavily on fast text search queries, we found that Elasticsearch is still essential.
So we kept it on for full-text searching.
But since its Query DSL is changing quite a lot over time, it is not suitable as a stable API.
We had to come up with an answer to this and created the ORI API service which can be found [here](https://id.openraadsinformatie.nl/).
The idea here was that every resource should have its own resolvable identifier that would stay the same.

## Open Data

Several applications now actively use the freely available data from Open Decision Making:

### OpenBesluitvorming.nl

We have built the [primary open-source search application (Dutch)](https://openbesluitvorming.nl) for users to discover and explore what is being said in their local government.
The beauty of open data is that everyone can build their own applications based on the same API.

![openbesluitvorming.nl](/images/posts/ori/zoek-openraadsinformatie.jpg "openbesluitvorming.nl")

&nbsp;

### Raadstalk.nl

VNG is the Dutch umbrella organization for all municipalities and has been the driving force in this project.
They've asked us to create a [widget](https://raadstalk.nl) to see what words are trending for all municipalities in each month.

![raadstalk.nl](/images/posts/ori/raadstalk.jpg "Raadstalk.nl")

&nbsp;

### Waaroverheid.nl

[WaarOverheid](https://waaroverheid.nl) shows all documents that mention a location in your neighborhood and district.
It provides a map interface to navigate from neighborhood to neighborhood to find documents in streets located there.

![waaroverheid.nl](/images/posts/ori/waaroverheid.jpg "Waaroverheid.nl")

&nbsp;

### 1848.nl

[1848](https://1848.nl) gathers political information from different sources to search and receive alerts to be able to stay on top of things.

![1848.nl](/images/posts/ori/1848.jpg "1848.nl")

&nbsp;

## Going forward

It all started with municipality data, but the scope has now broadened to provincial data and will soon include data from the national level as well.
We are working with the Dutch parliament on integrating their brand new API.

We believe that this is just a preview of what is possible with open government data.
If we can convince governments to adopt a more data-focused information publishing strategy (using machine-readable formats, such as RDF, instead of PDF), we can create even more engaging web applications: imagine clicking on a representative to see which motions they submitted, or compare voting behavior between political parties.

Working on this project has been challenging, we've overcome quite some hurdles that come with managing such amounts of data and systems.
Along the way, we have learned a great deal about ETL processes, Elasticsearch, Kafka and Kubernetes, to name a few.

[Contact us](/contact) to know how we can help you achieving your goals in modeling and transforming information to linked data.


We've aggregated meeting data from over 250+ Dutch governmental organisations and are now publishing it as linked data.

Publishing Dutch governmental meetings as linked data


Ontologies are resources that describe concepts and how several classes and properties in a certain domain relate to each other.
For example, if you're creating a social network app, you might want to use the Friend of a Friend ([FOAF](http://xmlns.com/foaf/spec/)) ontology, which describes things like ["Person"](http://xmlns.com/foaf/spec/#term_Person) and ["knows"](http://xmlns.com/foaf/spec/#term_knows).
When creating linked data applications, you're likely to work a lot with ontologies.
Since ontologies, their classes and their properties all use URLs as identifiers, that means that you might write a lot of long strings.
This is not only bothersome, but also a source of unnecessary typos and a possible performance bottleneck.

That's why we created the [@ontologies](https://github.com/ontola/ontologies) project.
This is an open source repository that converts RDF ontologies into JS exports, pubslihed in NPM for easy re-use.
If your IDE is smart enough, it will auto-import the ontologies and provide and useful tooltips with ontological descriptions for each item.

## How to use it

```sh
## Add it to your project
yarn add @ontologies/core @ontologies/schema
```

```js
// Initialize it once (./init.js)
import { setup } from "@ontologies/core";
setup();
```

```js
// Use it
import "./init";
import { name } from "@ontologies/schema";

document.getElementById("app").innerHTML = `
<h1>Schema URL: ${name.value}</h1>
`;
```

Check out a [running example on CodeSandbox](https://codesandbox.io/s/ontologies-41zgg).

We've currently added ActivityStreams, DCElements, DCTerms, DCMIType, FOAF, OWL, PROV-O, RDF, RDFS, Schema, SHACL, SKOS and XSD.

PRs for new ontologies are welcome!


Our new @ontologies NPM package helps developers write RDF applications in Javascript.

Using ontologies in Javascript apps


Contrary to some other data models, RDF is not bound by a single serialization format.
Triple statements (the data atoms of RDF) can be serialized in many ways, which leaves developers with a possibly tough decision: how should I serialize my [linked data](/blog/what-is-linked-data)?

## To answer the title's question: _it depends_

Skip to the [TL;DR](#tldr) if you're feeling hasty.
The order in which they appear is **chronological / historical and does not reflect preference**.

## RDF/XML

The first and perhaps most well-known RDF serialization format is [RDF/XML](https://www.w3.org/TR/rdf-syntax-grammar/).
It's also the most [despised](https://github.com/mhausenblas/rdfxml.info/blob/master/input/RDF-XML%20sucks%20-%20praise%20and%20damnation.txt).
Many systems were able to parse, store and serialize XML when RDF was invented almost 20 years ago, so RDF/XML seemed like a logical default.

Unfortunately, RDF/XML is a weird mixture of two fundamentally different concepts: a tree-like document, and a triple-based graph.
This makes RDF/XML conceptually difficult and quite verbose, compared to other standards.
For XML developers, it might look familiar, but since it does not clearly reflect the triple model, it will probably cause confusion.

Use it only if you need to work with XML.

```xml
<?xml version="1.0"?>

<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:schema="http://schema.org/">

<rdf:Description rdf:about="https://www.w3.org/People/Berners-Lee/">
  <schema:birthDate>1966-06-08</schema:birthDate>
  <schema:birthPlace rdf:resource="http://dbpedia.org/resource/London"/>
</rdf:Description>

</rdf:RDF>
```

## RDFa

[RDFa](https://www.w3.org/TR/rdfa-core/) is RDF inside HTML.
By adding attributes to HTML elements, you can give semantic context to the content inside your webpages.
Google parses it (amongst [JSON-LD](#JSON-LD), [Microdata](https://w3c.github.io/microdata/#microdata-and-rdf), Microformats and Pagemaps) to [enhance their search previews](https://developers.google.com/custom-search/docs/structured_data), although they [recommend using JSON-LD](https://www.searchenginejournal.com/google-structured-data-preference/297479/).
W3C's own Respec documentation tool [dropped support for RDFa](https://github.com/w3c/respec/issues/1503),
mainly because the adoption of RDFa was too low, the required code was messy and even Google didn't parse it correctly.

RDFa is fundamentally different from the other mentioned formats: it combines RDF with view data (HTML).
This means that it makes your HTML documents a bit larger and more complicated, and parsing it for triples will be more costly than parsing an RDF only format like N-Triples.
This makes it less useful if your application relies on a lot of RDF data.

Use RDFa if you want to make your existing website/ blog / HTML based application more semantic.

```html
<div about="https://www.w3.org/People/Berners-Lee/">
  <p>
    Tim is born on
    <span property="http://schema.org/birthDate">1955-06-07</span>
    in
    <a
      property="http://schema.org/birthPlace"
      href="http://dbpedia.org/resource/London"
    >
      London
    </a>
  </p>
</div>
```

## Notation3 (.n3)

Tim-Berners Lee wanted something better than RDF/XML, and came up with [N3](https://www.w3.org/TeamSubmission/n3/).
Contrary to RDF/XML, N3 closely resembles the RDF Subject / Predicate / Object model.
This makes N3 very easy on the eyes and helps to understand how RDF works.
By using `@prefix`es, N3 can be quite compact.

However, N3 is relatively costly to serialize, which could hinder performance.
It's also quite feature-heavy since it supports [RDF rules](http://www.ninebynine.org/RDFNotes/RDFFactsAndRules.html), which makes it harder to parse.

Unless you need the reasoning/rules features of N3, use its more popular (and very similar) successor Turtle.

```
@prefix tim: <https://www.w3.org/People/Berners-Lee/>.
@prefix schema: <http://schema.org/>.
@prefix dbpedia: <http://dbpedia.org/resource/>.

<tim> schema:birthDate "1955-06-08"^^<http://www.w3.org/2001/XMLSchema#date>.
<tim> schema:birthPlace <dbpedia:London>.
<tim> schema:birthPlace <dbpedia:London>.
```

## Turtle (.ttl)

[Turtle](https://www.w3.org/TR/turtle/), the _Terse RDF Triple Language_, is a subset of N3.

It strips some of the syntactic sugar en features of N3, which makes parsing Turtle a bit simpler.
This, in turn, made Turtle more popular, which means that it's easier to find libraries for it.

Unfortunately, it's still quite costly to parse compared to [N-Triples](#N-Triples).

Turtle is highly human-readable and is, therefore, a good candidate if you need to edit RDF by hand.

```
@prefix tim: <https://www.w3.org/People/Berners-Lee/>.
@prefix schema: <http://schema.org/>.
@prefix dbpedia: <http://dbpedia.org/resource/>.

<tim> schema:birthDate "1955-06-08"^^<http://www.w3.org/2001/XMLSchema#date>.
<tim> schema:birthPlace <dbpedia:London>.
```

## N-Triples (.nt) and N-Quads (.nq)

[N-Triples](https://www.w3.org/TR/n-triples/) is a very simple subset of Turtle, which in turn is a simple subset of N3.
N-Triples does not support `@prefixes` or any fancy features.
This makes N-Triples trivial to parse / serialize.
Therefore, many libraries for N-Triples are available and you can easily write one yourself.
It also makes serialization and parsing highly performant.

```
<https://www.w3.org/People/Berners-Lee/> <http://schema.org/birthDate> "1955-06-08"^^<http://www.w3.org/2001/XMLSchema#date>.
<https://www.w3.org/People/Berners-Lee/> <http://schema.org/birthPlace> <http://dbpedia.org/resource/London>.
```

However, the lack of prefixes and shorthands makes the format lengthy and a bit tough to read.
The lengthy URLs also mean that you'll need some form of compression (e.g.g-zip) if you don't want to waste precious bandwidth or storage capacity, so make sure to enable that in your server.

Since writing a parser / serializer for N-Triples is so simple, it's a good idea to support this pretty much always.
And since N-Triples is a subset of Turtle and N3, it means that Turtle / N3 parsers know how to deal with N-Triples, too.

[N-Quads](https://www.w3.org/TR/n-quads/) are like N-Triples, but they have an optional fourth column, which can be used to denote a graph label.
The graph label often refers to the source of the data, e.g. the URL of the HTML document or some external RDF resource.

## JSON-LD (.jsonld)

JSON is, without a doubt, the most popular way to serialize data in web applications.
[JSON-LD](https://json-ld.org/spec/latest/json-ld/) is an extension of JSON and is valid JSON as well.
You can turn your regular plain old JSON into RDF by adding [`@context`](https://json-ld.org/spec/latest/json-ld/#the-context).
This object mainly serves as a mapping, so your plain keys get turned into fancy links to RDF Classes and Properties.
You can add context either by adding an `@content` header in your HTTP response, by including the link in your JSON body, or by adding the entire `@context` object to you JSON.
This means that if you want to upgrade your JSON API to JSON-LD, you get to keep your serializers.

```json
{
  "@context": {
    "dbpedia": "http://dbpedia.org/resource/",
    "schema": "http://schema.org/"
  },
  "@id": "https://www.w3.org/People/Berners-Lee/",
  "schema:birthDate": "1955-06-08",
  "schema:birthPlace": {
    "@id": "dbpedia:London"
  }
}
```

JSON-LD is **easy to read**, and will feel familiar even to those new to RDF and linked data.
Because it's still valid JSON, it's easily usable for those who don't want to deal with URLs.
JSON arrays are converted to [RDF Lists](/blog/ordered-data-in-rdf/) (but can also be converted to [other sequence types]()).
I recommend spending some time in the [JSON-LD playground](https://json-ld.org/playground/) to get familiar with how it works.

Unfortunately, JSON-LD [difficult and costly to parse](http://www.dr-chuck.com/csev-blog/2016/04/json-ld-performance-sucks-for-api-specs/) if you need the RDF data instead of the JSON object.
Parsing JSON-LD often involves requesting data from the internet, and needs clever caching to be performant.
This complexity in parsing limits how many (bug-free) JSON-LD parsers are available, makes libraries heavy, and it also means that parsing JSON-LD takes relatively long.

JSON-LD is a compromise.
It supports RDF, it supports JSON, and it does both _okay_.
Use JSON-LD if you already have a RESTful JSON API, and if performant RDF parsing is not crucial.

## JSON-AD (Atomic Data)

_Disclaimer: I'm the creator of JSON-AD_.

[JSON-AD](https://docs.atomicdata.dev/core/json-ad.html) (JSON Atomic Data) only supports a strict subset of RDF, so it is an odd one in this list.
In other words: it cannot be used to serialize all existing RDF data.
However, because it allows only a strict subset, it also has a couple of advantages over other formats:

- Very **low parsing cost / high performance**. It can be parsed as plain JSON, which means that many efficient and performant parsers are available.
- **Easy to understand**. Since it supports only a subset of RDF, it doesn't suffer from some of the [quirks of RDF](https://docs.atomicdata.dev/interoperability/rdf.html) such as predicate-non-uniqueness
- It supports **native JSON arrays** as collections, which prevents the complexities of dealing with [ordered data in RDF](https://ontola.io/blog/ordered-data-in-rdf/).
- **Type safety**. Each key in a JSON-AD object should resolve to a [Property](https://atomicdata.dev/classes/Property), which describes [datatype](https://atomicdata.dev/properties/datatype) and [description](https://atomicdata.dev/properties/description).

Here's the `description` Property from above, serialized as JSON-AD:

```json
{
  "@id": "https://atomicdata.dev/properties/description",
  "https://atomicdata.dev/properties/datatype": "https://atomicdata.dev/datatypes/markdown",
  "https://atomicdata.dev/properties/description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.",
  "https://atomicdata.dev/properties/isA": [
    "https://atomicdata.dev/classes/Property"
  ],
  "https://atomicdata.dev/properties/shortname": "description"
}
```

Because these properties also have [`shortnames`](https://atomicdata.dev/properties/shortname), we can easily [serialize Atomic Data to plain JSON](https://docs.atomicdata.dev/interoperability/json.html), without the long URLS:

```json
{
  "@id": "https://atomicdata.dev/properties/description",
  "datatype": "https://atomicdata.dev/datatypes/markdown",
  "description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.",
  "is-a": [
    "https://atomicdata.dev/classes/Property"
  ],
  "shortname": "description"
}
```

## HexTuples

[HexTuples](https://github.com/ontola/hextuples) is an [NDJSON](http://ndjson.org/) (Newline Delimited JSON) based RDF serialization format.
It is designed to achieve the best possible performance in a JS context (i.e. the browser).
It uses plain JSON arrays, in which the position of the items denote `subject`, `predicate`, `object`, `datatype`, `lang` and `graph`.

```json
["https://www.w3.org/People/Berners-Lee/", "http://schema.org/birthDate", "1955-06-08", "http://www.w3.org/2001/XMLSchema#date", "", ""]
["https://www.w3.org/People/Berners-Lee/", "http://schema.org/birthPlace", "http://dbpedia.org/resource/London", "http://www.w3.org/1999/02/22-rdf-syntax-ns#namedNode", "", ""]
```

HexTuples is designed by Thom van Kalkeren (a colleague of mine) because he noticed that parsing / serialization was unnecessarily costly in our stack, even when using the relatively performant `n-quads` format.
Since HexTuples is serialized in NDJSON, it benefits from the [highly optimized JSON parsers in browsers](https://v8.dev/blog/cost-of-javascript-2019#json).
It uses NDJSON instead of regular JSON because it makes it easier to parse concatenated responses (multiple root objects in one document).
As an added plus, this enables streaming parsing as well, which gives it another performance boost.
Our JS RDF libraries ([link-lib](https://github.com/fletcher91/link-lib/), [link-redux](https://github.com/fletcher91/link-redux/)) have an internal RDF graph model which uses these arrays as well, which means that there is minimal mapping cost when parsing Hex-Tuple statements.
This format is especially suitable for real front-end applications that use dynamic RDF data.

## HDT

[HDT](http://www.rdfhdt.org/what-is-hdt/) (Header, Dictionary, Triples) is a compact data structure and binary serialization format for RDF, so it's more than just a way to serialize RDF.
Its data structure saves space and bandwidth (it's half the size of gzipped N-Triples).
Its design has indexing built-in, which means it can be searched or browsed efficiently.
Check out the impressive [technical specification](http://www.rdfhdt.org/technical-specification/) if you want to learn more about how it works.
HDT compression is a costly process, so it's not that attractive for highly dynamic data / data that changes over time.
Although some really useful libraries for HDT exist, be sure to check if there exists libraries that work with your stack.

## RDF Binary Thrift

[RDF Binary thrift](https://afs.github.io/rdf-thrift/rdf-binary-thrift.html) is an encoding of RDF data that uses Apache Thrift.
It is used in the Apache Jena RDF store.
It's a binary format, and therefore it's cheap to parse and serialize.
I've never used it, but it sounds interesting, although you'll need Thrift tooling for decoding it.

## Let your users choose a format

Choosing an RDF serialization format for your application or service might be a false dilemma.
Since you control your application and probably have an internal model, you can offer multiple serialization options.
Therefore, you can implement a serialization library (e.g. our [rdf-serializers](https://github.com/ontola/rdf-serializers) gem for Rails) and use [HTTP Content negotiation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation), so your project can handle all kinds of formats.

## TL;DR

- Use Hex-Tuples if you want high performance in JS with dynamic data.
- Use JSON-AD if you don't have to support existing RDF data, but do value JSON compatibility and type safety.
- Use HDT if you have big, static datasets and want the best performance and compression.
- Use N-Triples / N-Quads if you want decent performance and high compatibility.
- Use JSON-LD if you want to improve your existing JSON API, and don't need performant RDF parsing.
- Use Turtle if you want to manually read & edit your RDF.
- Use Notation3 if you need RDF rules.
- Use RDFa to extend your existing HTML pages.
- Use RDF/XML if you need to use XML.
- If you can, support all of them and use content negotiation.


Since RDF has a wide variety of serialization formats, it's hard to pick the best one. This post will help you to find out.

What's the best RDF serialization format?


[JSON-HAL](http://stateless.co/hal_specification.html) and [JSON-LD](https://json-ld.org/) are pretty popular formats for desiging APIs. They both have [their merits](https://sookocheff.com/post/api/on-choosing-a-hypermedia-format/), but they aren't compatible with each other. However, with some additions, we can turn JSON-HAL into valid JSON-LD, without changing any of the original keys. This way, you can keep compatibility with HAL clients and benefit from the [advantages of linked data](/blog/what-is-linked-data), such as conversion to other RDF formats.

Let's start with a JSON-HAL body object from the [docs](https://tools.ietf.org/html/draft-kelly-json-hal-08).

```json
{
  "_links": {
    "self": { "href": "/orders/523" },
    "warehouse": { "href": "/warehouse/56" },
    "invoice": { "href": "/invoices/873" }
  },
  "currency": "USD",
  "status": "shipped",
  "total": 10.20
}
```

To identify the main resource, we add an `@id` value:

```json
{
  "@id": "https://example.com/orders/123",
  "_links": {
    "self": { "href": "/orders/523" },
    "warehouse": { "href": "/warehouse/56" },
    "invoice": { "href": "/invoices/873" }
  },
  "currency": "USD",
  "status": "shipped",
  "total": 10.20
}
```

Let's add an `@context` object to map our keys to URIs:

```json
{
  "@id": "https://example.com/orders/123",
  "@context": {
    "currency": "https://example.com/ns/shipping#currency",
    "status": "https://example.com/ns/shipping#status",
    "total": "https://example.com/ns/shipping#total",
    "warehouse": "https://example.com/ns/shipping#warehouse",
    "invoice": "https://example.com/ns/shipping#invoice"
  },
  "_links": {
    "self": { "href": "/orders/523" },
    "warehouse": { "href": "/warehouse/56" },
    "invoice": { "href": "/invoices/873" }
  },
  "currency": "USD",
  "status": "shipped",
  "total": 10.20
}
```

If we try to [parse this as JSON-LD](https://json-ld.org/playground/#startTab=tab-nquads&json-ld=%7B%22%40id%22%3A%22https%3A%2F%2Fexample.com%2Forders%2F123%22%2C%22%40context%22%3A%7B%22currency%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23currency%22%2C%22status%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23status%22%2C%22total%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23total%22%2C%22warehouse%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23warehouse%22%2C%22invoice%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23invoice%22%7D%2C%22_links%22%3A%7B%22self%22%3A%7B%22href%22%3A%22%2Forders%2F523%22%7D%2C%22warehouse%22%3A%7B%22href%22%3A%22%2Fwarehouse%2F56%22%7D%2C%22invoice%22%3A%7B%22href%22%3A%22%2Finvoices%2F873%22%7D%7D%2C%22currency%22%3A%22USD%22%2C%22status%22%3A%22shipped%22%2C%22total%22%3A10.2%7D), we only get three valid triples. That's because the items in `_links` are not yet parsed. To do this, we need to let the JSON-LD parser know three things:

- The object in `_links` describes the same resource as the root object. We do this by adding an `@id` value in the `_links` object with a value identical to the `@id` of the root object.
- The `href` values in the objects in `_links` are actually the URI values of these properties. We do this by adding an2 `@context` object to the `_links` body that binds `href` to `@id`.
- The `_links` object contains useful information, don't skip it, parser! For the JSON-LD playground parser, we can do this by mapping `_links` to some arbitrary URL. Since documenting such a hacky solution might be useful, let's link to this article. This is the most hacky part of my solution, so let me know if you find something better!

```json
{
  "@id": "https://example.com/orders/123",
  "@context": {
    "currency": "https://example.com/ns/shipping#currency",
    "status": "https://example.com/ns/shipping#status",
    "total": "https://example.com/ns/shipping#total",
    "warehouse": "https://example.com/ns/shipping#warehouse",
    "invoice": "https://example.com/ns/shipping#invoice",
    "_links": "https://ontola.io/blog/json-hal-ld"
  },
  "_links": {
    "@id": "https://example.com/orders/123",
    "@context": { "href": "@id" },
    "self": { "href": "/orders/523" },
    "warehouse": { "href": "/warehouse/56" },
    "invoice": { "href": "/invoices/873" }
  },
  "currency": "USD",
  "status": "shipped",
  "total": 10.20
}
```

If we [parse this as JSON-LD](https://json-ld.org/playground-dev/#startTab=tab-nquads&json-ld=%7B%22%40id%22%3A%22https%3A%2F%2Fexample.com%2Forders%2F123%22%2C%22%40context%22%3A%7B%22currency%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23currency%22%2C%22status%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23status%22%2C%22total%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23total%22%2C%22warehouse%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23warehouse%22%2C%22invoice%22%3A%22https%3A%2F%2Fexample.com%2Fns%2Fshipping%23invoice%22%2C%22_links%22%3A%22https%3A%2F%2Fontola.io%2Fblog%2Fjson-hal-ld%22%7D%2C%22_links%22%3A%7B%22%40id%22%3A%22https%3A%2F%2Fexample.com%2Forders%2F123%22%2C%22%40context%22%3A%7B%22href%22%3A%22%40id%22%7D%2C%22self%22%3A%7B%22href%22%3A%22%2Forders%2F523%22%7D%2C%22warehouse%22%3A%7B%22href%22%3A%22%2Fwarehouse%2F56%22%7D%2C%22invoice%22%3A%7B%22href%22%3A%22%2Finvoices%2F873%22%7D%7D%2C%22currency%22%3A%22USD%22%2C%22status%22%3A%22shipped%22%2C%22total%22%3A10.2%7D), we get all six triples:

```
<https://example.com/orders/123> <https://example.com/ns/shipping#currency> "USD" .
<https://example.com/orders/123> <https://example.com/ns/shipping#invoice> <https://json-ld.org/invoices/873> .
<https://example.com/orders/123> <https://example.com/ns/shipping#status> "shipped" .
<https://example.com/orders/123> <https://example.com/ns/shipping#total> "1.02E1"^^<http://www.w3.org/2001/XMLSchema#double> .
<https://example.com/orders/123> <https://example.com/ns/shipping#warehouse> <https://json-ld.org/warehouse/56> .
<https://example.com/orders/123> <https://ontola.io/blog/json-hal-ld> <https://example.com/orders/123> .
```

Awesome! Just by adding a few keys, we converted JSON-HAL into JSON-LD, without breaking anything. However, the resulting object is not that pretty and quite verbose. Only use this is you really _have_ to support both JSON-HAL and JSON-LD. If you haven't invested in JSON-HAL yet, but do need a linked data API, I recommend you go with JSON-LD plus [Hydra](https://www.hydra-cg.com/spec/latest/core/), which also offers standardization for things like actions, collections and pagination.

## TL;DR

- Add an `@id` to both the root object and the `_links` object with the main resouce URL.
- In the root `@context`, add your usual JSON-LD mapping.
- In the root `@context`, map`_links` to some URL.
- In the `_links` `@context`, map `href` to `@id`.


JSON-HAL and JSON-LD can be made to be fully compatible with each other. We'll discuss how in this article.

Upgrade JSON-HAL to JSON-LD


The internet started off as a place for linked documents.
However, where humans are perfectly capable of understanding HTML documents through a browser, this does not apply for machines that need a specific piece of information.
That’s why every decent web service has an API, and probably uses it for their website, app, integrations and external services.
Unfortunately, too many APIs are unnecessarily hard to use and unintuitive.
In this article, I'll give some practical advice on designing a RESTful, hypermedia API that follows web conventions.

## Use URLs as IDs
Every thing in your API, every individual resource, should have its own URL. The URL should serve as both an _identifier_ as well as a _locator_: it is the identity of the thing and it provides a way to fetch information about that thing. This makes URLs really useful.

Firstly, URLs make your responses _far easier to navigate_. Imagine a JSON object representing a social media post that references some author with an identity of `18EA91FB19`. From this information, you wouldn't know where to find that author. You need to read the API docs, discover the endpoint for authors and compose your request. If the ID was a URL, you would instantly know where to send that request to. This is not just great for humans, but also for machines, since they can't read your API docs - but they can navigate URLs.

Secondly, URLs are always unique. URLs are not just unique identifiers in a single system, but also unique across different systems. The domain name takes care of that. This means that you can use your data across multiple systems without worrying about name collisions in identifiers. This is one of the properties [that makes linked data awesome](https://ontola.io/blog/what-is-linked-data).

Make sure that your URLs are stable. [Cool URIs don’t change](https://www.w3.org/Provider/Style/URI). If they really have to change, make sure the old URLs redirect to the new ones. Nobody likes broken links.

## Your API endpoint is your website
You don't need a subdomain for your API, like `api.example.com` or a sub-path, like `example.com/api`. Your endpoint should be the root of your webpage: `example.com`.

This is useful, because as discussed above the URL should be both the identifier as the locator of a single resource. Whether someone is looking for a HTML version or for example a JSON representation of a resource, he should be able to use the same URL. This makes your API easier to use, because someone who navigates your website can know at any time how to access the same resource in some other format.

But if the URL does not change across format, how do you request the right one? This is where [HTTP content negotiation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept) comes in handy. A client can send preferences about what kind of content it wants to receive in the `Accept` HTTP header. The default header for web browsers is `text/HTML`, but for most APIs, a machine readable setting such as `application/json` is more suitable.

But what about API versioning? We want our URLs _not to change_, so we should not use different URLs for different API versions. The solution, again, is to use a HTTP header. Use an `api-version` header or a [specific Mime type](https://developer.github.com/v3/media/) in your requests.

## Use sensible hierarchy in URL paths
Having a URL hierarchy that makes sense is not just important for your website, but also for your API - especially if your API structure resembles your website structure. Try to come up with a sensible URL strategy, discuss it with your colleages and do all of this early in the development process.

A few things to consdier:
* Move from large to small, from generic to specific.
* The user should be able to remove the last part of the URL and arrive at a parent resource.
* Let the hierarchy reflect the UX of navigating the website.
* Try to keep URLs as short as possible.
* Human readable URLs are easier to understand and share, and they are great for SEO.
* [Cool URIs don't change](https://www.w3.org/Provider/Style/URI). Leave out anything that might change, such as author, file name extensions, status or subject.

## Use query parameters correctly
The [URI spec](https://tools.ietf.org/html/rfc3986#section-3.4) tells us to use query parameters only for _non-hierarchical data_.

Don't use query parameters to identify a resource; use a path.

* **Bad**: `example.com/posts?id=123`
* **Good**: `example.com/posts/123`

Use query parameters for _optional_ things like limiting, sorting, filtering and other modifiers:

* **Good**: `example.com/posts?limit=30`
* **Good**: `example.com/posts/123?show_hidden=true`

## Use HTTP methods
Instead of having a bunch of endpoints for various types of actions, use _a single URL for every single resource_ in your application. Distinguish between actions using [HTTP methods](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods).

* **Bad**: `GET example.com/showPost/123`
* **Bad**: `GET example.com/removePost/123`
* **Good**: `GET example.com/posts/123`
* **Good**: `DELETE example.com/posts/123`

There is a big difference between requests that aim to read content, create content or edit content. Make sure to use the GET, POST, PUT and PATCH HTTP methods correctly. The GET and PUT operations are _idempotent_, which means that a request can be repeated multiple times without side effects. This distinction is important, because it tells the client whether it can try again if an error occurs. It also helps with caching, since only GET request should be cacheable.

If you want to offer a form to delete or edit a resource, _that form will be a different resource from the original item_, so it will need a seperate URL. A nice convention is to nest that form resource below the original item. This way, the user just adds `/edit` to the a URL if he wants to edit that resource.

* **Good**: `GET example.com/posts/123/remove`
* **Good**: `GET example.com/posts/123/edit`

## Use HTTP status codes
Pretty much all types of error messages can be categorized in the existing [HTTP status codes](https://developer.mozilla.org/nl/docs/Web/HTTP/Status). These are not just useful to humans, but especially to machines. Status codes can be parsed far more quickly than a body text. Another advantage is that they are standardized, so the client library is likely to know what the status code represents. You don’t have to support every single one, but at the very least make sure that you use the five categories:
* **1xx: informational** - just letting you know
* **2xx: successful** - everything’s OK
* **3xx: redirection** - your content is somewhere else
* **4xx: client error** - you’re doing something wrong
* **5xx: server error** - we’re doing something wrong

## Add context to your JSON
Assuming you use JSON as a serialization format, you can use [@context](https://json-ld.org/spec/FCGS/json-ld/20180607/#the-context). The @context object is a nifty little idea to make your API more self-descriptive. It describes what the various keys in your JSON actually represent. It provides links to where the definition can be found.

Make sure all your IDs are actually links, and your context is included. Now all your JSON has become JSON-LD, which is [linked data](https://ontola.io/blog/what-is-linked-data). That means that your JSON data is now convertible to other RDF formats (Turtle, N3, N-triples, etc.), which means it becomes far more reusable.

Keep in mind that the links that you use should preferably resolve to some document that explains what your concept represents. A good starting point to find relevant concepts is [schema.org](https://schema.org).

## Offer various serialization options
Be as flexible as possible in your serialization options. Someone who uses your API might be good at dealing with JSON, but may not know how to work with XML. Serializing an in-memory object to various serialization options is often not that complicated. If you use Ruby on Rails, check out our [rdf-serializers library](https://github.com/argu-co/rdf-serializers) which supports JSON-LD, RDF/XML, N3, N-triples and Turtle. Use the [aforementioned](#your-api-endpoint-is-your-website) HTTP accept header to handle content negotiation.

## Standardize index pages and pagination
You’re probably going to need index pages with pagination. How to deal with that? Pagination is not a trivial problem, but luckily for you, you’re not the first to encounter it. Don’t try to reinvent the wheel; use something that already exists, such as [W3C activity stream collections](https://www.w3.org/TR/activitystreams-core/#collections) or [Hydra collections](http://www.hydra-cg.com/spec/latest/core/#collections).

## Don’t require an API key
Your default API (the HTML one) doesn’t need one, so your JSON API [shouldn’t need one](https://ruben.verborgh.org/blog/2013/11/29/the-lie-of-the-api/#api-keys-are-a-lie) as well. Use rate limiting to make sure your servers don’t fry. You can still use API keys or authentication to give access to special parts of your API, of course.

## Use a `doc.` subdomain for API docs
Here's [a clever little idea](https://medium.com/@fletcher91/semantic-documentation-1177d563783c): make your API documentation available at `doc.example.com`. If a user wants to know how your api works for a certain page, he just adds `doc.` in front of his current URL. Show the user a page that tells something useful about how to use the API at that route.

## Use your own API
Finally, and perhaps most importantly: eat your own dog food. Make your API a first-class citizen by using it as the only way to access information from that system. API-driven development forces you to make your API actually work. It helps you document your API properly, since your colleagues need to use it as well. Besides, you'll make your application more modular and gradually realize a microservice architecture, which has its own set of benefits.

[Let me know](mailto:joep@argu.co) if I'm missing something, or if you want help with API design!


How do you design great RESTful APIs? This artricle provides do's and dont's for designing an API.

Best practices for RESTful API design

What is linked data, and what makes it useful? In this article, we'll take a piece of information and upgrade its data quality step by step, until it's linked data.

A brief introduction to linked data


## Onze democratie en het internet

Twee van onze favoriete uitvindingen.
Jammer dat ze nog niet zo goed samen werken.
Er zijn twee problemen die we moeten oplossen:

### 1. Het internet kan een betere rol spelen in onze democratie

We wonen met miljoenen mensen in een land, maar alle grote beslissingen worden gemaakt door kleine groepjes in oude houten zaaltjes.
Talloze waardevolle inzichten en ideeen worden hierdoor overgeslagen.
Met het internet zouden mensen vanaf allerlei plekken hun ideeen kunnen delen, kunnen meedenken en hun mening geven.

### 2. Social media zijn niet ontworpen voor het oplossen van problemen

Social media zijn meestal ontworpen om mensen naar advertenties te laten kijken, niet om samen problemen op te lossen, of om een genuanceerd blik op iets te krijgen.
Online discussies zijn vaak onoverzichtelijk, kortaf en daarmee niet goed bruikbaar.
Begrijpelijk dat veel organisaties dan ook bang zijn om het internet in te zetten.

<BigImage image="cases/argu_abstract_nl.png" />

## Onze oplossing

Dus zijn we aan de slag gegaan om een online democratie en community platform te bouwen.
Ontworpen om burgers zo makkelijk mogelijk te laten participeren, en om ambtenaren zo goed mogelijk vragen te laten stellen.
We lanceerden Argu in 2015, en hebben onderhand meer dan 150 participatietrajecten afgerond.
We hebben talloze [functionaliteiten](https://argu.co/info/functionaliteiten) ingebouwd, zoals digitaal stemmen, reageren, dynamische formulieren, vragenlijsten, kaartweergaven, notificaties, rechtenbeheer, groepen, custom domeinnamen, export mogelijkheden, custom theming, en nog veel meer.

<BigImage image="cases/argu_body.png" />

## Technologie

Argu is met afstand ons grootste project, en een belangrijke bron van onze inkomsten.
Het is een SAAS oplossing die we al jaren doorontwikkelen en live houden voor diverse klanten.
Het ontwikkelen van Argu heeft ons geleerd om webapplicaties te maken die onderhoudbaar, flexibel en schaalbaar zijn.

De stack van Argu bestaat uit diverse diensten, waarvan er twee in het bijzonder relevant zijn:

### Back-end: Apex

De basis van onze back-end van Argu is gebouwd op basis van [Ruby on Rails](/nl/tech/ruby-on-rails).
Gezien we de wens hadden om de data als [Linked Data](/nl/blog/what-is-linked-data) beschikbaar te stellen, hebben we hier een Open Source library voor geschreven: [Linked_Rails](https://github.com/ontola/linked_rails).
Dit geeft Apex een aantal unieke kwaliteiten:

- **Alles is linked data**. Zelfs de knopjes, formulieren, en acties zijn beschreven als data. Dit geeft ons een grote flexibiliteit, zodat we dezelfde broncode kunnen gebruiken voor zeer uiteenlopende usecases.
- **De API is gelijk aan de website**. Als je een pagina kan bekijken, kan je de data van die pagina opvragen met dezelfde URL. Eigenlijk ontzettend logsich (zie ons [artikel over API design](/blog/api-design)), maar gek genoeg vrij zeldzaam.

### Front-end: Libro

Libro is een webapplicatie geschreven in [React](/nl/tech/react) + [Typescript](/nl/tech/typescript).
Libro staat voor _linked data browser_.
Dit is wat Libro uniek maakt: het is in de kern niet een gewone app, maar een _data browser_.
Onder de motorkap werkt libro met de [Link libraries](https://github.com/rescribet/link-redux), ontwikkeld door onze CTO Thom van Kalkeren.
Libro communiceert exclusief met Apex door linked data (RDF) uit te wisselen.
We zijn een van de eerste SAAS ondernemingen die linked data zo ver in haar stack heeft gestopt.
Dat heeft ons [enkele unieke uitdagingen opgeleverd](/blog/full-stack-linked-data/), zoals een performance bottleneck waarvoor we een [nieuw RDF serialisatieformaat hebben ontwikkeld](https://github.com/ontola/hextuples).

## Gemaakt met

<TechWrapperSmall>
  <TechPill technology={'react'} />
  <TechPill technology={'typescript'} />
</TechWrapperSmall>

## Open Source

De gehele Argu broncode is open-source te [vinden op Gitlab](https://gitlab.com/ontola/argu/).


Een open source e-democracy en community platform, om organisaties te helpen hun achterban te bereiken.

Argu


## Het internet opnieuw decentraliseren

In 1989 publiceerde Tim Berners-Lee de eerste website.
Hij had daarbij een visie van een decentraal internet, een web, waarbij mensen hun eigen websites zouden hosten op hun eigen servers.
Nu zijn we meer dan 30 jaar verder, en het web is meer gecentraliseerd dan ooit.
Een handjevol techreuzen beheert onze data, bepaalt hoe onze digitale interacties er uit zien, en in welke volgorde we ons nieuws lezen.
Dat moet anders kunnen.

Techreuzen houden ons aan boord door onze eigen gegevens te koppelen.
Zo werkt je Google Calendar perfect samen met je GMail, en kan je je Google Drive bestanden snel als bijlage gebruiken.
Als techgigant heb je de data in handen om dit allemaal aan elkaar te koppelen, en zo gebruikers afhanklijk te maken van jouw diensten.
Gelukkig is er ook een andere manier om dit op te lossen: **standaardiseren**.
Met goede standaarden kunnen we onze data verplaatsen van App naar App, waardoor je niet vast komt te zitten in de ecosystemen van de techreuzen.
Daarnaast kan je door veel meer te standaardiseren dingen doen die je nu niet kan, zoals zoeken door al je data, zien wie wat wanneer heeft gewijzigd, dezelfde data openen in verschillende apps, en meer...
Maar hoe komen we daar?
Waar moeten standaarden aan voldoen, voordat we kunnen leven in deze tech-utopie?

## Van Linked Data naar Atomic Data

Tim Berners-Lee kreeg 10 jaar na de eerste website een belangrijk inzicht: dezelfde linkjes die we gebruiken voor websites, kunnen we ook gebruiken in _data_.
Zo ontstond _Linked Data_.
Linked Data maakt het mogelijk om het hele web te gebruiken als een grote, verbonden database.
Daarmee zou je apps kunnen maken die volledig decentraal zijn, die veel beter samenwerken, waarbij mensen zelf volledige controle hebben over hun data.

Althans, dat is de theorie.
In de praktijk is het nog erg lastig om apps te bouwen met Linked Data.
Wij kunnen dat weten, want als Ontola zijn we enkele jaren geleden [all-in](/blog/full-stack-linked-data) gegaan op deze technologie.
We herschreven ons e-democracy platform [Argu](https://argu.co) en gebruikten Linked Data voor letterlijk ieder knopje en menu.
Dat was een flinke klus, en we hebben talloze tools moeten maken om dit doel te bereiken.
Uit dit jarenlange ontwikkeltraject hebben we geleerd hoe we Linked Data nog beter - en vooral ook veel _makkelijker_ kunnen maken.

Onze CEO Joep Meindertsma begon in 2020 in zijn vrije tijd met het uitwerken van een nieuwe vorm van Linked Data.
Het doel: een makkelijkere, krachtigere abstractie vinden waarmee ontwikkelaars sneller apps kunnen maken die goed met elkaar kunnen samenwerken.
De beste dingen pakken uit Linked Data, maar verbeteren waar het mogelijk is.
Na een paar maanden stond er een [uitgebreide documentatie](https://docs.atomicdata.dev/) van Atomic Data online, en niet veel later de eerste server, libraries en browser.
Hierdoor is Atomic Data snel een behoorlijk volwassen systeem geworden, met degelijke implementaties en diverse tools.

## Waarom Atomic Data

- **Sneller standaardiseren**. Het afstemmen van hoe je een bepaald model vormgeeft is te vaak een tijdrovende klus. Daardoor wordt er _te weinig gestandaardiseerd_, en dat beperkt hoe goed applicaties samen kunnen werken. Atomic Data is ontworpen om snel en eenvoudig datamodellen te delen.
- **Sneller ontwikkelen**. Door de bestaande tools (zoals Atomic-Server) kan je snel aan de slag, en hoef je het wiel niet opnieuw uit te vinden. Opslag, zoeken, versiebeheer, rechtenbeheer... Dat is allemaal al geregeld.
- **Converteerbaar naar vrijwel alles**. Je kunt Atomic Data naar vrijwel ieder ander formaat exporteren: JSON, XML, JSON-LD, RDF, Turtle, N-Triples...
- **Makkelijk deelbaar**. Alle objecten hebben linkjes, en deze linkjes kan je opzoeken - net als websites. De data beschrijft aan de hand van die linkjes zelf waar je meer informatie kan vinden. Hierdoor is Atomic Data zeer goed deelbaar.
- **Verifieerbaar en corrigeerbaar**. Atomic Data introduceert [Commits](https://docs.atomicdata.dev/commits/intro.html), waarmee iedere wijziging wordt opgeslagen. Hierdoor ontstaat een geschiedenis van ieder object, zodat je altijd kan bepalen wie wat wanneer gedaan heeft.
- **Authenticatie en authorizatie**. Behee wie er toegang heeft tot welk stukje data.

## Het beste van linked data, JSON en type safety

- **De eenvoud van JSON**. Ontwikkelaars kennen het, en het werkt overal.
- **Type safety**. Je kunt zeker weten dat je data een bepaald type heeft.
- **Linked data**. Verbind je data aan elkaar - ook al staat het op andere machines.

<img src="/images/cases/atomic_venn.svg" />

## Wat we hebben gebouwd

- [**Atomic-Server**](https://crates.io/crates/atomic-server): een supersnelle database + server, geschreven in Rust.
- [**Atomic-Data-Browser**](https://github.com/joepio/atomic-data-browser): front-end applicatie voor beheer en weergave van Atomic Data.
- [**Atomic-CLI**](https://crates.io/crates/atomic-cli): handige command-line interface tool
- [**Atomic-lib**](https://crates.io/crates/atomic_lib): library voor Rust
- [**@tomic/react**](https://www.npmjs.com/package/@tomic/react): library voor React, met handige template
- [**@tomic/lib**](https://www.npmjs.com/package/@tomic/lib): library voor javascript / typescript
- [**Documentatie**](https://docs.atomicdata.dev/): meer dan 80 pagina's aan technische specificaties en voorbeelden.

## Atomic Server

Atomic Data is een modulaire specificatie, wat betekent dat je er stukjes uit kan pakken die je handig vind, maar dat je niet alles hoeft te gebruiken.
Als je snel aan de slag wilt, raden we je aan om onze Atomic-Server te gebruiken, welke alle specficiaties heeft geimplementeerd.

Atomic-Server is een krachtige, extreem snelle database + server, voorzien van ongelofelijk veel features.
Helemaal gratis en open source.
Geschreven in de programmeertaal Rust.
De front-end is geschreven in [Typescript](/nl/tech/typescript) met [React](/nl/tech/react).

Atomic-Server is ontworpen om de persoonlijke server van de toekomst te worden: een lichtgewicht applicatie die je thuis eenvoudig kan draaien op een goedkope computer, zoals een Raspberri Pi of een oude smartphone.

[Lees hier hoe je Atomic-Server kan installeren](https://github.com/joepio/atomic-data-rust/).

<video controls src="https://user-images.githubusercontent.com/2183313/139728539-d69b899f-6f9b-44cb-a1b7-bbab68beac0c.mp4"/>

## Gemaakt met

<TechWrapperSmall>
  <TechPill technology={'react'} />
  <TechPill technology={'rust'} />
  <TechPill technology={'typescript'} />
</TechWrapperSmall>


Een ambitieus, volledig open source initiatief om data eigenaarschap te vergroten en apps beter samen te laten werken.

Atomic Data


## De vraag

Als politieke partij zijn je leden je belangrijkste instrument om te groeien - zij zijn je campaginiers.
Een belangrijk onderdeel van hun activiteiten, is van deur tot deur gaan.
Ze hebben een route nodig, waar gaan ze aanbellen?
Welke gebieden hebben de meeste aandacht nodig, welke hebben het meeste potentieel?
En vooral - wat doen ze met de resultaten?
GLAPP is voor dit doel ontwikkeld.

## Onze oplossing

Wij ontwikkelden de GLAPP, de GroenLinks App.
Deze webapplicatie maakt het voor vrijwilligers makkelijk om mee te doen aan de campagne en voor campagneleiders om alles goed te laten verlopen.

## Resultaten verwerken

Resultaten van de gesprekken die vrijwilligers voeren tijdens het huis-aan-huis gaan, worden ingevoerd in de GLAPP.
Hierdoor wordt het potentieel in die buurt bijgewerkt, zodat andere vrijwilligers weten in welke gebieden zij de meeste impact hebben.
De gegevens worden altijd omgerekend naar een gemiddelde op straatniveau, dus er worden geen persoonlijke gegevens opgeslagen.

## Privacy eerst

Voor onze klant is privacy een belangrijk punt.
Daarom hebben we bij elke stap van de ontwikkeling privacy in ons achterhoofd gehouden.
Het filteren van onze logs, het beperken van toegang tot data uit bepaalde gebieden, two factor authentication, het aggregeren van resultaten in plaats van het opslaan per adres: alles om ervoor te zorgen dat er nooit persoonlijke data wordt opgeslagen of getoond aan een ongeautoriseerde gebruiker.

## Multilevel dashboards

We hebben meerdere dashboards gemaakt zodat verschillende mensen met verschillende rollen toegang hebben tot de gegevens die ze nodig hebben.
De gegevens kunnen gefilterd en geëxporteerd worden zodat ze voor elk gebruik geschikt zijn.

## Gamificatie

Het uitvoeren van repetitieve taken kan vermoeiend zijn.
Daarom hebben we de app niet alleen zo gebruiksvriendelijk mogelijk gemaakt, maar ook leuk om te gebruiken! Zowel campagnemanagers als vrijwilligers worden aangemoedigd om door te gaan met hun inspanningen door het tonen van vrolijke teksten en het toekennen van badges voor hun harde werk.

<BigImage image="cases/glapp_body.png" />

## Created with

<TechWrapperSmall>
  <TechPill technology={'react'} />
  <TechPill technology={'typescript'} />
</TechWrapperSmall>


Een webapplicatie voor Groenlinks, waarmee de online en offline campagnes zijn gemoderniseerd.

De GLAPP


## Aanvulling op online democratie

In 2015 begonnen wij [Argu](/nl/cases/argu) om mensen meer invloed te geven op onze democratie.
Met dit e-democracy platform hadden we een nieuwe manier van inspraak ontwikkeld, waardoor het makkelijker was dan ooit tevoren om van achter je computer mee te denken over de besluiten die in jouw gemeente genomen worden.
Maar we merkten dat er een probleem was: de online participatie in Argu liep _parralel_ aan de formele besluitvorming binnen gemeenten.
Hierdoor werd er niet genoeg teruggekoppeld naar inwoners over wat er met hun ideeën gebeurde.

Wij wilden daarom de data van de formele besluitvorming binnen gemeenten gebruiken om de participatie in Argu te verbeteren.
We zagen voor ons hoe we de agenda's van raadsvergaderingen konden koppelen aan vragen aan inwoners, en hoe we de besluiten en stemmingen tijdens vergaderingen met inwoners konden delen om ze optimaal op de hoogte te houden.

## Van Open Raadsinformatie naar Open Besluitvorming

We namen contact op met de [Open State Foundation](https://openstate.eu/nl/) (OSF), een organisatie die zich inzet voor open data in Nederland.
Samen lobbyden we bij de Tweede Kamer om hun data open te maken, en als eersten van Nederland mochten we aan de slag met hun gegevens.
OSF was niet geheel toevallig toen ook bezig met de VNG om de data van gemeenteraden open te maken, in het project Open Raadsinformatie.
Nadat OSF de eerste koppelingen met vergadersystemen hadden gemaakt, kregen wij van de VNG de opdracht om de zoekmachine te verbeteren en het project op te schalen.
In een jaar gingen we van 7 naar bijna 100 gemeenten die waren aangesloten.
Hiermee werd Open Raadsinformatie een van de grootste open data projecten van Nederland.
Vervolgens was OSF ook bezig met een aantal provincies. Dit project heette Open Stateninformatie.
Omdat we beide databronnen in een interface wilden tonen, kozen we er in 2019 voor om het geheel onder de naam OpenBesluitvorming te houden.
In de jaren daarna zorgden we er voor dat er nieuwe gemeenten werden toegevoegd, en onderhielden we de koppelvlakken met de vergadersystemen.

## Van OpenBesluitvorming naar PLOOI

In 2022 trad de Wet Open Overheid in werking, wat gemeenten verplicht om onder andere Raadsinformatie open te stellen.
De overheidsorganisatie KOOP kreeg de opdracht om hiervoor software te maken, genaamd PLOOI.
Het is de bedoeling dat [PLOOI](https://www.koopoverheid.nl/voor-overheden/rijksoverheid/plooi-platform-open-overheidsinformatie) binnenkort OpenBesluitvorming gaat vervangen.

## Techniek

OpenBesluitvorming bestaat uit de volgende componenten:

- Extract, Load & Transform (ETL) pipeline, geschreven in Python. Dit systeem haalt de data van de verschillende bronnen (Notubiz, Parleaus, iBabs en meer) op en zet deze om naar een gestandaardiseerd formaat.
- Zoek back-end, gebaseerd op Elasticsearch. Dit maakt het mogelijk om snel te zoeken in de data.
- Front-end, gemaakt met [React](/nl/tech/react). Dit is de interface die gebruikers zien.

Lees meer over de technologie in ons [blog](/nl/blog/publishing-dutch-governmental-meetings-as-linked-data/).

De code staat [open source op Github](https://github.com/ontola/openbesluitvorming).

<video controls src="https://user-images.githubusercontent.com/2183313/208661302-385fb00d-5be2-4c50-8159-591c7805e51c.mov"/>

## Gemaakt met

<TechWrapperSmall>
  <TechPill technology={'react'} />
  <TechPill technology={'typescript'} />
  <TechPill technology={'kubernetes'} />
</TechWrapperSmall>


Een van de grootste open data projecten van Nederland. Gestandaardiseerde vergaderdata en API van meer dan 270 gemeenten en provincies.

Software Development

Full-service

Cases