Gatsby JS Source Plugin Tutorial Step by Step for Beginners

Last Updated: 29 September 2022

This article is aimed at helping developers get started with creating Gatsby JS Source plugins.

It took me a while to understand the hows and whys behind creating a plugin , so hopefully I can save you some time! I was building my gatsby-source-gravityforms plugin when I found that the Gatsby plugin documentation – although great – was missing the smaller nuances and explanations that I need when learning.

What I love about Gatsby JS is how much it has simplified and sped up the development process. Part of that speed is down to Source plugins taking raw data and turning it into usable GraphQL data.

If you want to know more about GraphQL, check out the website.

TLDR: https://github.com/robmarshall/gatsby-source-randomcat

What is a Source Plugin?

A source plugin is a single purpose tool to integrate a specific third party system with Gatsby. Single purpose is very important, as a plugin should only integrate with one third party system. This allows for easy mixing and matching without confusion.

A good example is the Gatsby Source WordPress plugin. It connects with the WordPress Rest API and pulls the data into Gatsby which is then converted into GraphQL nodes.

The ability to create custom source plugins for specific APIs is a massive benefit if you want to use Gatsby to its full potential. This is especially the case if you are depending on many micro services, rather than using just one back end. My article on The Rise of the Content Mesh goes into this in a little more detail.

Creating your own plugin not only makes sense when using custom data, but also due to the fact that the ecosystem is so young you may have to create your own plugin. There is a chance that what you need doesn’t exist!

What Are We Making?

The Gatsby eco-system is missing one massive thing. A plugin that pulls in cat images. An essential!

As a basic set of requirements, I have decided that this plugin must:

Connect to the thecatapi.com API
Get predetermined number of cats images from API
Add them to GraphQL to be used in a Gatsby site

If you want to look at the finished product, take a look at this repo: https://github.com/robmarshall/gatsby-source-randomcat

Installation

Gatsby CLI

We will be using the Gatsby CLI to set up and run Gatsby for this tutorial. Install it globally using the following:

npm install -g gatsby-cli

Development Project

You will also need a project to run the development from. For this we will be using the Gatsby Starter project. It includes a bare bones set up, perfect for testing with!

gatsby new my-project-name https://github.com/gatsbyjs/gatsby-starter-default

Now move into the folder and install all dependencies. We will be using Yarn from now on.

cd my-project-name
yarn

Now this is complete move back to your root.

cd ../

Starter Source Plugin

The next step is setting up a the source plugin. To help get you started I have created a basic repository containing the essential files needed to start a source plugin: https://github.com/robmarshall/gatsby-source-starter.

This should be cloned into its own project folder outside of the tester project. This means that you can work on it by itself and later include it in your other local projects.

git clone https://github.com/robmarshall/gatsby-source-starter.git my-plugin-name

I have written a full explanation of this starter plugin here: https://robertmarshall.dev/blog/gatsby-source-plugin-starter-example-breakdown/

Folder Layout

Now you should have a folder layout like this:

projects parent folder
- Gatsby starter folder
- source plugin folder

Everything is now installed, the final thing is to link the plugin to the project folder. This is only needed for development. Once the plugin is rolled out it can be installed as a normal NPM package.

Linking Your Plugin For Development

Rather than building the plugin inside a current project, we are building it separately. This allows it to be used with multiple local projects without the need of duplicating. This is possible by using NPM Link, or Yarn Link.

These two commands work slightly differently from each other (I won’t go into now) and I have always found Yarn link more reliable.

Before linking your plugin, make sure that the package.json file of your source plugin has its name updated to what you want it to be.

Yarn Link

To connect your plugin folder to your tester folder you need to do the following:

Navigate to your plugins directory
```
cd gatsby-source-starter
```
Run the Yarn link command to create a global package
```
yarn link
```
Navigate to your Gatsby tester site folder
```
cd gatsby-source-starter
```
Run Yarn link to create a symlink to the plugin folder you just created. This will allow you to move or rename the plugin folder and the link will automatically update.
```
yarn link "gatsby-source-starter"
```

This process can be reversed by:

cd /gatsby-source-starter
yarn unlink "gatsby-source-starter"

# Then remove the package:
cd /gatsby-source-starter
yarn unlink

Now your plugin is connected to your test site. Any changes made to the plugin will be reflected automatically within the project.

Lets Make Something!

To start, move into the plugin folder.

cd gatsby-source-starter

Once in the folder, you will need to install some useful packages.

Install Packages

As the plugin gets data from a REST API, you will need to use either Fetch, Axios or similar technology. For ease I have opted for using Axios.

On top of this, I like to see what is going on when the my plugins are running. To make the console log prettier I am also including Chalk.

Install both of these packages:

yarn add axios chalk

gatsby-node.js

This is where the magic happens. The code in this file is run during the process of building the site. It is used to add nodes in GraphQL as well as responding to events during the build lifecycle.

Open the file in your editor, you should see something along the lines of this:

let activeEnv =
    process.env.GATSBY_ACTIVE_ENV || process.env.NODE_ENV || 'development'

if (activeEnv == 'development') {
    process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0'
}

exports.sourceNodes = async (
    { actions: { createNode }, createContentDigest, createNodeId },
    { plugins }
) => {
    // Do your stuff!
}

The top section 6 lines make sure that node will run without any errors if we are using a fake SSL certificate within the development environment.

The bottom 6 lines introduce the sourceNodes function, which is what you will need to use to add data to GraphQL.

The sourceNodes function exposes several functions that are needed later. These include:

createNode
createContentDigest
createNodeId

For an in-depth explanation of these, take a look at the source plugin starter overview.

To get started, add these lines to the top of the page.

const axios = require("axios");
const chalk = require("chalk");
const log = console.log;

This imports the required packages, and simplifies the use of console.log throughout the plugin.

Next add the following code inside the sourceNodes function, where there is the line “// Do your stuff”.

log(chalk.black.bgWhite("Starting Random Cat Source plugin"));
if (!api) {
  log(
    chalk.bgRed("You seem to be missing API details in your gatsby-config.js")
  );
  return;
}

Get The Cats

Now that is set up, we can interact with the cat API. This is handled by the following code.

let result;

try {
  result = await axios.get(
    `https://api.thecatapi.com/v1/images/search?limit=${limit ? limit : "5"}`,
    {
      responseType: "json",
      headers: { "x-api-key": api }
    }
  );
} catch (err) {
  log(chalk.bgRed("There was an error"));
  log(err);
}

Now we have the data as an object in the “results” variable. The final step is to loop through each cat object and add it as a node to GraphQL. To do this we will use the three functions brought in by the sourceNode function.

if (result.data) {
  for (const [key, value] of Object.entries(result.data)) {
    let catObj = result.data[key];
    let newCatNode = {
      id: createNodeId(`random-cat-${catObj.id.toString()}`),
      internal: {
        type: "Random__Cat",
        contentDigest: createContentDigest(catObj)
      },
      catId: parseInt(catObj.id),
      url: catObj.url,
      width: catObj.width,
      height: catObj.height,
      breeds: catObj.breeds
    };
    createNode(newCatNode);
  }
}

This code checks that we have data to loop through. If we do then we loop through the keys of the object containing the cats. The newCatNode is used to package up each cat, and pass to the createNode function.

To create a node the object has to include:

id (using the createNodeId function)
internal
- type (A unique nodetype name)
- contentDigest (using the createContentDigest function)
Any data you want to add to the node

The ID must be completely unique. Because of this I use the plugin name, plus the ID of that particular API node. This ensures a completely unique key.

The purpose of the contentDigest is to stop Gatsby from recreating nodes if the data has not changes. It takes all of the node to be created and hashes it. If this data changes, the hash changes and Gatsby knows to update.

That’s just about it

Now this was only a very small plugin, and you may be about to build something far larger. However the process is still the same. Everything needs to be passed through the createNode function, with unique type names and IDs.

For the final plugin: https://github.com/robmarshall/gatsby-source-randomcat

For an example of it in use: https://github.com/robmarshall/gatsby-randomcat-frontend

Let me know your thoughts on this article on Twitter: @robertmars. If you want a bit more insight into a section let me know and I will get it added!