Programming with APIs

10.2 Remote APIs

# Fetch

So, now that you are aware of all the parts of the world of fetch, let's start to use the fetch() method and talk to some real APIs.

There are a few ways that the initial fetch method call can be made depending on how much information you need to pass and what needs to be customized.

//1. send a string to the fetch method
let urlString = 'http://www.example.com/api';
fetch(urlString);
//uses GET as the default method. No extra headers set. No data being sent to the server

//2. send a URL object to the fetch method
let url = new URL();
fetch(url);
//same as version 1

//3. send a Request Object that contains the url
let req = new Request(url);
//Request object can also have an options param with data and headers and non-GET method
fetch(req);

1
2
3
4
5
6
7
8
9
10
11
12
13
14

If you are only requesting to receive data from a web server (a GET request), and the server does not need an API key or any authorization headers, and no data is being uploaded then any of these can be used.

It is only if you start to upload data or customize headers that you need to add a Headers object or define the body contents.

# IDE, API, Library, Framework, and SDK

Ever wonder what the difference is between these five things - IDE, API, Library, Framework, and SDK?

While there are different interpretations with different types of programming and disagreements between developers about the finer points, here is a general reference to what each is:

IDE is an Integrated Development Environment. This is a text editor on steroids. It has features that help developers write and compile their code, as well as, manage all their projects. Usually, they have integrations with Source Control (Git) too.

API is an Application Programming Interface. This is a group of functions or server-side endpoints that let another program access a service or data-source.

Library is one or more files that use the same programming language as your project. They can be included in your project to provide pre-tested code that will speed up your development work.

Framework is similar to a library but will typically also include UI components, design assets, best practice guidance, and conventions to follow.

SDK is a software development kit. It tends to be the largest of all of these. It will often include a library or framework. Most importantly, it will include tools that you need in order to develop for your target platforms, such as a compiler or testing tools.

# JSONPlaceHolder

This is a free website that developers often use to test that their code is working.

http://jsonplaceholder.typicode.com/ (opens new window)

They have a static series of datasets that will be returned when you make fetch calls to their endpoints.

Users http://jsonplaceholder.typicode.com/users (opens new window): 10 users
Albums http://jsonplaceholder.typicode.com/albums (opens new window): 100 albums
Photos http://jsonplaceholder.typicode.com/photos (opens new window): 5000 photos
Todo http://jsonplaceholder.typicode.com/todos (opens new window): 200 todos
Posts http://jsonplaceholder.typicode.com/posts (opens new window): 100 posts
Comments http://jsonplaceholder.typicode.com/comments (opens new window): 500 comments

You can make a GET request to any of those endpoints and you will get the same response each time.

If you want to pretend to do an upload a new object of one of those types then make a POST request and include the data you want to upload in the body of your request. You will get a 201 success message from the server. It doesn't actually add anything to the dataset. It just pretends.

If you want to get the details of a single item from one of those sets then just add the id of the item you want at the end of the endpoint and make a GET request.

You can also make DELETE requests with the id to pretend to delete one or PUT or PATCH requests with the id to pretend to do an update.

# RandomFox

https://randomfox.ca/ (opens new window)

Need a random image of a fox? We got you fam!

Make a fetch GET request to this endpoint https://randomfox.ca/floof/ (opens new window) and you will be sent a JSON response that looks like this:

{
  "image": "https://randomfox.ca/images/8.jpg",
  "link": "https://randomfox.ca/?i=8"
}

1
2
3
4

# RandomDog

https://random.dog/ (opens new window)

Need an image of a random dog? Search no more!

Send a GET request with fetch to this endpoint https://random.dog/woof.json (opens new window).

You will get a JSON response that looks like this:

{
  "fileSizeBytes": 95056,
  "url": "https://random.dog/1f3fcc44-3b7c-4268-92a9-a6faa6f75547.jpg"
}

1
2
3
4

# Creating Dynamic Content

Once you have a grasp on how to match a fetch request to a web server, then it is time to start looking at how to update your webpage content with the data that came back from the webserver.

//make a request to an API that will return some JSON data
fetch(url)
  .then((response) => {
    if (!response.ok) throw new Error('Data request failed');
    return response.json();
  })
  .then((body) => {
    //body will be the object with the data from the server
    //Where do you want to add the new data?
    //Are you replacing old HTML or adding to existing HTML?
    //Is there a template to use in building the new HTML?
    //Which approach do you want to use when building the HTML?
    //Does `body` contain an array to loop through?
  })
  .catch((err) => {
    //handle the error somehow
    //tell the user
    //write a message about the failure...
  });

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

We discussed best practices for dynamically adding new HTML from an array /modules/week5/dom-2/#dynamic-html-best-practices in week 5.

# Reddit REST API

One of the cool features built into the Reddit website, is that you can take nearly every URL from any subreddit and just add .json to the end of the URL to load a JSON file version of all the information on the page.

This means that you can make fetch() calls to the home page of a subreddit and load a JSON file with a current list of all the posts to that subreddit.

Say, for example you took the LearnJavaScript subreddit page - https://www.reddit.com/r/learnjavascript/ and then replace the final / with .json at the end of that string. Also switch the www to api. Then you would have the url to use in the fetch call.

const url = `https://api.reddit.com/r/learnjavascript.json`;
fetch(url)
  .then((response) => {
    if (!response.ok) throw new Error('Unable to fetch the URL');
    return response.json();
  })
  .then((data) => {
    //data will be the JavaScript object created from the JSON string returned from reddit.com
  })
  .catch((err) => {
    console.error(err.message);
  });

1
2
3
4
5
6
7
8
9
10
11
12

Try just pasting this URL https://api.reddit.com/r/learnjavascript/.json into your browser and see the results that are loaded.

# Github REST API

Github Public Repo REST APIs (opens new window)

Github has a number of public APIs that we can use to fetch information about Repositories or Users or more. You can make a fetch call to a url like https://api.github.com/users/prof3ssorSt3v3/repos to get a JSON file with a list of the all the repos for Steve Griffith's Github account. The URL https://api.github.com/users/maddprof1/repos returns a JSON file with a list of all the repos for Tony Davidson.

So, as long as you know the username, you can get a list of all that person's public repos.

Try loading either of those URLs into the browser and look at the JSON results that are displayed.

Github also provides their own JavaScript library called Octokit (opens new window), which can be used to make calls to the Github APIs.

With the Octokit library, instead of calling fetch, you would import the library, create an Octokit object and then call the request() method, with the desired endpoint URL.

import { Octokit } from 'https://cdn.skypack.dev/octokit';
//import the Octokit function
//then create an instance of the Octokit object
const octokit = new Octokit({}); //the {} object allows for passing in of options

octokit
  .request('GET /repos/{owner}/{repo}', {
    owner: 'octocat', //will become the {owner} part of the url
    repo: 'Spoon-Knife', //will become the {repo} part of the url
    sort: 'updated',
  })
  .then((response) => {
    console.log(response.status); //status code
    console.log(response.data); //same as the data object you get in fetch from response.json()
  });

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Http Cat

Looking for a fun way to learn and remember the Http Status Codes? https://http.cat/ (opens new window) will give you the full list.

If you want a single status code with it's corresponding image, then just add the status code to the end of the url. Eg:

https://http.cat/404 (opens new window)

# The Cat API

The Cat API (opens new window) is an API that you can use to retrieve lots of cat images. You can search for cats based on breed or category of picture. You can retrieve random images or download them in sets called pages.

You need to request an API key from the company. Go here (opens new window) to request a free API.

Here is the official documentation for using the API (opens new window).

# Authentication

Every request that you make for cat images needs to include the API key in either your Request Headers OR in the query string.

//the header version
let headers = new Headers();
header.append('x-api-key', 'your api key goes here');
let url = new URL(`https://api.thecatapi.com/v1/images/search?`);
let req = new Request(url, {
  headers: header,
});

//the querystring version
let url = new URL(`https://api.thecatapi.com/v1/images/search?api_key=YOUR_API_KEY`);
let req = new Request(url);

1
2
3
4
5
6
7
8
9
10
11

When doing a search for images, if you want to filter by category then you first need to get the list of possible category ids.

//url to get the list of categories
let url = `https://api.thecatapi.com/v1/categories`;

1
2

The resulting JSON returned from the server will be an Array of objects. Each object will contain a name and the category id.

# Search Params

When your make a request for images you need to provide parameters for the search in your querystring, as part of the request url. These are your possible querystring parameters.

limit Eg: ?limit=20
page Eg: ?page=0
order Eg: ?order=ASC or DESC or RAND
has_breeds Eg: has_breeds=0
breed_ids Eg: ?breed_ids=beng,abys
category_ids Eg: ?category_ids=3,5,15

You can find more information about these on the Basics: Getting Images.

# Example Results

After you make your HTTP Request for the search you will get a JSON response from the server.

[
  {
    "id": "ebv",
    "url": "https://cdn2.thecatapi.com/images/ebv.jpg",
    "width": 176,
    "height": 540,
    "breeds": [],
    "favourite": {}
  }
]

1
2
3
4
5
6
7
8
9
10

Your JSON data will be an array of objects. Each object will have an id which will also be the name of the image. Unfortunately, there are no names associated with each image. So, if you want names then you need to generate those yourself. However, you can use the id value as the key for an image in an object.

# Download Progress

The old XMLHttpRequest object had a progressevent that let you measure the progress of your upload of data to the server. This cannot be done with fetch. However, we are able to measure the progress of a download of any file from a server.

Note: This will not work if the server is not sending the content-length header with it's response. If you are building a server-side API, it is a good idea to include content-type and content-length headers with all your responses.

The response.body object that we get as a response to calling the fetch() method is a ReadableStream object. This means that you are actually getting a stream of data coming from the web server.

When the first then() method is triggered by fetch() it means that we have a Response object. The Response object will contain all the headers from the server. However, we don't necessarily have all the contents of the downloaded file in the body property.

This is why we have to call response.json() or response.text() or response.blob(), which are all asynchronous methods that return a Promise. They are ALL waiting for the rest of the body to be downloaded before they can extract the content and trigger the next then() method.

fetch(url)
  .then((response) => {
    //we now have a Response Object
    //it has all the Headers
    //If cached, we could have the whole file
    //If not cached we are still waiting for the contents of the response.body
    //call an async method to extract the contents from response.body
    return response.json();
    //this method does not resolve until all the contents have been downloaded and extracted.
  })
  .then((content) => {
    //NOW we have all the contents extracted from the body.
  })
  .catch((err) => {
    //handle errors
  });

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

So, to determine the progress of your download from the server we need to know both the total filesize to download PLUS the amount that you have downloaded so far.

This example will be specifically for a text file, like JSON, XML, or HTML. A Binary file like an image would have a few differences with the TypedArray.

First we will access the Reader object to get each chunk of data from the stream as it is downloaded. Then we go to the Content-Length header to find the size being downloaded.

//inside the first then()
//get the reader
const reader = response.body.getReader();
//get the total size of the file from the `Content-Length` header
const totalBytes = +response.headers.get('Content-Length');
// the unary plus operator in front of response.headers.get will convert the value to a Number

1
2
3
4
5
6

MDN Reference for Unary Plus operator (opens new window)

Next we want to start reading the chunks and add each of their byte sizes to a current size variable. The current size is the number of bytes downloaded so far in the stream. We also will need an array to hold all the chunks downloaded so far. After the loop we will combine all the chunks into a single block which will be our file body.

We will use the read() method that returns an object with two properties - done and value. The done value is a Boolean indicating if you have reached the end of the Stream yet. The value is the latest chunk of data from the server.

Add the value.length to our total bytes so far and add the latest chunk to our array of chunks.

let currentBytes = 0;
let chunks = []; // our Array of data that makes up the whole body
//create a loop that will keep looping until you tell it to stop
while (true) {
  const { done, value } = await reader.read();
  if (done) {
    break; //exit the loop if the reader's `done` property is true
  }
  chunks.push(value); //add the newest chunk to our Array
  currentBytes += value.length; //figure out the new amount downloaded so far.

  console.log(`Received ${currentBytes} of ${totalBytes}`);
  //output the percentage or values somehow on your page
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14

Once the loop has been exited it means that we have the whole file. We need to take all our chunks from our Array and put them into a single 8-bit TypedArray. This will be the actual file that we can then call methods on like json() or text().

//after the loop
let wholeFile = new Uint8Array(totalBytes); // TypedArray
let position = 0; //position in the TypedArray
for (let chunk of chunks) {
  wholeFile.set(chunk, position); // place a chunk into the wholeFile TypedArray
  position += chunk.length; //move to the next position in the TypedArray
}

1
2
3
4
5
6
7

When the for of loop is completed, we will have a wholeFile TypedArray which will contain all the binary data for our Response.body.

The last step is to tell the browser how to read the file. In other words, is it a UTF-8 text file or a binary file, etc. Say that we are dealing with a JSON file. We need to turn the contents of the TypedArray binary data into a UTF-8 string. Then we can parse the String from JSON to a JS Object.

//turn the TypedArray into a utf-8 String
let str = new TextDecoder('utf-8').decode(wholeFile);
//parse the string as a JSON string and return to the next then()
return JSON.parse(str);

1
2
3
4

The returned object created by calling JSON.parse() on our utf-8 string will be passed to the next then() method and become that content variable.

# Using for await of

There is a new variation of for of called for await of. The new version still loops through iterable objects returning all the values, but the iterables can be asychronous. This means you could be using Promises or fetch calls to get the values. The for await of loop will wait for each value to resolve before it increments to the next one.

Normal loops like for, for in, while and for of all aim to finish as quickly as possible. If the values that they are looping over are asynchronous Promises then the values you see in your loop will all be unresolved Promise. Which isn't very useful.

So, in the progress example above with the ReadableStream Response.body, you could use a for await of loop instead of the while loop.

Here is the new version of all the code from above using a for await of loop.

//with for await (of)
fetch(url)
  .then(response=>{
    if(!response.ok) throw new Error(response.statusText);
    const reader = response.body.getReader();
    let currentBytes = 0;
    const totalBytes = +response.headers.get('Content-Length');
    let wholeFile = new Uint8Array(totalBytes); // TypedArray
    let position = 0;
    //loop while getting chunks
    for await ({done, value} of reader.read()) {
      //output the percentage or values somehow on your page
      currentBytes += value.length; //figure out the new amount downloaded so far.
      console.log(`Received ${currentBytes} of ${totalBytes}`);
      // add the chunk into the wholeFile TypedArray
      wholeFile.set(value, position);
      position += value.length; //move to the next position in the TypedArray

      if (done) {
        break; //exit the loop if the reader's `done` property is true
      }
    }
    //parse the string as a JSON string and return to the next then()
    let str = new TextDecoder('utf-8').decode(wholeFile);
    return JSON.parse(str);
  })
  .then((contents) => {
    //we have the contents of the file as a JS Object
  })
  .catch((err) => console.warn);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

The last two lines inside the first then() are assuming that we are dealing with a text file, specifically a JSON file.

If you were dealing with a binary file like an image then we would want to create a Blob object instead of a string.

fetch(url)
  .then((response) => {
    // ...all the other code remains the same
    let blob = new Blob(wholeFile);
    return blob; //pass the binary large object to the second then()
    // If we weren't monitoring progress then we could just use
    // return response.blob();
  })
  .then((blob) => {
    //blob is the BLOB object with the image data
    let img = document.querySelector('#targetImage');
    //use URL.createObjectURL to turn the binary data into something that can be loaded by an <img>
    img.src = URL.createObjectURL(blob);
  })
  .catch((err) => console.warn);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

If your fetch is going to be retrieving both text or binary items then you can add a test to see what kind of file is being downloaded.

fetch(url).then((response) => {
  if (!response.ok) throw new Error(response.statusText);
  //get the content-type header
  const fileType = response.headers.get('Content-type');
  //do different things depending...
  if (!fileType) {
    //missing a content-type header...
  } else if (fileType.includes(`image/`) || fileType.includes(`video/`) || fileType.includes(`audio/`)) {
    //we are dealing with an image, audio or video file
  } else if (fileType.includes('application/json') || fileType.includes('text/')) {
    //json, html, txt, xml
  } else {
    //not one of the desired types
  }
});

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# What to do this week

TODO Things to do before next week.

Read all the content from Modules 10.1, 10.2, and 11.1.
Continue working on the Hybrid Exercises

← 10.1: AJAX and Fetch 11.1: Dynamic Data →

# Fetch

# IDE, API, Library, Framework, and SDK

# JSONPlaceHolder

# RandomFox

# RandomDog

# Creating Dynamic Content

# Reddit REST API

# Github REST API

# Http Cat

# The Cat API

# Authentication

# Get the Category List

# Search Params

# Example Results

# Download Progress

# Using for await of

# What to do this week