MAD9014
Programming with APIs

11.1 Fetch

# More APIs

Regardless of which APIs you are dealing with the process of building dynamic webpages with the data from the API will be pretty much the same.

  • Wait for your page to finish loading
  • Add event listeners to your interface that will let the user interact with your page
  • Do you need the user to fill out a form before getting the data?
  • Are you getting information from the querystring or hash value to add to the request?
  • Does the API need any special querystring or header values like an API key?
  • Do you need to register with the API for a key?
  • Build a Request object that contains all the necessary information and send the request
  • When the Response returns check the status code and content-type of the response.
  • Extract the text() or json() from the response object.
  • Use a common reusable function for building your new HTML content.
  • Does your new content need an event listener? Add it to the containing element.
  • Do you need to embed things like ids in the HTML? Use data- properties.
  • Which approach do you want to use for building the HTML? createElement? HTML String? map() through Array?
  • Is your new content replacing old content or being added?
  • If adding new content, should it be at the top or bottom of the containing element?

Answer these questions and build your code so it runs generally in this order.

The way you organize this code can be done in a variety of ways, with or without modules. Namespaces are a good idea either way.

# TMDB

The MovieDataBase (opens new window) is a free API that you can use to get data about TV Shows, Movies, and Actors. The home page for the https://www.themoviedb.org/ (opens new window) site.

The API requires you to sign up for a developer key. The unique key that they give you allows you to make AJAX calls to the API to search for movies, get movie recommendations, get movie reviews, and lots of other images and information about movies.

Readme file about how to get started with theMovieDatabase.org API: https://gist.github.com/prof3ssorSt3v3/bd984b862b71ebbe0a65aee4f2e6746d (opens new window)

# TMDB Base URLs

When you are making API calls to the data from TMDB then your URL endpoints will begin with https://api.themoviedb.org/3/. The part that comes after this defines which type of search you are doing.

All requests from the API need to include an API Key and it must be part of the query string. ?api_key=<<api_key>> The <<api_key>> part gets replaced with your own API Key.

# TMDB Images

Depending on whether you are trying to get an image for a movie or an actor there are different properties that hold the name of the image.

The URL for all images starts with https://image.tmdb.org/t/p/. Then you need to specify one of the specific widths. Different kinds of images are available in different sizes. An example size is w185. After the base url and the width property, you need to add the

To get a list of the possible values for the width property make a call to this URL- https://api.themoviedb.org/3/configuration?api_key={apikey}. Put your own API key in the query string.

# Common API Endpoints

https://developers.themoviedb.org/3/movies (opens new window) reference for getting details about a movie based on its ID.

https://developers.themoviedb.org/3/search/search-movies (opens new window) reference for doing a search for movies based on a keyword that matches against the title of the movie.

https://developers.themoviedb.org/3/search/search-tv-shows (opens new window) reference for doing a search for tv shows based on a keyword that matches against the title of the show.

https://developers.themoviedb.org/3/search/search-people (opens new window) reference for doing a search for people based on a string that matches against their name.

https://developers.themoviedb.org/3/people/get-person-combined-credits (opens new window) reference for doing a search for all TV Shows and Movies that are connected with an actor.

https://developers.themoviedb.org/3/people/get-person-movie-credits (opens new window) reference for doing a search for all Movies that are connected with an actor.

# Datasets about Movies

When you make API searches for Actors, Movies, and TV Shows, the resulting JSON data will have an array called results. Inside that array will be a series of objects. These objects could contain objects that are referencing either a TV Show or a Movie. It is important to note the destinction between them because they have some different properties. Watch for a property called media_type set to either movie or tv. You can also watch for release_date vs first_air_date. The release_date is for movies. TV Shows have name properties and Movies have title properties.

# OpenWeather API

The OpenWeather API (opens new window) is another API, with a free usage tier, that you can use to get the current weather, historical weather data, and forecasts. The API page (opens new window) shows the MANY different APIs that you can use.

Like most APIs, you need to sign up for a key (opens new window) before you can start using the data.

The API Key needs to be included, for any API call, in the querystring as the value of the appid property.

let url = `http://api.openweathermap.org/data/2.5/forecast?id=524901&appid=${APIKEY}`;
//the actual URL will vary depending on whether you are using the free or paid service
//and depending on which API you are calling.
1
2
3

A good API to start with is the one called ONE CALL 3.0 (opens new window).

The One Call API provides the following weather data for any geographical coordinates:

  • Current weather
  • Minute forecast for 1 hour
  • Hourly forecast for 48 hours
  • Daily forecast for 8 days
  • National weather alerts
  • Historical weather data for 40+ years back (since January 1, 1979)

# Random Data API

This is a great Canadian-made free API that let's you generate lots of data for testing purposes. This includes user data. Website (opens new window)

All API calls use a base URL of https://random-data-api.com/api/v2/

Then add the resource that you want - /users, /addresses, /banks, /beers, /appliances, /credit_cards, or /blood_types.

Then to your queryString add size=20 where the value is the number of objects you want in the JSON returned. While JSON is the default return type, you can ask for XML by adding response_type=xml to the queryString.

# RoboHash API

The Robohash API (opens new window) is a separate API but it used by the Random Data API as the source for avatar images within the users dataset.

The API accepts any string as a parameter. That string gets hashed into a repeatable value, so that each time you pass the same string you will get the same result.

There are parameters that you pass in the query string and based on those values, the API will return a robot, an alien, a monster, or a kitten image that can be used as an avatar.

# Google Maps

The Google Maps API for Web (opens new window) has versions for Android, iOS, and the web. While you do need to register an account and include a payment method, it has a free usage tier for accessing map data and directions.

The Google API is a bit different than the other ones that we have looked at so far.

It still works by sending HTTP Requests and handling HTTP Responses.

However, there is also a Google Maps script that we have to use, and you have to have an account that you register and projects that you define and manage through a web dashboard console.

First you would register for an account with Google Maps.

Then you would create a project in the Dashboard console (opens new window). Here are the instructions (opens new window) for setting up your account and project.

After the setup through Google is completed, then you can start building your website.

You need to include a link to the Google Maps script in your page(s).

<script async defer src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&callback=initMap"></script>
1

Put your own API KEY in the link to the script.

Finally, you will use the functions and objects defined in the Google Maps API script to build, move, zoom, and interact with the map images. So, instead of using fetch() to get the images, we call on the Google Maps API methods and them them figure out which map image tiles should be loaded in the browser.

Here is the Getting Started (opens new window) script introduction for working with Google Maps in your JS files.

Their basic script looks like this:

let map;

function initMap() {
  map = new google.maps.Map(document.getElementById('map'), {
    center: { lat: -34.397, lng: 150.644 },
    zoom: 8,
  });
}

window.initMap = initMap;
1
2
3
4
5
6
7
8
9
10

Note that map is a global variable. It will be the map object that you use for adding any event listeners or changing properties of the map.

They are calling a method new google.maps.Map() to get started. They also add the function initMap as a property of the window object and with the same name.

This means that in the google maps JS file that you loaded before writing your own script there is a namespace object called google. It has a property called maps and inside maps is a function called Map.

Roughly it would look like this:

const google = {
  maps: {
    Map: function (DOMElement, props) {
      //creates the Map object and returns a reference to it
      //Do NOT call this method until after the DOMContentLoaded event
    },
  },
};
1
2
3
4
5
6
7
8

The Map() function needs to be passed a DOMElement (usually a <div>) which will be where on the page your Map will be appended. Use CSS to size your containing element. Always set the height for your <div> to 100% in your CSS.

Map also accepts a props object. There are default values for many properties but the required ones are center and zoom. center is an object with lat and lng properties. These are the latitude and longitude values that will be used as the center of the map. The zoom value is an integer between 1 and 17. It indicates how much to zoom in. 1 showing the whole world. 17 showing a building or two.

You can rename the function if you want. You just need to change the callback=initMap value in the querystring that is part of the src of the <script> tag used to load the google maps JS.

Full Google Maps Playlist (opens new window)

# Uploading Data with Fetch

When you need to send data from the browser to some web server (remote API), you need to package the information in a way that can be read when it gets to the server.

The data that you want to upload can be put into a few places.

  • Cookies: Usually small identifier strings like a session id.
  • Headers: One custom header for each value.
  • QueryString: Name value pairs. Limited to ~2000 characters.
  • HTTP Request Body: Any time you have a large amount of text, or files, or don't want to show the value in the querystring.

Here are a few ways that the data can be sent in the body.

  • JSON String
  • A www-form-urlencoded string
  • A FormData object
  • A plain text string

When you are creating forms in your HTML pages you can do the last 3 of these like this:

<form enctype="application/x-www-form-urlencoded" action="/endpoint" method="POST"></form>
<form enctype="multipart/form-data" action="/endpoint" method="POST"></form>
<form enctype="text/plain" action="/endpoint" method="POST"></form>
1
2
3

Basically the encoding type and method attributes tell the browser how to format the information that will be send to the server. The action attribute defines the URL where the data will be sent.

If you need to upload a file then the method must be POST and the enctype must be multipart/form-data.

In JavaScript, we can create a FormData object and append name-value pairs of text or even append files. We can just set the value of the Request body to be the FormData object, or a JSON String, or a plain text string, or a x-www-form-urlencoded string. This last one means that you are writing all the data you want to upload to look like the QueryString. The full FormData notes are in week 10.

Whichever one of these methods you choose in JavaScript, you should add a Content-Type header to your Request as well. The code sample below shows all the different values for the header. Choose ONE of these.

let h = new Headers();
h.append('Content-Type', 'plain/text');
h.append('Content-Type', 'application/json');
h.append('Content-Type', 'application/x-www-form-urlencoded');
h.append('Content-Type', 'multipart/form-data');

let req = new Request(url, {
  headers: h,
  method: 'POST',
  body: YourDataVariableGoesHere,
});
1
2
3
4
5
6
7
8
9
10
11

By setting this header you are telling the web server how it should parse the data you are sending.

# REST

REST stands for REpresentational State Transfer. It is a term used to describe what we are calling AJAX or fetch. We are using HTTP requests to ask a web server to send us (a web client) information.

REST needs there to be a client and a server.

The client makes the requests and the server sends the responses.

The State refers to the information, rather the current state of the information that you are requesting.

the Transfer is the sending of information between the client and server.

One very important aspect of the process is that neither side maintains a memory of previous requests and responses. We gain a lot of efficiency by the server not having to keep track of whether or not a particular client has made the same request previously. This lack of memory of state meant that HTTP was very durable and resilient.

When you make an HTTP Request, the request is broken up into small packets. Each packet is numbered and keeps a record of the address that it came from and where it is going. Not all packets have to follow the same path. Routers all over the world are constantly trying to find the most efficient way to get information from point A to point B.

When all the packets reach their destination they are reassembled in order.

Now, because there is no maintaining of history of requests on the server or client, it has lead to the development of lots of other services. There are session management techniques in all the server-side programming languages to let developers keep track of users. There are also Proxy Servers and CDN (content delivery networks) that cache copies of requests and distribute copies of files across different zones so that it will be more efficient when requests are made.

For example, Netflix does not have a single server that is waiting for requests for video files. They have redundant servers with copies of the content spread all over the world. When a video is trending more copies of that video file are transferred to the servers that provide content to the areas where the video is trending. This is an example of a CDN.

# Service Workers

A recent feature added in browsers are Service Workers. These are scripts that run in the browser and act like Proxy Servers. They handle all the outgoing and incoming traffic to and from your browser for a specific domain. They are able to cache copies of files in case the browser is offline.

If you need to build a web app that can still run some functionality when offline then you need a Service Worker.

If you need to build a web app that can coordinate user activities across multiple tabs then you probably need a Service Worker.

If you want to manage caching of files on the client-side for improved performance then you should have a Service Worker.

# Adding Service Workers to a Website

When you add a service worker from your script, it will typically manage all the webpages loaded from that same origin. While it is possible to use multiple service workers for a single website by putting them each in charge of their own folders (scope), this can be difficult to manage.

Your service worker will be a single JS file called sw.js, placed at the root of your website.

From your own main.js file, you can register the service worker to be in control of the web pages for the current origin. Any time a web page from that origin is loaded on the user's browser, the service worker will automatically start up and oversee all communications from the web page to any web server.

//main.js

if ('serviceWorker' in navigator) {
  //always check for support first
  navigator.serviceWorker
    .register('sw.js')
    .then(() => {
      //successfully registered the service worker
    })
    .catch((err) => {
      //failed to register the worker.
      //likely a script error in the worker.
    });
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Basic Events in Service Workers

Inside the Service worker file sw.js there are four events that we will listen for - install, activate, fetch, and message.

Since service workers are a different execution context than web pages themselves, they don't use the window or document objects. To add the event listeners we use self to refer to the worker.

//sw.js

self.addEventListener('install', (ev) => {
  console.log('service worker has been installed.');
});

self.addEventListener('activate', (ev) => {
  console.log('service worker has been activated.');
});

self.addEventListener('fetch', (ev) => {
  console.log('web page has requested a file.');
  //this event fires for EVERY FILE REQUEST
  // images, html, json, css, fonts, etc
});

self.addEventListener('message', (ev) => {
  console.log('service worker has received a message.');
});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

If you want to learn more about Service Workers you can watch this playlist (opens new window). (15 videos)

If you want to learn more about Messaging between windows, tabs, and service workers, then you can watch this playlist (opens new window). (3 videos)

See the notes in week 13 about the Cache API to learn about how your Service Worker could save and manage files locally.

Note: There is a library from Google called WorkBox which is commonly used to manage service workers and their caching of files. Learn more here (opens new window)

# What to do this week

TODO Things to do before next week.

  • Read all the content from Modules 11.1, 11.2, and 12.1.
  • Finish working on the Hybrid Exercises
Last Updated: 6/9/2023, 6:25:41 PM

← 11.2: Fetch Security →