Programming with APIs

10.1 Fetch and AJAX

AJAX

AJAX stands for Asynchronous Javascript And Xml.

In the late nineties, it was not possible to download new data and apply that to the current webpage. If you wanted updated results you had to refresh the whole page.

What we will be discussing over the new few weeks is the process of how to make requests from webpages for new data or new files from a remote webserver.

It is with JavaScript that we will be requesting the new content and then injecting it into your pages.

XMLHttpRequest

The original way to make requests to a remote server to have files sent was an object called XMLHttpRequest.

let xhr = new XMLHttpRequest();
xhr.open('POST', url);
xhr.addEventListener('readystatechange', function (ev) {
  //this function runs each time readystate changes
});
xhr.addEventListener('error', function (ev) {
  //this function runs if an error happens when called xhr.send()
  //xhr.readyState - 2: sent, 3: partial response, or 4: complete
  //xhr.status - the http status code like 200, 404, 401, 500, etc
});
xhr.send(data);
//data is the data we are sending to the server - FormData or JSON string, etc

The limitations of the XMLHttpRequest object were mostly to do with security.

Along with ES6, and the addition of Promises allowed for a reworking of the whole process with a new method called fetch.

Fetch

The new method that replaces XMLHttpRequest. The good news here is that if you are already familiar with the syntax for Promises then you already know most of the syntax for fetch.

let url = 'https://www.example.com/api/';
fetch(url)
  .then((response) => {
    //we have a response from the server
    if (!response.ok) throw new Error(response.statusText);
    //if we got a bad response then we can trigger the catch
    //if the response was a JSON file then we can extract the contents with:
    return response.json();
    //extract the JSON String from the body of the response
    //convert the JSON String into an object
  })
  .then((data) => {
    //data will be the Object that was extracted from the JSON String
  })
  .catch((err) => {
    //this code runs if there was a network error preventing sending the request
    //OR if the response gave a bad status code
  });

When we send a request to a webserver it can be for ANY type of file - html, xml, json, css, js, png, jpg, avi, mpg, mp3, or anything else. When you make the request you should be aware of what kind of file will be send back.

In the example above we used request.json() to extract the JSON data from the response. If we were getting some other kind of text file, like CSS, HTML, or XML then we should use response.text(). If we were getting a binary file like an image or audio file then we would use response.blob() to get the binary data. Blob stands for Binary Large OBject.

Regardless of whether you are using fetch or XMLHttpRequest to get the file from the server we are working with the same technology. We are making an HTTP Request, which contains headers and, potentially, a body. The request is sent to a web server over the internet and then we get back an HTTP Response, which contains headers and has a body.

Think of making Request like mailing a letter to someone.

• The headers are the things that get written on the outside of the envelope.
• The body is the contents inside the envelope.
• Once the letter is sealed and addressed you stick it in a mail box. The postal service will take your letter, interpret what you wrote on the envelope and figure out how to get your letter to the correct address.
• The street address and city and province help you understand where you are sending your letter.
• The postal code is what the postal service actually uses to figure our how to route your letter.
• You don't need to understand how the letter gets to the other address.
• The envelope should have a return address on the outside of the envelope so that a response can be sent back to you.

JSON

JSON - JavaScript Object Notation is the most popular format for sending data between clients and servers. It is called JSON because it uses a JavaScript-compatible syntax for encoding the information inside the file.

Official JSON website (opens new window)

However, it is NOT JavaScript. It is just a text file with a single long string. For this exact reason, we cannot save things like functions or DOM elements inside of JSON files. We can only save String, Number, Boolean, and null (Primitive values) plus Array literals and Object literals.

The JSON file format is used by localStorage and sessionStorage to hold data in the browser. More on this next week.

The primary differences between a JS object and a JSON object are:

• All object keys must be wrapped in double quotes.
• All string values must be wrapped in double quotes.
• No trailing commas are allowed after array or object values.
• No comments are allowed in the file.
• No variable declarations.

Here is a JavaScript Object:

let obj = {
  name: 'Joanne',
  id: 123,
  active: true,
  courses: ['HRT100', 'HRT200', 'HRT300'],
};

and here is the same information as JSON:

{
  "name": "Joanne",
  "id": 123,
  "active": true,
  "courses": ["HRT100", "HRT200", "HRT300"]
}

Notice all the double quotes around all the string values. No quotes around the number or boolean values.

XML

XML - eXtensible Markup Language, created in 1998, was the first file format that was used for client-side web development for the transfer of data between clients and servers. As the name suggests, it is a MarkUp language. Angle brackets < > are used to wrap the tag names which are used to label and describe the information in the file.

The most important rule for writing XML files is Human Readable.

This one rule meant that XML rapidly became a very popular format with the thousands of new developers who started working in web development in the late 90s and early 2000s. The format was adopted by nearly all major software providers and is still widely used today.

An example of the widespread support for XML was the Microsoft adoption of it as a wrapper for all their MS Office files in Office 2007. With this release file formats changed from .doc to .docx and .xls to .xslx and so on. The name change reflected that XML had become a core part of the file format. A .docx file is really just a .doc file, wrapped inside of an XML file and then zipped. All the new features for MS Word have been added via the XML portion of the file.

JSON overtook XML as the most popular web development format during the last decade because it was Developer Readable and because the file size was noticeably smaller than XML.

Here is the same data as above, as an XML file.

<?xml version="1.0" encoding="utf-8" xmlns="https://com.algonquincollege/student">
<student>
  <name>Joanne</name>
  <id>123</id>
  <active>true</active>
  <courses>
    <course>HRT100</course>
    <course>HRT200</course>
    <course>HRT300</course>
  </courses>
</student>

You can see how much more typing is required to output that small amount of information.

Request Objects

When you make a fetch call, very often you are only providing a URL to the method. However, the fetch method will actually create a new Request() object on your behalf, using all the default values plus your URL.

If you need to you can create your own Request object.

let request = new Request();
//fetch also accepts a Request object instead of a URL object or URL string.
fetch(request)
  .then((response) => {})
  .then((body) => {})
  .catch(console.warn);

Inside the Request object you can define what the headers are and what the body is.

MDN reference for Request Object (opens new window). You can use this reference to find all the properties and methods of a Request object.

Response Objects

Typically you will be working with a response object that gets returned to the first then method after a fetch.

If you need to, like in a Service Worker when you want to send something to the browser that you are creating in response to a request, you can create your own Response Object.

let response = new Response();

MDN reference for a Response Object (opens new window). You can use this reference page to find all the properties and methods for a Response Object.

Body Objects

The Body object is the container for any file or large block of data being transferred between a client and a server. Both the Response and the Request objects have a body property that is used to access the Body object.

MDN reference for the body property of the Response object (opens new window)

The body property can contain one of the following data types:

• Blob (opens new window) - for binary files like images
• BufferSource (opens new window) - like an array of data but binary
• FormData (opens new window) - generally used for the information from a form
• ReadableStream (opens new window) - large chunk of text or numerical data
• URLSearchParams (opens new window) - Query String formatted string
• USVString (opens new window) - an encoded string

json(), text(), and blob() Methods

When a Response comes back to the browser, to either your main script or a service worker, the most common datatypes that we receive are:

• a json file
• a text file (CSS, XML, or HTML)
• an image (Blob)

Because of that, there are three specific methods that we can use to extract the contents of those files, from the body of the Response. We use the Response.json() method to convert the contents of the JSON file into a JavaScript Object. We use the Response.text() method to read the contents of the CSS, XML, or HTML file into a string. We use the Response.blob() method to extract the binary data from a binary file (like an image) into a Blob object.

All three of the methods are asynchronous and return a Promise. So, we use them inside a then() method and return the Promise that they create. That way it gets passed to the next then() in the chain. The next then() will receive the Object, String, or Binary content from the method.

fetch(request)
  .then((response) => {
    //only the first return actually runs
    return response.text();
    return response.blob();
    return response.json();
  })
  .then((body) => {
    //body is the formatted contents returned from one of those methods
  });

If we want to use the Blob as the source for an image element on our page then we need to use the URL.createObjectURL() method.

document.getElementById('dynamicImage').src = URL.createObjectURL(blob);

It is worth noting that there is also a formData() method that will extract the text from the body as if it were a FormData object. Also, you have an arrayBuffer() method available to use if you want the file contents as an ArrayBuffer instead of a Blob.

A Response object can only be used for one purpose - providing content to the webpage or saving it to the Cache API. If you need multiple copies of a response object you can use the clone() method to create that copy.

Header Objects

Inside your HTTP Request and HTTP Response, the Head holds the meta information about the request or response. What address is it being sent from, what address it is being sent to, whether it is encrypted, what type of file is contained in the body, the file size of the body, the encoding type, if it is compressed, cookies, what is the expiry date of the response, and much more. All the values in the Head are called Headers.

The QueryString is one of the Headers. It is a string that contains a series of name value pairs. There is an = sign between each name and value. And there is an & ampersand between each of the pairs. The values in the QueryString need to be URL encoded to avoid causing issues with other programs, through special characters, who might read the string.

The official object for a QueryString is a URLSearchParams (opens new window) object.

The Method verb (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS) is one of the Headers. The Method is important on the Server because it gives the server an idea of how to handle the request. You will learn more about this in MAD9124.

The Body is the container for the file that is being uploaded or downloaded. The Body can be empty if there is no file or data being uploaded or downloaded. With a GET Request the Body is always empty.

The two places that you can put information to send with a Request or Response are in the QueryString or in the Body. The QueryString is limited to about 2000 characters. The Body is limited in total size on most servers to files that are roughly 20MB. This limit can be changed through server-side settings.

The Header object is like an array of name value pairs. Many of the headers will be created by the browser and sent automatically. These are values that you are not allowed to edit. Things like the ip address cannot be altered in your script.

List of restricted header names (opens new window)

In addition, you are not allowed to programmatically edit the set-cookie or set-cookie2 headers in a Response object.

The Header object has an append method that we can use to add or edit header values.

let h = new Headers();
h.append('content-type', 'application/json');
h.append('content-length', 13234); //file size in bytes

When we are uploading a file, it means that we are adding a file to the Body and we should include headers that indicate the type and size of the attached file.

Here is a list of possible headers (opens new window)

Many of the existing headers can also be accessed through the .headers property on the Request and Response objects and its .get() method.

fetch(url).then((response) => {
  response.headers.get('Content-Type');
  response.headers.get('Content-Encoding');
  response.headers.get('Age');
  response.headers.get('Date');
  response.headers.get('X-my-custom-header');
});

The other methods on the Header object include has(), entries(), and delete(). Here is the link to the get method (opens new window).

It is worth noting that not all headers are accessible through JavaScript. This is for security reasons. Some are not able to be changed or deleted through script and some are not allow to be read.

URL and Location Objects

URLs can be thought of as just strings but they do have clearly defined parts: the domain, the path, the hash, the querystring(search), the protocol, and the port.

JavaScript has a top level object window.location that contains all these parts. If you console.log the location object you will see all the different components that make it up. One way that you could built a url is to take or create a Location object and set the values for its different components and then use the full value of the href property string.

MDN reference for Location Object (opens new window)

Another approach is to use the URL object to create one. It takes two parameters that make it work a lot more like a typical development situation, where you have a base url and then an endpoint with or without a QueryString.

const BASEURL = 'http://somedomain.com';
//a global variable that holds the base url for your API

let endpoint = '/api/people/23?api-key=8768374823';
let url = new URL(endpoint, BASEURL);
//first we have a url object to pass to a Request or fetch call.

let str = url.toString();
//here we have a String built from the url object

let req = new Request(url);
// OR
req = new Request(str);
//here we have a Request object with the url inside it

fetch(url).then((resp) => {
  //we can pass a url directly to fetch
  console.log('fetch with url', resp.status);
});

fetch(req).then((resp) => {
  //we can pass the request object that was given a url or string
  console.log('fetch with url in request', resp.status);
});

fetch(str).then((resp) => {
  //we can pass the string to the fetch
  console.log('fetch with url as string', resp.status);
});

The URL property of a Request or Location object, just like the URL object itself has a bunch of properties that you can access to directly read the different parts of a URL. (Otherwise you would have to parse the string yourself.)

• hash is the part that begins with #.
• host is a hostname with a port (if specified).
• hostname is the host without the port.
• href is the entire url string with all the parts.
• origin includes the scheme, domain, and port.
• pathname is the path and filename starting with /.
• port is the port number part of the url. Eg: 80 or 5500.
• protocol is the scheme, like http:, ftp:, https:, blob:, or file:.
• search is the querystring for the url and is either an empty string or includes the ? at the start.
• searchParams is a read-only SearchParams object with the elements in the querystring.

MDN reference for URL (opens new window)

FormData Objects

The FormData object is a great way to bundle an entire HTML form, including any hidden input elements, into a format that can be set as the value for a Request body.

let myform = document.getElementById('myform'); //a form element on the page
let fd = new FormData(myform);
//this will append all the name value pairs for the elements in the form.

2 lines of code and your entire form is bundled and ready to be uploaded.

Important

You must give each of your form input, select, and textarea elements a name attribute. Without the name attribute it will not get added to the FormData object.

You should always do validation on the data and not blindly upload it.

Now, you can also use the FormData object to bundle data that is not part of a form, or a mix of form data and other variables.

let fd = new FormData();
fd.append('name', 'value');
fd.append('email', document.getElementById('email').value);

This FormData object with the two name-value pairs can now be passed to the body property of a Request or options object for a fetch call.

URLSearchParams

The URLSearchParams object is the object that can be used to hold or build a QueryString. It can also be used as an alternative to FormData when bundling data to upload to the server. It can be used as part of the URL, in the headers, or in the body.

MDN reference for URLSearchParams (opens new window)

It is named as URLSearchParams because it represents the value held in the search property of the Location object, which is the top-level object that holds all the information about the webpage's url, hash, protocol, port, domain, path, etc.

MDN Location reference (opens new window)

It works much the same way as a Headers or FormData object do.

let search = new URLSearchParams();
search.set('key', 'value');
search.set('name', 'Karim');
search.has('key'); //true
search.get('name'); //Karim
search.sort(); //sorts the existing values by their key
search.forEach((val, key) => console.log(key, val));
search.delete('key');
search.toString(); //gives you the full string (without a `?`)

There is also an append method that works like set but will allow for duplicate entries with the same key.

It is an iterable object, which means it can be used in a for...of loop.

When you want to add a URLSearchParams string to the URL or to the body, use the toString method. Remember to add the ? in front of it if you are using it as part of the URL.

USVString

A USVString is a Unicode Scalar Value String. Basically it is a string designed for efficient text processing. It uses UTF-16 instead of UTF-8 for the holding of string values. This means that code-points that need 16-bit values to be represented can be saved as a single character. Chinese characters and Emojis both fall into this category. JavaScript will internally handle the conversion between UTF-8 strings and UTF-16 strings when you use the USVString.

When you see that a USVString is being used for an API, you can just think of it as a string that is encoded to work well in fetch calls and URLs.

Fetch All Together

Now that you know all the parts of an HTTP Request, you can build your own Request object and pass it to a fetch call.

The following example shows how to combine all the different parts into a Request object that gets passed to the fetch method. Only one value is being appended or added to the Headers, FormData, or URLSearchParms object for brevity's sake.

let head = new Headers(); //`append` what you need
head.append('x-custom-header', 'Memento');

let fd = new FormData(); //`append` what you need
fd.append('name', 'value');

let search = new URLSearchParams(); //`set` what you need
search.set('api-key', 'some-value');

let baseURL = `https://www.example.com`;
let relativePath = `./api/endpoint`;
let url = new URL(`${relativePath}?${search}`, baseURL);
// or new URL(path+querystring, baseurl).toString();
//call the toString method to create the DOMString to pass to the Request object

let req = new Request(url, {
  headers: head,
  method: 'POST',
  mode: 'cors',
  body: fd,
});

fetch(req)
  .then((response) => {
    //response to the fetch
  })
  .then()
  .catch();

What to do this week

TODO Things to do before next week.

• Read all the content from Modules 9.1, 9.2, and 10.1.
• Continue working on the Hybrid Exercises