This is a follow up to the second tutorial on PWA. You can also follow this tutorial if you haven’t followed the second one or my first tutorial about PWA. We are going to focus on some new Web APIs, such as:

We add these APIs to our existing PWA for taking “selfies”. With face detection we predict your emotion, your gender and your age.

To record an accompanying text (“Speech to text”) for your “selfie”, you can easily use the Web Speech API.

Experimental Web Platform features

The above APIs only work if you have enabled “Experimental Web Platform features” in your Chrome browser via the url: chrome: // flags

figure 1

As a starting point for the tutorial, clone the following Github repository:

Then in your terminal move to the following directory:  

and install the dependencies through:

Open your app on:  http:// localhost:8080

figure 2

Public url for your mobile

There are many ways to access our localhost:8080 from a remote mobile device. You can use ngrok for this (https://ngrok.com/).

Install ngrok via:

And run the following command in your terminal:

This command generates a public url for you. Then browse to the generated url on your mobile in Chrome.

Face recognition with JavaScript

Face detection / face recognition is one of the most used applications of Artificial Intelligence. The use of face recognition has increased in recent years.

In this tutorial we will expand the existing app with face recognition that even works in the browser. We predict your emotion, your gender and your age based on your “selfie”. We are using the Face-api.js here.

Face-api.js contains a JavaScript API for face recognition in the browser. This API is implemented on top of the tensorflow.js API.

The output of this app looks as shown below:

figure 3

The steps to implement the above are as follows:

Step 1 face-api

Face-api.js contains a JavaScript API for face recognition in the browser. This face-api is already available in the folder: public/src/lib

Step 2 models

Models are the trained data that we will use to detect features of your “selfie”. These models are already available in the folder: public/src/models

Step 3 index.html

In the index.html file we import:

  • the existing facedetection.css for the styling (see listing 1);
  • face-api.min.js, this is the Face detection API for processing the model data and extracting the features (see listing 3);
  • facedetection.js where we will write our logic.

 First, import the styling into the index.html

listing 1

Place the code below in the index.html file, directly below the tag: <div id=”create-post”>

We use the existing video tag to be able to take a “selfie”(see listing 2). And a “result container” for predicting your emotion, your gender and your age.

listing 2

Place the code below at the bottom of the index.html, so that we can use the Face-detection API:

listing 3

Step 4 import models into the PWA

Here we create a separate function in feed.js to start the video streaming (listing 4).

Move the following existing code from the initializeMedia()-function into a separate function called startVideo() (see result code: listing 4)

This function is responsible for the video streaming:

listing 4

In the existing feed.js we use Promise.all to load the models for the face API asynchronously. Once these models are properly loaded, we call the created startVideo() function (see listing 5).

Place the following code at the bottom of the initializeMedia() function:

listing 5

Step 5 Implement the facedetection.js

Below are the functions of the Face detection API, which we use in our app:

                                            faceapi.detectSingleFace                                               

faceapi.detectSingleFace uses the SSD Mobilenet V1 Face Detector. You can specify the face recognition by passing the videoPlayer object an options object. To detect multiple faces, replace the detectSingleFace feature with detectAllFaces.

                                                                      withFaceLandmarks                                                                         

This function is used to detect 68 face landmarks.

                                                                       withFaceExpressions                                                                       

This function detects all faces in an image and recognizes face expressions from each face and returns an array.

                                                                          withAgeAndGender                                                                       

This function detects all faces in an image, estimates the age and gender of each face and returns an array.

Place the code below in the existing file with the name: facedetection.js, under the existing code (see listing 6).

The above functions are called in here to perform face recognition.

First, a playing event handler is added to the videoPlayer, which responds when the video camera is active.

The variable videoPlayer contains the HTML element <video>. Your video tracks will be rendered in this.

Then a canvasElement is created under the name canvasForFaceDetection. This is used for face recognition. This canvasForFaceDetection is placed in the container faceDetection.

The setInterval() function repeats the faceapi.detectSingleFace function at a time interval of 100 milliseconds. This function is called asynchronously using async / await and finally the results of the face recognition are displayed in the fields: emotion, gender and age.

listing 6

The interface we are going to build for the Web Speech API will look like the one shown below (see figure 4). As you can see, the screen contains an input field to enter text. But it is also possible to record text using “Speech to text”. For this you have to click on the microphone icon in the input field.

Under this input field you can also choose a desired language via a select box.

figure 4

The steps to implement the above are as follows:

Step 1 index.html

In the index.html file we import:

  • the existing speech.css for the styling (see listing 7);
  • speech.js where we will write our logic for the “Speech to Text” feature (see listing 9).

First, import the styling into index.html

listing 7

Place the code below in the index.html, directly below the tag: <form> (see listing 8)

In the <div id =”info”> – section the info texts are placed that can be displayed as soon as this API is used.

With the startButton onclick-event, you can start and use the API.

Finally, with updateCountry onchange-event you can select a desired language via a select box.

listing 8

Place the code below at the bottom of index.html, so we can use the Web Speech API (see listing 9):

listing 9

Step 2 Implement the Web Speech API

This code in the speech.js initializes the Web Speech Recognition API (see listing 10):

{
let interim_transcript = ”;
for (let i = event.resultIndex; i {
….
};

recognition.onend = () => {

};
}” data-lang=”text/javascript”>

listing 10

First it is checked whether the API of the ‘webkitSpeechRecognition‘ is available in the window object. The window object represents the browser window (Javascript is part of the window object).

If the “webkitSpeechRecognition” is available in the window object, then a “webkitSpeechRecognition” object is created via: recognition = new webkitSpeechRecognition ();

We then set the following properties of the Speech API:

                                                                                          recognition.continuous = true                                                                 

This property determines whether results are continuously returned for each recognition.

                                                                                        recognition.interimResults = true                                                           

This property determines whether the Speech recognition returns intermediate results.

Event handler

                                                                                                 recognition.onstart(event)                                                                 

This event handler is executed when the SpeechRecognition API is started (see Listing 10).

The following info text is displayed: “Speak now”. Then an animation of a pulsating microphone is shown: mic-animate.gif (see image)

                                                                                           recognition.onresult(event)                                                                    

This event handler is executed when the SpeechRecognition API returns a result.

The SpeechRecognitionEvent “results property” returns a 2-dimensional SpeechRecognitionResultList. The isFinal property determines in a “loop” whether a result text from the list is “final” or “interim“. The results are also converted to a string with the transcript property.

 

                                                                                      recognition.onend(event)                                                                         

This event handler is executed when Speech Recognition is terminated.

No info text is displayed, only the default icon of a microphone.

 

                                                                                  recognition.onerror(event)                                                                         

This event handler handles the errors. A matching info text is also shown with the error message.

Starting the speech recognition

Add the following code at the top of the already existing speech.js, to start the Web Speech API, using the startButton event (see listing 11):

listing 11

This code starts the Web Speech Recognition API using recognition.start(). This start() function triggers a start event and is handled in the event handler recognition.onstart() (see Listing 10).

Furthermore, the selected language is set with recognition.lang. And finally, a pulsating microphone image is shown.

Conclusion

The Web is getting more sophisticated day-by-day. More native features are being brought on board, this is due to the fact that the number of web users is far greater than native app users. The native experience users see on native apps is brought to the web to retain them there without the need to go back to the native apps.

After this introduction, you can continue with an extensive tutorial that you can find in: https://github.com/petereijgermans11/progressive-web-app/tree/master/pwa-workshop. 

See also my previous article. In this article I will discuss some advanced PWA features that provide access to your hardware APIs, like: Media Capture API, Geolocation API and the Background Sync API.

And take a look into my first article. In this article I discuss some basics behind building PWAs and then provide a comprehensive tutorial on creating your first PWA.

Follow or like me on twitter https://twitter.com/EijgermansPeter





Source link

Write A Comment