This article covers

  • Returning the response headers needed to enable the SharedArrayBuffer in Firefox.
  • Accessing and modifying the pixel information from an image file directly in your web page.
  • Creating a WebAssembly module that uses pthreads (POSIX threads).

WebAssembly modules leverage several browser features in order to support pthreads: The SharedArrayBuffer, web workers, and Atomics.

The SharedArrayBuffer is similar to the ArrayBuffer that WebAssembly modules normally use, but this buffer allows multiple threads to share the same block of memory. Each thread runs in its own web worker and Atomics are used to synchronize data between the threads in a safe way.

I won’t cover Atomics in this article so. If you’d like to learn more, you can visit the following check out Mozilla’s docs.

In January 2018, the Spectre/Meltdown vulnerabilities forced browser makers to disable support for the SharedArrayBuffer. Since then, browser makers have been working on ways to prevent the exploit. By October 2018, Chrome was able to re-enable it for desktop versions of its browser by using site isolation.

Firefox chose a different approach to prevent the exploit. Rather than site isolation, they only allow access to the SharedArrayBuffer if two response headers are provided. This new approach went live with Firefox 79 that was released on July 28th, 2020.

NOTE: At the time of this article’s writing, the response header approach isn’t needed by Chrome, or Chromium-based browsers, like Edge, because the desktop versions use site isolation. According to the following article, Chrome will require the response headers shown in this article in the near future too: https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/_0MEXs6TJhg

In this article, you’re going to learn how to enable the SharedArrayBuffer in Firefox so that you can use pthreads in a WebAssembly module. You’ll learn how to load an image file from the device and access the pixel information so that you can adjust the image in the browser. Finally, you’ll see how pthreads can be used to speed up the processing.

Suppose you have a web service that lets your users upload an image to your server and download a modified version with various filters now applied. The web page works fine but is a little slow if the images are large because of all the data being uploaded and then downloaded once the modifications are complete. Using all that bandwidth also costs your customers money, so you’d like to move the processing from the server to the device.

Rather than jumping in with both feet, you decide that it would be best to create a prototype in order to compare the speed of using JavaScript directly, using a WebAssembly module but without using pthreads, and then using a WebAssembly module with pthreads.

To keep things simple for this test, the image will be converted to grayscale and then the web page will display each image along with how long it took to modify them as shown in the following image:

image customization performance

As shown below, the steps for building this web page are:

  1. Modify your web server to return the necessary response headers to enable the SharedArrayBuffer in Firefox.
  2. Create the web page and add the ability to load an image file from your user’s device.
  3. Adjust the image using JavaScript for a comparison to the WebAssembly versions.
  4. Create a WebAssembly module that modifies the image without using threads and with threads to see the difference between the two.

application workflow
As the following image shows, your first step towards building this web page is to modify your web server.
Modifying the web server

1. Modify the Web Server

In order to enable the SharedArrayBuffer in Firefox, you need to specify two response headers:

When you use the require-corp value and try to load a document from a cross-origin location, like a CSS file from a CDN for example, that location will need to support Cross-Origin Resource Sharing (CORS). If you trust that location, you also need to mark that file as loadable by including the crossorigin attribute. You’ll see the crossorigin attribute used later in this article.

For this article, I’m going to use Python as the web server, but you can use any web server you’re comfortable with.

Create a frontend folder for the web page files that you’ll create in this article.

If you choose to use your own web server, feel free to skip to the end of this section and continue on with section “2. Create the web page” once you’ve adjusted your web server to return the response headers with the required values.

You will need to modify the wasm-server.py file that was created in the “Extending Python’s Simple HTTP Server” article. If you didn’t follow along with that article, the files can be found here:

Place the wasm-server.py file in the frontend folder and then open it with your favorite editor.

In the end_headers method, there’s a comment showing the syntax necessary if you wanted to include a CORS header. This is where you’ll add the COOP and COEP headers.

Delete the two comments above the SimpleHTTPServer.SimpleHTTPRequestHandler.end_headers(self) line of code and replace them with the following:

Your class should now look like the following code snippet:

Save the wasm-server.py file.

As shown in the following image, your next step is to build the HTML file that will allow a user to open an image file. The HTML file will have four canvas tags with one to show the original image and three to show the grayscale images along with how long the different approaches take to complete.

Creating the web page

2. Create the Web Page

Most of the HTML for the web page is boilerplate code, so I’ll only point out key items and will present the full file at the end of this section.

You’ll be using the Bootstrap web development framework because it offers a professional-looking web page, which is faster than styling everything manually. The files needed for Bootstrap will loaded from a CDN rather than having to download the libraries.

Because you’ll be linking to files from a CDN, they’re not coming from the same origin as your web page and will be blocked by default because you specified the require-corp value for the COEP header. You can include the crossorigin attribute in the links for the CDN files in order to allow them to be downloaded. As an example, the following JavaScript link specifies the crossorigin attribute because it’s hosted on a Google server:

WARNING: You only want to include the crossorigin attribute for files that you know are safe because you are not in control of the server that they’re coming from.

As shown in the following code snippet, the body tag will be given an onload attribute so that the function you specify, initializePage in this case, will be called when the page first loads. You’ll use this function to wire up an event handler so that you can respond when the user selects a file.

For the file upload control, you’ll use the input tag with the type file. Rather than the standard file upload control with a browse button and label indicating which file was selected, as shown below, you’ll wrap the control in a label styled as a button and hide the input control.

browsing for files button
Note that hiding the input control, wrapping it in a label, and styling the label as a button is optional. The file upload will work just fine if you don’t make any changes to the input control so long as the input control is of type file.

You’ll also include the accept attribute for the input tag to ensure only image files are selected. The upload button’s code is shown in the following snippet:

Your web page will have four canvas tags. The canvas tag allows you to draw 2D or 3D graphics on your web page and can even be used for animations. For this article, you’ll use it to display the selected image on the first canvas and then the modified images on the other three canvasses. If you’d like to learn more about the canvas tag, you can visit the following web page: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/canvas

Finally, the HTML for the web page will end with two JavaScript file links. The first JavaScript file, pthreads.js, you’ll create in a moment. The other JavaScript file will be created by Emscripten at the same time as it creates the WebAssembly module. That file handles loading in the WebAssembly module for you, has a number of helper functions to make working with the module easier, and supports various features that might have been enabled when the module was compiled.

Create a file called pthreads.html, copy the following HTML into it, and then save the file:

Now that you’ve created the web page, you need to write the JavaScript that responds to the user choosing a file.

Create the JavaScript to Load an Image From Your User’s Device

In your frontend folder, create a js folder.

In the js folder, create a file called pthreads.js and open it with your favorite editor.

The first thing you need to do is create the initializePage function that will be called when your web page loads. In this function, you’ll attach to the file input control’s change event so that when the user chooses a file, your processImageFile function will be called. Add the following code snippet to your pthreads.js file:

Next, you need to define the processImageFile function. You’ll create a FileReader object to read in the selected file as a data URL. Once the file’s contents have been loaded, you’ll pass the data URL that was generated to the renderOriginalImage function. Add the contents of the following code snippet to your pthreads.js file after the initializePage function:

The next function that you’re going to create is renderOriginalImage. This function will first draw the image that the user selected to the first canvas, and then it’ll display the dimensions of the image below the canvas. Next, it will pull the pixel data from the canvas and pass that off to be adjusted and displayed on the other canvasses.

The full version of the renderOriginalImage function will be shown in a moment but first, the aspects of the function’s code will be explained.

As shown in the following snippet, the first step to drawing the image onto the canvas is to create an instance of an Image object and have it load the data URL by setting the src property. You then respond to the onload event:

Within the onload event, you’ll get the 2D context from the canvas and use the context’s drawImage function to draw the image that was loaded. The instance of the Image object will also contain information like the width and height of the image that you’ll display below the canvas as shown in the following snippet:

The final portion of code within the onload event of the Image instance is shown in the following code snippet. The code will grab the pixel data from the canvas using the context’s getImageData function and will pass that off to the adjustImageJS and adjustImageWasm functions to modify and display the results.

One thing to note about the following code snippet is that the adjustImageJS and adjustImageWasm functions are asynchronous and will finish at some point after the onload event completes. The functions are asynchronous so that the JavaScript code isn’t blocking the browser while the modifications are being made. All three functions will execute at the same time and the canvasses that are ready will be drawn when the data is received rather than in the sequence that the functions were called. The browser will also remain responsive to user input.

If you did want to wait for the functions to complete before exiting the onload event, you can pass the result of each function to a variable (for example: const promise1 = functionCall();). Using this approach will allow each function to execute concurrently and then you can await the variables (for example: await promise1;). The following web page has more information on the async and await keywords if you’d like to learn more: https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous/Async_await

The full renderOriginalImage function is shown below. Add it after the processImageFile function in your pthreads.js file:

Now that you’re able to display the image that the user selects, the next step is shown in the following image where you’ll adjust the image data and display the results using only JavaScript. This will give you a comparison to see what the difference is between the JavaScript approach and the two WebAssembly approaches.

Adjusting images using javascript

3. Adjusting the Image Using JavaScript

The renderOriginalImage function that you created calls the adjustImageJS function to have the user’s selected image adjusted using JavaScript. In the adjustImageJS function, you’ll create a copy of the original image data using the a Uint8ClampedArray which ensures each value is an integer in the range of 0 to 255. If a value is not an integer, it’s rounded to the nearest integer. More information on this array can be found here if you’re interested: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8ClampedArray

Once you have a copy of the image data, you’ll pass that off to the adjustPixels function telling it to loop from the first pixel to the last. The function will adjust the pixels in the Uint8ClampedArray instance that you pass in.

Before and after the adjustPixels call, you’ll grab the current date and time to determine how long the function takes to execute.

Finally, you’ll call the renderModifiedImage function to have the modified pixels rendered on the desired canvas.

Add the adjustImageJS function shown in the following code snippet after the renderOriginalImage function in your pthreads.js file:

Each pixel in the image data has four bytes (one for each color and the alpha channel). The adjustPixels function will loop from the first index specified to one less than the last index specified and will step through the data in increments of four. Each time through the loop, the adjustColors function is called to adjust the colors at that index.

Add the adjustPixels function, that’s shown in the following snippet, after the adjustImageJS function in your pthreads.js file:

The adjustColors function grabs the Red, Green, and Blue values and averages them out. Then it applies the calculated color to the Red, Green, and Blue values to create the grey. The alpha channel isn’t adjusted.

Add the adjustColors function, from the following code snippet, after the adjustPixels function in your pthreads.js file.

Now, to have the modified image data rendered to a canvas, you’ll create the renderModifiedImage function. The function will get the context of the destination canvas and, from that context, it will get the image data of that canvas. Next, it will overwrite that image data with the modified data and then put the new image data back into the canvas to have it drawn.

Lastly, the function will display how long the calling code took to execute the modifications.

Add the renderModifiedImage function, shown in the following snippet, after the adjustColors function in your pthreads.js file:

Now that you have the code that adjusts the original image using JavaScript, the last bit of JavaScript code that you need to create is the adjustImageWasm function. This function will pass the original image data to the WebAssembly module, have the module modify the image, and then retrieve the modified data from the module to be displayed on the desired canvas.

The full version of the adjustImageWasm function will be shown in a moment. I’ll explain the sections of the function’s code first.

The first thing the function needs to do is allocate a portion of the module’s memory to hold the image data. Then you copy the image data to that location in the module’s memory as shown in the following snippet:

The next step is to call the desired function based on the destinationCanvasId parameter that’s passed to the function as shown in the following snippet:

The code copies the modified image data from the module’s memory and then tells the module that it can release the memory that was allocated for the image data as shown in the following snippet:

Finally, the renderModifiedImage function is called to display the results of the modification to the appropriate canvas.

The following code snippet shows the whole adjustImageWasm function that you need to place after the renderModifiedImage function in your pthreads.js file:

Save the pthreads.js file.

With the web page now created, your next step as shown in the following image, is to create the WebAssembly module.

Adjusting the image using WebAssembly

4. Create the WebAssembly Module

To create the WebAssembly module, you’re going to write some C++ code and compile it to WebAssembly using Emscripten.

Create a source folder that’s at the same level as your frontend folder.

In the source folder create a file called pthreads.cpp and then open it with your editor.

You’ll start the pthreads.cpp file with the headers needed for the uint8_t data type (cstdio), the std::chrono library (chrono) to help track how long the image manipulation takes, pthread.h for pthread support, and emscripten.h for Emscripten support. You’ll also add an extern “C” block around the code so that the compiler doesn’t adjust the function names.

Add the code in the following snippet to your pthreads.cpp file.

Add the following global variable within the extern “C” block in your pthreads.cpp file. The variable will be set once execution completes and will be returned when the GetDuration function is called.

After the execution_duration global variable, and within the extern “C” block of your pthreads.cpp file, add the functions in the following code snippet that will allocate space in the module’s memory and free that memory respectively:

After the FreeBuffer function, and within the extern “C” block of your pthreads.cpp file, add the following function that will tell the caller how long it took for the code to execute:

Aside from slight syntax differences, the following two functions are the same as the JavaScript versions you created earlier. The AdjustColors function adjusts the colors for a specific index and the AdjustPixels function loops through a range of indexes calling AdjustColors for every fourth index.

Add the code in the following snippet to your pthreads.cpp file after the GetDuration function and within the extern “C” block:

The next function that you’ll create is the AdjustImageWithoutUsingThreads function. This function will grab the current time, call the AdjustPixels function telling it to modify all the pixels in the image, and then it will grab the current time again in order to calculate the execution’s duration. The duration is then placed in the execution_duration global variable.

Add the code in the following snippet to your pthreads.cpp file after the AdjustPixels function and within the extern “C” block:

Your next step is to define an object (thread_args) that you’ll use to pass information to the threads that you create. This will hold a pointer to the image data, the index for where to start adjusting the image, and an index for where to stop.

Following the definition of the thread_args object, you’ll create the thread function itself (thread_func). The thread_func function will call the AdjustPixels function passing it the values it receives from the thread_args parameter value.

After your AdjustImageWithoutUsingThreads function, and within the extern “C” block, add the code in the following snippet to your pthreads.cpp file:

The final function that you’re going to create is the AdjustImageUsingThreads function. For the threading in this function, you’ll create four pthreads because there are four bytes per pixel (RGBA). You can use any number of threads so long as you divide up the chunks so that each grouping keeps that in mind.

At the beginning of this article it was mentioned that WebAssembly pthreads make use of existing browser features. Each pthread will run in a web worker. Something to be aware of is that web workers have overhead and take some time to start up. It’s not usually noticeable if you only have a couple of web workers but the startup time becomes noticeable as the number of threads increase.

As you’ll see in a moment, when you compile this code, you’ll tell Emscripten how many threads you want. When the WebAssembly module is being instantiated, all of the threads that you asked for are spun up and placed into a thread pool for use when you’re ready for them.

You’ll want to be as precise as possible with how many threads you request because it wastes device resources if some are spun up and never used. Also, depending on how many threads you request, you may notice a short delay before your module is ready to be interacted with.

My recommendation is that you test to see what you feel is the right balance between startup time and processing power.

The full version of the AdjustImageUsingThreads function will be shown in a moment.

As shown in the following snippet, the AdjustImageUsingThreads starts off the same as the AdjustImageWithoutUsingThreads function:

Next, you’ll declare a few variables:

  • The first variable is an array of pthread_t that will hold the thread ids of each thread that’s created.
  • The second variable is an array of thread_args that will tell each thread function which grouping of indexes to modify.
  • The third variable holds the number of bytes that each thread is to modify.

The next step after declaring the variables is to create a loop that will set the values for the thread_args array at that index. Then the loop will create the thread. At the end of the loop, the next loop’s start index is the index where the current loop stopped.

The following snippet shows the variable declaration and thread creation loop:

Next, the function will loop again but this time to wait for each of the threads to finish as shown in the following snippet:

The function finishes off the same as the AdjustImageWithoutUsingThreads function does by calculating how long it takes the code to execute.

The full code for the AdjustImageUsingThreads function is shown in the following code snippet. Add the following code after the thread_func function, and within the extern “C” code block of your pthreads.cpp file:

Save the pthreads.cpp file.

With the C++ file created, your next step is to compile it into a WebAssembly module.

Compiling the code into a WebAssembly module

The Emscripten version used for this article was 1.39.20. If you don’t already have Emscripten installed on your machine, you can download it from the following web page by clicking on the green Code button and then clicking Download ZIP: https://github.com/emscripten-core/emscripten

The installation instructions for Emscripten can be found here: https://emscripten.org/docs/getting_started/downloads.html

Some of the C++ features used in the code you just wrote, like the uint8_t data type, require a minimum of C++11. By default, Emscripten’s front-end compiler uses C++98 but this can be changed by specifying the -std=c++11 command line flag.

Memory growth is slow but you need to allow the memory to grow (-s ALLOW_MEMORY_GROWTH=1 command line flag) because you don’t know what image sizes your users will try to upload. What you can do though is try to pick a large enough initial memory size that seems reasonable and, if the user’s file exceeds that, then let the memory grow. Perhaps display a warning to the user if the file is larger than the initial memory size because you’ll know how many bytes the file has before you ask the module to allocate the memory for it.

To specify an initial amount of memory, as bytes, you’ll use the -s INITIAL_MEMORY flag. By default, this value is 16 MB (16,777,216 bytes). For this module, you’ll set the initial memory to 64 MB (67,108,864 bytes).

To enable pthread support you need to specify the -s USE_PTHREADS=1 flag. You also want to use 4 pthreads so you need to tell Emscripten that by using the -s PTHREAD_POOL_SIZE=4 flag.

There are various levels of optimization that are available. You’ll use the -O3 level (O is not a number, it’s a capital o).

The last item that you’ll specify is what type of output you want and where you’d like it to be created by using the -o flag. You’ll have Emscripten create its JavaScript code and the WebAssembly module in your fontendjs folder.

To compile your pthreads.cpp file into a WebAssembly module, open a command prompt, navigate to your source folder, and then run the following command (note that the line wraps here but it should be all one line at the command prompt):

You’ll likely see a warning about the use of the ALLOW_MEMORY_GROWTH flag but there shouldn’t be any errors and you should now have three new files in your frontendjs folder:

  • emscripten_pthread.js
  • emscripten_pthread.wasm
  • emscripten_pthread.worker.js

Now that your web page and WebAssembly module are created, it’s time to test the web page to see the results.

Viewing the results

If you’re using the Python web server extension that you modified earlier, open a command prompt, navigate to your frontend folder, and then run the following command:

Open Firefox 79 or higher and type http://localhost:8080/pthreads.html into the address box to see your web page:

running application
Click the Upload button to launch a File Upload window similar to the following image. Select an image and press the Open button.
uploading file from file system

As shown in the following image, the web page will display the original image, the modified images, and the execution duration for each method used.

uploading image and measuring performance

Based on these results, you can see that the WebAssembly non-threaded version is twice as fast as its JavaScript counterpart. The WebAssembly threaded version is five times faster than the JavaScript version.

Summary

As you learned in this article, as of Firefox 79, it’s now possible to use WebAssembly pthreads so long as you specify the Cross-Origin-Opener-Policy (COOP) response header with the value same-origin and the Cross-Origin-Embedder-Policy (COEP) response header with the value require-corp.

Because of the COEP response header’s
require-corp value, if you want to include resources from another server that you trust, you need to include the
crossorigin attribute.

Although, at the time of this article’s writing, Chrome and Chromium-based browsers like Edge didn’t require the COOP and COEP response headers in order to enable the SharedArrayBuffer, they will require it in the near future.

WebAssembly will create a web worker for each thread you request. The web workers are created when the module is instantiated and, if you request a lot of threads, the startup time for your module may become noticeable.

Source Code

The source code for this article can be found in the following github repository: https://github.com/cggallant/blog_post_code/tree/master/2020%20-%20July%20-%20WebAssembly%20threads%20in%20Firefox



Source link

Write A Comment