Introduction to khadga

This book is a small guide on how to create a web application using react, typescript, and webassembly generated from rust. It will take you from knowing nothing about how to create wasm to how to generate both the front and back end code for your app.

The application that is generated here is a small chat application that will set up WebSocket connections between two or more clients and the central web server.

If you're wondering where the name khadga comes from, it's a Sanskrit word meaning sword. It is often referred to in spiritual or mystical concepts as the sword that cuts away illusion.

Why?

One might ask why go through this? If the main point of this app is to do a chat application with video, there's a dime-a-dozen ready made apps for that.

Basically, I wanted to write a non-trivial application from top to bottom. A truly full-stack application where the front end, the back end, the database, the CI deployments, integration with IoT sensor data, and the data analysis is all done by one developer (that'd be me, and hopefully you the reader as well).

Yes, it's a tall order. Although the primary purpose is to use this project as a vehicle for learning how to do deep learning, my intention is to use khadga as a learning tool. Not just for myself but for others as well. What I have learned is that most books don't walk you through a non-trivial project from beginning to end. Or, they might show you how to use some framework, but not always why.

My hope is that by me forging ahead and suffering the learning pains, others can follow along and avoid the mistakes I made. Because of the pedantic nature of this project, I will endeavor to do the following:

  • Write documentation for all my code
  • Keep this book up to date
  • Show you every step of building a project (as much as possible)
    • Writing unit and integration tests
    • Using CI/CD to build, test, and deploy an application (on openshift)
  • Try not to cut corners (ie, I will try to use .unwrap() or .expect() as little as I can)

What you will learn

This guide will walk you through everything required to develop, test and deploy both the front and backend application. This includes:

  • How to write an asynchronous web server using rust's warp framework
  • How to use wasm-bindgen, web-sys and js-sys crates to create a wasm npm module and publish it
  • How to serve your single page app from the async web server
  • How to create a react+redux front end, using WebRTC and WebSockets modules written in webassembly
  • How to deploy your app to Openshift Online using docker
  • How to set up unit and integration tests for the front and back end
  • How to set up a CI pipeline between your deployment and your tests using travis-ci

What the app does

We will build the functionality of the app slowly in order to make it useful early on, but to build up functionality as we go.

  • Chat application
  • WebRTC for cam-to-cam video conferencing
  • Video recognition

First, we will start with a relatively simple chat application. It will also cover things like setting up a database of users and saved chat history. It will also showcase how to do user authentication and authorization.

Next, we will write a webassembly module that interacts with WebRTC and WebSocket APIs. We will use this so that the webassembly can quickly and efficiently store data into tensor format that we can hand back to tensorflowjs.

Then, we will enhance the app so that it will do video as well as text based chatting. In this step, we will add a signaling service to the bacnd, so peers can find one another. Chats can either be saved locally or stored on the discovery server. This step will also show how to encrypt the streams for end to end encryption.

Lastly, we will build on the video streams enabled by the WebRTC to do image recognition. We will use this as a project to detect faces that are displayed and see if it is a known user.

Prerequisites

This chapter will cover how to scaffold and generate the initial wasm-pack code as well as a basic asynchronous web server but it assumed that the reader is already familiar with:

  • basic rust
  • basic html
  • basic css
  • basics of docker

The reader's rust skills should be at a basic level. One of the goals of the book is to explain some of the more tricky rust concepts (or at least tricky to the author). A basic understanding of rust ownership and lifetimes, and how closures work will be required. Understanding how Traits work, especially AsRef and From/Into will also be useful. The async/await portions will be described in detail however.

The reader should have a basic level of HTML5 understanding. This includes knowing what the DOM is and the basics of common tags. Most of the front end will be calling the web-sys bindings to the Web APIs, so it is useful but not required to know javascript. Because we are creating a single page app, knowledge of CSS will be useful, as the book will not spend a lot of time explaining much of the CSS content.

For deployment, we will be using Openshift Online. There is a free tier available and you can use this to deploy the application. In order to get the server to the cloud, we will need to create a container and therefore a docker image. This will not be advanced, but there will not be a lot of explanation of what the docker file does.

Caveats

The biggest caveat is the author is new to this himself. The decision to write the book was to help others so they do not have to learn the hard way like the author did. Also, the author is at a basic to intermediate level understanding of rust. There could very well be a better way to write the code and if so, please make a PR and contribute!

The second caveat was that this project made an opinionated stance on the technology used. First and foremost was the desire to use the new async/await syntax. This lead to several problems. For example, since async/await is still new, documentation is scarce.

Why not yew?

Seasoned developers might ask why yew was not used for this project. The simple answer here is that there was a desire to use wasm-bindgen, web-sys and js-sys crates to create the app.

The author has no working knowledge of yew, and it was considered initially. Afterall, it seems to tick off a lot of the right boxes:

  • Built in virtual DOM
  • Macros to generate JSX like code
  • Concurrency
  • Safe state management

However, yew is built on a crate called stdweb instead of wasm-bindgen, web-sys and js-sys. The main difference is that those libraries are created and maintained by the official Web Assembly Working Group. It will therefore be more up to date and have "official" support. It was also designed to be language agnostic and the bindings are auto-generated from the WebIDL schema.

Why not percy?

There is another framework that looked promising called percy. It also has a virtual DOM and macro generator to create some JSX-like code. Unlike yew, it is using wasm-bindgen and web-sys and js-sys crates. The problem with percy is that because of some of the macros, it required a nightly toolchain.

Although nightly is great for individual learning and experimentation, it's not the best for teaching others. The brittleness of nightly means that what may compile one day for one person may not compile for another person (or the same person!) on another day. It can also be hard sometimes to find a nightly build that allows all dependency crates to be built successfully.

Why not seed?

I discovered seed very recently, and while it seems to fit the bill for writing the entire application in rust, upon some consideration, I felt that it would be more beneficial to write the front end in react and typescript, and use wasm only for speed (or safety) sensitive areas of the code.

This also has the advantage of being able to slowly convert existing applications to use wasm, rather than have an all-in-one greenfield project created from scratch.

Local Setup

Now that you know what you will be building, let's get started setting up all the development dependencies. You'll need to have rust and npm set up as well as a few cargo tools

  • rustup
  • npm (recommend using nvm)
  • cargo-generate
  • cargo-edit

You can also checkout the github repository of khadga itself

Installing rustup

If you haven't already, install rustup by following the directions. If you already have rustup installed, make sure it's at the latest and greatest (at the time of writing, this is 1.40). To update your rustup, do

rustup self update
rustup update

Next, you need to set up the wasm32 target so that rustc can compile to the wasm32-unknown-unknown target triple

rustup target add wasm32-unknown-unknown

Other rustup goodies

While we are configuring rustup, we can install some other components as well

rustup component add llvm-tools-preview rustfmt clippy rls rust-analysis

C(++) toolchain

Some rust crates have native dependencies. For example the openssl crate will use and link to a native ssl lib on your system. Because of this, it's sometimes necessary to have a C(++) toolchain on your system as well.

It's beyond the scope of this book to show how to do this, since each operating system will do it a bit differently. For Windows, it's recommended to install one of the free Visual Studio C++ compilers. For linux debian based distributions, you can usually get away with something like this:

sudo apt install build-essential

For Fedora, you can do this:

sudo dnf groupinstall "Development Tools"
sudo dnf groupinstall "C Development Tools and Libraries"

Adding cargo tools

Although cargo is automatically installed by rustup, we are going to install some cargo additions.

cargo install cargo-generate cargo-edit

cargo-generate is a tool that will auto generate a template for you (and is used by wasm-pack) and cargo-edit is a nice little cargo command that lets you add a dependency to your Cargo.toml (think npm install).

Setting up vscode

We'll be using the Microsoft VS Code editor, since it has good support for rust and is relatively lightweight. Because we are using some bleeding edge crates, we'll also have to specify some additional configuration in the rust extension.

First, install vs code itself. Once you have code installed, we need to install the rust extension. You can either do this from the command line, or through VS Code itself.

code --install-extension rust-lang.rust

While we are installing extensions, let's install a couple others that will make our lives easier:

  • crates: To make it easier to see what the latest (stable) crate version is at
  • lldb debugger: So we can debug our code
  • toml: so we can get syntax highlights and coloring for our toml files
code --install-extension bungcip.better-toml
code --install-extension vadimcn.vscode-lldb
code --install-extension serayuzgur.crates

Install npm (and nvm)

Since we are building a front end web app, we will be making use of some npm tools. It's highly recommended that you use the Node Version Manager (nvm) for this if you are on linux or MacOS. For windows users, you'll probably need to use chocolatey to install node (and npm).

linux and macos

For linux and macos users, you can follow the directions here to install nvm. Once you install nvm, you'll need to actually install a node version.

nvm install 13
nvm use 13

Windows

For windows users, if you don't have chocolatey already, install that. Then you can install node (and therefore npm) with:

choco install nodejs  # make sure you run from an elevated command/powershell shell

Installing wasm-pack

For this project, we will be using wasm-pack which will generate a template for us, as well as set up a webpack config which will automatically compile our rust code to wasm.

You can install wasm-pack here.

Alternatively, you can install wasm-pack via cargo:

cargo install wasm-pack

Setting up the project

Now that we have all our dependencies out of the way, it's time to actually create our project. Unlike many rust projects you will see on tutorials, we are going to have a fairly advanced cargo setup. We will be using a feature of cargo called workspaces to work on the front and back end.

Creating your initial project

First, create a directory then create a Cargo.toml file in it:

mkdir -p ~/Projects/weblearn
cd ~/Projects/weblearn
touch Cargo.toml

The Cargo.toml file will need to be edited to look something like this:

[workspace]

members = [
  "backend",
  "frontend"
]

Generating the machine learning project

Since this is a (partially) isomorphic web projectrunning rust on the backend, and a combined webassembly/typescript front end (and a forthcoming rust IoT microcontroller sensor), we need a way to generate the the frontend project so that our rust code compiles to wasm, and all the glue clode to call to/from javascript functions can be done.

To set up the front end project you can go into your weblearn directory and run the following:

npm init rust-webpack mllib

This will generate a wasmpack style project that contains the code necessary for a combined rust and javascript project. Later, we will go into what files were generated, and how to build this workspace, but for now, you can browse this new frontend directory.

Using typescript

However, we are using rust, so why would we want to use javascript with it's dynamic types? We will enhance our project build code by allowing us to use typescript.

We will use a project called create-base-ts for this purpose. It will scaffold a project for use with typescript.

npm init base-ts frontend
npx tsc

The above commands will install typescript as a dev-dependency in our package.json file, and the npx tsc command will build our base project

We will tweak some of the configuration parameters later, once we build the project for the first time.

Generating the backend project

The backend will be a typical rust web server using actix-web, so we can just use cargo for this:

cd /path/to/weblearn
cargo new backend

Using kubernetes for testing and deployment

Ultimately we are creating a Google Cloud Product, and therefore we need to create a kubernetes project. As we develop the project and have something to actually test and deploy, we will go into more detail.

However, we can go over the actual build container. At the beginning of the chapter, we went over how to setup all your local development dependencies. However, we also need a reproducible way to build our project. While installing local tools is nice, what we really want to do in a production environment are many things:

  • Automate the process of building all the artifacts our project needs
  • As the artifacts are built, unit test them
  • Once all the artifacts have been built, run integration tests
  • If all the unit and integration tests pass, create a docker image
  • Build your Kubernetes Objects (deployments, services, etc) locally
  • Run minikube and apply all your config files to run your app
  • Run end to end tests against the app
  • If the tests pass
    • Tag the build(s) of your image(s)
    • push the image to a container repo like docker hub or GCR
  • Tell GCP to use the latest images and reploy

Yes, that's a rather large set of things to do!! This is called CD or continuous deployment. This is slightly different from CI which stands for continuous integration. The main difference is that the latter will actually push out the end application/project to a public release, whereas the former is more about building and testing the artifacts, but not necessarily pushing the artifact to deployment.

Where does kubernetes fit in?

Why kubernetes?

We want a way to make reproducible builds. In the olden days, developers would have their local build environments, which often was some configuration of an IDE. When the developer was done with their feature or bug fix, they would click some button on their IDE to build the application, and maybe they might run some tests (yeah, this was before TDD, BDD, CI, etc). Assuming it looked good, said developer of yore would make a CVS or perforce commit and announce to all the coworkers that new source code was ready.

Very often, a coworker would checkout a copy of the new source code, and lo and behold, he couldn't even get the project to build! What trickery was this? Were gremlins sabotaging code in the ancient CVS revision control system? The fastidious engineer might have compared the source code to make sure it was indeed correct and identical to what the developer who originally built it successfully had on his or her system.

Well, the problem was often that one developer's configuration of his build tools was a little bit different. Or perhaps the build required configuration files to be present in a particular location, and in a particular state. It was also highly likely that the local dependencies on one developers workstation was different from another. For you youngsters who have only been programming a few years, you may have wondered where the terms DLL Hell, Classpath Hell, npm hell (pre yarn or package-lock.json days) etc all came from. Well, one of the symptoms of the above problems is when either your program has some kind of dependency conflict (package A requires package B of version 1.0, while package C requires package B at version 2.0), or it could also mean that the dependencies on your system are missing or the incorrect version.

Whole dependency management systems were written to try to tackle this problem. For example, pkg-config for C(++) shared libraries/object, ant/maven/gradle/ivy for Java dependency loading and classpath/modulepath configuration, etc etc.

When I first started learning docker, I first struggled with the concept until I realized it was just a kind of abstraction around a process which carried around a file system with it. But I also tended to think of it in terms of running applications. Afterall, one of the selling points was that an image was self-contained: it had all the dependencies and runtimes needed to run the application. Ever built a python, java, javascript, etc app at work or school, and wanted to have other people use it? Well, first, you had to tell them "oh yeah, before you run my application, you need to install python. How do you do that? Well, first you have to....". Now imagine some of your dependency libraries are tucked away in a private repository. Like maybe you have JFrog serving up a private npm, maven, pypi etc repo. Now you have to tell your users how to access the private repo.

Ughhh, what a pain.

So that's the first thing that containers solved. Of course, it's also why we are seeing a resurgence of "build to native" languages, like rust, go, nim, crystal and even haskell. To be honest, I always felt like docker containers were a bit of a hack to get around the popularity of interpreted languages (and yes, I am looking at you java, your bytecode still needs to be fed into a JVM to spit out the actual machine code). But containers still have their uses even with "build to native" languages.

There are 2 other use cases that come to mind:

  • Creating a container that handles building a project
  • Applications that require more than just a binary

Your build environment as a container

In order to build a rust project, what do you need?

  • The rustc compiler
  • Tools like cargo

You may think, ha! That's all I need. But is it? Some rust libraries link to native libraries. The most notable example of this is the openssl crate which leverages the system openssl.lib. Some crates will also attempt to link or build C code. In that case, you now need a C(++) compiler and set of tools like make, autoconf, pkg-config, ld, nm, ar, etc.

Or, what if you have a webassembly project? Now you need to install wasm-pack (well, technically you don't, but good luck setting it up manually). If you have a webassembly project, you will need npm installed too. Oops, to install npm, I need node.

Maybe you're getting the picture. There's a lot of stuff that you have to install and get right. We can automate this process by creating a Dockerfile that builds our...well, build environment.

Let's do that now by creating a Dockerfile, but we're going to put it somewhere special:

FROM fedora:31 as builder

RUN dnf update -y \
    && dnf upgrade -y \
    && dnf install -y curl openssl-devel \
    && dnf groupinstall -y "Development Tools" \
    && dnf groupinstall -y "C Development Tools and Libraries" \
    && dnf clean all -y \
    && mkdir -p /apps/vision

# TODO: create a khadga user and run as that user.  We don't need to run as root

# Install rust tools
# Strictly speaking this isn't necessary for runtime.  Adding these will create 
# a container that lets us build khadga
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > rustup.sh \
    && sh rustup.sh -y \
    && source ~/.cargo/env \
    && rustup update \
    && cargo install wasm-pack

# Install node tools
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.2/install.sh | bash
RUN source ~/.bashrc \
    && nvm install 13 \
    && nvm use 13 \
    && mkdir -p /apps/bundle \
    && mkdir -p /src

# TODO: either clone the remote repo, or copy the entire khadga project over.
# In the "build locally" version, we assume that we've built khadga on our dev machine
# but we could also build it in the docker container itself.  Instead of doing a COPY
# we could do a MOUNT
COPY ./khadga /src/khadga
COPY ./vision /src/vision
COPY ./noesis /src/noesis
COPY ./build-docker.sh /src
WORKDIR /src

# TODO: If we need to publish a new noesis library, we need to 

RUN source ~/.bashrc \
    && source ~/.cargo/env \
    && ./build-docker.sh \
    && ls -al

RUN rm -rf /src

FROM fedora:31

RUN mkdir -p /apps/vision

COPY --from=builder /apps/vision /apps/vision

WORKDIR /apps/vision

CMD [ "./khadga" ]

We want to put this Dockerfile in a dockers/build folder. So your folder should look something like this:

khadga/
- docker/
	- build/
		- Dockerfile
- khadga/
- noesis/
- vision/

We don't actually have any source code yet so we can't run this yet. But I'll show you how to run this.

cd /path/to/top-level/khadga
sudo docker build -f ./docker/builder/Dockerfile -t stoner/khadga .

So for example, if my khadga is in /home/stoner/Projects/khadga, I would do:

cd /home/stoner/Projects/khadga
sudo docker build -f ./docker/builder/Dockerfile -t stoner/khadga .

Running this command will tell docker to use the special Dockerfile we just created and to tag it with stoner/khadga. If you don't use the -f option, docker will look in the current directory for a Dockerfile. Since we are going to have another Dockerfile in the root khadga directory, that's why we created and saved this Dockerfile somewhere special.

What does it do?

Basically the above Dockerfile will create a Fedora31 base image, update it with the latest goodies, and then install our tooling needed to build all the khadga subprojects (including the khadga backend, the noesis webrtc wasm library, and the vision front end).

Of note is that we use the FROM <base-image> as <phase-name>. Usually, when you see the FROM command in a Dockerfile, you don't see the as part. We use this when we have a multiphase build. Notice that further down almost at the bottom, we say again FROM fedora:31. This means that at this point docker can throw away the preceeding layers and start with the resulting image as a base image. However, docker still "remembers" the old image. That's why you later see the command:

COPY --from=builder /apps/vision /apps/vision

The --from=builder part says, "from our previous phase called builder, copy the /apps/vision directory to our current image's /apps/vision directory. If we didn't use this phased approach, the resulting image would have been about 2.6GB in size. But by using the phased approach, the size was less than 1/10th the size at 203MB.

The other thing to note is the build-docker.sh script that gets called. This is what actually builds the subprojects. It will look like this:

ls -al
cd noesis
wasm-pack build
if [ "${PUBLISH}" = "true" ]; then
  wasm-pack test --firefox
	echo "Deploying noesis to npm"
	npm login
	wasm-pack publish
fi

cd ../vision
rm -rf node_modules
npm install
npm run clean
npm run build
npx jest

cd ../khadga
cargo build --release
# KHADGA_DEV=true cargo test

# Copy all the artifacts
cd ..
echo "Now in ${PWD}"
ls -al khadga
cp -r vision/dist /apps/vision/dist
cp -r khadga/config /apps/vision/config
cp -r khadga/target/release/khadga /apps/vision

As we build up our code, we will go into more detail on what's happening here.

Running kubernetes

Here, we will show how to set up minikube and use kubectl

Installing minikube

You can follow the directions to install minikube here

Using minikube

minikube start --vm-driver=kvm2
minikube status

Creating deployment files

TODO: Go over all the deployment files we need

Installing kubectl

You can follow the directions here to install kubectl

Setting up kubectl

To get kubectl to use the minikube k8s enviroment by creating a new namespace, run this command:

kubectl config set-context --namespace=localdev minikube

Kubectl commands

This command will expose an already running pod named khadga, and attach a LoadBalancer service with a name of khadga-service

kubectl expose deployment khadga --type=LoadBalancer --name=khadga-service

This command uses the kompose file, and will run the get command on everything in the kompose

kubectl get -k .

Will display all the services running in the kubernetes environment (for the default namespace)

kubectl get service
kubectl get deployment
kubectl get pod
kubectl get pod -o wide

Shows how to delete various kubernetes objects like Deployments or Services

kubectl delete deployment khadga
kubectl delete service khadga-service

Uses the kompose file to apply all the config files. Looks for a kompose file

kubectl apply -k .

Creates a new namespace. Namespaces isolate the kubernetes environment

kubectl create namespace localdev 

kubectl config get-contexts

Setting up gcloud

You can follow the directions here to install the gcloud SDK.

Once the gcloud SDK is installed, you will also need to initialize it.

gcloud init

kubectl commands

Allow gcloud to use docker

gcloud auth configure-docker

To set up auth for logging in, run this command.

gcloud auth login

To set the project in google cloud to work with, run this command

gcloud config set project khadga-dev

To show a list of images in GCR run this command, where the argument to repository is the URL for the GCR repo (eg gcr.io, eu.gcr.io, etc), a slash, and then the name of the project

gcloud container images list --repository=gcr.io/khadga-dev

To show the tags of an image use this command

gcloud container images list-tags gcr.io/khadga-dev/khadga

docker-credential-gcloud configure-docker

To get credentials 

gcloud container clusters get-credentials standard-cluster-1 --zone us-central1-a --project khadga-dev

General workflow for testing and deployment

  • docker build your container images
  • tag your images
  • push the images to gcr.io
  • Update the config file for the updated image tag

For testing

  • create a localdev namespace and use it
  • Create the secret so that localdev knows how to use it
  • patch the serviceaccount to use the imagePullSecrets key

For the first step, you need to create a new namespace if you haven't already:

kubectl create namespace localdev

If you have run this step before, you don't need to again. You can go to step 3.

You can either append --namespace=localdev to all kubectl commands, or you can make it your default, where the final argument is the name of your context. To view your contexts:

kubectl config get-contexts

Then to set your namespace to the context you want:

kubectl config set-context --namespace=localdev minikube

Next, you can set up a new secret key by following there directions here

Then you need to patch your serviceaccount to use the key that you downloaded.

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}'

The main client interface

First, let's start off with a very basic webpage. Our end goal with this app is not to create something beautiful, but simply as a way to get data from some agent (a human, being, an IoT sensor, or another remote service for example) so that we can work on this data in real time.

Our initial goal is to provide a user interface for a human user (or some AI agent) to post messages similiar to slack. It's sort of half slack/gitter, half webchat in that the messages should be persistenet and editable, but they should also be seen in a public forum (or privately).

A second goal is to provide video camera access to allow for video using WebRTC.

The SPA running on the users browser has 2 initial services:

  • Collect messages from an agent and do text classification on it
  • Perform face recognition and eye tracking

General Layout

Before we get to actually coding up anything, let's think about what we want to show the user, and how we want the interface to be presented.

  • Navbar
    • Google Login
    • Settings for app
      • video/audio enumeration
      • chat and appearance
  • Chat messages window
    • Public message window
    • Message threads
    • Private message window
  • Logged in users
  • Video (Draggable and Resizable div component)

State

What are some of the things we need to keep track of for our app?

  • Is user logged in already? (ie, session is current and active?)
  • What users have connected and logged in that we can chat with?
  • Do we need to show modal for user to sign up? login?
  • Max message/text limit for chat message windows before removing from DOM
  • Is there a video chat session?
  • SDP data for webrtc peer connection for both offer and reply

Navbar

TODO: Go over how the navbar is implemented in bulma.

Chat message window

TODO: Go over how to split up into a main public column, and another column for message threads

Public messages

TODO: Explain what the public messages are vs. threaded

Message threads

TODO: Allow replies to a message that follows that specific message

Private messages

TODO: How do we add new containers for private messages?

Logged in users

TODO Show logged in users

Video

React and state management

For this application we will be using react and redux for state management on the front end side of things at least in the beginning. Once we start getting into tensorflow, we will start using rust data structures to capture some of the data, however the state of the application itself will be handled with redux.

Why react

One could ask why we are using react here. There are other popular and not so popular front end frameworks out there including but not limited to angular, vue,

Why redux

Hooking a component into redux

  • Add a new Action Creator with some new state to pass along
  • Add a reducer that that does something with the action type and new state
  • Add the new reducer to the combineReducers function
  • Add a mapPropsToState on component and only pass the data needed (that the reducer takes)
  • Add a mapDispatchToState if needed
    • Used if component needs to change state that another component needs to react to

Navbar component

Google Authentication component

Mongo Database

We need a way to store information that the app either needs or collects. For example, we need to know information about a user that signs up, and what permissions he or she has. We also want a way to persist information about messages we receive so that we can operate on them later if needed.

In this chapter, we will go over setting up a mongodb database, and in later chapters we will revisit it to shore up security and create some new indexes to speed up searches. This chapter will also go over a docker set up to make it easier to set up our static backend files, and setup up the necessary networking glue to get our rust actix server and databse to talk to each other

Using Docker

In order to test our new code changes as well as deploy our final app, we need a way to deliver a usable artifact that we can either run tests against, or actually use in production.

Although rust is very nice in that it produces standalone binaries, our product here is actually a set of services. For example, khadga needs a database and we can't stuff a database inside our binary (well, we could, but where would you persist the data?). Other parts that probably aren't good to stuff inside the binary are configuration files. Many times you want to dynamically configure how an application runs based on a file that it can read. If we statically link in the file to the binary, it's stuck with that configuration. If you allow your app to read the config file at runtime, where and how do you bring along the configuration file(s)?

To solve these kinds of problems, we will use docker and docker-compose. If you are not that familiar with docker, that's fine. Here, I will explain what is going on, and why we are doing what we are doing.

Installing docker

Depending on your operating system, this can be a tricky affair, so I will direct you to the docs on the docker site.

Creating a Dockerfile

The Dockerfile will be our recipe to create a container that contains our backend server and the configuration files that it needs.

First, we create a file named Dockerfile in the root of our project. It will look like this:

FROM ubuntu:bionic

RUN apt update \
    && apt upgrade -y \
    && apt install -y curl \
    && mkdir -p /apps/vision/dist

# TODO: create a khadga user and run as that user.  We don't need to run as root

# Copy the dist that was generated from wasm-pack and webpack to our working dir
# then, copy the executable to the vision directory.  This is because the binary
# is serving files from ./dist
WORKDIR /apps/vision
COPY vision/dist ./dist
COPY khadga/config ./config
COPY target/debug/khadga .

CMD [ "./khadga" ]

Explanation

So what's this Dockerfile file doing? For those not familiar with docker, the Dockerfile is the default name for the file that the docker CLI will look for when running certain subcommands like build. It is a file that contains instructions for how to build a docker image step-by-step.

As a quick aside, do not confuse docker images, from docker containers. A container is really a kind of fancy process with special restrictions. An image is like an executable binary. From one binary, you can launch many processes. The docker image is the executable that a container is spawned from. The command docker ps will list running containers, while docker images will list docker images on your system.

Each line of code is actually creatig a new layer in a docker image (which is why you frequently see certains commands like RUN using && to concatenate commands to a single layer). This layering is also important to understand when you make changes to your Dockerfile. As each command in the Dockerfile is encountered, docker will create a layer which it caches so that it doesn't need to be done again. If you only ever add new steps at the bottom of the Dockerfile, that's fine. However, if you change a step at the beginning (for example), all steps after your new change must be executed again, since the layer system is immutable. This can increase your build times significantly.

Our base image

So our first step is a FROM command, which basically specifies what base image to use. Here, we are saying that the base image we will use is an ubuntu bionic release. Notice the use of the colon here. To the left of the colon is the base image name (ubuntu in this case), and to the right of the colon is a tag. The colon and tag name are optional, but if you do not use them, there is typically a default tag (usually :latest).

What this does is download from the docker hub repository an image named ubuntu, with the tag of bionic. This is our base image. From this base image, we will add more to it.

Adding new components to our base image

Next, we encounter the RUN command. RUN takes a shell command and executes it within a docker daemon (which is actually executing our base image). If you look at the shell commands it is running, you can see it is updating the operating system, adding curl and creating some directories.

Copying files from dev machine to docker image

A critical thing to understand in docker is the difference between our development machine and the docker image itself. We need to get the files from our dev machine onto the docker image. Another important thing to understand is the idea of a context which is like a build directory.

When you actually build your docker image, you specify a build context argument. For example:

sudo docker build -t <username></tag> .

The . at the end tells docker "use the current directory as the build context". The build context is the point of view from your development machine. If you use your current directory as your build context, any COPY operations in the Dockerfile will assume that the source is from that build context.

For the sake of the example, let's assume that you are currently in /path/to/my-project. And that you then run

The WORKDIR command tells the docker daemon "use this directory as our working directory from the docker image point of view". In the Dockerfile, we are using WORKDIR /app/vision. Any command that copies files for example, will use this directory as the destination directory by default.

So the next command COPY vision/dist ./dist is basically saying "Copy the files from ${BuildContext}/vision/dist to ${WORKDIR}/dist". So, if you set your build context to "." during the docker build command, and that you are currently in the /path/to/my-project directory, that this will copy our project's vision/dist directory to the docker image's /app/vision/dist directory.

Hopefully the next COPY commands are clear now. We also copy our local khadga/config on our dev machine to the image's /app/vision/config directory. The last COPY is interesting. Here, we will copy the binary generated from cargo (in this case the debug version) to our docker image in /app/vision. Eventually, we'll want to somehow dynamically set that based on whether we want to use the docker container for testing or release it to production.

Executing our process

A docker container executes what's in the CMD operation. The first arg is our command or executable, and subsequent elements in the array are any arguments the executable needs.

Setting up mongodb with docker-compose

This chapter is about mongodb, but so far we haven't touched mongodb at all. Creating the Dockerfile was necessary however for our next piece, the docker-compose file, which will be explained in the next section

TODO: docker-compose

create mongodb container

TODO: schemas for data types

Yes, this is mongo, but schemas are not a bad thing.

TODO: setup initial db.rs code

Security

Authenticating a user

For khadga, we will be using the Google OAuth mechanism.

Setting things up

There are actually 2 parts to this:

  • Setting up a google account with OAuth Credentials
  • The front end client

Setting up google account for OAuth

The first thing we will need to do is to set up a google account. Go to console.developers.google.com and select or create a project. Since the UI is always changing, it's hard to say what to look for, but currently, in the left hand sidebar, there is a link that says Credentials.

If you click on Credentials, a new layout appears that shows your current credentials for your project:

  • API Keys
  • OAuth2 credentials
  • Service Accounts

At the top, there is a button to + Create Credentials. In the drop down that appears, select the OAuth client ID option. In the next screen, you can select Web Option. A new section appears and you can optionally give it a name.

There will also be a field that specifies authorized javascript origins. For demo purposes, we can put in http://localhost:7001 which is where our khadga backend, and therefore where our web page's window.location will be.

Setting up the front end client

The first thing we need to do is load the javascript. Unfortunately, google does not release an npm package, so we need to load this in a <script> tag in our index.html.

  <head>
    <meta charset="UTF-8" name="viewport" content="width=device-width, initial-scale=1">
    <title>Grid test web app!</title>
    <script defer src="https://use.fontawesome.com/releases/v5.3.1/js/all.js"></script>
    <script src="https://apis.google.com/js/api.js"></script>
    <link rel="stylesheet" href="./styles.css"><link>
  </head>

Once we do this, we will have access to the gapi object from our window.navigator. Since we are using typescript, adding objects to the global window object is a bit hacky. We will do a simple approach which is not type-safe, and then improve on it later.

function latestName(conn, user) {
	let baseName = user;
	let re = new RegExp(`${baseName}-(\\d+)`);
	let index = 0;
	for (let name of conn) {
		if (name === baseName) {
      console.log(`${name} in list matches current of ${baseName}`)
			let matched = name.match(re);
			if (matched) {
				index = parseInt(matched[1]) + 1;
				console.log(matched);
				console.log(`Got match, index is ${index}`);
				baseName = baseName.replace(/\d+/, `${index}`);
			} else {
				index += 1;
				baseName = `${baseName}-${index}`;
			}
		} else {
      console.log(`${name} does not equal ${baseName}`)
		}
		console.log(`baseName is now ${baseName}`)
	}
	return baseName;
}

latestName(["sean", "sean-1", "sean-2"], "sean")

Letencrypt TLS certs

If you have an https server, you'll need a real TLS server. Creating self-signed certs is ok for testing purposes, but even then, unless you have good automation and CI/CD pipelines, there's a danger of accidentally deploying a server with a self-signed cert to production.

Here, we will talk about how to create a Letsencrypt TLS cert that you can use for the khadga backend

Certbot

TODO: Show how to use certbot and docker to autogenerate and autorenew a TLS cert that we can deploy

The backend

Even single page apps have to get downloaded to the user's browser somehow. This chapter will talk about how to set up a web server written in the warp framework for rust.

We will also go over the endpoints we'd like to establish, although in later chapters, we will flesh out some additional endpoints we would like to make.

Warp

TODO: Creating https in warp

Discuss nginx load balancer as an option

Authentication

We don't just want anyone to come in and use our service. We could allow guests to access some parts of the system, but not everything. The first step is having a user sign up and register themselves to the service.

Once a user is registered, we need a way to validate that the entity accessing the service, is who they actually say they are. This is authentication. This is not to be confused with authorization, which means that we have know you are John Doe, but we need to know what permissions John Doe has to our system.

Classic database authentication

We will be implementing a classic password + database security. While perhaps not the best system, we can later augment this.

TODO: Discuss how to send data from web page, to backend, and then to the mongodb.

2FA

TODO: using 2FA or TPM

Talk about stragies for stronger authN

Oauth

TODO: Talk about possibly implementing OAuth with google

Authorization

It's not good enough to authenticate the user is who he or she is, but we also need to restrict access to service to those who are authorized for those services.

TODO: Describe JWT tokens and how to use rust to generate them, and to have the khadga backend make use of them.

Backend Server

We need a server to supply a couple of services we need:

  • Authentication service to provide JWT tokens
  • Signaling service so peers can send WebRTC media to each other
  • Websocket service for chatting

Noesis

Noesis in Greek means, "insight", "to understand" or "to see". I thought it was a good name for a webassembly module intended to grab data from different sources and as a helper for tensorflowjs.

What it does

Noesis will be a slowly growing webassembly module that will do the following:

  • WebRTC handling, including getting MediaStreams and MediaDevices
  • Tensor'izing data from websockets
  • Tensor'izing data from MediaStreams

How it works

TODO: Explain how to use wasm-pack, wasm-bindgen, web-sys and js-sys