How to Install a Face Recognition Model at the Edge Device with AWS IoT Greengrass

Editor Note: This post is co-authored by Gong Xinyue, one of Global Accounts Solution Architects.

You might already know how to use AWS IoT Core and AWS IoT Greengrass for remote device communication and control. With AWS IoT Greengrass machine learning (ML) inference, you can run machine learning models on your local devices without any transmission delay. In this blog post, I will show you how to use AWS IoT Greengrass ML inference on a Raspberry Pi to perform local facial recognition for home surveillance.

Using an Amazon Echo Dot, which is connected to the Alexa Voice Service, as the control device for the Raspberry Pi’s camera, you’ll be able to take a photo of people outside your door and, using the photo, perform facial detection and comparison with a local dataset using the pretrained ML model deployed to the Raspberry Pi. Although the comparison results can also be used with your door lock or other smart devices, these use cases are not covered in this post.

Prerequisites

Install the AWS IoT Greengrass Core software on your Raspberry Pi device. Create an IoT Greengrass group and core. For instructions, see Getting Started with AWS IoT Greengrass in the AWS IoT Greengrass Developer Guide. The group and core are used in deployments.

In this post, I use a pretrained face-detection model, train it with TensorFlow, and then deploy it to the Raspberry Pi with AWS IoT Greengrass. You can use the same model, or you can use Amazon SageMaker to train one of your own. For information about how to prepare a model with Amazon SageMaker, see Get Started in the Amazon SageMaker Developer Guide.

Install the following dependencies.

• OpenCV 3.3
• TensorFlow
• Numpy
• Scipy
• Scikit-learn
• Dlib
• All image processing libraries

Use the following commands to install the image processing dependencies.

sudo apt-get install build-essential
sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev pkg-config graphicsmagick

Use the following commands to install OpenCV 3.3.

git clone https://github.com/Itseez/opencv.git
git clone https://github.com/Itseez/opencv_contrib.git
cd opencv
git checkout 3.1.0
cd ../opencv_contrib/
git checkout 3.1.0
cd ..
cd opencv
mkdir release
cd release
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-D BUILD_EXAMPLES=ON ..
make -j4
sudo make install
sudo ldconfig

The ML model used in this post is TensorFlow. Use the following commands to install TensorFlow on your Raspberry Pi.

# Install Wheel
wget https://github.com/samjabrahams/tensorflow-on-raspberry-pi/releases/download/v1.0.1/tensorflow-1.0.1-cp34-cp34m-linux_armv7l.whl
sudo pip install tensorflow-1.0.1-cp34-cp34m-linux_armv7l.whl
sudo pip install mock
#Install Pip3
sudo apt install libatlas-base-dev
pip install tensorflow

Use the following commands to install Numpy, Scipy, and Scikit-learn.

pip install numpy scipy scikit-learn

Use following commands to install Dlib.

#Modify your swapfile
sudo nano /etc/dphys-swapfile

change CONF_SWAPSIZE = 100 to CONF_SWAPSIZE=1024
change virtual memory from 100M to 1G

sudo /etc/init.d/dphys-swapfile restart

sudo raspi-config
#Change you boot Options
1)Boot Options => Desktop / CLI => Console Autologin
2)Advanced Options => Memory Split => change GPU to 16

#Doanload your Dlib
download Dlib to your device : http://dlib.net/
sudo python setup.py install

ML models and Lambda functions

You will find two ML models on GitHub.

• Face Detection Model
  |—-haarcascade_frontalface_default.xml
• Face Recognition Model
  |—–train_faces.model-500.data-00000-of-00001
  |—–train_faces.model-500.index
  |—–train_faces.model-500.meta
  |—–checkpoint

You will also find notes to that describe the Lambda functions used in this post.

FaceDetection Lambda function on AWS IoT Greengrass

#open camera
cam = cv2.VideoCapture(0)
while True:
# capture an image from video stream
_, img = cam.read()
# convert RBG image to grayscale image
gray_image = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# detect face
dets = detector(gray_image, 1)
if not len(dets):
# detect face failed
key = cv2.waitKey(30) & 0xff
if key == 27:
sys.exit(0)
# start recognition
for i, d in enumerate(dets):
x1 = d.top() if d.top() > 0 else 0
y1 = d.bottom() if d.bottom() > 0 else 0
x2 = d.left() if d.left() > 0 else 0
y2 = d.right() if d.right() > 0 else 0
face = img[x1:y1, x2:y2]

face = cv2.resize(face, (size, size))
# compare image with dataset and compose JSON format message
record_data = {}
record_data['result'] = is_my_face(face)
prepared_record_data = json.dumps(record_data)
# send MQTT message to “record/webcam” topic
client.publish(topic = 'record/webcam', payload = prepared_record_data)
break;

Alexa trigger Lambda function

def get_welcome_response():
# public a message to IoT core
client.publish(
topic='listening/record',
payload="Hi",
qos=1
)
# return message of this Alexa Skill
session_attributes = {}
card_title = "Welcome"
speech_output = "Sure. I will check the camera now."
# If the user either does not reply to the welcome message or says something
# that is not understood, they will be prompted again with this text.
reprompt_text = "Sure. I will check the camera now."
should_end_session = True
return build_response(session_attributes, build_speechlet_response(
card_title, speech_output, reprompt_text, should_end_session))

Get started

The architecture of the example described in this post is shown here. The facial recognition model and datasets, which are used to create AWS Lambda function for recognition, have been uploaded to an Amazon S3 bucket. AWS IoT Greengrass synchronizes the required files to the Raspberry Pi. Echo Dot runs as a trigger. When Echo Dot listens to a command such as,“Alexa, open Monitor,” it calls an Alexa skill to send a message to AWS IoT Core. AWS IoT Core invokes the recognition Lambda function, which is deployed on Raspberry Pi local storage, and if the Lambda function recognizes the identity of the guest, the door opens.

Let’s prepare the resources you will deploy on the Raspberry Pi.

First, create an AWS Lambda function that will capture the frame from the Raspberry Pi camera, convert it to a JPG file, and then invoke the local ML model for analysis. The AWS Lambda function will send the analysis result in JSON format to AWS IoT Core.

In the AWS Lambda console, choose Create Function, and then choose Author from scratch. Give your AWS Lambda function a name like greengrassFaceRecognization. For Runtime, choose Python 2.7. Because this Lambda function will be deployed on edge devices, you can choose any role without impacting its function on the Raspberry Pi.

In the function code area, choose the option to upload a ZIP file, and then upload the ZIP package provided for this Lambda function from GitHub. From Actions, publish a new version. AWS IoT Greengrass doesn’t support $LATEST as the version for Lambda aliases, so make sure you assign a version number (for example, version 1) to your Lambda function.

When the Lambda function runs, it will call the local ML model for facial recognition. Use the following values to configure the Lambda function:

The Lambda function needs to invoke some local devices on your Raspberry Pi. Add those devices to your AWS IoT Greengrass resources.

In the AWS IoT Core console, choose AWS IoT Greengrass. Choose Groups, and then choose your Greengrass group (for example, greengrassFaceReco). In the left navigation pane, choose Resources. On the Local tab, choose Add a local resource.

Because the example in this post uses the Raspberry Pi camera, add two devices to the Greengrass group:

• videoCoreSharedMemory
• videoCoreInterface

Complete the fields as follows:

When you’re done, your device configuration should look like this:

Add your ML model to this Greengrass group. In the left navigation pane, choose Resources, choose Machine Learning, and then choose Add machine learning resource.

In this example, the model is stored in an S3 bucket. If you use Amazon SageMaker to train your model, choose Amazon SageMaker as your model source. The local path is the directory on your edge device where you will store the model. Configure this path carefully. Otherwise, your model cannot be used on your edge device. You can see your model has been added to your Greengrass group:

Now create a subscription so that the local analysis result can be sent to the AWS IoT Cloud for processing, such as storing the result in Amazon DynamoDB and performing result analytics in Amazon EMR. If you have another local device connected to Raspberry Pi, you can use the AWS IoT Greengrass core to control it through AWS Lambda based on the analysis result. In this example, the result is sent to the AWS IoT Cloud through an MQTT message.

You can now deploy the Greengrass group. From Actions, choose Deploy. Deployment times vary, depending on the size of the model. When the deployment is complete, the results appear in the console, as shown here.

Your Raspberry Pi should now be able to recognize the face photo captured by the Pi camera. Use Echo Dot to voice control the Pi camera to get the image.

Create another Lambda function to trigger the AWS IoT Greengrass local face detection Lambda function through an MQTT message. The trigger Lambda function will send an MQTT message to the Greengrass core through AWS IoT Core. AWS IoT Greengrass will process this message through the local face detection Lambda function and then trigger the photo analysis. You can find this Lambda code based on Python 2.7 on GitHub. Make a note of the ARN for this Lambda function. You will use it when you configure Alexa skills.

Open the Amazon Developer Portal. Choose Alexa, choose Your Alexa Consoles, choose Skills, and then choose Create Skill. For information about creating your own Alexa skills, see Build Skills with the Alexa Skills Kit.

Finally, build a smart home surveillance system by using your Echo Dot to send a voice control message to your Raspberry Pi to start the local face recognition process. The result will be sent back to the AWS IoT Cloud through an MQTT message.

Conclusion

By using the example in this post, you can build a small home surveillance system on your Raspberry Pi device with AWS IoT Greengrass. Then, you can explore further by training other ML models and deploying them to your AWS IoT Greengrass devices.