Building a Linux edge-computing solution with AWS Snowball Edge and Amazon EC2

There are many situations where you must run data inference close to the data sources. Often times these are located at remote, disconnected locations. Let’s consider the following examples:

• A remote oil drilling platform has numerous sensors that generate data. Critical components have to be monitored for wear and tear, or failure, and replacements must be done proactively.

• An agriculture farm having a variety of sensors to monitor moisture, humidity, and PH values of the soil, must use inference to deliver water and nutrients at the right time to maximize health and growth.

• When the military is deployed to forward bases where there is no connectivity, they must automate the supplies and logistics.

• Autonomous vehicles generate large amounts of data on a daily basis, these have to be off-loaded, tagged, and pre-processed for anomalies at a central location, daily.

• An assembly line in a modern factory must move and optimize component deliveries efficiently.

• At a concert venue, the production company must aggregate footage from multiple cameras and then convert to different formats.

All these scenarios require running compute, storage, and a network close to the data source. You can solve these problems using an AWS Snowball Edge device, a ruggedized device that does not require built up data center space to run. You can run cloud native services like Amazon S3, Amazon EC2, Amazon EBS, AWS IoT Greengrass, and a Network File System (NFS) interface for data ingestion.

In this blog post, I explain how to launch a Linux-based edge-computing solution with AWS Snowball Edge devices, focusing specifically on Snowball Edge Compute Optimized devices.

Ordering process

To use a compute instance on a Snowball Edge device, create a job and specify your AMIs. You can do this from the AWS Snowball Management Console, with the AWS Command line Interface (AWS CLI), or with one of the AWS SDKs. Typically, there are some prerequisites that you must perform before creating your job:

1. Log in to the AWS Management Console and use the documented procedure to create an Amazon Machine Image (AMI) and add the same to the Snowball Edge order. AMIs also can be created using VM Export/Import.
2. Receive your Snowball Edge device from the shipping provider.

NOTE: Install any software packages and tools that you need; for example: Apache, PostgreSQL, SQLite, PHP, and Python. You may also want to install the Snowball client and AWS CLI (1.16.14 is the recommended version). This is useful if you are planning to use the CLI from the EC2 instance on Snowball Edge.

You can order Snowball Edge devices through the AWS Management Console. You order them with a pay-as-you-go model for short-term jobs, or with discounted pricing for longer-term commitments. Snowball Edge devices come in two variants: Compute Optimized or Storage Optimized. While it can be used for edge computing, the Snowball Edge Storage Optimized device is more commonly used for data transfer as it has twice the S3 storage capacity and half the compute resources of a Snowball Edge Compute Optimized device.

A Snowball Edge Compute optimized device currently offers 52 vCPUs, 208 GB of RAM, and a GPU option. They also have 7.68 TB of NVMe storage for creating performance-optimized Amazon EBS block storage volumes, as well as 39.5 TB of usable HDD storage capacity for object storage. For additional information, please refer to AWS Snowball Edge specifications.

Supported AMIs

Today, AWS Snowball supports CentOS 7 (x86_64) – with Updates HVM, Ubuntu Server 14.04 LTS (HVM), and Ubuntu 16.04 LTS – Xenial (HVM) AMIs (Amazon Machine Images) for Snowball Edge. We are continuously adding capabilities in response to customer needs, and will support additional operating systems in the future.

Snowball Edge site preparation

1. Place the Snowball Edge near a power source and connect the power cable. The network connections can be made using the built-in RJ45 Ethernet port with a CAT-6 cable, Directly Attached Copper (DAC) cables, or SFPs to connect to your network infrastructure.

2. The RJ45 supports 1 Gbs or 10 Gbs. The SFP port supports 10 Gbs or 25 Gbs. The QSFP port supports 25 Gbs or 40 Gbs speeds, and the latest Snowball Edge device even supports 100 Gbs on this port. The SFPs or the DAC cables are not supplied with the Snowball Edge device. Please refer to the Snowball Edge specifications for the latest supported network connections on your device.

3. The following network ports must be opened from your workstation to the Snowball Edge device:

4. Access the front panel of the Snowball Edge device, where the Kindle display is located. Press the power button. The device may take 15–20 minutes to self-validate and fully boot up.

5. From the Snowball Edge control panel select:

1. Connection, then select DHCP or Static.
2. For static IP selection, enter the IP address, Netmask, and Default Gateway.
3. Apply the changes once done.

Preparing a Linux administrative workstation

In this example, we are using a Linux machine as an administrative workstation. You may use Microsoft Windows or MAC. However, the command lines have to be modified accordingly.

This workstation performs the common Snowball Edge administrative tasks such as unlocking, launching services, and monitoring the device.

Launching the EC2 instance

The following diagram illustrates a Snowball Edge deployment with three EC2 instances. Each instance has its own private IP address. You are creating a virtual network interface with a public IP address matching your current address scheme. Security groups allow you to block or allow specific type of network traffic.

Configuration

Prior to configuring the Snowball Edge device, you must obtain the manifest file and unlock code from your AWS Management Console.

You can use AWS OpsHub or the AWS CLI to manage the Snowball Edge. Once you install OpsHub, you can unlock, configure network interfaces, and launch EC2 instances with few clicks. AWS OpsHub is available at no charge wherever Snowball Edge is available. Check out this demo video to see AWS OpsHub in action:

To learn more and to get started with AWS OpsHub, visit the Snowball resources page.

Start configuration of AWS Snowball Edge

You can also use the AWS CLI procedure below to configure the device. To run the steps, you have to download and install the Snowball Edge client and the AWS CLI.

1. Log in to your administrative workstation. Ping the Snowball Edge IP address, and confirm a response.

2. Run configure and create a Snowball Edge profile. A profile makes the management easy when multiple Snowball Edge devices are in use.

prompt> snowballedge configure –profile snow1
Configuration will stored at /home/user1/.aws/snowball/config/snowball-edge.config
Snowball Edge Manifest Path: /home/user1/SNOW/ID14a49650-53e6-47bc-82a9-c27a5133f56e_manifest.bin
Unlock Code: 7a1f2-cbb39-5b02c-597f4-c2b99
Default Endpoint: https://192.168.1.116 
prompt>

3. Unlock the Snowball Edge.

prompt> snowballedge unlock-device --profile snow1
The Snowball Edge unlock status is: UnlockSnowballResult(status=UNLOCKING)
prompt>

4. Check Snowball Edge device status, and also note down the physical network interface ID that you have used for connecting.

prompt> snowballedge describe-device --profile snow1
  "DeviceId" : "JID14a49650-53e6-47bc-82a9-c27a5133f56e",
  "UnlockStatus" : {
    "State" : "UNLOCKED"
  },
  "ActiveNetworkInterface" : {
    "IpAddress" : "192.168.1.116"
  },
  "PhysicalNetworkInterfaces" : [ {
    "PhysicalNetworkInterfaceId" : "s.ni-98c7e5d691a07a62e",
    "PhysicalConnectorType" : "RJ45",
    "IpAddressAssignment" : "STATIC",
    "IpAddress" : "192.168.1.116",
    "Netmask" : "255.255.255.0",
    "DefaultGateway" : "192.168.1.2",
    "MacAddress" : "aa:bb:cc:11:dd:de"
  }, 
--lines removed -- 
}
prompt> 

Launch an Amazon EC2 Instance

1. List the services.

prompt> snowballedge list-services --profile snow1
{
  "ServiceIds" : [ "s3", "ec2", "fileinterface" ]
}

2. Get the access key and secret key. The access key and secret key are used for authentication to the local Snowball Edge EC2/S3 service endpoints.

prompt> snowballedge list-access-keys --profile snow1
{
  "AccessKeyIds" : [ "AKIAABOURMPPK784AG4I" ]
}
prompt>

prompt> snowballedge get-secret-access-key --access-key-id AKIAABOURMPPK784AG4I –profile snow1
[snowballEdge]
aws_access_key_id = AKIAABOURMPPK784AG4I
aws_secret_access_key = jqiTyUvyZM+t2+AeGJ12algPdqKDp9CIXADBX
prompt>

3. Create a new service profile using the AWS configure command.

prompt> aws configure –profile compute1
AWS Access Key ID [None]: AKIAABOURMPPK784AG4I 
AWS Secret Access Key [None]: jqiTyUvyZM+t2+AeGJ12algPdqKDp9CIXADBX
Default region name [None]: snow
Default output format [json]: 
prompt> 

4. Create a virtual network interface.

prompt> snowballedge create-virtual-network-interface --physical-network-interface-id s.ni-98c7e5d691a07a62e --ip-address-assignment STATIC --static-ip-address-configuration IpAddress=192.168.1.172,Netmask=255.255.255.0 –profile snow1
{
  "VirtualNetworkInterface" : {
    "VirtualNetworkInterfaceArn" : "arn:aws:snowball-device:::interface/s.ni-8c184704e9c0b1872",
    "PhysicalNetworkInterfaceId" : "s.ni-98c7e5d691a07a62e ",
    "IpAddressAssignment" : "STATIC",
    "IpAddress" : "192.168.1.172",
    "Netmask" : "255.255.255.0",
    "DefaultGateway" : "192.168.1.2",
    "MacAddress" : "22:b4:94:23:dc:78"
  }
}
prompt>

5. You can use the following command to list all virtual network interfaces configured on the Snowball Edge.

prompt>snowballedge describe-virtual-network-interfaces --profile snow1
{
  "VirtualNetworkInterfaces" : [ {
    
  } ]
}
prompt>

6. List images on Snowball Edge and note the AMI ID.

prompt> aws ec2 describe-images  --endpoint http://192.168.1.116:8008 --profile compute1
{
    "Images": [
        {
            "Description": "CentOS 7 image for SnowballEdge",
            "EnaSupport": false,
            "ImageId": "s.ami-05a690772e39d6ff0",
            "State": "AVAILABLE",
            "Public": false,
            "Name": "CentOS7_SBE_Image"
        },
        {
            "Description": "Windows 2012R2 Image for SnowballEdge",
            "EnaSupport": false,
            "ImageId": "s.ami-09a19c1583173d1d6",
            "Platform": "windows",
            "State": "AVAILABLE",
            "Public": false,
            "Name": "Windows_2012R2_SBE_1"
        }
    ]
}
prompt>

7. Launch an EC2 Instance on the Snowball Edge from the image.

prompt> aws ec2 run-instances --image-id s.ami-05a690772e39d6ff0 --endpoint http://192.168.1.116:8008 --instance-type sbe1.medium --profile compute1
{
    "Instances": [
        {
            "SourceDestCheck": false,
            "InstanceId": "s.i-84f09ff791b6828b0",
            "EnaSupport": false,
            "ImageId": "s.ami-09a13c1583275d1a6",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "AmiLaunchIndex": 0,
            "InstanceType": "sbe1.small"
        }

Checking on your Amazon EC2 Instance

1. Check status of the launched instances

prompt> aws ec2 describe-instances  --endpoint http://192.168.1.116:8008 --profile compute1
{
    "Reservations": [
        {
            "Instances": [
                {
                    "SourceDestCheck": false,
                    "InstanceId": "s.i-84f09ff791b6828b0",
                    "EnaSupport": false,
                    "ImageId": "s.ami-05a690772e39d6ff0",
                    "State": {
                        "Code": 16,
                        "Name": "running"
                    },
                    "EbsOptimized": false,
                    "LaunchTime": "2018-08-22T00:48:32.909Z",
                    "AmiLaunchIndex": 0,
                    "InstanceType": "sbe1.small",
                    "PrivateIpAddress": "34.211.108.9"
                }
            ],
            "ReservationId": "s.r-8bc47a1364f21ebab"
        }
    ]
}
prompt>

2. Associate IP addresses to the instance.

prompt> aws ec2 associate-address --public-ip 192.168.1.172 --instance-id s.i-84f09ff791b6828b0 --endpoint http://192.168.1.116:8008 --profile compute1
prompt>

3. Connect to the EC2 instance using the key that you have saved from AWS Management Console, before the creation of the AMI.

If you have completed all the preceding steps, you should now be able to successfully log in to the Linux instance running on your Snowball Edge device.

prompt> ssh ubuntu@192.168.1.172 -i MySshKey1.pem
The authenticity of host192.168.1.172 can't be established.
ECDSA key fingerprint is SHA256:fingerprint
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘' (ECDSA) to the list of known hosts.
Last login: Wed Mar 25 23:36:08 2020 from 192.168.1.100
$

Summary

In this post, I showed you how to order a Snowball Edge device for running an edge compute use case. During the ordering process, I covered adding your own custom AMI, receiving the Snowball Edge device, connecting, and unlocking it using the snowballedge command line. I then discussed creating a virtual network interface that has an IP address scheme matching your local network, along with detailing the retrieval of the API credentials from the device. Additionally, I showed you how to launch an EC2 instance and attach a virtual network interface to it using AWS CLI. Lastly, I covered connecting to the EC2 instance using Secure Shell (SSH).

Through the steps I just outlined and covered throughout this post, you can deploy a powerful edge compute platform with AWS Snowball Edge. With that platform ready, you can build your real-life applications to process, analyze, and store data close to your data sources. Snowball Edge has made running data inference in remote edge locations easier than ever. Not only do the devices require no data center space to operate, but they also enable you to run compute, storage, and a network close to data sources in challenging environments.

For more information about this feature, read the AWS Snowball Edge documentation. Thank you for reading this blog post, please leave a comment in the comments section with any questions or feedback.