Today we have a guest contribution of Kayode Alade. Kayode is one Ioot Solutions architects with a hand to get things going. He sees himself as a constant learner who is always enthusiastic about acquiring new knowledge, be it from a technical point of view or business perspective.
You can find Kayode Github And LinkedIn.
Embedded DevOps changes how embedded teams create software and transfer updates to customer devices. In the constantly developed landscape of IoT, companies like Golioth And Memfault are groundbreaking efforts to optimize the device management. Seamless Provision To Over-the-air (OTA) Updates, their progress change the interaction with connected devices. For many engineers, these processes could look like an overkill. However, in order to ensure the success of IoT depreciation, it is important to send software to customer devices in a quick, efficient and reliable way. In this article you will learn how to create a serverless Ci/CD workflow with Github campaigns, AWS tools and ESP IDF framework. Our approach will be a cloud approach for the device, that is, we set up the firmware aspect on the device and then integrate into the cloud infrastructure.
Set up the ESP IDF in the visual studio code
The espressional IoT development Frame(ESP IDF) is a robust development framework for the development of solutions using the ESP32. It is intended for the development of IoT applications (Internet-of Things) with Wi-Fi. BluetoothElectricity management and several other system functions.
The ESP IDF easily integrates into Visual Studio code (VScode) We will use VScode as our IDE selection.
While the application code is the most important part of the device pages setup, the CMakeLists.txt The file plays an important role in the management of the project’s creation process, the integration of components and the definition of essential configurations. The project version variable helps to create the project version number in the code. This serves as an identifier for the current device firmware version. In order to ensure an exact version test and make firmware updates easier, it is important to increase the version number for any solvable build.
cmake_minimum_required(VERSION 3.5)
set(EXTRA_COMPONENT_DIRS
$ENV{IDF_PATH}/examples/common_components/protocol_examples_common)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
# Use execute_process command to get version information from Git
execute_process(
COMMAND git describe --tags
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
OUTPUT_VARIABLE GIT_VERSION
OUTPUT_STRIP_TRAILING_WHITESPACE
ERROR_QUIET
)
# Check if Git command was successful
if(NOT GIT_VERSION)
# Set default values if Git is not available or command failed
set(BUILD_TAG "default")
set(BUILD_INFO "")
message("Git command failed: ${GIT_ERROR}")
return()
endif()
string(REPLACE "-" " " VERSION_LIST ${GIT_VERSION})
list(GET VERSION_LIST 0 BUILD_TAG)
# Print the version information
message("Build tag: ${BUILD_TAG}")
set(PROJECT_VER ${BUILD_TAG})
string(REPLACE "." "_" FILE_VER ${BUILD_TAG})
set(FILE_NAME "firmware_esp32_${FILE_VER}")
project(${FILE_NAME})
The version number is called dynamically from git tags with the command GIT described. This enables seamless integration of the project version, improvement in automation and reducing manual efforts. The version obtained, saved in the BUILD_TAG Variable is used to construct the firmware details.
Device-update support
A resilient CI/CD pipeline is only as robust as the ability to orchestrate updates on the device side. The effectiveness of a serverless OTA workflow depends on the complicated orientation between the Cloud infrastructure and the IoT devices it rules. This aspect includes the mechanisms according to which devices query the backend, determine whether an update is required and download the update. In order to call up the OTA process, the device uses an HTTP query to connect to the backend. The query contains the current version that is executed on the device. This helps the device confirm whether a new update is available for download. The device uses the esp_ota_ops Library for extracting the currently installed firmware version. This library offers a number of processes for the management of firmware updates on ESP32 devices, including calling up the current version, during esp_http_client The library helps with the management of HTTP events and ensures a robust and reliable interaction between the device and the backend.
extern const uint8_t ClientCert_pem_start() asm("_binary_certificate_pem_start");
extern const uint8_t ClientCert_pem_end() asm("_binary_certificate_pem_end");
static const char *TAG = "OTA-UPDATE";
static void client_post_rest_function() {
esp_app_desc_t running_app_info;
if (ESP_OK ==
esp_ota_get_partition_description(
esp_ota_get_running_partition(), &running_app_info)
)
{
char complete_url(256);
snprintf(complete_url,
sizeof(complete_url),
"https://stgquzvr3h.execute-api.us-east-2.amazonaws.com/dev/firmwares?rawVersion=%s",
running_app_info.version);
esp_http_client_config_t config_get = {
.url = complete_url,
.method = HTTP_METHOD_GET,
.cert_pem = (const char *)ClientCert_pem_start,
.event_handler = client_event_get_handler
};
esp_http_client_handle_t client = esp_http_client_init(&config_get);
esp_err_t err = esp_http_client_perform(client);
if (err == ESP_OK)
{
ESP_LOGI(TAG, "HTTPS GET request successful");
}
else
{
ESP_LOGE(TAG, "HTTPS GET request failed");
}
esp_http_client_cleanup(client);
}
else
{
ESP_LOGE(TAG, "Failed to get the running firmware info");
}
}
As soon as an update is required, a URL is returned to the device. This URL is used to download the latest firmware version. If no new update is available, the device continues its normal operation.
void ota_task(const char *otaURL) {
printf("OTA URL: %s\n", otaURL);
// Check if the URL starts with "https://"
if (strstr(otaURL, "https://") == NULL)
{
ESP_LOGE(TAG, "Response does not contain 'https://': %s", otaURL);
printf("RES: Update cannot be done. Invalid URL!\n");
}
else
{
perform_https_ota(otaURL);
}
}
Serleess infrastructure setup
The concept of “serveress” enables developers to concentrate more on writing code, while the cloud provider manages the infrastructure and automatically scales the resources as required. We will use AWS available services to set up the infrastructure. The entire OTA -Cloud workflow is presented with the Serverless framework. Since this project uses AWS tools, it is a prerequisite for having an AWS account to ensure that everything works properly. Create a AWS account And create a user with programmatic access that you can use to provide Cloud infrastructure on AWS.
Clone the Repository For this project and install all dependencies via the terminal with:
npm installAs soon as the dependencies are installed, create a serverless account Here. Access ID Buttons should be used to set up the settings for the providers so that the serverless framework can be accessed to your AWS account for the provisional purposes in the infrastructure.
We will use A serverless.yml File to orchestrate the provision of the serverless infrastructure.
org: thathardwareguy
app: cicd-serverless-app
service: cicd-serverless-app
plugins:
- serverless-webpack
- serverless-iam-roles-per-function
provider:
name: aws
runtime: nodejs16.x
stage: ${opt:stage, 'dev'}
region: ${opt:region, 'us-east-2'}
environment:
PROJECT_NAME: secure-ota-cicd
FIRMWARE_TABLE: firmware-builds3-${self:provider.stage}
S3_BUCKET_NAME: esp32-cicd-firmwares-${self:provider.stage}
functions:
# configuire write s3 event to dynamoDB lambda
LogFirmwareData:
handler: lambda/firmwareData.handler
events:
- s3:
bucket: !Ref AttachmentsBucket
event: s3:ObjectCreated:*
rules:
- suffix: .bin
existing: true
iamRoleStatementsName: ${self:provider.environment.PROJECT_NAME}-firmware-data-role-${self:provider.stage}
iamRoleStatements:
- Effect: Allow
Action:
- dynamodb:PutItem
Resource: arn:aws:dynamodb:${self:provider.region}:*:table/${self:provider.environment.FIRMWARE_TABLE}
GetDownloadUrl:
handler: lambda/getDownloadUrl.handler
events:
- http:
method: get
path: firmwares
cors : true
iamRoleStatementsName: ${self:provider.environment.PROJECT_NAME}-get-download-url-role-${self:provider.stage}
iamRoleStatements:
- Effect: Allow
Action:
- dynamodb:Query
Resource: arn:aws:dynamodb:${self:provider.region}:*:table/${self:provider.environment.FIRMWARE_TABLE}
resources:
Resources:
FirmwareTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: deviceType
AttributeType: S
KeySchema:
- AttributeName: deviceType
KeyType: HASH
BillingMode: PAY_PER_REQUEST
TableName: ${self:provider.environment.FIRMWARE_TABLE}
AttachmentsBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:provider.environment.S3_BUCKET_NAME}
CorsConfiguration:
CorsRules:
- AllowedOrigins:
- '*'
AllowedHeaders:
- '*'
AllowedMethods:
- GET
- PUT
- POST
- DELETE
- HEAD
MaxAge: 300
PublicAccessBlockConfiguration:
BlockPublicAcls: true
IgnorePublicAcls: true
BlockPublicPolicy: true
RestrictPublicBuckets: true
BucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
PolicyDocument:
Id: MyPolicy
Version: '2012-10-17'
Statement:
- Sid: PublicReadForGetBucketObjects
Effect: Deny
Principal: '*'
Action: 's3:GetObject'
Resource: 'arn:aws:s3:::${self:provider.environment.S3_BUCKET_NAME}/*'
Bucket: !Ref AttachmentsBucket
The setup contains two decisive Lambda functions that serve as a backend infrastructure that are executed on AWS:
LogFirmwareData: This function records and logs firmware-related data (file name, firmware version, etc.) in Dynamodb. It reacts to S3 events that have been triggered especially for new firmware binary files (.binFiles) are added to the specified S3 -Bucket. The associated IAM role ensures that the Lambda has the permissions to carry out dynamodb operations.GetDownloadUrl: This function is responsible for generating and serving URLs for firmware updates and reacts to HTTPGETInquiries on/firmwareS end point. It uses DynamoDB for querying firmware data and returning the corresponding download link.
Navigate to the backend directory and use the sls Command to provide the Lambda functions. You should receive a similar edition if the command is called up:
sls
Running “serverless” from node_modules
The “app” in this service does not yet exist in your Organization.
? What would you like to do? Create ‘cicd-serverless-app’ in ‘thathardwareguy’ org
✓ Your project is ready to be deployed to Serverless Dashboard (org: “thathardwareguy”,app: “cicd-serverless-app”)
? Do you want to deploy now? Yes
Deploying cicd-serverless-app to stage dev (us-east-2, “kay-admin” provider)
Packaging (0s)
(Webpack) Building with webpackAs soon as the resources have been successfully provided, you will receive an edition that contains the URL with which the device can query the backend.
✓ Service deployed to stack cicd-serverless-app-dev (145s)
Dashboard: https://xxxx/apps/cicd-serverless-app/cicd-serverless-app/dev/us-east-2
Endpoint:GET - https://xxxx.execute-api.us-east-2.amazonaws.com/dev/firmwares
functions:
LogFirmwareData: cicd-serverless-app-dev-LogFirmwareData (4.1 MB)
GetDownloadUrl: cicd-serverless-app-dev-GetDowmloadUrl
Setting up Github actions
The goal of setting up a CI/CD -ota workflow is to ensure that new functions are continuously led to customer devices as soon as they are available. This is only possible if a build can be triggered automatically if new code is pressed. Used this system Github As a repository and version version system of the selection and github campaign of CI/CD to automate our build and to provide the compiled firmware for the S3 -Bucket. To create the project, we will use the available ESP IDF campaign Here. The workflow that triggers the build .github/workflows List in the root of your project and add Yaml Files with the desired workflow configurations. Each YAML file defines specific jobs such as build and test tasks and is placed in the .github/workflows Directory. The project you have previously cloned has a workflow setup.
Github actions requires a Yaml file that activates the build based on configured triggers.
name: Serverless CICD
on:
push:
tags:
- 'v*.*.*'
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Get latest release version number
id: get_version
uses: battila7/get-version-action@v2
- name: Convert Version Format
id: convert_version
run: |
version="${{ steps.get_version.outputs.version }}"
ver="${version//./_}"
echo "Formatted Version: $ver"
echo "::set-output name=formatted_version::$ver"
- name: Install ESP-IDF and Build project
uses: rmshub/esp-idf-action@v5
with:
esp_idf_version: v4.4.4
esp_idf_target: esp32
- name: Archive build output artifacts
uses: actions/upload-artifact@v3
with:
name: build
path: |
./build/firmware_esp32_${{steps.convert_version.outputs.formatted_version}}.bin
- name: Upload file to bucket
uses: zdurham/s3-upload-github-action@master
Env: FILE: ./build/firmware_esp32_${{steps.convert_version.outputs.formatted_version}}.bin
AWS_REGION: 'us-east-2'
S3_BUCKET: ${{ secrets.S3_BUCKET_NAME}}
AWS_ACCESS_KEY_ID: ${{secrets.AWS_KEY_ID}}
AWS_SECRET_ACCESS_KEY: ${{secrets.AWS_SECRET_ACCESS_KEY}}
You also have to configure secrets for authentication if you provide the compiled project for the S3 -Bucket.
To trigger a build, the workflow is activated when a new day is pressed on Github.
git add .
git commit -m ‘Edit: Change Connection string’
(main 58604ce) Edit: Change Connection string
1 file changed, 2 insertions(+), 2 deletions(-)
git tag v1.4.8
git push origin main --tags
Enumerating objects: 38, done.
Counting objects: 100% (38/38), done.Click on the actions tab in your repo and you should see the workflow executed
When the creation process is successfully completed, the file is successfully provided in the S3 -Bucket in which the device can download it.
The .bin The file that is the output of the project construction is displayed in the S3 bucket below.
Devices -ota test
Change the version of the project by testing the proper work operation of the code by creating a new day by creating a new day git tag Command:
git tag v1.4.9Compile and download the code into your ESP32. As soon as this is done, create a GIT day with a higher version number and put it in your repository. Enter your ESP32 and you should receive a similar edition:
Client HTTP_EVENT_ON_DATA: {“Response”:https://xxxxxxxxxxxx.amazonaws.com/firmware_esp32_v1_4_10.bin”}
I (7468) OTA-UPDATE: HTTPS GET request successful
Extracted URL: https://xxxxxxxxxxxx.amazonaws.com/firmware_esp32_v1_4_10.bin
OTA URL: https://xxxxxxxxxxxx.amazonaws.com/firmware_esp32_v1_4_10.bin
I (12468) esp_https_ota: Writing to partition subtype 16 at offset 0x110000
(12488) OTA-UPDATE: Running firmware version: v1.4.9
I (86218) esp_image: segment 0: paddr-00110020 vaddr=3f400020 size-1adbch (110012) map
I (86258) ́esp_image: segment 1: paddr=0012ade4 vaddr=3ffb0000 size-038d8h ( 14552) I (86258) esp_image: segment 2: paddr=0012e6c4 vaddr=40080000 size=81954h ( 6484)
I(86268)esp_image: segment 3: paddr=00130020 vaddr=400de020 size=8fd98h (589288) map
I(86458) esp_image: segment 4:paddr=001bfdc0 vaddr=40081954 size=133e0h (78816)
I(86498) esp_image: segment 0: paddr-00110020 vaddr=3f400020 size-1adbch (110012) map
I (86538) esp_image: segment 1: paddr=0012ade4 vaddr=3ffb0000 size-038d8h ( 14552) I (86538) esp_image: segment 2: paddr=0012e6c4 vaddr-40080000 size-01954h ( 6484)
I (86548) esp_image: segment 3: paddr-00130020 vaddr-400d0020 size-8fd98h (589208) map
I (86738) esp_image: segment 4: paddr=001bfdc0 vaddr=40081954 size-133e0h ( 78816)
I (86848) OTA-UPDATE: ESP_HTTPS_OTA upgrade successful. Rebooting…
I (87848) wifi:state: run -> init (0)
I (87848) wifi:pm stop, total sleep time: 54048973 us / 86176568 us
The edition shows the current version (V1.4.9 above) and the answer from the server. The file that is available in the S3 bucket is V1.4.10. After a successful update, the device starts automatically. If an update is checked again, the backend returns an answer “Device is up to date”, since the current version matches what is available in the S3 -Bucket.
I (6275) wifi:<ba-add>idx:0 (ifx:0, 9a:74:da:c6:3b:35) tid:0, ssn:18, winSizw
Client HTTP_EVENT_ON_DATA: {“Response”: “Device up to date”}”
I (6626) OTA-UPDATE: HTTPS GET request successful
Diploma
In this tutorial we can integrate a serverless architecture to achieve seamless OTA updates on IoT devices. This is just a starting point. A production system must take additional functions into account, e.g. B. Rederving downloads. Staged rolloutsand security. The next tutorial deals with:
- Integration of custom scripts that support project compilation using Github campaigns for various boards.
- The use of precisely signed URLs and private keys directly on different ways to make the OTA update process safer.
- Integrate AWS IoT jobs and Mqtt To remove the request for word cycle, the HTTP uses to check a more efficient way for updates.