The Internet of Things (IoT), combined with the Industrial Internet of Things (IIoT), is a fast growing emerging market. It has already spread out in many different parts of our live and is no longer the buzzword of startup ecosystems. The global number of network or Internet-connected devices is increasing worldwide and has exceeded 8 billion so far.
In this blog article, I will give you a short overview of the AWS IoT Core control service and how it can be used to create serverless IoT backend solutions, based on some example scenarios.
All these IoT applications usually involve backend services accessible by the ecosystem. The cloud infrastructure and platform are among the key drivers behind all IoT services and solutions. Data ingestion, transformation, storage, processing, analysis, and integration with third-party services are the key capabilities of these platforms. IoT cloud platforms are offering real-time stream analytics and batch processing services. This seamless integration between device management and data platforms makes public cloud IoT services attractive for enterprises and developers.
AWS IoT provides multiple services from the edge to the cloud that allows you to design architectures for various IoT workloads. It provides:
The device software is used to securely connect your devices to the cloud, collect data from various sources and take local actions, even without an Internet connection.
Control services enable you to control and manage a large number of different devices. You can connect to devices, ingest data, and interact with other AWS services. With device management tools, you can onboard new devices, group your device fleet, as well as monitor and search for devices based on any combination of device attributes or states. You can define security configurations for your devices to detect unusual behaviors based on continuous monitoring of various security metrics.
Data services enable you to extract value from IoT data by running sophisticated analytics on massive volumes of data. With AWS IoT Analytics, you are able to automatically collect, process, store, and analyze IoT data using analysis and machine-learning tools at massive scale. You can clean, filter, transform or enrich this data and store it in a time-series data store for further processing. You can analyze data using an SQL-based query language to extract specific information, like calculating an average distance or threshold violations. You can apply mathematical transformations to convert unit values or run time-series analysis to validate the behaviour of your devices over time.
IoT applications include many distributed devices, that need to securely connect and interact with cloud-based services to ingest, process, analyze, and store data generated by connected devices. As described above, AWS IoT provides a huge set of services for all layers in the application. Each of these services is rather complex and would need its own article to describe its capabilities and use cases. Thus, I will focus on the base services provided by AWS IoT first and give you some insides of AWS IoT Core.
AWS IoT Core is a platform that enables you to connect your IoT devices to various AWS services and other devices. It contains various components used to connect and register your devices to the AWS cloud, transport data to other backend services, or perform actions on specific data messages.
AWS IoT Core has a strong authorization and authentication model, which provides mutual authentication and encryption at all connection points. AWS IoT Core supports device authentication based on X.509 certificates, TLS encryption for data in transit, and IAM policies to authorize fine-grained access to your AWS services and resources. You can create, deploy, and manage certificates or access policies for devices using the management UI console or accessing the API. To ensure security, each connected device must provide credentials to access the message broker or the Device Shadow service. Additionally, IP traffic to and from AWS IoT must be encrypted with TLS.
The Device Gateway is used to manage a device’s connection and supports the secure communication with AWS IoT Core. Devices can send and receive messages at any time with low latency using the MQTT, HTTP, or WebSockets protocols. The Device Gateway is fully managed and scales automatically to support a stable connection to a very large number of devices. The devices connected to AWS IoT are maintained in the AWS IoT Registry, which allows you to keep a record of all registered devices. The registry establishes a unique identity for each device and tracks metadata information like attributes and capabilities.
The high throughput pub/sub Message Broker is used to securely transmits messages to and from IoT devices and applications. It supports the MQTT protocol to publish IoT messages and subscribe for them. You can define access controls at the topic level to manage the permissions of individual connections. This ensures that your devices and applications will only send and receive authorized data messages.
With the AWS IoT Device SDK, you can connect and authenticate your devices as well as exchange messages with IoT Core platform using the MQTT, HTTP, or WebSockets protocols.
AWS IoT Rules enables you to process and analyze messages from your connected devices to interact with various AWS services. Your rules can process MQTT messages that pass through the message broker of IoT core. With an SQL-based query language, you can select data from the message payload, process it, and trigger an action to send data to other AWS services, based on business rules you have defined. A rule can validate data from one or many devices. It can also trigger different actions in parallel. You can define actions to write data to a DynamoDB database, a Kinesis stream, IoT Analytics, or to invoke a Lambda function, which gives you great flexibility to process device data.
The AWS IoT Device Shadow service maintains a device state representation in the cloud for each device you have connected to AWS IoT. Such a state is described by a JSON document, which is used to persist the last reported state of the device plus its desired future state. Devices are expected to report their current states, and the AWS IoT service takes care of synchronizing property values between a connected device and its shadow. You can use the shadow to get and set the state of a device over MQTT or HTTP, which enables cloud and mobile applications to easily interact with the connected devices registered in AWS IoT Core.
So now that we have an overview of the services and features of AWS IoT Core, let’s take a look at how we can actually use them to interact with our devices.
Let’s assume we have a sensor which is constantly sending various measurements to the cloud, where they need to be processed and analyzed. If a critical threshold is exceeded the user should be notified immediately with a push notification to his mobile phone. How can this common use case be realized with a serverless backend architecture?
To send data from your device to the IoT backend, you need to publish messages via MQTT to the message broker of AWS IOT Core, using a predefined topic (1).
You can also update the device shadow (2) to store the new state information and send a message to all subscribers with the difference between desired or reported state (3).
After the message has been received by the message broker, it gets processed by the rule engine (4).
The rule triggers an action to invoke AWS services and pass data from the thing’s message to that service. In our example, the rule will trigger a Lambda function (5), which will analyze the current sensor value and compare it to a predefined threshold. The threshold value is defined in a JSON document stored in DynamoDB.
The Lambda function requests the value from the database (6) and evaluates whether a critical limit value has been exceeded.
In this case, the Lambda function sends a message to SNS via a given topic (7).
The SNS service is used to send the push notification to the user’s mobile device to report the issue (8).
We have seen how to receive and process IoT data from a device in the cloud and notify a mobile client in critical cases. The next scenario shows how we can send data from a mobile client to our device.
Let’s assume we want to update the configuration of our device using an application running on a mobile phone.
In order to process HTTP requests, we can use the AWS API Gateway and create a specific REST endpoint for the device configuration. The application sends a request to this configuration endpoint of our API containing the updated configuration in the JSON body (1).
This request will trigger a Lambda function (2), which is used to process the JSON and publish the new configuration via MQTT to the message broker of AWS IOT Core via a given topic (3).
The device has subscribed to that topic, directly receives the message, and handles the configuration update accordingly (4).
Alternatively, we can also use the device shadow and send the delta of the specific configuration by updating the device shadow topic “/shadow/update/delta” (5).
There are many use cases where the value proposition of an IoT application lies in collecting huge amounts of telemetry information of many connected devices. This data can be a valuable source of information when it gets processed, analyzed, and visualized in a scalable, cost-efficient way. Engineers can monitor the device state and activity as well as performance. The marketing and management teams can check for usage patterns and process statistics. Such a huge data source could be used to optimize certain processes with machine learning algorithms used for predictive maintenance or anomaly detection.
The following example solution describes a scenario where we collect telemetry data from many IoT devices in real-time. This data gets processed and analyzed to create specific metrics, which should be visualized in real-time. It should also be persisted for future processing.
So how can we build such an application with AWS services in a highly available and reliable serverless architecture at a huge scale, where we need to ingest large volumes of real-time data from many distributed IoT devices?
The telemetry data from all our IoT devices is sent to the backend over MQTT to the same topic, and all messages are received within AWS IOT Core (1). The message broker is able to scale automatically, and it handles the complete message volume of all devices.
Each received message triggers a rule that is configured for the specified topic name (2). The rule engine allows for selecting data from message payloads, processing it, and sending it to other services.
The executed rule triggers an action which sends the message to a Kinesis Firehose delivery stream (3). Firehose is used to decouple the processing logic from the data ingestion. This enables us to asynchronously consume messages by multiple independent consumers and to consolidate the continuous data stream into batches for further processing.
The incoming data from the Firehose delivery stream is forwarded to Kinesis Analytics, where we can process and analyze the data in real time using standard SQL queries (4). We can extract specific data sections from the incoming data stream and perform real-time ETL on it. In our scenario, we are processing the telemetry data to transform them to specific device metrics.
We are using Amazon Quicksight to visualize the metrics in an interactive dashboard, which makes it easy to build visualizations, perform ad-hoc analysis, and display business insights from the data. Quicksight supports many data sources.We are using an S3 bucket to provide the metics in form of a flat JSON file. To provide this file, we are first sending the processed data from Analytics to a second Firehose delivery stream. There the data is batched into the JSON file within a desired time frame (5).
When using the data transformation within Firehose, incoming data is buffered, and a specified Lambda function is asynchronously invoked with each buffered data batch. The JSON file is then stored in an S3 bucket (6) from where it can be retrieved by Quicksight (7).
Additionally to the real-time visualization, we want to store the metrics within a database for future processing and analytics. For this, we can send the metrics data to an Kinesis data stream (8).
The benefit here is that you can configure a Lambda function to be a consumer of this data stream. With that mapping, Lambda will automatically read all records from the data stream and invoke the configured function synchronously with an event that contains the stream records (9).
From within the Lambda function, you can access the metrics and send them to various AWS services for further processing, like DynamoDB, SNS, or other third party services. In our case, we are sending the metrics to a DynamoDB table (10).
In this scenario, you have seen how to build an IoT analytics application to ingest, process, transform, visualize, and persist device data in near real time. For this, we are entirely using AWS managed services to create a scalable, reliable, and cost-efficient serverless architecture.
The backend infrastructure is a key aspect behind all IoT applications and solutions. Public cloud provider like AWS offer a huge amount of services, which can be used to create highly scalable, reliable, and secure IoT platforms for every kind of IoT workload. With easy to use serverless services, you can quickly and effectively create a serverless IoT backend to collect, process, analyse, and visualize data from IoT devices. The services from AWS can be combined in a very flexible way to create powerful custom solutions. The pay-as-you-go pricing model for AWS IoT services allows you to build application without any upfront costs and pay only for what you actually use. All these aspects lead to increasing popularity of cloud providers. They are probably the best choice for creating new and innovative IoT products and solutions.