PSHS Internship 2023

2023-06-27T02:36:19Z

Lawrence Quizon:

PSHS Internship 2023

2023-06-26T08:42:24Z

Lawrence Quizon:

PSHS Internship 2023

2023-06-23T09:10:54Z

Lawrence Quizon: more resources

2022-11-04T07:57:24Z

Lawrence Quizon: Checkpoint 2, saving to secure backup

The use of artificial intelligence (AI) in extreme edge devices such as wireless sensor nodes (WSNs) will greatly benefit its scalability and application space. AI can be applied to solve problems with clustering, data routing, and most importantly it can be used to reduce the volume of data transmission via data compression or making conclusions from data within the node itself <ref>Alsheikh, Mohammad Abu, et al. "Machine learning in wireless sensor networks: Algorithms, strategies, and applications." ''IEEE Communications Surveys & Tutorials'' 16.4 (2014): 1996-2018.</ref>.

At the moment, potential use cases for ML in WSN include, but are not limited to:

# General Pattern Classification (including recognizing speech commands, voice recognition for authentication, bio-signal processing for medical sensors)
# Image Processing (including presence detection a.k.a. visual wake words, car counting for traffic analytics)
# Node routing and scheduling to improve network scalability and lifespan (through the use of RL, see more in [[Clustering and routing|clustering]]).

However, since devices in the extreme edge are constrained to work with extremely low amounts of energy <ref>Ma, Dong, et al. "Sensing, computing, and communications for energy harvesting iots: A survey." ''IEEE Communications Surveys & Tutorials'' 22.2 (2019): 1222-1250.</ref>, even the simplest AI models are difficult to execute with typical sequential processors. WSNs have memories in the order of kB and clock speeds in the order of kHz to MHz due to energy constraints, rendering them unable to run state-of-the art AI applications.
{| class="wikitable"
|+
!Reference WSN Project
!Device
!CLK
!Memory
!Secondary Memory
!Processor
|-
|ReSE2NSE v1
|Digi XBee-Pro, TI MSP430F213
|16MHz
|512B
|8KB+256B
|16-bit MSP-430
|-
|ReSE2NSE v2
|ATSAMR21
|<48MHz
|32KB
|256KB
|ARM32-Cortex M0
|}
To contrast with the above specs, the TinyML recommended model to classify CIFAR-10 has 3.5MB parameters.

Apart from the above constraints, WSNs running AI are placed in potentially changing environments that are potentially different from the environment in which the AI was trained. This is a problem known as concept drift. AI algorithms running on WSN need to be able to work regardless of continuously changing patterns using concepts such as online learning.

Zhou '21 describes edge AI as using widespread edge resources to gain AI insight. This means not only running an AI algorithm on one node, but potentially cooperatively running inference and training on multiple nodes. There are several degrees to the concept, ranging from training and inference in the server, training in the server but inference in the node, and running both training and inference on the node itself. Ideally, for the highest scalability and lowest communication overhead, training and inference ideally take place in the nodes.
[[File:Concept Map of Distributed Machine Learning for WSN 2.png|center|frame]]

=== TinyML and accepted benchmarks for Edge AI ===
The [https://www.tinyml.org/ TinyML organization] describes tinyML as the "field of machine learning technologies and applications including hardware, algorithms and software capable of performing on-device sensor data analytics at extremely low power, typically in the mW range and below, and hence enabling a variety of always-on use-cases and targeting battery operated devices".

MLPerfTiny<ref>Banbury, Colby, et al. "Mlperf tiny benchmark." ''arXiv preprint arXiv:2106.07597'' (2021).

</ref> is widely accepted as the common benchmarking requirement for TinyML works. It prescribes basic accuracy targets along with suggested models (if benchmarking hardware) for 4 selected basic use cases for TinyML as shown in the table below.
{| class="wikitable"
|+
!Use Case
!Dataset
!Suggested Model
!Quality Target
|-
|Visual Wakewords
|MSCOCO
|MobileNetv1
|80% (top-1)
|-
|Keyword Spotting
|Google Speech Commands
|DS-CNN
|90% (top-1)
|-
|Image Classification
|CIFAR-10
|ResNet
|85% (top-1)
|-
|Anomaly Detection
|DCASE2020
|FC-Autoencoder
|85% Area under Curve
|}
Visual Wakewords is a classification task of telling whether or not a person is in a picture. This covers similar tasks where complex high-resolution data is provided but the conclusions required are simple.

Anomaly detection is a similar task, where . This task covers cases where both the conclusions required and the input data are likely to be simple, but stakes are slightly higher where false positives and negatives must be minimized.

=== State of the Art: Efficient Vision Models ===
Traditional ML techniques struggle most with high-dimension image classification tasks. This is the main task for which heavily parametrized resource-heavy neural networks are used. Additionally, similar architectures are applied to reinforcement learning (such as the one for AlphaZero)<ref>Silver, David, et al. "A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play." Science 362.6419 (2018): 1140-1144.</ref>, and so similar efficient architectures are needed for running RL clustering models.

Shown below is a table summarizing the most efficient known vision models.
{| class="wikitable"
|+
!Notes
!Architecture
! colspan="2" |Applied to
!Parameters
!#FLOPs
|-
| rowspan="3" |Deep Residual Learning
| rowspan="3" |ResNet-18
|CIFAR-10
|85% (Wan et al.)
| rowspan="3" |11M
| rowspan="3" |1800M
|-
|CIFAR-100
|82.3% (SAMix Augmented Data)
|-
|ImageNet
|72.33%/91.8% (SAMix Augmented Data)
|-
|MobileNet
|MobileNetv2
|ImageNet
|72%
|3.5M
|300M
|-
| rowspan="2" |Baseline Effnet
| rowspan="2" |EfficientNetB0
|CIFAR-10
|'''93.52% (Main Paper) (Transfer Learning)'''
| rowspan="2" |5.3M
| rowspan="2" |390M
|-
|ImageNet
|'''77.1%/93.3% (Main Paper)'''
|-
| rowspan="2" |Grouped Pointwise Convolutions
|kEffNet-B0 16ch
|CIFAR-10/100
|92.24%/71.92%
|0.64M
|129M
|-
|kMobileNet Large 16ch
|CIFAR-10/100
|92.74%/71.36%
|'''0.40M'''
|'''81M'''
|-
|Patch-based Inference
|MCUNetv2M4
|ImageNet
|64.90%
|0.47M
|119M
|-
|Patch-based Inference
|MCUNetv2H7
|ImageNet
|71.80%
|0.67M
|256M
|}

=== Principles of Efficient Inference ===
Convolutional neural networks (CNNs) are widely used for any task with data that come with spatial relations, such as images or time-based data.

Empirically, deeper networks (with more layers) are known to perform better on more complicated tasks <ref>He, Kaiming, et al. "Deep residual learning for image recognition." ''Proceedings of the IEEE conference on computer vision and pattern recognition''. 2016.</ref>. However, making networks deeper had a limit where training them is no longer possible after a certain depth is reached. Models such as '''ResNet''' with sets of layers that skip connections (known as residual blocks) between layers was introduced to solve this problem allowing for extremely deep networks. <ref>He, Kaiming, et al. "Deep residual learning for image recognition." ''Proceedings of the IEEE conference on computer vision and pattern recognition''. 2016.</ref>

Improving upon that, it was found with '''MobileNetv2''' that models with residual blocks that reduce and then expand the number of channels (known as bottleneck residuals or inverted residuals) allows networks to keep good performance with a much lower parameter count and number of operations.<ref>Sandler, Mark, et al. "Mobilenetv2: Inverted residuals and linear bottlenecks." ''Proceedings of the IEEE conference on computer vision and pattern recognition''. 2018.</ref>

'''EfficientNet''' was developed when researchers attempted to analyze how performance changes in the former networks as you vary parameters of the architecture, like the number of channels or the network depth. They found both ways to improve accuracy by scaling the network up and also ways to improve efficiency by scaling the network down in a way that preserves accuracy.<ref>Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." ''International conference on machine learning''. PMLR, 2019.</ref>

Works this year ('''kEffNet, kMobileNet''') have found that applying old principles like grouped convolutions and pointwise convolutions to the recent efficient networks further reduce the needed parameters and float operations. <ref>Schuler, Joao Paulo Schwarz, et al. "Grouped Pointwise Convolutions Reduce Parameters in Convolutional Neural Networks." ''MENDEL''. Vol. 28. No. 1. 2022.</ref>

Apart from those, NAS is an approach where heuristic algorithms are used to find accurate models with parameter count constraints ('''ProxylessNAS, MCUNet''') by efficiently estimating the probability of high accuracy (on a specific task) from a candidate model before training and optimizing search spaces.<ref>Lin, Ji, et al. "Mcunet: Tiny deep learning on iot devices." ''Advances in Neural Information Processing Systems'' 33 (2020): 11711-11722.</ref><ref>Cai, Han, Ligeng Zhu, and Song Han. "Proxylessnas: Direct neural architecture search on target task and hardware." ''arXiv preprint arXiv:1812.00332'' (2018).</ref>

=== State of the Art: Models demonstrated in microcontrollers ===
Microcontrollers are devices that can be used for IoT and also as wireless sensors. Works in the TinyML field tend to demonstrate their works on resource-constrained microcontrollers, the best of whom are summarized here.

The table below summarizes the best demonstrations submitted to [https://mlcommons.org/en/inference-tiny-07/ MLCommons: Tiny Inference]. The most efficient work by far is by a digital accelerator, part of the open division, which used custom models (not the prescribed models) and special high level synthesis (HLS) on an FPGA, and is work by CERN. <ref>Borras, Hendrik, et al. "Open-source FPGA-ML codesign for the MLPerf Tiny Benchmark." ''arXiv preprint arXiv:2206.11791'' (2022).</ref>
{| class="wikitable"
|+
| rowspan="2" |'''Application'''
| rowspan="2" |'''Accuracy/AUC/Dist'''
| rowspan="2" |'''Inference Latency'''
| rowspan="2" |'''Energy Per Inference'''
| rowspan="2" |'''Software Stack'''
| colspan="5" |'''Device, Specs'''
|-
|'''Device'''
|'''CLK'''
|'''Memory + Secondary'''
|'''Processor'''
|'''Voltage'''
|-
|VWW
|80
|151.63ms
|4.03mJ
| rowspan="4" |X-Cube-AI v.7.1.0
| rowspan="4" |Nucleo-U575ZI-Q
| rowspan="4" |160 MHz
| rowspan="4" |768k + 2M
| rowspan="4" |ARM32CM33
| rowspan="4" |1.8V / SMPS
|-
|CF10
|85
|158.13ms
|4.15mJ
|-
|Speech Commands
|90
|54.81ms
|1.48mJ
|-
|ToyADMOS Car
|0.85
|5.73ms
|0.152mJ
|-
|VWW
|80
|186ms
|1.721mJ
| rowspan="4" |Silicon Labs Gecko SDK/ TFLite Micro
| rowspan="4" |Silicon Labs xG24-DK2601B
| rowspan="4" |78 MHz
| rowspan="4" |256k + 1.5M
| rowspan="4" |ARM32CM33 w/ FPU + DSP + TrustZone
| rowspan="4" |1.8V
|-
|CF10
|85
|240ms
|2.248mJ
|-
|Speech Commands
|90
|63.1ms
|0.611mJ
|-
|ToyADMOS Car
|0.85
|5.41ms
|0.045mJ
|-
|'''CF10'''
|'''84.5'''
|'''1.5ms'''
|'''2.535mJ'''
|FINN
| rowspan="3" |Xilinx PynqZ2
| rowspan="3" |100 MHz / 650 MHz
| rowspan="3" |+ 128M
| rowspan="3" |Dual Core ARM, Cortex-A9 MPCore
|?
|-
|'''Speech Commands'''
|'''82.5'''
|'''0.033ms'''
|'''0.0537mJ'''
|FINN
|?
|-
|'''ToyADMOS Car'''
|'''0.83'''
|'''0.019ms'''
|'''0.03mJ'''
|hls4ml
|?
|}
In addition to the above, [https://hanlab.mit.edu/ MIT's Han Lab] has a lot of MLPerfTiny-passing works demonstrated on very small microcontrollers, as tabulated below.
{| class="wikitable"
|+
!
!
!
!
!Accuracy
!Latency
!Software Stack
!Device
!CLK
!Memory + Secondary
!Processor
!Voltage
|-
| rowspan="8" |MCUNet1
| rowspan="8" |MCUNet
| rowspan="4" |INT4
| rowspan="4" |ImageNet
|62
|
| rowspan="11" |TinyEngine
|STM32F412
|
|256kB + 1M
|
|
|-
|63.5
|
|STM32F746
|216 MHz
|320k + 1M
|ARM32CM7
|1.7V-3.6V
|-
|65.9
|
|STM32F765
|
|512k + 1M
|
|
|-
|70.7
|
|STM32H743
|
|512k + 1M
|
|
|-
| rowspan="3" |INT8
| rowspan="3" |VWW
|92
|880ms
| rowspan="4" |STM32F746
| rowspan="4" |216 MHz
| rowspan="4" |320k + 1M
| rowspan="4" |ARM32CM7
| rowspan="4" |1.7V-3.6V
|-
|88.7
|200ms
|-
|87
|100ms
|-
|INT8
|Speech Commands
|96
|1105ms
|-
| rowspan="3" |MCUNet2
| rowspan="3" |MCUNet2
| rowspan="3" |INT8
| rowspan="3" |ImageNet
| rowspan="2" |71.8
| rowspan="2" |
|STM32H743
|
|512k + 2M
|
|
|-
| rowspan="2" |STM32F412
| rowspan="2" |
| rowspan="2" |256kB + 1M
| rowspan="2" |
| rowspan="2" |
|-
|64.9
|
|}

=== Federated Learning ===
A recent approach known as federated learning<ref>McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." ''Artificial intelligence and statistics''. PMLR, 2017.</ref> was put forth to solve privacy issues when training deep neural network (DNN) models, but can also be applied. By aggregating local updates to model parameters in a central host, a powerful trained model can be created in the host which can then distribute the better model.

=== References ===

Distributed learning

2022-11-04T06:46:52Z

Lawrence Quizon: Under construction, saving for backup.

File:Concept Map of Distributed Machine Learning for WSN 2.png

2022-11-04T06:03:27Z

Lawrence Quizon:

Map showing the relationships of ideas involved with enabling ML for WSN, including the specific constraints they work on.

File:Concept Map of Distributed Machine Learning for WSN.svg

2022-11-04T06:00:26Z

Lawrence Quizon:

Map displaying the constraints involved with running distributed ML on WSN, along with potential concepts and solutions.