Distributed learning - Revision history

Alvionne Baquiran: Added Summary of survey and CIDR DLSF

2023-06-28T09:29:29Z

Added Summary of survey and CIDR DLSF

Lawrence Quizon: notes of efficient vision models

2022-12-06T01:41:25Z

notes of efficient vision models

← Older revision		Revision as of 09:41, 6 December 2022
Line 91:		Line 91:
	!#FLOPs		!#FLOPs
	\|-		\|-
	\| rowspan="3" \|~~Deep~~ Residual ~~Learning~~		\| rowspan="3" \|Residual Blocks & Skip Connections
	\| rowspan="3" \|ResNet-18		\| rowspan="3" \|ResNet-18
	\|CIFAR-10		\|CIFAR-10
Line 104:		Line 104:
	\|72.33%/91.8% (SAMix Augmented Data)		\|72.33%/91.8% (SAMix Augmented Data)
	\|-		\|-
	\|~~MobileNet~~		\|Linear Bottleneck Blocks
	\|MobileNetv2		\|MobileNetv2
	\|ImageNet		\|ImageNet
Line 111:		Line 111:
	\|300M		\|300M
	\|-		\|-
	\| rowspan="2" \|~~Baseline Effnet~~		\| rowspan="2" \|Efficiently Scaled Model Depth & Width
	\| rowspan="2" \|EfficientNetB0		\| rowspan="2" \|EfficientNetB0
	\|CIFAR-10		\|CIFAR-10
Line 121:		Line 121:
	\|'''77.1%/93.3% (Main Paper)'''		\|'''77.1%/93.3% (Main Paper)'''
	\|-		\|-
	\| rowspan="2" \|Grouped Pointwise Convolutions		\| rowspan="2" \|Grouped Pointwise Convolutions
	\|kEffNet-B0 16ch		\|kEffNet-B0 16ch
	\|CIFAR-10/100		\|CIFAR-10/100
Line 134:		Line 134:
	\|'''81M'''		\|'''81M'''
	\|-		\|-
	\|~~Patch-based Inference~~		\|Neural Architecture Search
	\|MCUNetv2M4		\|MCUNetv2M4
	\|ImageNet		\|ImageNet
Line 141:		Line 141:
	\|119M		\|119M
	\|-		\|-
	\|~~Patch-based Inference~~		\|Neural Architecture Search
	\|MCUNetv2H7		\|MCUNetv2H7
	\|ImageNet		\|ImageNet

Lawrence Quizon: Added simulation methodologies

2022-11-08T03:12:11Z

Added simulation methodologies

@@ Line 372: / Line 372: @@
 === Federated Learning ===
 A recent approach known as federated learning<ref>McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." ''Artificial intelligence and statistics''. PMLR, 2017.</ref> was put forth to solve privacy issues when training deep neural network (DNN) models, but can also be applied. By aggregating local updates to model parameters in a central host, a powerful trained model can be created in the host which can then distribute the better model.
 === References ===

Lawrence Quizon: /* State of the Art: Models demonstrated in microcontrollers */

2022-11-08T01:31:52Z

State of the Art: Models demonstrated in microcontrollers

← Older revision		Revision as of 09:31, 8 November 2022
Line 184:		Line 184:
	\|-		\|-
	\|VWW		\|VWW
	\|80		\|80%
	\|151.63ms		\|151.63ms
	\|4.03mJ		\|4.03mJ
Line 195:		Line 195:
	\|-		\|-
	\|CF10		\|CF10
	\|85		\|85%
	\|158.13ms		\|158.13ms
	\|4.15mJ		\|4.15mJ
	\|-		\|-
	\|Speech Commands		\|Speech Commands
	\|90		\|90%
	\|54.81ms		\|54.81ms
	\|1.48mJ		\|1.48mJ
	\|-		\|-
	\|ToyADMOS Car		\|ToyADMOS Car
	\|0.85		\|85%
	\|5.73ms		\|5.73ms
	\|0.152mJ		\|0.152mJ
	\|-		\|-
	\|VWW		\|VWW
	\|80		\|80%
	\|186ms		\|186ms
	\|1.721mJ		\|1.721mJ
Line 221:		Line 221:
	\|-		\|-
	\|CF10		\|CF10
	\|85		\|85%
	\|240ms		\|240ms
	\|2.248mJ		\|2.248mJ
	\|-		\|-
	\|Speech Commands		\|Speech Commands
	\|90		\|90%
	\|63.1ms		\|63.1ms
	\|0.611mJ		\|0.611mJ
	\|-		\|-
	\|ToyADMOS Car		\|ToyADMOS Car
	\|0.85		\|85%
	\|5.41ms		\|5.41ms
	\|0.045mJ		\|0.045mJ
	\|-		\|-
	\|'''CF10'''		\|'''CF10'''
	\|'''84.5'''		\|'''84.5%'''
	\|'''1.5ms'''		\|'''1.5ms'''
	\|'''2.535mJ'''		\|'''2.535mJ'''
Line 247:		Line 247:
	\|-		\|-
	\|'''Speech Commands'''		\|'''Speech Commands'''
	\|'''82.5'''		\|'''82.5%'''
	\|'''0.033ms'''		\|'''0.033ms'''
	\|'''0.0537mJ'''		\|'''0.0537mJ'''
Line 254:		Line 254:
	\|-		\|-
	\|'''ToyADMOS Car'''		\|'''ToyADMOS Car'''
	\|~~'''0.~~83~~'''~~		\|83%
	\|'''0.019ms'''		\|'''0.019ms'''
	\|'''0.03mJ'''		\|'''0.03mJ'''

Lawrence Quizon: title change

2022-11-04T11:33:44Z

title change

← Older revision		Revision as of 19:33, 4 November 2022
Line 1:		Line 1:
			{{DISPLAYTITLE:Distributed Machine Learning}}

	The use of artificial intelligence (AI) in extreme edge devices such as wireless sensor nodes (WSNs) will greatly benefit its scalability and application space. AI can be applied to solve problems with clustering, data routing, and most importantly it can be used to reduce the volume of data transmission via data compression or making conclusions from data within the node itself <ref>Alsheikh, Mohammad Abu, et al. "Machine learning in wireless sensor networks: Algorithms, strategies, and applications." ''IEEE Communications Surveys & Tutorials'' 16.4 (2014): 1996-2018.</ref>.		The use of artificial intelligence (AI) in extreme edge devices such as wireless sensor nodes (WSNs) will greatly benefit its scalability and application space. AI can be applied to solve problems with clustering, data routing, and most importantly it can be used to reduce the volume of data transmission via data compression or making conclusions from data within the node itself <ref>Alsheikh, Mohammad Abu, et al. "Machine learning in wireless sensor networks: Algorithms, strategies, and applications." ''IEEE Communications Surveys & Tutorials'' 16.4 (2014): 1996-2018.</ref>.

Lawrence Quizon: Q1 commit 1

2022-11-04T08:45:33Z

Q1 commit 1

← Older revision		Revision as of 16:45, 4 November 2022
Line 38:		Line 38:
	[[File:Concept Map of Distributed Machine Learning for WSN 2.png\|center\|frame]]		[[File:Concept Map of Distributed Machine Learning for WSN 2.png\|center\|frame]]

	=== TinyML and accepted benchmarks for Edge AI ===		== TinyML and accepted benchmarks for Edge AI ==
	The [https://www.tinyml.org/ TinyML organization] describes tinyML as the "field of machine learning technologies and applications including hardware, algorithms and software capable of performing on-device sensor data analytics at extremely low power, typically in the mW range and below, and hence enabling a variety of always-on use-cases and targeting battery operated devices".		The [https://www.tinyml.org/ TinyML organization] describes tinyML as the "field of machine learning technologies and applications including hardware, algorithms and software capable of performing on-device sensor data analytics at extremely low power, typically in the mW range and below, and hence enabling a variety of always-on use-cases and targeting battery operated devices".

Line 74:		Line 74:

	Anomaly detection is a similar task, where . This task covers cases where both the conclusions required and the input data are likely to be simple, but stakes are slightly higher where false positives and negatives must be minimized.		Anomaly detection is a similar task, where . This task covers cases where both the conclusions required and the input data are likely to be simple, but stakes are slightly higher where false positives and negatives must be minimized.

			= Efficient Software Models =

	=== State of the Art: Efficient Vision Models ===		=== State of the Art: Efficient Vision Models ===
Line 157:		Line 159:

	Apart from those, NAS is an approach where heuristic algorithms are used to find accurate models with parameter count constraints ('''ProxylessNAS, MCUNet''') by efficiently estimating the probability of high accuracy (on a specific task) from a candidate model before training and optimizing search spaces.<ref>Lin, Ji, et al. "Mcunet: Tiny deep learning on iot devices." ''Advances in Neural Information Processing Systems'' 33 (2020): 11711-11722.</ref><ref>Cai, Han, Ligeng Zhu, and Song Han. "Proxylessnas: Direct neural architecture search on target task and hardware." ''arXiv preprint arXiv:1812.00332'' (2018).</ref>		Apart from those, NAS is an approach where heuristic algorithms are used to find accurate models with parameter count constraints ('''ProxylessNAS, MCUNet''') by efficiently estimating the probability of high accuracy (on a specific task) from a candidate model before training and optimizing search spaces.<ref>Lin, Ji, et al. "Mcunet: Tiny deep learning on iot devices." ''Advances in Neural Information Processing Systems'' 33 (2020): 11711-11722.</ref><ref>Cai, Han, Ligeng Zhu, and Song Han. "Proxylessnas: Direct neural architecture search on target task and hardware." ''arXiv preprint arXiv:1812.00332'' (2018).</ref>

			= Running efficient software in small hardware =

	=== State of the Art: Models demonstrated in microcontrollers ===		=== State of the Art: Models demonstrated in microcontrollers ===
Line 278:		Line 282:
	\| rowspan="11" \|TinyEngine		\| rowspan="11" \|TinyEngine
	\|STM32F412		\|STM32F412
	\|		\|100 MHz
	\|256kB + 1M		\|256kB + 1M
	\|		\|ARM32CM4
	\|		\|1.7V-3.6V
	\|-		\|-
	\|63.5		\|63.5
Line 294:		Line 298:
	\|		\|
	\|STM32F765		\|STM32F765
	\|		\|216 MHz
	\|512k + 1M		\|512k + 1M
	\|		\|ARM32CM7
	\|		\|1.7V-3.6V
	\|-		\|-
	\|70.7		\|70.7
	\|		\|
	\|STM32H743		\|STM32H743
	\|		\|480 MHz
	\|512k + 1M		\|512k + 1M
	\|		\|ARM32CM7
	\|		\|1.7V-3.6V
	\|-		\|-
	\| rowspan="3" \|INT8		\| rowspan="3" \|INT8
Line 335:		Line 339:
	\| rowspan="2" \|		\| rowspan="2" \|
	\|STM32H743		\|STM32H743
	\|		\|480 MHz
	\|512k + 2M		\|512k + 2M
	\|		\|ARM32CM7
	\|		\|1.7V-3.6V
	\|-		\|-
	\| rowspan="2" \|STM32F412		\| rowspan="2" \|STM32F412
	\| rowspan="2" \|		\| rowspan="2" \|100 MHz
	\| rowspan="2" \|256kB + 1M		\| rowspan="2" \|256kB + 1M
	\| rowspan="2" \|		\| rowspan="2" \|ARM32CM4
	\| rowspan="2" \|		\| rowspan="2" \|1.7V-3.6V
	\|-		\|-
	\|64.9		\|64.9
	\|		\|
	\|}		\|}

			= Online learning & Learning with little data =
			As mentioned before, the AI model running on the WSN needs to be able to adapt to long-term changes in the sensing environment. To do this, one has to update the AI model based on new inputs. This puts special constraints on model requirements:

			# The AI must be able to do model updates in an unsupervised fashion, as it is infeasible to have labelled data on the edge.
			# The AI and online learning algorithm pair must not overfit, or erroneously update to misclassified data.
			# As training is known to be much more resource heavy than inference, the training algorithm must be efficient.

			Additionally, a major requirement of AI algorithms is the presence of large sets of data (labeled or unlabeled, depending on the application). For novel applications, the datasets may not yet exist such as for kinetic energy harvesting patterns for microelectromechanical (MEMS) harvesting nodes. There is a high work overhead with respect to deploying nodes to gather beforehand, and it will be hard to create effective AI that generalize from only loosely related public datasets.

			=== Transfer Learning & Classifier Fine Tuning ===
			By pre-training a heavy feature extractor and a much lighter classifier on a general but difficult task (ImageNet 1000-class classification for vision networks, for example), and then further using this pre-trained network on an easier task, it can be made to converge (for the easier task) much faster with less data.

			This scheme can be made to use much less resources by freezing (not training) parameters of earlier layers (or maybe the feature extractor as a whole, if not using a DNN), as later layers are empirically found to contribute much better to accuracy. This scheme is demonstrated in Han Lab's MCUNetv3 <ref>Lin, Ji, et al. "On-Device Training Under 256KB Memory." ''arXiv preprint arXiv:2206.15472'' (2022).</ref>. By analyzing marginal accuracy gains from freezing different parameters, they found heuristics that train models to near-original performances with tight resource constraints (under 256kB memory).

	=== Federated Learning ===		=== Federated Learning ===

Lawrence Quizon: Checkpoint 2, saving to secure backup

2022-11-04T07:57:24Z

Checkpoint 2, saving to secure backup

Show changes

Lawrence Quizon: Under construction, saving for backup.

2022-11-04T06:46:52Z

Under construction, saving for backup.

New page

The use of artificial intelligence (AI) in extreme edge devices such as wireless sensor nodes (WSNs) will greatly benefit its scalability and application space. AI can be applied to solve problems with clustering, data routing, and most importantly it can be used to reduce the volume of data transmission via data compression or making conclusions from data within the node itself <ref>Alsheikh, Mohammad Abu, et al. "Machine learning in wireless sensor networks: Algorithms, strategies, and applications." ''IEEE Communications Surveys & Tutorials'' 16.4 (2014): 1996-2018.</ref>.

At the moment, potential use cases for ML include, but are not limited to:

# Pattern Classification (including recognizing speech commands, voice recognition for authentication, bio-signal processing for medical sensors)
# Image Processing (including presence detection a.k.a. visual wake words, car counting for traffic analytics)
# Node routing and scheduling (see more in clustering)

However, since devices in the extreme edge are constrained to work with extremely low amounts of energy <ref>Ma, Dong, et al. "Sensing, computing, and communications for energy harvesting iots: A survey." ''IEEE Communications Surveys & Tutorials'' 22.2 (2019): 1222-1250.</ref>, even the simplest AI models are difficult to execute with typical sequential processors. WSNs have memories in the order of kB and clock speeds in the order of kHz to MHz due to energy constraints, rendering them unable to run state-of-the art AI applications.
{| class="wikitable"
|+
!Reference WSN Project
!Device
!CLK
!Memory
!Secondary Memory
!Processor
|-
|ReSE2NSE v1
|Digi XBee-Pro, TI MSP430F213
|16MHz
|512B
|8KB+256B
|16-bit MSP-430
|-
|ReSE2NSE v2
|ATSAMR21
|<48MHz
|32KB
|256KB
|ARM32-Cortex M0
|}
To contrast with the above specs, the TinyML recommended model to classify CIFAR-10 has 3.5MB parameters.

Apart from the above constraints, WSNs running AI are placed in potentially changing environments that are potentially different from the environment in which the AI was trained. This is a problem known as concept drift. AI algorithms running on WSN need to be able to work regardless of continuously changing patterns using concepts such as online learning.

Zhou '21 describes edge AI as using widespread edge resources to gain AI insight. This means not only running an AI algorithm on one node, but potentially cooperatively running inference and training on multiple nodes. There are several degrees to the concept, ranging from training and inference in the server, training in the server but inference in the node, and running both training and inference on the node itself. Ideally, for the highest scalability and lowest communication overhead, training and inference ideally take place in the nodes.
[[File:Concept Map of Distributed Machine Learning for WSN 2.png|center|frame]]

=== TinyML ===
The [https://www.tinyml.org/ TinyML organization] describes tinyML as the "field of machine learning technologies and applications including hardware, algorithms and software capable of performing on-device sensor data analytics at extremely low power, typically in the mW range and below, and hence enabling a variety of always-on use-cases and targeting battery operated devices".

MLPerfTiny<ref>Banbury, Colby, et al. "Mlperf tiny benchmark." ''arXiv preprint arXiv:2106.07597'' (2021).

</ref> is widely accepted as the common benchmarking requirement for TinyML works. It prescribes basic accuracy targets for 4 selected basic use cases for TinyML as shown in Table.
{| class="wikitable"
|+
!Use Case
!Dataset
!Specified Model
!Quality Target
|-
|Visual Wakewords
|MSCOCO
|MobileNetv1
|80% (top-1)
|-
|Keyword Spotting
|Google Speech Commands
|DS-CNN
|90% (top-1)
|-
|Image Classification
|CIFAR-10
|ResNet
|85% (top-1)
|-
|Anomaly Detection
|DCASE2020
|FC-Autoencoder
|85% Area under Curve
|}

=== Concept Drift ===
A recent approach known as federated learning<ref>McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." ''Artificial intelligence and statistics''. PMLR, 2017.</ref> was put forth to solve privacy issues when training deep neural network (DNN) models, but can also be applied. By aggregating local updates to model parameters in a central host, a powerful trained model can be created in the host which can then distribute the better model.