In the proto_lab, a continuous data pipeline has been designed for the transfer of sensor data from manufacturing machines to the Azure Cloud. When selecting suitable tools and services, the specific needs of an educational and research laboratory environment were taken into account to enable diverse usage. The result is a customized and cost-effective solution for the proto_lab!
The architecture enables the testing of latest products in the cloud. For example, performance data from a production facility can be exchanged via the Event Grid, which recently started supporting the MQTT protocol. From the Event Grid, the data flows into the Event Hub, a buffer station for large amounts of data that bundles messages in a resource-efficient manner and forwards them to Azure Function in batches. There, the messages are read, processed, and finally systematically stored in Azure Storage, a cost-effective repository for large data volumes. Here, sensor raw data at the lowest bronze level can initially be assigned to the respective production machines. In the next higher silver level, already processed and merged data is available, which can also be graphically represented. Data enhancement is made possible through a pipeline to Azure Synapse, which pulls data from Azure Storage, processes it, and deposits it back there in different processing stages (bronze, silver, or gold). In the highest gold level, the data is finally written to a SQL database, enabling targeted data queries. This allows for the evaluation of performance data, such as determining machine states (on, off, idle, processing), or analyzing and optimizing the energy consumption of machines or individual processes.