Development of a secure perception and decision-making system for a collaborative robot based on deep learning

Abstract

The study is devoted to the development of a system for safe perception and decision-making for a collaborative robot capable of operating in close proximity to a human without physical guarding while maintaining productivity in an unstructured work environment. The goal is to integrate deep learning algorithms into the safety loop so that the robot adaptively adjusts its speed and trajectory in a preventive manner rather than relying solely on emergency stops. Experimental validation was performed on a laboratory testbed for assembling large-scale structures using a six-degree-of-freedom cobot with a payload capacity of 10 kg, a network of four perimeter-mounted RGB-D cameras, and a solid-state lidar mounted on the flange; computations were implemented on an embedded GPU module under ROS 2, with training in PyTorch and inference via TensorRT. The training dataset was formed from real and synthetic data and includes more than 150 000 annotated frames with bounding boxes and segmentation masks; data augmentation was applied to simulate dust, vibrations, and extreme lighting conditions. Performance was evaluated using IoU and mAP, preprocessing and inference time, end-to-end latency, the number of false stops, and missed hazardous approaches; more than 5000 interaction episodes were analyzed. A nonlinear decrease in accuracy with reduced illumination was demonstrated: at 1050 lx, mAP reaches 94,12% (model B), while at 10 lx it drops to 72,45%, whereas model C degrades to 41,98%, justifying the inclusion of low-light scenes in the dataset. Transferring inference from CPU to embedded GPUs reduced total latency from 98,45 to 22,45 ms and decreased jitter, enabling tighter timeouts; when increasing the resolution to , latency increased to 49,87 ms. Kinematic calibration showed that at a speed of 1,0 m/s, the total stopping distance is 215,34 mm with a 30 ms reaction time and 286,01 mm with 100 ms, while at 1,5 m/s it increases to 457,56 mm, making early detection critical. Fusion of lidar and stereo camera data reduced depth error by 45%, and temporal filtering eliminated up to 80% of false triggers without loss of reaction speed; no incidents were recorded during testing. The study concludes that a hybrid architecture is promising, in which neural networks handle high-level prediction, while a lower-level layer verifies control actions to meet functional safety requirements.