Development of Multi-Robotic Arm System for Sorting System Using Computer Vision

Cong Duy Vo, Duy Anh Dang, Phuong Hoai Le

Abstract


This paper develops a multi-robotic arm system and a stereo vision system to sort objects in the right position according to size and shape attributes. The robotic arm system consists of one master and three slave robots associated with three conveyor belts. Each robotic arm is controlled by a robot controller based on a microcontroller. A master controller is used for the vision system and communicating with slave robotic arms using the Modbus RTU protocol through an RS485 serial interface. The stereo vision system is built to determine the 3D coordinates of the object. Instead of rebuilding the entire disparity map, which is computationally expensive, the centroids of the objects in the two images are calculated to determine the depth value. After that, we can calculate the 3D coordinates of the object by using the formula of the pinhole camera model. Objects are picked up and placed on a conveyor branch according to their shape. The conveyor transports the object to the location of the slave robot. Based on the size attribute that the slave robot receives from the master, the object is picked and placed in the right position. Experiment results reveal the effectiveness of the system. The system can be used in industrial processes to reduce the required time and improve the performance of the production line.

Keywords


Robot arm; Computer vision; Stereo vison; Modbus RTU; RS485.

Full Text:

PDF

References


R. Song, F. Li, T. Fu and J. Zhao, “A Robotic Automatic Assembly System Based on Vision,” Applied Sciences, vol. 10, no. 3, 2020, https://doi.org/10.3390/app10031157.

V. D. Cong, “Industrial robot arm controller based on programmable system-on-chip device,” FME Transactions, vol. 49, no.4, pp. 1025-1034, 2021, https://doi.org/10.5937/fme2104025C.

M. Z. U. Rahman, M. Usman, A. Farea, N. Ahmad, I. Mahmood and M. Imran, “Vision-Based Mobile Manipulator for Handling andTransportation of Supermarket Products,” Mathematical Problems in Engineering, vol. 7, 2022, https://doi.org/10.1155/2022/3883845.

F. Li, Q. Jiang, W. Quan, S. Cai, R. Song and Y. Li, “Manipulation Skill Acquisition forRobotic Assembly Based on Multi-ModalInformation Description,” IEEE Access, vol. 8, pp. 6282 – 6294, 2019, https://doi.org/10.1109/ACCESS.2019.2934174.

P. Guo, Z. Zhang, Y. Liu, Y. Liu, D. Zhu and C. Shao, “A Skill Programming Method Based on Assembly Motion Primitive for Modular Assembly System,” IEEE Access, vol. 9, pp. 101369-101380, 2021, https://doi.org/10.1109/ACCESS.2021.3080838.

V. D. Cong, L.D. Hanh, L.H. Phuong, D.A. Duy, “Design and development of robot arm system for classification and sorting using machine vision,” FME Transactions, vol. 50, no.1, 2022, https://doi.org/10.5937/fme2201181C.

T. Jiang, H. Cui, X. Cheng, W. Tian, “A Measurement Method for Robot Peg-in-Hole Prealignment Based on Combined Two-Level Visual Sensors,” IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-12, 2021, https://doi.org/10.1109/TIM.2020.3026802.

Q. Zhao, X. Li, J. Lu and J. Yi, “Monocular vision-based parameter estimation for mobile robotic painting,” IEEE Transactions on Instrumentation and Measurement, vol. 68, no. 10, pp. 3589-3599, Oct. 2019, https://doi.org/10.1109/TIM.2018.2878427.

F. Li, Q. Jiang, W. Quan, S. Cai, R. Song and Y. Li, “Manipulation skill acquisition for robotic assembly based on multi-modal information description,” IEEE Access, vol. 8, pp. 6282-6294, 2020, https://doi.org/10.1109/ACCESS.2019.2934174.

L. D. Hanh and V.D. Cong, “Implement contour following task of objects with unknown geometric models by using combination of two visual servoing techniques,” International Journal of Computational Vision and Robotics, vol. 12, no. 5, pp. 464–486, 2022, https://doi.org/10.1504/IJCVR.2022.10048788.

C. Cai, R. Qiao, H. Meng and F. Wang, “A novel measurement system based on binocular fisheye vision and its application in dynamic environment,” IEEE Access, vol. 7, pp. 156443-156451, 2019, https://doi.org/10.1109/ACCESS.2019.2949172.

O. I. Abdullah, W.T. Abbood and H.K. Hussein, “Development of automated liquid filling system based on the interactive design approach,” FME Transactions, vol. 48, no. 4, pp. 938-945, 2020, https://doi.org/10.5937/fme2004938A.

L. Caruana and E. Francalanza, “Safety 4.0 for Collaborative Robotics in the Factories of the Future,” FME Transactions, vol. 49, no. 4, pp. 842-850, 2021.

V. D. Cong, L.D. Hanh, “A new decoupled control law for image-based visual servoing control of robot manipulators,” International Journal of Intelligent Robotics and Applications, vol. 6, pp. 576–585, 2022, https://doi.org/10.1007/s41315-022-00223-5.

J. Wang, C. Hu, Y. Wang, Y. Zhu, “Dynamics Learning With Object-Centric Interaction Networks for Robot Manipulation,” IEEE Access, vol. 9, pp. 68277-68288, 2021, https://doi.org/10.1109/ACCESS.2021.3077117.

C. Hu, T. Ou, Y. Zhu and L. Zhu, “GRU-type LARC strategy for precision motion control with accurate tracking error prediction,” IEEE Transactions on Industrial Electronics, vol. 68, no. 1, pp. 812-820, Jan. 2021.

B. Ichter and M. Pavone, “Robot motion planning in learned latent spaces,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2407-2414, Jul. 2019.

X. Li, J. Xiao, W. Zhao, H. Liu, G. Wang, “Multiple peg-in-hole compliant assembly based on a learning-accelerated deep deterministic policy gradient strategy,” Industrial Robot: the international journal of robotics research and application, vol. 49, no. 1, pp. 54, 2022.

C. Weckenborg, K. Kieckhäfer, C. Müller, C. et al., “Balancing of assembly lines with collaborative robots,” Business Research, vol. 13, pp. 93–132, 2020, https://doi.org/10.1007/s40685-019-0101-y

F. Zha, Y. Fu, P. Wang, W. Guo, M. Li, X. Wang, H. Cai, “Semantic 3D Reconstruction for Robotic Manipulators with an Eye-In-Hand Vision System,” Applied Sciences, vol. 10, 2020, https://doi.org/10.3390/app10031183.

M. Intisar, M.M. Khan, M.R. Islam and M. Masud, “Computer Vision Based Robotic Arm Controlled Using Interactive GUI,” Intelligent Automation and Soft Computing, vol. 27, no. 2, pp. 533-550, 2021, https://doi.org/10.32604/iasc.2021.015482.

H. Wei, X.X. Chen and X.Y. Miao, “Vision‑guided fine‑operation of robot and its application in eight‑puzzle game,” International Journal of Intelligent Robotics and Applications, vol. 5, no.3, pp. 576–589, 2021, https://doi.org/10.1007/s41315-021-00186-z.

A. Zakhama, L. Charrabi and K. Jelassi, “Intelligent Selective Compliance Articulated Robot Arm robot with object recognition in a multi-agent manufacturing system,” International Journal of Advanced Robotic Systems, vol. 16, no. 2, 2019, https://doi.org/10.1177/1729881419841145.

H. M. Ali, Y. Hashim and G. A. Al-Sakkal, “Design and implementation of Arduino based robotic arm,” International Journal of Electrical and Computer Engineering (IJECE), vol. 12, no. 2, pp. 1411-1418, 2021, https://doi.org/10.11591/ijece.v12i2.

F. Fang, M. Shi, K. Qian, et al., “A human-aware navigation method for social robot based on multi-layer cost map,” International Journal of Intelligent Robotics and Applications, vol. 4, pp. 308–318, 2020, https://doi.org/10.1007/s41315-020-00125-4.

A. H. Wei, B.Y. Chen, “Robotic object recognition and grasping with a natural background,” International Journal of Advanced Robotic Systems, vol. 17, no. 2, pp. 1-17, 2020, https://doi.org/10.1177/1729881420921102.

M. M. Babr, M. Faghihabdolahi, D. Ristić-Durrant and K. Michels, “Deep Learning-Based Occlusion Handling of Overlapped Plants for Robotic Grasping,” Applied Sciences, vol. 12, no. 7, 2022, https://doi.org/10.3390/app12073655.

A. Atefi, Y. Ge, S. Pitla and J. Schnable, “Robotic Detection and Grasp of Maize and Sorghum: Stem Measurement with Contact,” Robotics, vol. 9, no. 3, 2020, https://doi.org/10.3390/robotics9030058.

M. Tahmasebi, M. Gohari and A. Emami, “An Autonomous Pesticide Sprayer Robot with a Color-based Vision System,” International Journal of Robotics and Control Systems, vol. 2, no. 1, pp. 115-123, 2022, https://doi.org/10.31763/ijrcs.v2i1.480.

M. N. Sofu, O. Er, M.C. Kayacan and B. Cetisli, “Design of an automatic apple sorting system using machine vision,” Computers and Electronics in Agriculture, vol. 127, pp. 395-405, 2016.

X. Wu, X. Ling and J. Liu, “Location recognition algorithm for vision-based industrial sorting robot via deep learning,” International Journal of Pattern Recognition and Artificial Intelligence, vol. 33, no. 07, Jun. 2019.

Q. Wu, M. Li, X. Qi, Y. Hu, B. Li, and J. Zhang, “Coordinated control of a dual-arm robot for surgical instrument sorting tasks,” Robotics and Autonomous Systems, vol. 112, pp. 1-12, Feb. 2019.

Y. Lin, H. Zhou, M. Chen and H. Min, “Automatic sorting system for industrial robot with 3D visual perception and natural language interaction,” Measurement and Control, vol. 52, no. 1, pp. 100-115, Jan. 2019.

K. Zhou, Z. Meng, M. He, Z. Hou and T. Li, “Design and Test of a Sorting Device Based on Machine Vision,” IEEE Access, vol. 8, pp. 27178-27187, 2020.

R. Song, F. Li, T. Fu and J. Zhao, “A Robotic Automatic Assembly System Based on Vision,” Applied Sciences, vol. 10, no. 3, 2020, https://doi.org/10.3390/app10031157.

W. Wan, F. Lu, Z. Wu and K. Harada, “Teaching robots to do object assembly using multi-modal 3D vision,” Neurocomputing, vol. 25, no. 9, pp. 85–93, 2017.

H. Chen, J. Xu, B. Zhang, T.A. Fuhlbrigge, “Improved parameter optimization method for complex assembly process in robotic manufacturing,” Industrial Robot, vol. 44, no. 1, pp. 21–27, 2017, https://doi.org/10.1108/IR-03-2016-0098.

L. Yang, M. Chong, C. Bai and J. Li, “A multi-workpieces recognition algorithm based on shape-SVM learning model,” Journal of Physics Conference Series, vol. 1087, no. 2, 2018, https://doi.org/10.1088/1742-6596/1087/2/022025.

L. Roveda, “Adaptive Interaction Controller for Compliant Robot Base Applications,” IEEE Access, vol. 7, pp. 6553–6561, 2019.

T. Dewi, Z. Mulya, P. Risma and Y. Oktarina, “BLOB analysis of an automatic vision guided system for a fruit picking and placing robot,” International Journal of Computational Vision and Robotics, vol. 11, no. 3, pp.315–327, 2021.

Q. Wei, C. Yang, W. Fan and Y. Zhao, “Design of demonstration-driven assembling manipulator,” Applied Sciences, vol. 8, no. 5, pp. 797, May 2018.

B. Jia, S. Liu and Y. Liu, “Visual trajectory tracking of industrial manipulator with iterative learning control,” Industrial Robot: An International Journal, vol. 42, no. 1, pp. 54-63, 2015.

Y.C. Du, T. Taryudi, C.T. Tsai, et al., “Eye-to-hand robotic tracking and grabbing based on binocular vision,” Microsystem Technologies, vol. 27, no. 4, pp. 1699–1710, 2021, https://doi.org/10.1007/s00542-019-04475-3.

Taryudi and M.S.Wang, “Eye to hand calibration using ANFIS for stereo vision-based object manipulation system,” Microsystem Technologies, vol. 24, no. 1, pp. 305–317, 2018, https://doi.org/10.1007/s00542-017-3315-y.

Y.C. Du, M. Muslikhin, T.H. Hsieh and M.S. Wang, “Stereo Vision-Based Object Recognition and Manipulation by Regions with Convolutional Neural Network,” Electronics, vol. 9, no. 2, 2020, https://doi.org/10.3390/electronics9020210.

R. Holubek and M. Vagaš, “Center of Gravity Coordinates Estimation Based on an Overall Brightness Average Determined from the 3D Vision System,” Applied Sciences, vol. 12, no. 1, 2022, https://doi.org/10.3390/app12010286.

C.A. Aguilera, C. Aguilera, C.A. Navarro and A.D. Sappa, “Fast CNN Stereo Depth Estimation through Embedded GPU Devices,” Sensors, vol. 20, no. 11, 2020, https://doi.org/10.3390/s20113249.

A. Mohamed, P. Culverhouse, A. Cangelosi, et al., “Active stereo platform: online epipolar geometry update,” EURASIP Journal on Image and Video Processing, vol. 54, 2018, https://doi.org/10.1186/s13640-018-0292-8.

W. Chen, T. Xu, J. Liu, M. Wang and D. Zhao, “Picking robot visual servo control based on modified fuzzy neural network sliding mode algorithms,” Electronics, vol. 8, no. 6, pp. 605, May 2019.

V.D. Cong, L.D. Hanh, L.H. Phuong, “Real-time Measurement and Prediction of Ball Trajectory for Ping-pong Robot,” 2020 5th International Conference on Green Technology and Sustainable Development (GTSD), pp. 9-14, 2020.

F. Šuligoj, B. Šekoranja, M. Švaco, B. Jerbić, “Object tracking with a multiagent robot system and a stereo vision camera,” Procedia Engineering, vol. 69, pp. 968-973, 2014.

V.D. Cong, L.D. Hanh, “Combination of two visual servoing techniques in contour following task,” 2021 International Conference on System Science and Engineering (ICSSE), pp. 382–386, 2021.

C.H.G. Li, Y.M. Chang, “Automated visual positioning and precision placement of a workpiece using deep learning,” International Journal of Advanced Manufacturing, vol. 104, pp. 4527–4538, 2019.

Z. Wang, Y. Xu, Q.He et al., “Grasping pose estimation for SCARA robot based on deep learning of point cloud,” International Journal of Advanced Manufacturing, vol. 108, pp. 1217–1231, 2020.

D. Zhao, F. Sun, Z. Wang et al., “A novel accurate positioning method for object pose estimation in robotic manipulation based on vision and tactile sensors,” International Journal of Advanced Manufacturing, vol. 116, pp. 2999–3010, 2021.

Z. Wang, Y. Xu, G. Xu, J. Fu, J. Yu and T. Gu, “Simulation and deep learning on point clouds for robot grasping,” Assembly Automation, vol. 41, no. 2, pp. 237-250, 2021.

S. Kumra, S. Joshi and F. Sahin, “GR-ConvNet v2: A Real-Time Multi-Grasp Detection Network for Robotic Grasping,” Sensors, vol.22, no. 16, 2022, https://doi.org/10.3390/s22166208.

T. Zhang, C. Zhang, T. Hu, “A robotic grasp detection method based on auto-annotated dataset in disordered manufacturing scenarios,” Robotics and Computer-Integrated Manufacturing, vol. 76, 2022, https://doi.org/10.1016/j.rcim.2022.102329.

K. Kleeberger, R. Bormann, W. Kraus, et al., “A Survey on Learning-Based Robotic Grasping,” Current Robotics Reports, vol. 1, pp. 239–249, 2020, https://doi.org/10.1007/s43154-020-00021-6.

M. Hegedus, K. Gupta and M. Mehrandezh, “Efficiently finding poses for multiple grasp types with partial point clouds by uncoupling grasp shape and scale,” Autonomous Robots, vol. 46, no. 4, pp. 749–767, 2022, https://doi.org/10.1007/s10514-022-10049-6.




DOI: https://doi.org/10.18196/jrc.v3i5.15661

Refbacks

  • There are currently no refbacks.


Copyright (c) 2022 Cong Duy Vo, Duy Anh Dang, Phuong Hoai Le

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

 

Journal of Robotics and Control (JRC)

P-ISSN: 2715-5056 || E-ISSN: 2715-5072
Organized by Peneliti Teknologi Teknik Indonesia
Published by Universitas Muhammadiyah Yogyakarta in collaboration with Peneliti Teknologi Teknik Indonesia, Indonesia and the Department of Electrical Engineering
Website: http://journal.umy.ac.id/index.php/jrc
Email: jrcofumy@gmail.com

Kuliah Teknik Elektro Terbaik