Sorting Line Assisted by A Robotic Manipulator and Artificial Vision with Active Safety

María F. Mogro, Fausto A. Jácome, Guillermo M. Cruz, Jonathan R. Zurita

Abstract


This article presents the design, implementation and evaluation of an object classification and manipulation system in industrial environments by integrating artificial vision and a MELFA RV-2SDB robotic manipulator. The central problem lies in the need to achieve rapid and accurate classification of objects for palletizing, while ensuring the safety of operators. To address this challenge, a machine vision system based on Logitech C920 HD Pro cameras and force and torque sensors was used on the robotic manipulator. The methodology focused on the use of object and person detection algorithms, as well as direct and inverse kinematics to calculate adaptive movements of the manipulator. The experiments covered evaluation of the system's accuracy and efficiency under various lighting and environmental conditions, as well as testing people detection and geometric shape classification. The results indicated that the system allowed precise and efficient manipulation, adapting in real time to the position and characteristics of the detected objects. The conclusions highlighted the effectiveness of the system in improving productivity and safety in collaborative industrial environments, highlighting the importance of integrating cutting-edge technologies to address automation challenges in the industry.

Keywords


Artificial Vision; MELFA RV-2SDB Robotic Manipulator; Object Classification; Active Safety, Inverse Kinematics.

Full Text:

PDF

References


A. Dzedzickis, J. Subačiūtė-Žemaitienė, E. Šutinys, U. Samukaitė-Bubnienė, and V. Bučinskas, “Advanced Applications of Industrial Robotics: New Trends and Possibilities,” Applied Sciences, vol. 12, no. 1, p. 135, Dec. 2021, doi: 10.3390/APP12010135.

J. Chen, Y. Fu, W. Lu, and Y. Pan, “Augmented reality-enabled human-robot collaboration to balance construction waste sorting efficiency and occupational safety and health,” J. Environ. Manage., vol. 348, p. 119341, Dec. 2023, doi: 10.1016/J.JENVMAN.2023.119341.

F. Tao, Q. Qi, A. Liu, and A. Kusiak, “Data-driven smart manufacturing,” J. Manuf. Syst., vol. 48, pp. 157–169, Jul. 2018, doi: 10.1016/J.JMSY.2018.01.006.

S. Barris and C. Button, “A review of vision-based motion analysis in sport,” Sports Medicine, vol. 38, no. 12, pp. 1025–1043, Oct. 2008, doi: 10.2165/00007256-200838120-00006/METRICS.

K. K. H. Ng, C. H. Chen, C. K. M. Lee, J. (Roger) Jiao, and Z. X. Yang, “A systematic literature review on intelligent automation: Aligning concepts from theory, practice, and future perspectives,” Advanced Engineering Informatics, vol. 47, p. 101246, Jan. 2021, doi: 10.1016/J.AEI.2021.101246.

B. M. Muir, “Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated systems,” Ergonomics, vol. 37, no. 11, pp. 1905–1922, 1994, doi: 10.1080/00140139408964957.

E. Hofmann and M. Rüsch, “Industry 4.0 and the current status as well as future prospects on logistics,” Comput. Ind., vol. 89, pp. 23–34, Aug. 2017, doi: 10.1016/J.COMPIND.2017.04.002.

R. Pozzi, T. Rossi, and R. Secchi, “Industry 4.0 technologies: critical success factors for implementation and improvements in manufacturing companies,” Production Planning & Control, vol. 34, no. 2, pp. 139–158, 2023, doi: 10.1080/09537287.2021.1891481.

I. El Makrini et al., “Working with Walt: How a Cobot Was Developed and Inserted on an Auto Assembly Line,” IEEE Robot Autom. Mag., vol. 25, no. 2, pp. 51–58, Jun. 2018, doi: 10.1109/MRA.2018.2815947.

P. Dias, F. J. G. Silva, R. D. S. G. Campilho, L. P. Ferreira, and T. Santos, “Analysis and Improvement of an Assembly Line in the Automotive Industry,” Procedia Manuf., vol. 38, pp. 1444–1452, Jan. 2019, doi: 10.1016/J.PROMFG.2020.01.143.

H. ElMaraghy, L. Monostori, G. Schuh, and W. ElMaraghy, “Evolution and future of manufacturing systems,” CIRP Annals, vol. 70, no. 2, pp. 635–658, Jan. 2021, doi: 10.1016/J.CIRP.2021.05.008.

I. P. Vlachos, R. M. Pascazzi, G. Zobolas, P. Repoussis, and M. Giannakis, “Lean manufacturing systems in the area of Industry 4.0: a lean automation plan of AGVs/IoT integration,” Production Planning & Control, vol. 34, no. 4, pp. 345–358, 2023, doi: 10.1080/09537287.2021.1917720.

P. M. Reyes, J. K. Visich, and P. Jaska, "Managing the Dynamics of New Technologies in the Global Supply Chain," in IEEE Engineering Management Review, vol. 48, no. 1, pp. 156-162, 2020, doi: 10.1109/EMR.2020.2968889.

B. Ferreira and J. Reis, “A Systematic Literature Review on the Application of Automation in Logistics,” Logistics, vol. 7, no. 4, p. 80, Nov. 2023, doi: 10.3390/LOGISTICS7040080.

D. Perkumienė, K. Ratautaitė, and R. Pranskūnienė, “Innovative Solutions and Challenges for the Improvement of Storage Processes,” Sustainability, vol. 14, no. 17, p. 10616, Aug. 2022, doi: 10.3390/SU141710616.

O. F. Odeyinka, O. G. Omoegun, O. F. Odeyinka, and O. G. Omoegun. Warehouse Operations: An Examination of Traditional and Automated Approaches in Supply Chain Management. Operations Management - Recent Advances and New Perspectives, 2023.

S. Paneru and I. Jeelani, “Computer vision applications in construction: Current state, opportunities & challenges,” Autom. Constr., vol. 132, p. 103940, Dec. 2021, doi: 10.1016/J.AUTCON.2021.103940.

P. Papcun and J. Jadlovský, “Optimizing industry robot for maximum speed with high accuracy,” Procedia Eng., vol. 48, pp. 533–542, 2012, doi: 10.1016/J.PROENG.2012.09.550.

C. Yang, S. Liu, H. Su, L. Zhang, Q. Xia, and Y. Chen, “Review of underwater adsorptive-operating robots: Design and application,” Ocean Engineering, vol. 294, p. 116794, Feb. 2024, doi: 10.1016/J.OCEANENG.2024.116794.

A. Perdana, W. E. Lee, and C. Mui Kim, “Prototyping and implementing Robotic Process Automation in accounting firms: Benefits, challenges and opportunities to audit automation,” International Journal of Accounting Information Systems, vol. 51, p. 100641, Dec. 2023, doi: 10.1016/J.ACCINF.2023.100641.

Z. Luo, W. Cheng, T. Zhao, and N. Xiang, “Intelligent sensory systems toward soft robotics,” Appl. Mater. Today, vol. 37, p. 102122, Apr. 2024, doi: 10.1016/J.APMT.2024.102122.

S. Y. Nof. Springer handbook of automation. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, doi: 10.1007/978-3-030-96729-1.

A. Billard and D. Kragic, “Trends and challenges in robot manipulation,” Science, vol. 364, no. 6446, Jun. 2019.

M. Rinaldi, M. Caterino, and M. Fera, “Sustainability of Human-Robot cooperative configurations: Findings from a case study,” Comput. Ind. Eng., vol. 182, p. 109383, Aug. 2023, doi: 10.1016/J.CIE.2023.109383.

Z. Ren, F. Fang, N. Yan, and Y. Wu, “State of the Art in Defect Detection Based on Machine Vision,” International Journal of Precision Engineering and Manufacturing-Green Technology 2021, vol. 9, no. 2, pp. 661–691, May 2021, doi: 10.1007/S40684-021-00343-6.

T. Benbarrad, M. Salhaoui, S. B. Kenitar, and M. Arioua, “Intelligent Machine Vision Model for Defective Product Inspection Based on Machine Learning,” Journal of Sensor and Actuator Networks, vol. 10, no. 1, p. 7, Jan. 2021, doi: 10.3390/JSAN10010007.

X. Wang and Z. Zhu, “Context understanding in computer vision: A survey,” Computer Vision and Image Understanding, vol. 229, p. 103646, Mar. 2023, doi: 10.1016/J.CVIU.2023.103646.

A. Ettalibi, A. Elouadi, and A. Mansour, “AI and Computer Vision-based Real-time Quality Control: A Review of Industrial Applications,” Procedia Comput. Sci., vol. 231, pp. 212–220, Jan. 2024, doi: 10.1016/J.PROCS.2023.12.195.

E. S. Kim, Y. Oh, and G. W. Yun, “Sociotechnical challenges to the technological accuracy of computer vision: The new materialism perspective,” Technol. Soc., vol. 75, p. 102388, Nov. 2023, doi: 10.1016/J.TECHSOC.2023.102388.

Z. Wei, J. Calautit, S. Wei, and P. W. Tien, “Real-time clothing insulation level classification based on model transfer learning and computer vision for PMV-based heating system optimization through piecewise linearization,” Build Environ., p. 111277, Feb. 2024, doi: 10.1016/J.BUILDENV.2024.111277.

S. Nawoya et al., “Computer vision and deep learning in insects for food and feed production: A review,” Comput. Electron. Agric., vol. 216, p. 108503, Jan. 2024, doi: 10.1016/J.COMPAG.2023.108503.

G. Alnowaini, A. Alttal, and A. Alhaj, "Design and simulation robotic arm with computer vision for inspection process," 2021 International Conference of Technology, Science and Administration (ICTSA), pp. 1-6, 2021, doi: 10.1109/ICTSA52017.2021.9406541.

S. M. Hussein and Y. I. Mohammed, “Modeling and Simulation of Industrial SCARA Robot Arm,” International Journal of Engineering and Advanced Technology (IJEAT), pp. 2249–8958, 2015.

S. Gollapudi. Learn computer vision using OpenCV. Apress, 2019.

S. Taheri, A. Vedienbaum, A. Nicolau, N. Hu, and M. R. Haghighat, “OpenCV.js: Computer vision processing for the openWeb platform,” Proceedings of the 9th ACM Multimedia Systems Conference, MMSys 2018, pp. 478–483, Jun. 2018, doi: 10.1145/3204949.3208126.

D. A. Bastidas, L. F. Recalde, P. N. Constante, V. H. Andaluz, D. E. Gallegos, and J. Varela-Aldás, “Non Immersive Virtual Laboratory Applied to Robotics Arms,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13343, pp. 898–906, 2022, doi: 10.1007/978-3-031-08530-7_76.

B. Araujo, J. Araujo, R. Silva, and D. Regis, “Machine vision for industrial robotic manipulator using raspberry Pi,” 2018 13th IEEE International Conference on Industry Applications, INDUSCON 2018 - Proceedings, pp. 894–901, Jul. 2018, doi: 10.1109/INDUSCON.2018.8627203.

M. Abdelaal, “A Study of Robot Control Programing for an Industrial Robotic Arm,” ACCS/PEIT 2019 - 2019 6th International Conference on Advanced Control Circuits and Systems and 2019 5th International Conference on New Paradigms in Electronics and Information Technology, pp. 23–28, Nov. 2019, doi: 10.1109/ACCS-PEIT48329.2019.9062878.

O. I. Borisov et al., “Human-free robotic automation of industrial operations,” IECON Proceedings (Industrial Electronics Conference), pp. 6867–6872, Dec. 2016, doi: 10.1109/IECON.2016.7793922.

O. I. Borisov et al., “Case study on human-free water heaters production for industry 4.0,” Proceedings - 2018 IEEE Industrial Cyber-Physical Systems, pp. 369–374, Jun. 2018, doi: 10.1109/ICPHYS.2018.8387686.

A. Maria Zanchettin, I. Dieter Uckelmann, and F. Ahmed, “Automation of an RFID measurement chamber with a robotic manipulator,” POLITesi, 2023.

B. G. F. Araujo. Sistema de visão de máquina para detecção e localização automática de peças utilizando raspberry pi. MS thesis. 2019.

E. Mujčić, S. Lonić, and M. Muminović, “Programming and Experimental Analysis of MELFA RV-2SDB Robot,” Lecture Notes in Networks and Systems, vol. 28, pp. 810–818, 2018, doi: 10.1007/978-3-319-71321-2_70.

P. Papcun and J. Jadlovský, “Mathematical Model of Robot Melfa RV-2SDB,” Advances in Intelligent Systems and Computing, vol. 316, pp. 145–154, 2015, doi: 10.1007/978-3-319-10783-7_16.

P. Papcun and J. Jadlovský, “Optimizing Industry Robot for Maximum Speed with High Accuracy,” Procedia Eng, vol. 48, pp. 533–542, Jan. 2012, doi: 10.1016/J.PROENG.2012.09.550.

Q. Wu, M. Li, X. Qi, Y. Hu, B. Li, and J. Zhang, “Coordinated control of a dual-arm robot for surgical instrument sorting tasks,” Rob Auton Syst, vol. 112, pp. 1–12, Feb. 2019, doi: 10.1016/J.ROBOT.2018.10.007.

R. Basu and S. Padage, “Development of 5 DOF Robot Arm -Gripper for sorting and investigating RTM Concepts,” Mater Today Proc, vol. 4, no. 2, pp. 1634–1643, Jan. 2017, doi: 10.1016/J.MATPR.2017.02.002.

X. Zhang et al., “Design and operation of a deep-learning-based fresh tea-leaf sorting robot,” Comput Electron Agric, vol. 206, p. 107664, Mar. 2023, doi: 10.1016/J.COMPAG.2023.107664.

E. Mujčić, S. Lonić, and M. Muminović, “Programming and Experimental Analysis of MELFA RV-2SDB Robot,” Lecture Notes in Networks and Systems, vol. 28, pp. 810–818, 2018, doi: 10.1007/978-3-319-71321-2_70/COVER.

A. Issa, M. O. A. Aqel, A. Tubail, Y. Alkayyali, A. Alay, and M. Ferwana, “Vision assisted SCARA manipulator design and control using arduino and LabVIEW,” Proceedings - 2017 International Conference on Promising Electronic Technologies, ICPET 2017, pp. 54–59, Nov. 2017, doi: 10.1109/ICPET.2017.16.

V. Kumar, Q. Wang, W. Minghua, S. Rizwan, S. M. Shaikh, and X. Liu, “Computer vision based object grasping 6DoF robotic arm using picamera,” Proceedings - 2018 4th International Conference on Control, Automation and Robotics, ICCAR 2018, pp. 111–115, Jun. 2018, doi: 10.1109/ICCAR.2018.8384653.

R. Reddy and S. R. Nagaraja, “Integration of robotic arm with vision system,” 2014 IEEE International Conference on Computational Intelligence and Computing Research, IEEE ICCIC 2014, pp. 373–378, 2015, doi: 10.1109/ICCIC.2014.7238302.

O. Hock and J. Sedo, “Inverse kinematics using transposition method for robotic arm,” 12th International Conference ELEKTRO 2018, 2018 ELEKTRO Conference Proceedings, pp. 1–5, Jun. 2018, doi: 10.1109/ELEKTRO.2018.8398366.

A. Loungthongkam and C. Raksiri, “A Development of Mathematical Model for Predictive of the Standard Uncertainty of Robot Arm,” 2020 IEEE 7th International Conference on Industrial Engineering and Applications, ICIEA 2020, pp. 281–286, Apr. 2020, doi: 10.1109/ICIEA49774.2020.9101979.

Y. Dai, C. Xiang, Y. Zhang, Y. Jiang, W. Qu, and Q. Zhang, “A Review of Spatial Robotic Arm Trajectory Planning,” Aerospace 2022, Vol. 9, Page 361, vol. 9, no. 7, p. 361, Jul. 2022, doi: 10.3390/AEROSPACE9070361.

M. Intisar, M. Monirujjaman Khan, M. Rezaul Islam, and M. Masud, “Computer Vision Based Robotic Arm Controlled Using Interactive GUI,” 2021, doi: 10.32604/iasc.2021.015482.

X. Hou and S.-K. Tsui, “A mathematical model for control of flexible robot arms,” Systems modelling and optimization, pp. 391–398, Mar. 2022, doi: 10.1201/9780203737422-51.

P. Sooraksa and G. Chen, “Mathematical modeling and fuzzy control of a flexible-link robot arm,” Math Comput Model, vol. 27, no. 6, pp. 73–93, 1998, doi: 10.1016/S0895-7177(98)00030-2.

T. Punnoose Valayil, V. Selladurai, and N. Rajam Ramaswamy, “KINEMATIC MODELING OF A SERIAL ROBOT USING DENAVIT-HARTENBERG METHOD IN MATLAB,” 2018, Accessed: Feb. 11, 2024. [Online]. Available: www.tagajournal.com

K. S. Gaeid, A. F. Nashee, I. A. Ahmed, and M. H. Dekheel, “Robot control and kinematic analysis with 6DoF manipulator using direct kinematic method,” Bulletin of Electrical Engineering and Informatics, vol. 10, no. 1, pp. 70–78, Feb. 2021, doi: 10.11591/EEI.V10I1.2482.

Z. Liao, G. Jiang, F. Zhao, X. Mei, and Y. Yue, “A novel solution of inverse kinematic for 6R robot manipulator with offset joint based on screw theory,” Int J Adv Robot Syst, vol. 17, no. 3, May 2020, doi: 10.1177/1729881420925645/ASSET/IMAGES/LARGE/10.1177_1729881420925645-FIG10.JPEG.

T. S. Lee and E. A. Alandoli, “A critical review of modelling methods for flexible and rigid link manipulators,” Journal of the Brazilian Society of Mechanical Sciences and Engineering, vol. 42, no. 10, pp. 1–14, Oct. 2020, doi: 10.1007/S40430-020-02602-0/METRICS.

D. Wang and Z. Zhang, “High-efficiency nonlinear dynamic analysis for joint interfaces with Newton–Raphson iteration process,” Nonlinear Dyn, vol. 100, no. 1, pp. 543–559, Mar. 2020, doi: 10.1007/S11071-020-05522-9/FIGURES/15.

A. Torres-Hernandez and F. Brambila-Paz, “Fractional Newton-Raphson Method and Some Variants for the Solution of Non-linear Systems,” Applied Mathematics and Sciences An International Journal (MathSJ), vol. 7, no. 1, pp. 13–27, Aug. 2019, doi: 10.5121/mathsj.2020.7102.

M. Heydarzadeh, N. Karbasizadeh, M. Tale Masouleh, and A. Kalhor, “Experimental kinematic identification and position control of a 3-DOF decoupled parallel robot,” https://doi.org/10.1177/0954406218775906, vol. 233, no. 5, pp. 1841–1855, May 2018, doi: 10.1177/0954406218775906.

R. Dou et al., “Inverse kinematics for a 7-DOF humanoid robotic arm with joint limit and end pose coupling,” Mech Mach Theory, vol. 169, p. 104637, Mar. 2022, doi: 10.1016/J.MECHMACHTHEORY.2021.104637.

Y. Miao, J. Y. Jeon, and G. Park, “An image processing-based crack detection technique for pressed panel products,” J Manuf Syst, vol. 57, pp. 287–297, Oct. 2020, doi: 10.1016/J.JMSY.2020.10.004.

M. Meenu, C. Kurade, B. C. Neelapu, S. Kalra, H. S. Ramaswamy, and Y. Yu, “A concise review on food quality assessment using digital image processing,” Trends Food Sci Technol, vol. 118, pp. 106–124, Dec. 2021, doi: 10.1016/J.TIFS.2021.09.014.

Nursalim, C. S. Octiva, S. Sri Wahyuningsih, M. Lukman Hakim, and N. Hasti, “Application of The Speed-Up Robust Features Method To Identify Signature Image Patterns On Single Board Computer,” Jurnal Sistim Informasi dan Teknologi, pp. 14–18, Oct. 2023, doi: 10.60083/JSISFOTEK.V5I4.312.

S. Sankaraiah and R. S. Deepthi, “Highly optimized OpenCV based cell phone,” 2011 IEEE Conference on Sustainable Utilization Development in Engineering and Technology, STUDENT 2011, pp. 47–52, 2011, doi: 10.1109/STUDENT.2011.6089323.

R. S. Deepthi and S. Sankaraiah, “Implementation of mobile platform using Qt and OpenCV for image processing applications,” 2011 IEEE Conference on Open Systems, ICOS 2011, pp. 284–289, 2011, doi: 10.1109/ICOS.2011.6079235.

G. A. Odesanmi, Q. Wang, and J. Mai, “Skill learning framework for human–robot interaction and manipulation tasks,” Robot Comput Integr Manuf, vol. 79, p. 102444, Feb. 2023, doi: 10.1016/J.RCIM.2022.102444.

R. Szczepanski, K. Erwinski, M. Tejer, A. Bereit, and T. Tarczewski, “Optimal scheduling for palletizing task using robotic arm and artificial bee colony algorithm,” Eng Appl Artif Intell, vol. 113, p. 104976, Aug. 2022, doi: 10.1016/J.ENGAPPAI.2022.104976.

G. D’Ago et al., “Modelling and identification methods for simulation of cable-suspended dual-arm robotic systems,” Rob Auton Syst, p. 104643, Feb. 2024, doi: 10.1016/J.ROBOT.2024.104643.

Y. Cheng, C. Su, and H. Li, “Design of power monitoring system for 33MW DC submerged arc furnace based on PLC,” 2021 3rd International Conference on Intelligent Control, Measurement and Signal Processing and Intelligent Oil Field, ICMSP 2021, pp. 403–407, Jul. 2021, doi: 10.1109/ICMSP53480.2021.9513331.

E. Decky et al., “Pick and place robot automation module for learning industrial motion, web server GUI, and vision sensor using Kalman filter,” AIP Conf Proc, vol. 2710, no. 1, p. 050016, Feb. 2024, doi: 10.1063/5.0145128.

A. M. Jazeel, “Theme Emerging Technologies for Industrial Sustenance Symposium Chair Editorial Board”, Accessed: Feb. 11, 2024. [Online]. Available: https://www.researchgate.net/publication/332190133

R. Badarinath and V. Prabhu, “Integration and evaluation of robotic fused filament fabrication system,” Addit Manuf, vol. 41, p. 101951, May 2021, doi: 10.1016/J.ADDMA.2021.101951.

Y. Murase, M. Hidaka, and R. Yoshida, “Self-driven gel conveyer: Autonomous transportation by peristaltic motion of self-oscillating gel,” Sens Actuators B Chem, vol. 149, no. 1, pp. 272–283, Aug. 2010, doi: 10.1016/J.SNB.2010.06.017.

M. Andrejiova, A. Grincova, and D. Marasova, “Monitoring dynamic loading of conveyer belts by measuring local peak impact forces,” Measurement, vol. 158, p. 107690, Jul. 2020, doi: 10.1016/J.MEASUREMENT.2020.107690.

I. Q. Habeeb, “Image Processing Based Ambient Context-Aware People Detection and Counting,” 2018, doi: 10.18178/ijmlc.2018.8.3.698.

A. Brunetti, D. Buongiorno, G. F. Trotta, and V. Bevilacqua, “Computer vision and deep learning techniques for pedestrian detection and tracking: A survey,” Neurocomputing, vol. 300, pp. 17–33, Jul. 2018, doi: 10.1016/J.NEUCOM.2018.01.092.

R. K. Tripathi, A. S. Jalal, and S. C. Agrawal, “Suspicious human activity recognition: a review,” Artif Intell Rev, vol. 50, no. 2, pp. 283–339, Aug. 2018, doi: 10.1007/S10462-017-9545-7/METRICS.




DOI: https://doi.org/10.18196/jrc.v5i2.20327

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 María Fernanda Mogro, Fausto Andrés Jácome, Guillermo Mauricio Cruz, Jonathan Raphael Zurita

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

 

Journal of Robotics and Control (JRC)

P-ISSN: 2715-5056 || E-ISSN: 2715-5072
Organized by Peneliti Teknologi Teknik Indonesia
Published by Universitas Muhammadiyah Yogyakarta in collaboration with Peneliti Teknologi Teknik Indonesia, Indonesia and the Department of Electrical Engineering
Website: http://journal.umy.ac.id/index.php/jrc
Email: jrcofumy@gmail.com

Kuliah Teknik Elektro Terbaik