پیوندهای مفید
اخبار و اعلانات
آمار
| تعداد نشریات | 161 |
| تعداد شماره‌ها | 6,533 |
| تعداد مقالات | 70,506 |
| تعداد مشاهده مقاله | 124,125,001 |
| تعداد دریافت فایل اصل مقاله | 97,233,503 |
کنترل هوشمند میکروربات به منظور ناوبری روی سطح سیال و شبیه‌سازی کاربرد آن برای از بین بردن میکروپلاستیک‌ها | ||
| مهندسی بیوسیستم ایران | ||
| دوره 54، شماره 3، مهر 1402، صفحه 75-94 اصل مقاله (1.87 M) | ||
| نوع مقاله: مقاله پژوهشی | ||
| شناسه دیجیتال (DOI): 10.22059/ijbse.2024.366451.665527 | ||
| نویسندگان | ||
| عمار صالحی1؛ سلیمان حسین پور* 1؛ نصرالله طباطبائی2؛ محمود سلطانی فیروز1 | ||
| 1گروه مهندسی ماشین‌های کشاورزی، دانشکده کشاورزی، دانشکدگان کشاورزی و منابع طبیعی، دانشگاه تهران، کرج، ایران | ||
| 2گروه نانوفناوری پزشکی، دانشکده فناوری‌های نوین پزشکی، دانشگاه علوم پزشکی تهران، تهران، ایران | ||
| چکیده | ||
| در سال‌های اخیر میکروپلاستیک‌های موجود در مواد غذایی و آشامیدنی به یک معضل جهانی تبدیل شده‌اند. میکروربات‌های مغناطیسی به‌عنوان یک رویکرد نوین در حل این مشکل، پتانسیل خوبی برای جداسازی و از بین بردن میکروپلاستیک‌ها نشان داده‌اند. بااین‌حال، هدایت و ناوبری خودکار، هوشمند و دقیق میکروربات‌ها برای اجرای چنین وظایفی، همچنان یک چالش اصلی محسوب می‌شود. روش‌های مرسوم برای دستیابی به چنین سطحی از کنترل، اغلب به مدل‌سازی‌های پیچیده‌ای از دینامیک میکروربات، محیط و سیستم تحریک نیاز دارند. به‌عنوان یک رویکرد جایگزین، در این پژوهش یک سیستم کنترل مبتنی بر الگوریتم یادگیری تقویتی عمیق بدون مدل برای کنترل میکروربات مغناطیسی روی سطح سیال ارائه شد. هدف سیستم، آموزش میکروربات برای هدایت آن از یک نقطه در محیط واقعی به سمت موقعیت هدف بود تا فرآیند ناوبری به سوی موقعیت میکروپلاستیک شناور روی سطح سیال شبیه‌سازی شود. برای کنترل موقعیت میکروربات، یک سیستم تحریک مغناطیسی شامل دو آهنربای ثابت و یک سیم‌پیچ هلمهولتز تک‌محوره ساخته شد. نتایج آموزش میکروربات نشان داد که میکروربات توانست با دقت و سرعت بالایی به موقعیت هدف برسد. نتایج ارزیابی مدل آموزش‌یافته نیز حاکی از موفقیت میکروربات در رسیدن به نقطه هدف با میانگین پاداش 02/39 از 40، و انحراف معیار 71/0 در تمام اپیزودها بود. این نتایج نشان می‌دهد که سیستم کنترل مبتنی بر الگوریتم یادگیری بدون داشتن هیچ‌گونه دانش قبلی از دینامیک محیط یا سیستم تحریک، یک سیاست بهینه را با استفاده از تعامل با محیط برای هدایت میکروربات کشف کرد. | ||
| کلیدواژه‌ها | ||
| کنترل؛ میکروپلاستیک؛ میکروربات؛ یادگیری تقویتی عمیق | ||
| عنوان مقاله [English] | ||
| Smart Control of a Microrobot for Navigation on Fluid Surface and Simulation of its Application in Microplastics Removal | ||
| نویسندگان [English] | ||
| Amar Salehi1؛ Soleiman Hosseinpour1؛ Nasrollah Tabatabaei2؛ Mahmoud Soltani Firouz1 | ||
| 1Department of Agricultural Machinery Engineering, Faculty of Agricultural, College of Agriculture and Natural Resources, University of Tehran, Karaj, Iran | ||
| 2Department of Medical Nanotechnology, School of Advanced Technologies in Medicine, Tehran University of Medical Sciences, Tehran, Iran. | ||
| چکیده [English] | ||
| Microplastic contamination of food and beverages has become a global concern in recent years. As a novel approach, magnetic microrobots offer promising potential to address microplastic separation and degradation. However, achieving precise, intelligent, and automated navigation control for microrobots in such tasks remains a significant challenge. This level of control is typically achieved by modeling the complex dynamics of microrobots, the environment, and the actuation system. In this study, an alternative approach was presented using a model-free deep reinforcement learning algorithm (DRL) to navigate a magnetic microrobot on fluid surfaces. In order to simulate the process of reaching a microplastic particle on the fluid surface, the DRL system was implemented to train the microrobot to autonomously navigate from an initial position within the real-world environment to a specified target position. A magnetic actuation system based on two permanent magnets and one-axis Helmholtz coils was constructed to manipulate the position of the microrobot. During the training phase, the microrobot demonstrated high accuracy and speed in achieving the desired position. The evaluation results of the trained model also confirmed the microrobot's success in all episodes, with an average reward of 39.02 out of 40 and a standard deviation of 0.71. These findings indicate that the control system could effectively learn an optimal policy by employing DRL without any prior knowledge of environmental dynamics or the actuation system. | ||
| کلیدواژه‌ها [English] | ||
| Control, Deep reinforcement learning, Microplastic, Microrobot | ||
| مراجع | ||
|
Agrahari, V., Agrahari, V., Chou, M.-L., Chew, C. H., Noll, J., & Burnouf, T. (2020). Intelligent micro-/nanorobots as drug and cell carrier devices for biomedical therapeutic advancement: Promising development opportunities and translational challenges. Biomaterials, 260, 120163. Amoudruz, L., & Koumoutsakos, P. (2022). Independent Control and Path Planning of Microswimmers with a Uniform Magnetic Field. Advanced Intelligent Systems, 4(3), 2100183. Bae, H., Paludan, M., Knoblauch, J., & Jensen, K. H. (2021). Neural networks and robotic microneedles enable autonomous extraction of plant metabolites. Plant Physiology, 186(3), 1435–1441. Behrens, M. R., & Ruder, W. C. (2022). Smart Magnetic Microrobots Learn to Swim with Deep Reinforcement Learning. Advanced Intelligent Systems, 4(10), 2270049. Beladi-Mousavi, S. M., Hermanová, S., Ying, Y., Plutnar, J., & Pumera, M. (2021). A Maze in Plastic Wastes: Autonomous Motile Photocatalytic Microrobots against Microplastics. ACS Applied Materials & Interfaces, 13(21), 25102–25110. Bellman, R. (1957). A Markovian decision process. Journal of Mathematics and Mechanics, 679–684. Borra, F., Biferale, L., Cencini, M., & Celani, A. (2022). Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number. Physical Review Fluids, 7(2), 023103. Cai, M., Wang, Q., Qi, Z., Jin, D., Wu, X., Xu, T., & Zhang, L. (2022). Deep Reinforcement Learning Framework-Based Flow Rate Rejection Control of Soft Magnetic Miniature Robots. IEEE Transactions on Cybernetics, 1–13. Campuzano, S., Orozco, J., Kagan, D., Guix, M., Gao, W., Sattayasamitsathit, S., Claussen, J. C., Merkoçi, A., & Wang, J. (2012). Bacterial Isolation by Lectin-Modified Microengines. Nano Letters, 12(1), 396–401. Choi, J., Hwang, J., Kim, J. young, & Choi, H. (2021). Recent Progress in Magnetically Actuated Microrobots for Targeted Delivery of Therapeutic Agents. Advanced Healthcare Materials, 10(6), 1–24. Colabrese, S., Gustavsson, K., Celani, A., & Biferale, L. (2017). Flow Navigation by Smart Microswimmers via Reinforcement Learning. Physical Review Letters, 118(15), 158004. Dan, J., Shi, S., Sun, H., Su, Z., Liang, Y., Wang, J., & Zhang, W. (2022). Micro/nanomotor technology: the new era for food safety control. Critical Reviews in Food Science and Nutrition, 1–21. de Jesus, J. C., Kich, V. A., Kolling, A. H., Grando, R. B., Cuadros, M. A. de S. L., & Gamarra, D. F. T. (2021). Soft Actor-Critic for Navigation of Mobile Robots. Journal of Intelligent & Robotic Systems, 102(2), 31. Ding, Z., Huang, Y., Yuan, H., & Dong, H. (2020). Introduction to Reinforcement Learning. In Deep Reinforcement Learning (pp. 47–123). Springer Singapore. Dong, X., & Sitti, M. (2020). Controlling two-dimensional collective formation and cooperative behavior of magnetic microrobot swarms. The International Journal of Robotics Research, 39(5), 617–638. Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021). Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9), 2419–2468. Gustavsson, K., Biferale, L., Celani, A., & Colabrese, S. (2017). Finding efficient swimming strategies in a three-dimensional chaotic flow by reinforcement learning. The European Physical Journal E, 40(12), 110. Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., & Levine, S. (2018). Learning to walk via deep reinforcement learning. ArXiv Preprint ArXiv:1812.11103. Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In J. Dy & A. Krause (Eds.), Proceedings of the 35th International Conference on Machine Learning (Vol. 80, pp. 1861–1870). PMLR. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., & Abbeel, P. (2018). Soft actor-critic algorithms and applications. ArXiv Preprint ArXiv:1812.05905. Huang, L., Rogowski, L., Kim, M. J., & Becker, A. T. (2017). Path planning and aggregation for a microrobot swarm in vascular networks using a global input. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 414–420. Jiang, J., Yang, L., & Zhang, L. (2023). DQN-based on-line Path Planning Method for Automatic Navigation of Miniature Robots. 2023 IEEE International Conference on Robotics and Automation (ICRA), 5407–5413. Jiang, J., Yang, Z., Ferreira, A., & Zhang, L. (2022). Control and Autonomy of Microrobots: Recent Progress and Perspective. Advanced Intelligent Systems, 4(5), 2100279. Khalesi, R., Yousefi, M., Nejat Pishkenari, H., & Vossoughi, G. (2022). Robust independent and simultaneous position control of multiple magnetic microrobots by sliding mode controller. Mechatronics, 84, 102776. Kober, J., Bagnell, J. A., & Peters, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11), 1238–1274. Lee, J., & Ha, J.-I. (2017). Direction Priority Control Method for Magnetic Manipulation System in Current and Voltage Limits. IEEE Transactions on Industrial Electronics, 64(4), 2914–2923. Leslie, H. A., van Velzen, M. J. M., Brandsma, S. H., Vethaak, A. D., Garcia-Vallejo, J. J., & Lamoree, M. H. (2022). Discovery and quantification of plastic particle pollution in human blood. Environment International, 163, 107199. Lynch, K. M., & Park, F. C. (2017). Modern robotics. Cambridge University Press. Magnetics, Magnetop. (2023). Neodymium Cube - 20mm x 20mm x 20mm - N42. https://www.magnetop.com.au/neodymium-cube-20mm-x-20mm-x-20mm-n42 Mayorga‐Martinez, C. C., Castoralova, M., Zelenka, J., Ruml, T., & Pumera, M. (2023). Swarming Magnetic Microrobots for Pathogen Isolation from Milk. Small, 19(6), 2205047. Medina-Sánchez, M., & Schmidt, O. G. (2017). Medical microbots need better imaging and control. Nature, 545(7655), 406–408. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. Nauber, R., Goudu, S. R., Goeckenjan, M., Bornhäuser, M., Ribeiro, C., & Medina-Sánchez, M. (2023). Medical microrobots in reproductive medicine from the bench to the clinic. Nature Communications, 14(1), 728. Parmar, J., Vilela, D., Villa, K., Wang, J., & Sánchez, S. (2018). Micro- and Nanomotors as Active Environmental Microcleaners and Sensors. Journal of the American Chemical Society, 140(30), 9317–9331. Pawashe, C., Floyd, S., Diller, E., & Sitti, M. (2012). Two-Dimensional Autonomous Microparticle Manipulation Strategies for Magnetic Microrobots in Fluidic Environments. IEEE Transactions on Robotics, 28(2), 467–477. Qiu, J., Huang, W., Xu, C., & Zhao, L. (2020). Swimming strategy of settling elongated micro-swimmers by reinforcement learning. Science China Physics, Mechanics & Astronomy, 63(8), 284711. Rahmer, J., Stehning, C., & Gleich, B. (2017). Spatially selective remote magnetic actuation of identical helical micromachines. Science Robotics, 2(3). Salmani, Y., Mousazadeh, H., Alimardani, R., Jafarbiglu, H., Omrani, E., Makhsoos, A., & Kiapey, A. (2018). Evaluation of a Navigation Algorithm for Robot Boat and Comparison to Simulation Results. Iranian Journal of Biosystem Engineering, 49(3), 353–366. (In Persian) Sehyuk Yim, Gultepe, E., Gracias, D. H., & Sitti, M. (2014). Biopsy using a Magnetic Capsule Endoscope Carrying, Releasing, and Retrieving Untethered Microgrippers. IEEE Transactions on Biomedical Engineering, 61(2), 513–521. Singh, B., Kumar, R., & Singh, V. P. (2022). Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review, 55(2), 945–990. Sitti, M. (2017). Mobile Microrobotics (R. Arkin (ed.)). The MIT Press. Sun, M., Chen, W., Fan, X., Tian, C., Sun, L., & Xie, H. (2020). Cooperative recyclable magnetic microsubmarines for oil and microplastics removal from water. Applied Materials Today, 20, 100682. Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press. TowerPro. (2023). MG996R. https://www.towerpro.com.tw/product/mg996r/ Tsang, Alan C. H., Demir, E., Ding, Y., & Pak, O. S. (2020). Roads to Smart Artificial Microswimmers. Advanced Intelligent Systems, 2(8), 1900137. Tsang, Alan Cheng Hou, Tong, P. W., Nallan, S., & Pak, O. S. (2020). Self-learning how to swim at low Reynolds number. Physical Review Fluids, 5(7), 074101. Urso, M., Ussia, M., & Pumera, M. (2021). Breaking Polymer Chains with Self‐Propelled Light‐Controlled Navigable Hematite Microrobots. Advanced Functional Materials, 31(28). Urso, M., Ussia, M., & Pumera, M. (2023). Smart micro- and nanorobots for water purification. Nature Reviews Bioengineering. Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). Vilela, D., Stanton, M. M., Parmar, J., & Sánchez, S. (2017). Microbots Decorated with Silver Nanoparticles Kill Bacteria in Aqueous Media. ACS Applied Materials & Interfaces, 9(27), 22093–22100. Villa, K., Vyskočil, J., Ying, Y., Zelenka, J., & Pumera, M. (2020). Microrobots in Brewery: Dual Magnetic/Light‐Powered Hybrid Microrobots for Preventing Microbial Contamination in Beer. Chemistry – A European Journal, 26(14), 3039–3043. Wang, L., Kaeppler, A., Fischer, D., & Simmchen, J. (2019). Photocatalytic TiO 2 Micromotors for Removal of Microplastics and Suspended Matter. ACS Applied Materials & Interfaces, 11(36), 32937–32944. Wang, Q., Li, T., Fang, D., Li, X., Fang, L., Wang, X., Mao, C., Wang, F., & Wan, M. (2020). Micromotor for removal/detection of blood copper ion. Microchemical Journal, 158, 105125. Wang, X., Hu, C., Pane, S., & Nelson, B. J. (2022). Dynamic Modeling of Magnetic Helical Microrobots. IEEE Robotics and Automation Letters, 7(2), 1682–1688. Watkins, C. J. C. H. (1989). Learning from delayed rewards. Wright, S. L., Thompson, R. C., & Galloway, T. S. (2013). The physical impacts of microplastics on marine organisms: A review. Environmental Pollution, 178, 483–492. Xu, T., Yu, J., Yan, X., Choi, H., & Zhang, L. (2015). Magnetic Actuation Based Motion Control for Microrobots: An Overview. Micromachines, 6(9). Yang, G.-Z., Bellingham, J., Dupont, P. E., Fischer, P., Floridi, L., Full, R., Jacobstein, N., Kumar, V., McNutt, M., Merrifield, R., Nelson, B. J., Scassellati, B., Taddeo, M., Taylor, R., Veloso, M., Wang, Z. L., & Wood, R. (2018). The grand challenges of Science Robotics. Science Robotics, 3(14). Yang, Y., Bevan, M. A., & Li, B. (2020a). Efficient Navigation of Colloidal Robots in an Unknown Environment via Deep Reinforcement Learning. Advanced Intelligent Systems, 2(1), 1900106. Yang, Y., Bevan, M. A., & Li, B. (2020b). Micro/Nano Motor Navigation and Localization via Deep Reinforcement Learning. Advanced Theory and Simulations, 3(6), 2000034. Yang, Y., Bevan, M. A., & Li, B. (2022). Hierarchical Planning with Deep Reinforcement Learning for 3D Navigation of Microrobots in Blood Vessels. Advanced Intelligent Systems, 4(11), 2200168. Yousefi, M., & Nejat Pishkenari, H. (2021). Independent position control of two identical magnetic microrobots in a plane using rotating permanent magnets. Journal of Micro-Bio Robotics, 17(1), 59–67. Zhang, H., & Yu, T. (2020). Taxonomy of Reinforcement Learning Algorithms. In Deep Reinforcement Learning (pp. 125–133). Springer Singapore. Zhou, H., Mayorga‐Martinez, C. C., & Pumera, M. (2021). Microplastic Removal and Degradation by Mussel‐Inspired Adhesive Magnetic/Enzymatic Microrobots. Small Methods, 5(9), 2100230. Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A., Kumar, V., & Levine, S. (2020). The ingredients of real-world robotic reinforcement learning. Zou, Z., Liu, Y., Young, Y.-N., Pak, O. S., & Tsang, A. C. H. (2022). Gait switching and targeted navigation of microswimmers via deep reinforcement learning. Communications Physics, 5(1), 158. | ||
|
آمار تعداد مشاهده مقاله: 588 تعداد دریافت فایل اصل مقاله: 499 |
||