Finding the identity of missing migrants using artificial intelligence

Every year, hundreds of thousands of people leave their homes and leave their country in search of a better life or to flee violence. Many are injured or killed on the way. Many others disappear without their loved ones knowing whether they are alive or dead, or what happened to them.

According to the project Missing migrants of the International Organization for Migration (IOM), we have lost track of 45 migrants worldwide since 000, including 2014 in the Mediterranean.

In 2020, the INSA (National Institute of Applied Sciences) alliance was approached by the trans-regional forensic team of the International Committee of the Red Cross (ICRC), which aimed to improve the process of identifying migrants died in the Euro-Mediterranean area. There are a lot of drownings here – 16 since 000. To our knowledge, this effort led by the anthropologist Jose Pablo Baraybar of the ICRC is the only one that confronts this problem in a transversal way in the region.

The INSA teams thus intervened to propose solutions to this essential identification work of the ICRC, which has to deal with a large number of cases, scattered or poor quality information on the missing persons.

After a pilot project led by INSA Lyon, which provided the ICRC with tools for managing information on recovered bodies, the partnership took shape. He integrated the Alliances program of the INSA Foundation.

This program mobilizes students and teacher-researchers on concrete cases for which NGOs, such as Handicap International or the ICRC, need scientific and technical expertise. In all, there are 37 students who, as part of their course, have developed seven projects combining methods and tools specific to engineering schools with the field knowledge of the ICRC.

Artificial intelligence at the service of humanitarianism

In theory, the process of identifying drowned people could easily be initiated by having the deceased recognized by their relatives using photographs. However, these documents are not always “showable”: either these photos are of poor quality, or the bodies are so damaged and the images so traumatic that they prevent any formal recognition.

This situation led us to explore the idea of ​​comparing the photos of deceased individuals with the photographs of people wanted by their relatives using facial recognition technologies.

This approach was explored in particular as part of Zacharie Hellouin's end-of-studies internship in 2020. His project consisted of using and then evaluating the contribution of facial recognition algorithms and models in the identification of the remains of people found drowned. .

In concrete terms, it involves adapting and using models of machine learning, an artificial intelligence technique allowing a program to learn, on its own, to recognize similarities and differences in data sets. By confronting it with repeated experiences, such as recognizing a person's identity, the program trains and improves its results. This work made it possible to validate the interest of this technique for the recognition of the missing.

To implement it, we compared photos of living migrants with those of deceased migrants in the hope of obtaining positive matches. For this, we have set up a similarity index based on a matching algorithm which makes it possible to obtain probable identity scores of the person in the form of percentages.

Everything has been integrated into a Web application intended for ICRC agents and those legally in charge of identifying mortal remains, such as forensic institutes. This application is under development and each of the projects aims to improve it.

The results obtained are encouraging. Thanks to this software, we were able to develop a complete prototype of facial recognition applied to missing migrants. However, to be able to offer truly reliable indicators of similarity between photos of living and dead people, thousands and thousands of photos would have to be obtained.

These limits having been set, the tool developed today offers ICRC agents the possibility of directing their searches by providing a list of probable matches, making the search certainly laborious, but humanly possible.

Constantly improving software

At the start of this project, in 2020, specifications had to be drawn up. The INSA students and their teacher Charles Dossal therefore translated into technical terms the automatic or non-automatic processing to be performed on these images: extract the face from the decor (a bag, the bottom of a boat, a table, etc.), center and align the image, lessen or eliminate wounds, remove foam from the mouth and bring a glimmer of life to the eyes.

Two students in 4^e year, Adam Hamidallah and Din Triem Phan, then programmed the algorithms that we had identified as the most relevant to solve these different problems. It will sometimes have been necessary to copy parts of healthy skin to "digitally heal" wounds or insert eyes from another face when these were too damaged. The results are encouraging, but we have also been able to measure that artificial intelligence (AI) could provide more complete answers.

During the summer of 2021, Zoé Philippon and Jeong Hwan Ko viewed these terrible images with the aim of seeing more precisely what AI can bring to this mission.

Zoé Philippon's objective was to test the limits of facial recognition algorithms based on artificial neural networks when applied to images of the faces of the deceased, mainly of African origin. These algorithms are effective on images similar to those used to calibrate them, here faces of living people, mostly white and male, with a small proportion of female or African faces.

She therefore carried out numerous tests, re-trained the AI ​​to be more effective on the images of missing persons. The results seem to indicate that these algorithms would benefit from being trained more specifically on faces of a population more representative of the missing and that recognition deteriorates significantly when the person to be recognized is dead. Access to a greater quantity of data could confirm these very encouraging initial results.

Digital makeup

Jeong Hwan Ko has tried to improve the results of "digital make-up" by using artificial neural networks, also pretrained, to fill in holes in images. These methods have proven to be extremely effective in erasing injuries, but to repair a mouth or eyes, it was necessary to use other neural networks capable of inserting part of an image into another.

For the moment the programmer chooses the image to insert, but in the future it will probably be more efficient to let the algorithm search for itself in a large database, the eyes, the mouth or the ears in good condition that most resemble that of the face to be identified. There is still work to be done and, here again, wider access to data would undoubtedly improve the quality of this facial reconstruction.

Today, the projects continue. We are always looking for data to further train machine learning programs. We are also looking for corporate sponsors willing to share their technology, time and support with us.

Finally, it should be noted that these same applications, developed to respond to the crisis of missing migrants, can also be used in other contexts such as disasters, conflicts or any situation that may lead to the non-identification of deceased persons.

This article was co-authored by Samuel Kenny, coordinator of the ICRC-INSA Alliance.

Sami Yangui, Teacher-Researcher in Computer Science, INSA Toulouse et Charles Dossal, Professor of Mathematics, INSA Toulouse

This article is republished from The Conversation under Creative Commons license. Read theoriginal article.

Image credit: Shutterstock.com/pabloavanzini