Automatic analysis of verbal and non-verbal behavior and provision of feedback in video selection interviews
| Responsable du projet |
Marianne Schmid Mast
Daniel Gatica Perez |
| Collaborateur |
Skanda Muralidhar
Eric Mayor Emmanuelle Kleinlogel |
| Résumé |
This project investigates social interaction in personnel selection
interviews enhanced by digital technology. We will create a
database of applicants participating in video interviews
(applicants receive a list of interview questions from a recruiter
online and then record themselves answering those questions), which
are a newly emerging interview format. We will develop automated
procedures for extracting relevant behavioral features from streams
of applicants’ verbal and nonverbal behavior in these interviews.
This information will be (1) linked to external criteria (e.g.,
hireability ratings by expert recruiters), (2) used to train
machine learning algorithms, and (3) fed back to the applicants. We
will assess applicants’ perceptions of this feedback, whether and
how they use it to improve their performance in a second video
interview a day later, and how they perceive data privacy issues
related to the use of their data. The project addresses three issues mentioned in the call. First, how is digitalization transforming social ties? The selection interview is the gateway to employment and thus the potential beginning of one of the fundamental social ties in modernity: the work relationship. We explore a new format by which selection interviews are conducted in an online, asynchronous manner. Second, how is digitalization transforming the economy? The selection interview is an important personnel selection procedure, which itself is an important component of strategic talent management. The digitalization of talent management is rapidly expanding in practice, but is currently poorly understood in research. Third, how is digitalization transforming our subjective experience? Video interviews are a novel experience for many applicants. Machine learning techniques can be used to extract the applicants’ behaviors recorded on the videos and to some degree infer their personality and social skills. This information can then be fed back to the applicants, potentially changing their subjective experience of the video interview. However, questions like how such feedback is best provided and how the applicant apprehends and uses it are largely unexplored. The study will yield four main sets of outputs. First, the primary data from the study will lead to publications in scientific journals or conference proceedings in human-computer interaction and organizational psychology or human resources. Second, the data will be used to adapt an existing data collection platform and to improve the quality of algorithms to infer verbal and nonverbal behavior from videos. Third, data about user experiences will inform the development of evidence-based coaching programs for improving applicants’ performance. Fourth, the rich set of data and experience generated will constitute fruitful avenues for further research by the applicant team. |
| Mots-clés |
Selection interview, nonverbal behavior, question-answering, video interview, algorithm, feedback |
| Type de projet | Recherche appliquée |
| Domaine de recherche | Psychologie du travail, intelligence artificielle |
| Source de financement | FNS |
| Etat | En cours |
| Début de projet | 1-12-2018 |
| Fin du projet | 31-5-2020 |
| Budget alloué | 150'000 |
| Contact | Adrian Bangerter |