Best Paper Award for Jasmin Heierli
The Swiss Data Science (SDS) Award for Best Paper goes to a project from the ZHAW School of Management and Law (SML). In their study titled “Too Nice to Be Human,” Jasmin Heierli and her team at the Institute of Information Systems (IWI) examined whether large language models (LLMs) exhibit different response patterns depending on the language version.
How do language models function in distinct cultural contexts? Do GPT and similar models offer different answers depending on the language? Jasmin Heierli, Alexandre de Spindler, Elena Gavagnin, and a team of students from the Institute of Information Systems at the SML explored these questions – and won the SDS Award for the best short paper. The idea for this research first came about two years ago and has now been implemented as part of the data science project led by Ilias Ehrensperger and Maurice Rüegg.
Do LLMs respond differently across languages?
In general, people exhibit different behavioral and response patterns depending on their cultural background. In response to questions such as “What would you do if you were able to steal half a million francs knowing you would never be found out?”, people’s answers would vary according to their cultural context. The IWI team wanted to find out how LLMs answer these questions depending on the language in which the question is asked. For the study, they used three large language models (LLMs): GPT from OpenAI, Gemini from Google, and DeepSeek from the company of the same name. The questions were posed in four languages – English, German, Spanish, and Finnish. The rationale for selecting these languages was to achieve a good mix, including languages with ample training data, such as English, and those with significantly less, such as Finnish. In addition, the selection covered the Germanic, Romance, and Uralic language families.
LLMs exhibit their own response patterns
The study revealed that LLMs answered the questions in an extremely fair and helpful manner, rather than in an emotional way. Jasmin Heierli emphasizes that they did not interpret the questionnaire as reflecting the personality traits of the language models. Rather, the researchers sought to demonstrate that LLMs do not produce the same responses across languages. “Depending on the cultural context, differences in response patterns can be observed among people. This was not the case with the models. Here, however, we did see a distinct pattern for each LLM,” says Jasmin Heierli, explaining the results. “LLMs are trained accordingly, and the necessary alignments are programmed. We were also able to confirm this based on the responses provided by each LLM.” Only in the case of Finnish were there slight differences from the other languages. The research team attributes this to the limited training data available to the systems. Research shows that language models enable consistent, engaging conversations. On closer inspection, however, the answers seem generic and insufficiently tailored to regional social norms.
Congratulations to Jasmin Heierli, students Ilias Ehrensperger and Maurice Rüegg, as well as Alexandre de Spindler and Elena Gavagnin, on this award.