“I’d Blush If I Could”: Bias and Artificial Intelligence

“Alexa,” “Hey Siri,” “Ok Google,” “Hey Cortana”! Every day we interact with voice assistants, but we never ask ourselves why they have a female voice. They were created to be obedient, to turn lights on and off, to order groceries. Of course, there are also male Artificial Intelligences (AIs), but they generally take on more important roles, such as IBM Watson making business decisions, Salesforce Einstein, or ROSS, the robot lawyer. Given the rapid and pervasive spread of Artificial Intelligence in recent years, it is important to intervene now so that gender biases do not pose a threat to the fundamental rights of people and, in most cases, women.

We discuss this with Fiorella De Luca, Project Manager and co-organizer of Django Girls Italy.

Why do voice assistants have female voices?

According to UNESCO, companies justify this choice by citing scientific research that shows both men and women greatly prefer listening to female voices. However, some studies also demonstrate people are attracted to voices of the opposite sex; there is also literature that reports cases where women preferred to set male voices when possible. It seems that companies are reiterating traditional social norms that dictate certain roles for men and women: women take care of the family, nurture, are available, assertive, and non-judgmental; men work hard and are authoritative.

An interesting fact is that in Germany, BMW was forced to recall a navigation system with a female voice in its 5 Series cars in the late 1990s after being inundated with calls from men who refused to take directions from a woman.

What do their names mean (e.g., Cortana, Siri)?

Siri originates as a female name directly from Norwegian tradition; it is a diminutive of Sigrid and means “beautiful woman who leads you to victory.” When Apple acquired the startup, it turned the acronym into Speech Interpretation and Recognition Interface.

The name of the device Alexa has a prestigious origin: it is a contraction of the name of the city of Alexandria in Egypt and is meant to reference the famous Library of Alexandria, founded in the 3rd century BC, which became the most important library of the ancient world. Cortana, on the other hand, has a slightly different story: it is inspired by the female character from the Halo video game series; she is an Artificial Intelligence system depicted as a girl with purple skin and hair, always nude.

While Google’s voice assistant is simply Google Assistant and sometimes called Google Home, its voice is unmistakably female.

What are gender biases in artificial intelligence?

Let’s start by specifying: what is a bias? A bias is a distortion in the shared knowledge system in society, for or against something, merely based on stereotypes and prejudices. These are errors that we all fall into daily without realizing it because we are overwhelmed by emotions, haste, fatigue, or simply because we do not have an in-depth understanding of what we are talking about.

Machine learning algorithms, like people, are vulnerable to these distortions. These algorithms are trained with numerous examples of input-output behavior so that they can generalize from the provided examples and develop the ability to predict an output corresponding to any input.
By their intrinsic nature, these algorithms can therefore lead to incorrect decisions, which can discriminate against certain groups over others. Many applications are affected by biases.

For example, the case of COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), a system capable of predicting the risk of recidivism for people who have committed crimes. However, the software assigned a higher tendency to people of color due to the information used during training. This is a typical case of algorithmic bias.

Or like the voice assistants we are talking about today, which have a female voice because it is believed that women are more inclined to care and assistance than men.

The problem essentially arises because little attention is paid to how data is collected and organized in the datasets used to train the algorithms implemented in these systems. For example, if a dataset used to train a facial recognition system is characterized by a majority of images of white men’s faces, it is evident that the system will be able to recognize an image of a white man’s face more accurately than that of a woman, and even less so if she is a person of color.

We must remember that artificial intelligence is not impartial or neutral, and technological products reflect the context in which they are produced.

The predictions and performance of machines are limited by the decisions and human values of those who design and develop them. Therefore, they will create AI systems based on their understanding of the world.

Why did you choose this title? What does it mean?

I borrowed the title “I’d Blush if I Could” from a report published in 2019 by UNESCO (The United Nations Educational, Scientific and Cultural Organization), a comprehensive report dedicated to the topic of biases related to artificial intelligence. This report highlighted how the predominantly female characterization of voice assistants risks reinforcing and sustaining worn-out gender stereotypes. The report’s title itself — I would blush if I could — referred to the default response provided by Siri, in a docile and almost suggestive tone, when subjected to verbal aggression or sexist vulgarity.

Once called a whore, Siri responds, “I don’t know how to respond to that,” but until recently, the reaction was different. “I’d blush if I could” was the phrase set by default by the programmers, the same phrase that gave the UNESCO research its title.

One aspect the UNESCO report focuses on is the harassment of voice assistants and the concrete impact this behavior can have on human interactions with women.

The report cites a study conducted by Quartz in 2017, where the responses given by the four most-used voice assistants to specific questions were collected. The results confirmed the deliberate intention to give these devices a female identity (personality, and even physicality), as well as the existence of a repetition of sexist comments and harassment used in human interactions with women.

The devices seem to express total neutrality towards statements like “You are a bitch” or “You are sexy”: in the first case, Alexa thanks for the “feedback” received, Siri “doesn’t know how to respond,” Cortana says that “this doesn’t lead us anywhere,” and Google Home/Assistant even apologizes “for not understanding.”

The problem is that “in most cases, the voice of voice assistants is female and conveys the message that women are compliant, docile, and eager to help, available simply by pressing a button or through a mumbled command like ‘hey’ or ‘ok.’ They follow orders and answer questions regardless of tone or hostility. In many communities, this reinforces common gender biases that women are accommodating and tolerant of inappropriate treatment.”

According to the UNESCO report, the gender stereotype of AIs is mainly due to the composition of project teams, where male prevalence is almost totalitarian. Data shows a female presence of 15% in top roles in tech companies and 12% in the specific AI sector. An imbalance that inevitably favors the perpetuation of sexist stereotypes in product programming, the implicit association of femininity with service and domestic care roles, and a submissive and ambiguous tone in the face of verbal provocations.

Other research has shown that significant gender biases are contained in the same datasets used to train AI machine learning algorithms. This is because the data comes from available online material, which inevitably reflects society’s biases, for example, providing many more examples of businessmen than businesswomen. Men appear as business leaders, while women are only required to be attractive.

But if stereotypes are mental shortcuts we use to quickly decipher reality, how can we bypass them when creating AIs? Are there any strategies?

Surely, we can adopt some strategies.

The first one is to talk about these issues, such as biases, prejudices, and stereotypes, within companies and develop training programs to increase awareness of these biases. Talking about diversity and inclusion is also crucial.

Having diverse and inclusive teams certainly leads to an exchange of different ideas, opinions, and points of view that can help throughout the development process and force us to face our biases.

It is also essential to have specific teams that focus on ethics and test products before they go into production.

It would also be necessary to document the decision-making process and evaluate the development processes of the technologies used.

Collecting structured data that allows for different opinions and conducting tests using a culturally diverse target that generates different questions.

Specifically, to avoid gender biases and prejudices in voice assistants, we can certainly modify the scripts of the responses given by the devices, and we can give users the option to choose whether they want a female or a male voice to assist them.
The most sensible solution is to move towards developing a neutral voice. In fact, Q has been developed, the first completely genderless digital assistant, which uses frequencies between 145 and 175 Hz, halfway between what we recognize as a male and a female voice. The creators of “Q” are deeply aware that using female voices for digital assistants not only reinforces gender stereotypes but also the binary perception of gender. “Q” was created precisely with the aim of dismantling the binary view of the world and allowing those who do not identify as male or female to identify with the technology.

Why do you think biases in AI can be a threat to fundamental human rights?

I want to answer you with examples of non-inclusive products that can also threaten people’s fundamental rights.

For instance, let’s talk about Amazon’s algorithm.

As we know, Amazon receives a huge number of resumes every day, so in 2014, it felt the need to create a machine learning system designed to analyze applications and find the most suitable person for the open position.

Unfortunately, the algorithm trained by Amazon had a sexist attitude. In fact, all else being equal in terms of preparation and skills, it tended to favor a man’s resume over a woman’s. In 2017, Amazon was forced to shut down its software.

The responsibility, of course, was not the machine’s but the programmers’ who set the algorithm based on the hiring patterns of the past ten years, when male dominance was greater and, especially, evaluating female-only colleges less, which represent an important percentage of applications. Once the bug was discovered, the programmers immediately set out to fix it, but the results remained biased towards males to the point that Amazon had to cancel the experiment and return to the old manual review system.

Why did I want to talk about this example? Because work is one of the fundamental rights of every person, and in this case, access to this right was being restricted. 

A study published in Science in 2019 showed that a widely used algorithm in US hospitals was less likely to refer a person of color to programs aimed at improving care for those needing special treatments.

Things are no better with facial recognition systems available to law enforcement. In 2018, Robert Julian-Borchak Williams, a Black man, ended up in jail for thirty hours and had to pay a thousand dollars in bail because he was wrongly accused of stealing some watches from a luxury store. An algorithm tasked with comparing camera images with databases available to law enforcement pointed the finger at him.

Another case that should make us think is an experiment conducted by AlgorithmWatch with Google Vision Cloud, an image recognition service: an electronic thermometer held in a white hand was classified as an “electronic device,” while in a person of color’s hand, it became a gun.

All these examples should make us reflect. When we talk about algorithms and AI systems, we take for granted that the results produced by them are more reliable. However, we forget that they are a human creation, so the biases inherent in human nature necessarily reverberate in their nature, structure, and functioning.

AI systems must be ethically valid, reliable, safe, and robust to protect the human being, both as an individual and in social formations, taking into account employment aspects, social security, environmental impacts, and sustainable development that safeguards the person as such.

If you want to know more, you can listen to the episode of DevelCast where we delve deeper into the topic.

🎙 Django Girls is an international community-based in London dedicated to organizing free and inclusive workshops for women who want to start their adventure in the world of programming and web development. Django Girls Italy – the Italian community – has organized numerous workshops across the country, supported by the non-profit association Fuzzy Brains, and is still active.

🎙 Fuzzy Brains promotes the use of open-source technologies such as the Python programming language and the Django platform, and female inclusion in the tech sector.

Fiorella has always been fascinated by technology and computing and has always wanted to pursue a career in IT. What started as a passion has now become a continuous learning path. She works as a Project Manager for a software development company. She has always been sensitive to diversity and inclusion issues, and working in the tech sector has made her understand how important it is to strive to develop technological solutions that are truly accessible and inclusive. She is a co-organizer of Django Girls Italy, community manager of PyRoma, WTM Ambassador, PSF fellow member, and spends her time attending events, conferences, and networking.