In the wave of platform society and intelligent media, the discipline of journalism and communication is facing the opportunity to introduce and localize the auditing method system. Is the technical operation of intelligence and algorithms really opaque and unknowable, or is it just a shield for journalists and researchers to avoid difficult problems, or a disclaimer for policymakers to do nothing?Auditing research in classical social science research has the potential to be a starting point for solving this problem. This paper attempts to introduce the approach of audit research and analyze how it finds its place in the discipline of journalism and communication.
The interdisciplinary trajectory of audit research
Audit research refers to the quasi-experimental approach to identifying systemic biases in society. Bias is a neutral word that includes both bias and discrimination, where bias is an affective bias at the level of attitudes, while discrimination is a behavioral bias based on bias.
Auditing originated in the United Kingdom and the United States in the mid-20th century and has been used in social science research for more than 60 years, with early audits focusing on systemic biases in housing and employment. In the United Kingdom, the first attempt was made in a housing audit study hosted by the Commonwealth National Commission for Immigration. In 1968, in order to assess the social impact of the Race Relations Act, a number of departments cooperated to conduct an audit of discrimination in housing, employment, and services for three groups: white native English speakers, white immigrants, and black job seekers. "Audit" was described at this time as "situational testing" to distinguish it from the more sophisticated methods of investigation, laboratory experiments, etc. In the United States, the study of auditing emerged during the Civil Rights Movement. In 1977, the U.S. Department of Housing and Urban Development conducted the first large-scale housing audit. In partnership with nonprofits that promote housing equity, the department conducted 3,264 audit studies on housing conditions in 20 cities. These findings reveal discrimination against blacks in the housing sales and rental market, which is manifested in the existence of discriminatory sales and rental terms for blacks and higher transaction brokerage fees.
In the early days, audit research was conducted face-to-face, but with the spread of the Internet in housing and employment, audit research began to be carried out by e-mail, and communication audits gradually took shape. Researchers have also moved beyond simplistic descriptions of bias to more in-depth exploration of the mechanisms, intentions, and conditions of bias. The world's first communications audit appeared in the UK in 1969. The researchers applied the resumes of 256 applicants to 128 job openings and found that whites born in the UK were treated more favourably in real-life recruitment than the other four groups of immigrants, revealing systemic racial discrimination in employment in the country. Traditional audit research focuses on issues such as race and discrimination. Since the beginning of the 21st century, the object of audit research has been further expanded to bias problems in the fields of medical health and marketing.
Traditional audit research contains concern for social fairness and public interest, which coincides with the long-standing value norms and legitimacy discourse of journalism, so audit results have become an important topic for news reporting. In recent years, platforms and algorithms have become new audit objects, which has also accelerated the interdisciplinary integration of audit research in social science and journalism and communication research. Journalists and academics consciously or unconsciously use audit ideas or methods to try to identify algorithmic biases and discover how decision-makers' positions, engineers' logic, user habits, and societal biases are embedded in the new "techno-social" chain.
Audit algorithm bias
Algorithmic bias is the focus of current audit research. An algorithm is a set of instructions and steps run by a computer to solve a specific problem or achieve a definite goal. The use of audit methods to systematically review the result bias of algorithms, also known as algorithm auditing, is often used to detect the class, race, and gender biases implicit in algorithms, and to analyze the interconstructive relationship between algorithms and society.
The object of algorithm audit is the interaction between different types of algorithms and social practices. China's "Provisions on the Administration of Algorithmic Recommendation for Internet Information Services" divides algorithm technology into five categories: personalized push, sorting and selection, retrieval and filtering, scheduling and decision-making, and generation and synthesis. The first four types of algorithms are primarily concerned with mining, filtering, optimizing, and scheduling information from existing data and resourcesGenerative synthesis algorithms are used to generate new texts and images, such as ChatGPT, which caused heated discussions in early 2023. In the field of journalism and communication, journalists and scholars have comprehensively used audit methods to investigate the bias contained in various algorithms and their social consequences, and included issues such as data bias and information gap into the scope of auditing.
Audit studies of search filtering algorithms are the most common. Marina Frasnu et al. of New York University used a combination of crawler audits and non-intrusive user audits to audit image search algorithms in 153 countries to verify gender bias in search algorithms. Published in the Proceedings of the National Academy of Sciences (PNAS)**, the paper reveals the cyclic proliferation of bias in society, artificial intelligence technology, and the multidimensional interactions of digital listeners. For another example, Chinese scholars Chen Changfeng and Shi Wen conducted an audit of the search of domestic mainstream search engines, and found that the interest relationship between search engines and business ** will lead to bias and challenge the public nature of the digital ecology.
The adoption and operation of scheduling decision-making algorithms in society are more secretive, which requires long-term and systematic audit research by professional journalists. The nonprofit Propublica's investigative story series titled "Machine Bias" was shortlisted for the 2016 Pulitzer Prize for Interpretive Journalism. The report found that black defendants are more likely to be misconvicted in the process, while white defendants are more likely to be missed, revealing the systemic bias in the judicial system's use of algorithms to assist in adjudication.
The methods of algorithmic auditing can be summarized into five categories. The first is the audit, which analyzes the platform source to find possible program errors and security vulnerabilities;The second is crawler audit, in which the auditor sends a query request to the platform to observe the data resultsThe third is a non-intrusive user audit, in which real users are invited to report the results of their interactions with the system, and then conduct system analysisFourth, the first audit, using a computer program to simulate the user to observe the output of the algorithm, which can be simply understood as the researcher using a robot to experiment with the algorithm system;Fifth, crowdsourcing audit, based on the user's wishes, directly intervenes in the user's account and collects data for auditing. However, the above-mentioned audit methods face unresolved ethical issues such as invasion of privacy and reproduction of false information, which are worthy of vigilance.
Positioning audit research
How should audit research be integrated into existing journalism and communication disciplines?It is analyzed from three aspects: practice, theory and method.
In practice, algorithmic auditing has the potential to become an independent news reporting line and expand the reporting space of public issues. For example, Sun Ping and other researchers found that algorithm platforms discipline the digital labor and emotional labor of riders, substitute drivers and other groups. A number of news systems, including People magazine, followed up and introduced algorithmic bias into the first reports and the public eye, so that the digital labor issue received widespread attention. Since 2020, the "Basket Project"** has continued to focus on the disadvantaged groups in the algorithm system, telling the stories of different professions such as data annotators and programmers. Another example is the data report, "The Markup", which examines the social impact of platform technology companies by obtaining a basic data set.
In terms of theory, audit research expands the ideas for constructing digital journalism theory, especially provides theoretical and methodological preparation for demonstrating the concept of "transparency". In the journalism theory of the mass ** era, the principle of "objectivity" of news reports to distinguish between facts and opinions has always been enshrined, but it has always faced unrealizable doubts and criticism of power vassals. In a post-truth context where professional content (PGC), user content (UGC) and algorithmic content (AGC) are highly mixed, the principle of "objectivity" is challenged as never before. Some scholars have proposed replacing objectivity with transparency. For example, Bill Kovacs and others argue that "the use of verification, especially the notion of transparency, is one of the most effective ways to address bias". More and more news is adopting transparency practices, such as disclosing source data at the end of a story, declaring how human-machine collaboration is divided. Using the audit method, researchers can evaluate the world's mainstream transparency practices, and introduce concepts such as transparency, auditability, and accountability into digital journalism research to expand the theoretical construction of digital journalism.
In terms of methodology, audit research is essentially a quasi-experimental method, which is simply an experiment on the system. The digital space of the Internet, where algorithm technology is deeply embedded, is not only an information recording space, but also a new social action space. Researchers can build experimental scenarios and simulate experimental conditions according to the research objectives on existing social platforms and algorithm platforms, and there may be systematic biases in the interaction between algorithms and human behavior. This method has the advantages of low cost and high external validity. Therefore, audit research can be seen as a new development of experimental methods under the condition of data and intelligence.
In summary, audit research aims to empirically examine the systemic biases that exist in society, and has accumulated a wealth of cases and feasible methods. Audit research can help improve the quality of news, ensure credibility, improve platform supervision, improve public media literacy, and promote relevant academic research.
This paper is a phased achievement of the National Social Science Youth Project "Research on Boundary Reconstruction and Value Leadership of News Production under the Condition of Artificial Intelligence" (19CXW008).
(Author's Affiliation: School of Journalism, Communication University of China;Institute of Survey and Statistics, School of Journalism, Communication University of China).
Authors: Qiu Yunqian, Wu Ao, Zhang Hongru **China Social Sciences Daily, Issue 2812, January 10, 2024.