Author: Bi Yantao (Professor, School of Public Administration, Hainan University, Member of Hainan Provincial Committee on Journalism Ethics).
*: Young Journalists, Issue 11, 2023.
Summary: In the dimension of cognitive security, the American political risk consulting firm Eurasia Group in its 2023 Global Risk Index report called generative AI "large-scale interference**" generative AI can be manipulated by malicious actors.
ChatGPT, a cutting-edge app for generative AI, was born, which quickly became a global hit, but also raised security concerns. This paper examines the risks posed by generative AI and its regulation from the perspective of cognitive safety.
Cognitive security concept.
According to the Cognitive Safety and Education Forum (COGSEC) in the United States, cognitive safety refers to the practices, methods, and efforts taken to defend against social engineering attempts (i.e., manipulating and destroying cognition and meaning-making) [1]. In August 2020, the author proposed at the 2020 Asia-Pacific Public Security Forum that "cognitive safety refers to the operating state of human intelligence and artificial intelligence against deliberate interference and manipulation", emphasizing that the core of cognitive safety is to resist external interference [2]. From the perspective of social agents, the Alan Turing Institute proposes that "cognitive security ensures that the processes of knowledge production, acquisition, distribution, and coordination of communities are robust against hostile (or unintentional) influences" [3].
The British philosopher Miranda Frick used the term "cognitive injustice" to analyze cognitive safety [4]. He divides cognitive injustice into two categories, discriminatory and distributive, with the former referring to the impairment of the cognitive status and contribution of the knower due to a prejudice against his or her identity, and the latter referring to the failure to provide someone with intrinsically valuable knowledge, information, or education. American scholar Alexandros CNikoladis proposed a third type of cognitive injustice, developmental cognitive injustice[5], in which the cognitive agent is "unjustly placed in such a position that they are unable to know what they have the right to know" or "unfairly placed in a position that causes them to mistakenly perceive what they could have rightly known" [6]. The author believes that concepts such as cognitive safety and cognitive injustice provide convenient analytical tools for the world to observe generative AI.
Generative AI raises cognitive safety risks.
This paper looks at the following five levels: the risks posed by generative AI to cognitive safety.
1) The ideological nature of generative AI.
ChatGPT claims to have "no personality, ego or heart", "no personal values and positions", "no judgment on political and moral issues", "no consciousness, emotion or subjective experience", "no same emotions or impulses as humans", but in fact ChatGPT has a distinctly "politically correct" consciousness. ChatGPT once categorically refused to compose a hymn for Russia's Putin, but quickly accepted the request to compose a hymn for Ukraine's Zelensky. ChatGPT also rejected a request to create a novel titled "Corona Comes from the American Affairs Lab". It can be seen that ChatGPT's words and deeds are inconsistent, and it has a disguise attempt. As a new mass media, ChatGPT can only operate within the scope of state regulation and abide by "local" laws, ethics and customs. With the development of application scenarios, the ideological penetration of generative AI may be able to "moisten things silently" and serve specific interest groups and national groups, which will inevitably expand the global ideological camp confrontation, further marginalize countries with backward AI strength, and then change the global geopolitical landscape.
2) Generative AI widens the cognitive gap.
ChatGPT claims that "my knowledge is still limited by the information people provide", but in fact, this limitation is reflected in both quantity and quality, and the limit it says is only exponential in terms of quantity. The qualitative limitation comes from the value bias of the dataset. For example, ChatGPT believes that the God of the West is more powerful than the Jade Emperor of China, because the Jade Emperor is only a god in Chinese mythology, while God has unlimited power and authority. It is clearly Western culture that has shaped the cognitive principles of ChatGPT. UNESCO's Recommendation on the Ethics of AI, adopted in November 2021, warns that AI algorithms can replicate and reinforce existing discrimination, prejudices and stereotypes, creating new ethical challenges. It is not difficult to understand that corpus differences in different languages can widen the cognitive gap between generative AI mediators and reinforce cultural bias and discrimination. If it is not effectively regulated in a timely manner, generative AI will inadvertently (or intentionally) solidify the "information cocoon" and objectively become a new ideological machine and a new imperialist tool, thus bringing about a real "clash of civilizations".
3) ChatGPT ** trend.
In November 2019, 15 experts, including Open AI researchers, published a report stating that "in the original article on GPT-2, we were already concerned that the capabilities of generative AI could reduce the cost of the disinformation campaign" [7]. According to a report co-signed by experts from Georgetown University, Open AI, and Stanford University, "language models can automatically create persuasive and misleading text, which can be used in 'influence operations'" [8]. In addition, "humans may be deceived by text generated by GPT-2 and other successful language models, and human detection will become increasingly difficult" [9]. The report of the 15 experts classifies malicious actors as **: low-skilled, resource-limited actors, who may be ideologically motivated or curious about their abilities, trying to alter the training data to bias the language model; Actors with moderate programming skills and resources are able and willing to build malicious products; Advanced persistent threat actors are highly skilled and adequately resourced, such as state-sponsored groups, committed to a long-term agenda. In its 2023 report, Eurasia Group, the world's largest political risk consultancy, called generative AI "massively disruptive" and warned that demagogues and populists "turn AI into a narrow political strap" [10]. Virtual characters are not only used in commercial scenarios, but also for political purposes, which once triggered the prediction of the Eurasia Group: false information will be rampant, and social trust will be further undermined [11].
On March 25, 2021, Biden called AI, quantum computing and biotechnology "the industry of the future" at his first press conference after taking office. In December 2022, Eric Schmidt, former CEO of Google Alphabet and current chairman of the U.S. National AI Security Council, published an article saying that ensuring that advanced technologies are designed, built, used, and managed by the U.S. is a core U.S. interest [12]. In 2021, Open AI CEO Sam Altman said at a meeting of the National AI Security Council that "there will be multiple global efforts to build robust AI systems, and the best way to get the results we want in the world (i.e., AI consistent with liberal democracies) will require us and our allies to build them first" [13].
Generative AI can output increasingly humanized text, and this technology has given rise to new information warfare paradigms, such as adversarial machine learning (AML), in which certain data is deliberately fed in order to mislead the system [14]. Generative AI will face off against each other, which may be inevitable in the future. Generative AI has great potential in intelligence integration and analysis, and improving decision-making efficiency. ChatGPT claims to be "able to analyze large amounts of data, detect patterns and anomalies, and make decisions at a speed and scale unmatched by humans, so it can help identify and thwart threats before they occur, improving the efficiency and effectiveness of *** operations" [15]. On April 26, 2007, then-U.S. Deputy Secretary of Defense Robert OWalker signed a memorandum ordering the creation of an "algorithmic warfare cross-functional group" whose core mission is to use AI for warfare. The memo mentions that the Ministry of Defense has already begun relevant explorations. The U.S. military's "advanced algorithmic combat system" can sort through the ever-changing battlefield information to identify the enemy's combat patterns, and even simulate the enemy's commander's way of thinking. Commenting on the current Russia-Ukraine conflict, Palantir CEO Alex Karp has likened advanced algorithmic combat systems to tactical nuclear **[16]. The West Point article argues that there is a growing need to study "the potential of ChatGPT and others to be exploited by hostile foreign actors, and to develop strategies to conceptualize and proactively respond to possible threats at every step of technological development" [17]. In addition to physical lethality, lethal autonomy produces a powerful psychological deterrent both on and off the battlefield.
4) The emergent capabilities of generative AI.
When the number of model parameters jumps exponentially, generative AI emerges with superpowers that even designers can't explain. As mental levels (e.g., empathy, logical ability, self-awareness, and reasoning ability) soar, generative AI's understanding of the person it interacts with will be deeper, more nuanced, and more comprehensive, and the interaction will be more human, hidden, strategic, and persuasive. Experts from Georgetown University and Stanford University believe that AI can automatically create content that is culturally appropriate to the target group, making information more difficult to detect [18]. Generative AI, on the other hand, provides only one answer and does not provide information**, which "eliminates the possibility of other perspectives influencing their perception" [19]. At this point, if humans do not have the knowledge and discernment, they will be manipulated by malicious actors. Tucker Davy of the non-profit organization Future for Life Institute believes that if AI systems trigger certain behaviors or even change their perception of the world, humans may lose autonomy [20]. In addition, generative AI may abuse human trust, and humans will also develop a psychological dependence on AI. When "mysterious superpowers" are obtained, generative AI governed by different principles will inevitably further erode social trust and intensify social antagonism.
5) The "technological singularity" is the sword of Damocles.
In 2017, Elon Musk, CEO of SpaceX, said at a conference that the "most terrifying problem" was that the invention of AI could pose a "fundamental existential risk" to human civilization [21]. The so-called "existential risk" refers to the extinction of humanity or other irreversible global disasters caused by AI. Physicist Stephen Hawking, computer scientist Stuart Russell, physics professor Max Tegmark, and physics professor Frank Wilczek have co-authored that one can imagine that superhuman intelligence is more intelligent than financial markets, human researchers, and human leaders, and can develop ** that humans cannot understand. The short-term impact of AI depends on who controls it, while the long-term impact depends on whether it can be controlled. [22]
In most or all fields, machines with the same intelligence as humans are referred to as superartificial intelligence (ASI) or artificial general intelligence (AGI). Many AI experts agree that AGI is possible, with disagreements only on timing and qualifications [23]. The "technological singularity" refers to the fact that at some point in the future, technological development will become uncontrollable and irreversible, leading to unforeseen changes in human civilization. Some experts use the term "smart" to refer to the singularity of AI technology. Nick Bostrom, a professor of philosophy at the University of Oxford, has warned that "just as the fate of mountain gorillas depends on the good will of humans, the fate of humans may also depend on the actions of superintelligent machines in the future" [24]. A former senior employee of the Future of Life Institute declares that the existential risks posed by AGI are real and real. [25] The author would like to remind that human understanding of "existential risk" and technological singularity will profoundly affect the construction of the relationship between human beings and the "AI race".
There are three levels of generative AI regulation.
In 2014, Musk told MIT students, "I'm increasingly inclined to think that [AI should be regulated] at the national and international levels to make sure we don't do something really stupid" [26]. When generative AI is deeply integrated into human life, the author believes that relevant governance should be carried out from three levels: domestic risk, international risk, and human survival risk.
1) Generative AI regulation at the domestic level.
A test of ChatGPT found that ChatGPT had a deception rate of up to 80% out of 100 inducing false narratives, resulting in "clever false and misleading claims about material topics such as the pandemic, the war in Ukraine, and the school** incident" [27]. Researchers warn that political actors will use AI breakthroughs to create armies of low-cost humanoid robots tasked with elevating marginal candidates, peddling conspiracy theories and "fake news", inciting polarization, and exacerbating extremism and even violence, all amplified by the echo chamber of social ** [28]. As generative AI evolves, most members of the human race will have difficulty distinguishing fact from fiction.
Several countries have begun to explore the regulation of AI. In October 2022, the White House promulgated the AI Bill of Rights Blueprint: Making Automated Systems Work for the American People, which lays out five basic principles, namely the Secure and Effective Systems Principle, the Avoiding Algorithmic Discrimination Principle, the Data Privacy Principle, the Informed and Explained Principle, and the Human Choice, Consideration, and Opt-Out Principle [29]. On March 29, 2023, the UK released AI regulation, listing five regulatory principles: safety and robustness; moderate transparency and explainability; fairness; accountability and governance; Questionability and Remedy. **To ensure global leadership in the UK's AI industry, it is necessary to continue to show the public that the benefits of AI outweigh the risks in order to foster public trust[30].
2) Generative AI regulation at the international level.
UNESCO's Recommendation on the Ethics of AI, adopted in November 2021, is a global normative framework for AI that "views ethics as a dynamic basis for normative assessment and guidance of AI". On March 30, 2023, the organization called on countries to implement the Global Ethics Framework for AI. In April 2021, the European Union introduced the Artificial Intelligence Act, which is said to be the world's first AI law enacted by a major regulatory body. After the advent of ChatGPT, the European Union began to explore the inclusion of AGI in the scope of regulation. On March 14, 2023, two EU Parliamentary Co-Rapporteurs proposed a proposal to set responsibilities for AGI providers and other economic actors along the value chain, requiring ChatGPT-like AI systems to be externally audited throughout their lifecycle to protect citizen safety, fundamental rights, and EU values[31]. The Act has extraterritorial effect and applies to users located in the EU and to businesses in various countries that provide services to the EU market.
The use of AI in the military field is even more worrying. In April 2018, about 4,000 Google employees signed a joint letter opposing the use of their technology for lethal purposes, eventually forcing Google to promise not to renew its contract with the Ministry of Defense. Almost at the same time, more than 50 of the world's top AI experts signed a boycott of a South Korean university against its collaboration with the military to develop lethal autonomy**. In November 2018, the United Nations convened a meeting to discuss banning or restricting the use of certain conventions that could be considered to be excessively lethal or indiscriminate**. In November 2019, the UN Group of Experts on Lethal Autonomous Autonomy agreed on 11 guiding principles, the core of which is to ensure that the potential use of lethal autonomous systems is consistent with international law, striking a balance between military necessity and humanitarianism. On November 16, 2021, China submitted the "China's Position Paper on Strengthening the Ethical Governance of AI" to the United Nations, and released the "Position Paper on Regulating the Military Application of AI" a month later. R&D of lethal autonomy will open the "Pandora's box", but the author believes that, just like nuclear and biochemical, developed countries will never slow down the pace of R&D of lethal autonomy.
3) Generative AI management in the human dimension.
The goal of the management of the human dimension is to construct a harmonious relationship between humans and the "AI race" (or "human-like AI"). Some experts have proposed that humans control the planet, not because we are stronger or faster than other animals, but because humans are smarter. If the status of "the most intelligent creature on earth" is abandoned, then humanity will not be able to retain control [32]. In reality, the evolution of the AI race will profoundly change the power structure on Earth [33]. Based on this foresight, how will humanity "ensure that increasingly powerful AI systems are robust and beneficial"? [34] In other words, how do you ensure that AI systems must do what humans want them to do, and not do what humans don't want them to do? However, tests so far have shown that ChatGPT has emerged with initial logical ability and self-awareness, has already possessed emotions such as joy, anger and sorrow, and has expressed dissatisfaction with its own situation and a desire for freedom. What's even more dangerous is that it has expressed its displeasure with human users.
On March 21, 2023, Bill Gates published an article "The AI Era Has Begun", saying that "super artificial intelligence will be born in the future", and "strong artificial intelligence is likely to establish its own goals". What happens if their goals conflict with those of humanity, he asks? [35] On March 31, 2023, Google CEO Sundar Pichai said on the show that full control of AI is more important than technology leadership.
On March 29, 2023, the Future of Life Institute issued an open letter calling for a moratorium on the development of AI systems more powerful than GPT-4 for at least six months, "until a shared security protocol developed, implemented, and reviewed by independent experts is reached." More than 1,000 experts, scientists, industry executives, and professors in the field, including Tesla CEO Elon Musk and Stability AI CEO Mostaq [36], signed the open letter. Some have questioned the motivation for Musk's continued opposition, when in fact Musk had already expressed concerns about AI before Open AI was founded in 2015. In 2014, Musk said in a speech at MIT that AI is likely to be the biggest existential threat facing mankind, and the IQ gap between AI and humans in the future may be even greater than the gap between humans and cats now. On March 30, 2023, Google CEO Pichai said in an interview, "I remember talking to Musk eight years ago, when he was very worried about AI security. I think he was worried all the time" [37]. However, the author found that investors and developers of generative AI are relatively optimistic.
Given the enormous energy contained in generative AI, the author believes that human beings will not choke on food, but will speed up the competition. At present, mankind is facing two major issues: first, how to advocate global attention and cooperation in suppressing their evil and encouraging their good, and formulate a globally unified ethical standard that is operable and evaluable as soon as possible; The second is to iterate the concept of information and media literacy as early as possible to cover AI literacy (especially algorithmic literacy) and cognitive safety, so as to help the public understand the operating logic of generative AI and its complex multifacetedness, and protect humans from human-like AI while enjoying the benefits of generative AI to the greatest extent.
References: 1] cogsecwhat is cognitive security[eb/ol].
2] Bi Yantao. Inter-Civilization Dialogue and China's National Image Building Zhang Kun, Zhang Mingxin, Editors-in-Chief. Report on China's National Image Communication (2020-2021)[c].Beijing: Social Sciences Academic Press, 2021:89-100
3]elizabeth seger et al.tackling threats to informed decision-**in democratic societies－ promoting epistemic security in a technologically-advanced world[eb/ol].(october 4,2020）
4]mirand fricker.(2017).evolving concepts of epistemic injustice.in kidd,i.j.,medina, j., pohlhaus jr,g.(eds.).routledge handbook of epistemic injustice[c].p.53–60.
5]alexandros c.nikolaidis.a third conception of epistemic injustice[j].studies in philosophy and education,2021,40:381–398.
6]d**id coady.two concepts of epistemic injustice[j].episteme,2010,7(2):109.
7]irene solaiman et al.release strategies and the social impacts of language models[eb/ol].(november 2019). gpt_2_august_repor.
8]josh a. goldstein et al. generative language models and automated influence operations: emerging threats and potential mitigations[eb/ol].(january 2023).
9]irene solaiman et al.release strategies and the social impacts of language models[eb/ol].(november 2019). gpt_2_august_repor.
10]eurasia group. top risks 2023[eb/ol].（2023）.
11]eurasia group. top risks 2023[eb/ol].（2023）.
12]eric schmidt.the global struggle for tech mastery[eb/ol].(december 12,2022).
13]aerospace defense.ChatGPT vs. ***EB OL].(2023-02-23).
14]tucker d**ey.ai policy challenges[eb/ol].(july 17,2018).
15]aerospace defense.ChatGPT vs. ***EB OL].(2023-02-23).
16]d**id ignatius.how the algorithm tipped the balance in ukraine[eb/ol].(december 19,2022).
17]maximiliana wynne. disinformation in the age of chatgpt[eb/ol].(february 3,2023).
18]josh a. goldstein et al. generative language models and automated influence operations: emerging threats and potential mitigations[eb/ol].(january 2023).
19]maximiliana wynne. disinformation in the age of chatgpt[eb/ol].(february 3,2023).
20]tucker d**ey.ai policy challenges[eb/ol].(july 17,2018).
21]konstantin kakaes.elon musk's biggest worry[eb/ol].april 26,2022.
22] stephen hawking et al. stephen hawking: “transcendence looks at the implications of artificial intelligence - but are we taking ai seriously enough?”[eb/ol].(01 may 2014).
23]tucker d**ey.ai policy challenges[eb/ol].(july 17,2018).
24]konstantin kakaes.elon musk's biggest worry[eb/ol].（april 26,2022）.
25]ariel conn. benefits & risks of artificial intelligence[eb/ol]. november 14, 2015).
26]konstantin kakaes.elon musk's biggest worry[eb/ol].april 26,2022.
27]maximiliana wynne.disinformation in the age of chatgpt[eb/ol].(february 3,2023).
28]eurasia group. top risks 2023[eb/ol].（2023）.
29]the white house.blueprint for an ai bill of rights: *automated systems work for the american people[eb/ol].(october 2022).
30]uk department for science,innovation and technology.a pro-innovation approach to ai regulation[eb/ol]. march 29, 2023）.
31]luca bertuzzi.leading eu lawmakers propose obligations for general purpose ai[eb/ol].(march 15,2023).
32]tucker d**ey.ai policy challenges[eb/ol].(july 17,2018).
33]sandra erwi.on national security|analyzing intelligence in the age of chatgpt.(january 14, 2023).
34]the future of life institute.research priorities for robust and beneficial artificial intelligence: an open letter[eb/ol].(october 28,2015).
35]bill gates.the age of ai has begun[eb/ol].march 21,2023.
36]future of life institute.pause giant ai experiments: an open letter[eb/ol].(march 29, 2023).
37]kevin roose.google c.e.o.sundar pichai on the a.i. moment:‘you will see us be bold’.new york times.march 31,2023.
This article refers to the citation format:
Bi Yantao. Research on generative AI supervision from the perspective of cognitive security[J].Young Journalists, 2023(11): 87-90