Chen PeiwenRecently, the AI data poisoning tool Nightshade of the University of Chicago research team was officially released, and it is said that the number of ** exceeded 250,000 in just 5 days after it was launched, and its popularity far exceeded the research team's expectations.
I was looking at Nightshade a few months ago when the team announced that it was going to be released, as it was different from the usual idea of adding watermarks or logos to AI-generated content.
Nightshade allows artists to upload their own original drawings and then embed toxic data in the paintings that are identified by machine learning algorithms as misinformation, such as a car with four legs. Therefore, if an AI company uses dozens of such images to train an AI image generation system without the artist's permission, it is equivalent to being poisoned, and the accuracy of the system's output will drop dramatically.
It can be seen that due to the artist's strong dissatisfaction with the random use of artworks for AI training, with the advent of Nightshade, the artist picked up the AI data poisoning tool and began to turn defense into offense.
As a result, there will definitely be some new changes in the development of AI.
AI developers will definitely have to increase the workload in data cleaning and security verification, and the cost and complexity of AI model development will further increase. In the process of AI application operation, the technical link of data verification will continue to be introduced to identify and filter poisoning data to ensure the reliability of AI model operation.
This could drive a shift in the AI industry toward more copyright-respecting, transparent access to training data. At the same time, there may also be development initiatives and action guidelines that require the healthy development of AI technology not to have a potential impact on the fundamental rights and interests of individuals.
The emergence of Nightshade, an AI data poisoning tool, is an important event. Not only does it reflect the tension between AI development and copyright protection, but it also marks the imminent arrival of a second type of risk posed by AI.
From the perspective of industry observation, I believe that the risks brought about by AI development can be divided into three main categories. The first category is that AI is used for various malicious purposes, such as counterfeiting, false advertising, fraud, cyber attacks, etc.; The second category is the risk of AI utilization that has evolved in various confrontations between attack and defense; The third category is the risk that AI will develop too fast and exceed human expectations.
Among them, the first category is known risks, which existed before the advent of AI, but the application of AI has increased the harmfulness of such risks. The various AI risk prevention requirements that are being promoted today also focus on this.
Categories 2 and 3 are unknown risks. Even with careful research and deduction, it is difficult to know the full picture of these risks. For example, before the advent of Nightshade, the industry was following the development idea of AI-generated content watermarks, but the emergence of such a poisoning adversarial tool may become the main tool for content protection. The risks posed by such confrontations have not been easily taken into account in previous risk analyses.
In addition to Nightshade, Anthropic has also recently published research results on embedded backdoors in the AI model training process, and it is expected that we will also see more and more AI countervailing offensive and defensive risks surfaced.
Finally, as for the risk of AI developing too fast and exceeding human expectations, researchers such as Hinton and Sutskever, who are at the forefront of AI, have discussed it, and its connotation has risen to the level of values and development concepts, which cannot be talked about in one or two articles. However, I still maintain the basic judgment in the previous article, the development of AI will be faster and faster, and the changes, opportunities, risks, and challenges it will bring will be greater than now expected.