AI: System vulnerabilities, safety and the human factor
Humanity is now well on the way to the fast development of freely accessible, godlike AI technology that can be immensely beneficial but dangerous.
Concerns about safety and AI's existential threat to humanity have been dismissed by some, likely shaped by fear that these concerns could dampen or break the financial AI bubble in the U.S. economy as well as the fear that the U.S. will fall behind in the AI development race with China.
Many others including Geoffrey Hinton, the Nobel-winning computer scientist who helped create AI, believe AI has a tremendous upside, but have expressed three concerns about dangerous areas of related risk: harm-causing inappropriate responses; loss of human control via lack of stringent safety guardrails to prevent the system from going rogue; and maliciously motivated human and/or state bad actors that elude stringent security measures.
Not mentioned is a fourth risk area, humans who unwittingly make fatally bad decisions either via hubris or by their careless training of AI agents that have a propensity to go rogue as well as undetected. The first risk area can be resolved by engineering design. However, the second risk area involves troubling, if not impossible design objectives, while the third and fourth risk areas are in the domain of human factors, the primary focus of this essay.
A lesson applicable to AI systems, learned back in 1958 from the late Mitford Matthews, the National Security Agency's Assistant Director for Research and Development, was that no matter how secure the encryption of a data transmission, the overall system will still be vulnerable to human factors. It formed the basis for a May 15, 2023, Daily Herald Guest View, “The human factor in America's security systems.” The Guest View told why technically “perfect” security fails; an unbreakable encryption code does not equate to an unbreakable system since the primary point of failure is not technical, but human.
Similarly, closed AI systems are not perfectly closed. Like crypto systems, AI systems rely on individuals who have been granted security clearances. When insiders are untrustworthy, careless, or malicious, they can bypass all technical safeguards and take control of the system.
Unless and until AI companies can effectively manage and mitigate the human element, even the most expensive and advanced systems will remain deeply vulnerable.
This is not only a formidable task, but, over time, likely an impossible task as well since human nature is what it is. Clearly, there is an urgent need for stringent background investigations coupled with periodic vetting of all individuals holding sensitive company positions by a responsible independent agency that must be held accountable for any breach of security. Costly though it may be, it's the price that must be paid to help minimize the vulnerability of America's AI systems.
AI leaders will also face the AI dilemma, to wit: not pursuing the enormous benefits of godlike technology on the one hand and pursuing those benefits at the risk of triggering catastrophic events or even humanity's extinction on the other. According to AI ethicists at the Center for Humane Technology, resolving this dilemma requires an approach that rejects the false binary of total capitulation or complete shutdown, so AI systems must be subjected to exhaustive safety testing.
To that end, U.S. AI system providers will do their utmost to engineer safe products so as to avoid liability lawsuits and prosecution under the Computer Fraud and Abuse Act of 1986 (18 U.S. 1030), especially if their officers and boards of directors are subject to civil liability for the systems their company deploy. Nonetheless, these providers cannot be trusted to self regulate. To enhance trust in an AI system's safety before deployment, it should also be subjected to multiple third-party testing by organizations such as MIRI and METR.
There can be little doubt that fast-paced AI technology development will continue by all parties that are incentivized by their quest for market dominance, ultimately by money and geopolitical power. President Trump is working to boost AI to shore up support for data centers and to buoy the U.S. economy — employing a strategy of minimal regulation to stay ahead of China in the AI race. According to a recent AP poll, most Americans think AI is developing too fast.
In conclusion, although the price of not being first in the race with China could be very high, there is likely a higher price associated with AI's human-based risks: malicious intent and unwitting-fatally-bad decisions. Understanding that both human-based risks can lead to catastrophic events should prompt AI CEOs and world leaders to be ultra vigilant, recognize the peril ahead and slow down their fast-paced race to attain market and/or geopolitical dominance.
Therein is an eerie reminder of the ill-fated Titanic, said to be the safest ship afloat with its state-of-the-art safety features, that was commanded by a veteran captain who pursued a speedy race across the North Atlantic in April 1912, despite multiple warnings of icebergs ahead.
• Frank G. Splitt, of Mount Prospect, is a former McCormick Faculty Fellow at Northwestern University and was a Vice President Emeritus of Nortel Networks. He is author of the book “An Odyssey of Reform Initiatives: 1986-2015” and two sequels.