FCS Perspective：机器学习领域先驱者Thomas G. Dietterich教授观点分享
本期FCS Perspectives栏目，美国俄勒冈州立大学电气工程和计算机科学学院Thomas G. Dietterich教授受FCS主编之邀撰文。
Dietterich教授是机器学习领域的先驱者之一。他最著名的工作是在机器学习中的集成方法，包括发展纠错输出编码，他还发明了重要的强化学习算法，包括用于分层强化学习的MAXQ方法；他也是国际机器学习学会创始主席、国际人工智能学会（AAAI）前主席、美国白宫《国家人工智能研究与发展策略规划》起草人之一。
Dietterich教授认为，AI技术增加了人类组织和行动的失误被技术放大，并造成破坏性后果的风险。为避免这些灾难性的失误，人类和AI组织的协作必须具备高可靠性。高可靠性组织的工作指出了技术开发和政策制定的重要方向。非常关键的问题是，我们必须立即遵循这些方向；并且，确保AI的利用只能在保持高可靠性的组织内进行。
全文推出此篇国际重量级学者的观点分享，以飨读者。
Frontiers of Computer Science,
https://doi.org/10.1007/s11704-018-8900-4
Thomas G. DIETTERICH
Thomas G. Dietterich (AB Oberlin College 1977; MS University of Illinois 1979; PhD Stanford University 1984) is Professor Emeritus in the School of Electrical Engineering and Computer Science at Oregon State University, USA where he joined the faculty in 1985. Dietterich is one of the pioneers of the field of Machine Learning and has authored more than 190 refereed publications and two books. His research is motivated by challenging real world problems with a special focus on ecological science, ecosystem management, and sustainable development. He is best known for his work on ensemble methods in machine learning including the development of errorcorrecting output coding. Dietterich has also invented important reinforcement learning algorithms including the MAXQ method for hierarchical reinforcement learning.
Dietterich has devoted many years of service to the research community. He is Past President of the Association for the Advancement of Artificial Intelligence, and he previously served as the founding president of the International Machine Learning Society. Other major roles include Executive Editor of the journal Machine Learning, co-founder of the Journal for Machine Learning Research,and program chair of AAAI 1990 and NIPS 2000. He currently serves as moderator for cs. LG, the machine learning category on ArXiv. Dietterich is a Fellow of the ACM, AAAI, and AAAS.
Robust artificial intelligence and robust human organizations
Thomas G. DIETTERICH
The more powerful technology becomes, the more it magnifies design errors and human failures. An angry man who has only his fists, cannot hurt very many people. But the same man with a machine gun can kill hundreds in just a few minutes. Emerging technologies under the name of “artificial intelligence”(AI) are likely to provide many new opportunities to observe this “fault magnification” phenomenon. As society contemplates deploying AI in self-driving cars, in surgical robots, in police activities, in managing critical infrastructure, and in weapon systems, it is creating situations in which errors committed by human users or errors in the software could have catastrophic consequences.
Are these consequences inevitable? In the wake of the Three Mile Island nuclear power plant failure, Perrow[1] published his book “Normal Accidents” in which he argued that in any sufficiently complex system, with sufficiently many feedback loops, catastrophic accidents are “normal”—that is, they can not be avoided.
Partly in reaction to this, Todd LaPorte, Gene Rochlin, Karlene Roberts and their collaborators and students launched a series of studies of how organizations that operate high-risk technology manage to avoid accidents [2]. They studied organizations that operate nuclear power plants, aircraft carriers, and the electrical power grid. They summarized their findings in terms of five attributes of what they call high reliability organizations (HROs):
1)Preoccupation with failure. HROs believe that there exist new failure modes that they have not yet observed. These failure modes are rare, so it is impossible to learn from experience how to handle them. Consequently, HROs study all known failures carefully, they study near misses, and they treat the absence of failure as a sign that they are not being sufficiently vigilant in looking for flaws. HROs encourage the reporting of all mistakes and anomalies.
2)Reluctance to simplify interpretations. HROs cultivate a diverse ensemble of expertise so that multiple interpretations can be generated for any observed event. They adopt many forms of checks and balances and perform frequent adversarial reviews. They hire people with non-traditional training, perform job rotations, and engage in repeated retraining. To deal with the conflicts that arise from multiple interpretations, they hire and value people for their interpersonal skills as much as for their technical knowledge.
3)Sensitivity to operations. HROs maintain at all times a small group of people who have deep situational awareness. This group constantly checks whether the observed behavior of the system is the result of its known inputs or whether there might be other forces at work.
4)Commitment to resilience. Teams practice managing surprise. They practice recombining existing actions and procedures in novel ways in order to attain high skill at improvisation. They practice the rapid formation of ad hoc teams to improvise solutions to novel problems.
5)Under-specification of structures. HROs empower every team member to make decisions related to his/her expertise. Any person can raise an alarm and halt operations. When anomalies or near misses arise, their descriptions are propagated throughout the organization, rather than following a fixed reporting path, in the hopes that the person with the right expertise will see them. Power is delegated to operation personal, but management is completely available at all times.
Scharre [3], in his book Army of None, reports that the US Navy adopted HRO practices on Aegis cruisers after the Vicennes incident in which the Vincennes, an Aegis cruiser, accidentally shot down an Iranian civilian airliner resulting in the death of all passengers and crew. The Aegis system is an autonomous ship defense system. Scharre suggests that the safe deployment of autonomous weapon systems requires that the organization using the systems be a high-reliability organization.
There are at least three lessons for the development and application of AI technology. First, our goal should be to create combined human-machine systems that become high reliability organizations. We should consider how AI systems can incorporate the five principles listed above. Our AI systems should continuously monitor their own behavior, the behavior of the human team, and the behavior of the environment to check for anomalies, near misses, and unanticipated side effects of actions. Our AI systems should be built of ensembles of diverse models to reduce the risk that any one model contains critical errors. They should incorporate techniques, such as minimizing down-side risk, that confer robustness to model error [4]. Our AI systems must support combined human-machine situational awareness, which will require not only excellent user interface design but the creation of AI systems whose structure can be understood and whose behavior can be predicted by the human members of the team. Our AI systems must support combined human-machine improvisational planning. Rather than executing fixed policies, methods that combine real time planning (receding horizon control or model-predictive control) are likely to be better-suited to improvisational planning. Researchers in reinforcement learning should learn from the experience of human-machine interactive planning systems [5]. Finally, our AI systems should have models of their own expertise and models of the expertise of the human operators so that the systems can route problems to the right humans when needed.
A second lesson from HRO studies is that we should not deploy AI technology in situations where it is impossible for the surrounding human organization to achieve high reliability. Consider, for example, the deployment of face recognition tools by law enforcement. The South Wales police have made public the results of 15 deployments of face recognition technology at public events. Across these 15 deployments, they caught 234 people with outstanding arrest warrants.They also experienced 2,451 false alarms—a false alarm rate of 91.3% [6]. This is typical of many applications of face recognition and fraud detection. To ensure that we achieve 100% recall of criminals, we must set the detection threshold quite low, which leads to high false alarm rates. While I do not know the details of the South Wales Police procedures, it is easy to imagine that this organization could achieve high reliability through a combination of careful procedures (e.g., human checks of all alarms, looking for patterns and anomalies in the alarms, continuous vetting of the list of outstanding arrest warrants and the provenance of the library face images). But now consider the proposal to incorporate face recognition into the “body cams” worn by police in the US. A single officer engaged in a confrontation with a person believed to be armed would not have the ability to carefully handle false face recognition alarms. It is difficult to imagine an organizational design that would enable an officer engaged in a firefight to properly handle this technology.
A third lesson is that our AI systems should be continuously monitoring the functioning of the human organization to check for threats to high reliability. As AI technology continues to improve, it should be possible to detect problems such as over-confidence, reduced attention, complacency, inertia, homogeneity, bullheadedness, hubris, headstrong acts, and self-importance in the team. When these are detected, the AI system should be empowered to halt operations.
In summary, as with previous technological advances, AI technology increases the risk that failures in human organizations and actions will be magnified by the technology with devastating consequences. To avoid such catastrophic failures, the combined human and AI organization must achieve high reliability. Work on high-reliability organizations suggests important directions for both technological development and policymaking. It is critical that we fund and pursue these research directions immediately and that we only deploy AI technology in organizations that maintain high reliability.
References
1. Perrow C. Normal Accidents: Living with High-Risk Technologies.
Princeton: Princeton University Press, 1984
2. Weick K E, Sutcliffe K M, Obstfeld D. Organizing for high reliability:
processes of collective mindfulness. Research in Organizational
Behavior, 1999, 21: 23–81
3. Scharre P. Army of None: Autonomous Weapons and The Future of
War. New York: W. W. Norton & Company, 2018
4. Chow Y, Tamar A, Mannor S, Pavone M. Risk-sensitive and robust
decision-making: a CVaR optimization approach. In: Proceedings of
the 28th International Conference on Neural Information Processing
Systems. 2015, 1522–1530
5. Bresina J L, Morris P H. Mixed-initiative planning in space mission
operations. AI Magazine, 2007, 28(2): 75–88
6. SouthWales Police.
https://www.south-wales.police.uk/en/advice/facialrecognition-
technology/Accessed November 12, 2018
请长按下方二维码识别，阅读该文。
也可点击左下方“阅读原文””，了解更多文章信息。
Frontiers of Computer Science
长按二维码关注Frontiers of Computer Science公众号
相关内容
微信扫码咨询专知VIP会员