The AI race in the security and defence sector is on. Amid mounting geopolitical tensions, research and development in this area will only increase as states seek to gain advantage over adversaries through cutting-edge technology. At the same time, there is growing concern at the influence and power the private sector wields over military AI.
A striking example of this close relationship came when Alex Karp, the chief executive of Palantir, became the first western executive to visit President Volodymyr Zelenskyy after war broke out in Ukraine. Palantir, which describes itself as building ‘world-class software for data analytics and data-driven decision making’, has recently signed a three-year deal with the UK’s Ministry of Defence to aid military operations and intelligence worth £75 million.
Its reputation is less than sparkling. Following the outrage over the US Immigration and Customs Enforcement’s policy of separating immigrant families under the Trump administration, Palantir was reported to have enabled such practices by gathering information on undocumented immigrants, mapping out family relations and selecting targets.
Profoundly disruptive effects
Palantir also holds multi-million pound contracts with NHS England, partnership agreements with Scuderia Ferrari and was implicated in the Cambridge Analytica scandal in which data was harvested from Facebook users.
Palantir attests to two worrying realities. First, while the private sector has always played a key role in international security and defence, the lines between the military and the civilian realms are being blurred more than ever.
This is exacerbated by the rise in dual-use technologies, which benefit both civilian and military communities, and which many states including NATO members and China are pursuing. Such developments have both operational and legal implications, which require clarification to ensure they comply with applicable laws, ethical standards and policies.
Second is the amount of data the company holds. Such a concentration of data, and ultimately power, in a handful of private companies practically puts them on par with states.
Concerns arise with regards to the accountability and lack of democratic oversight of these companies. Opacity in the defence sector is understandable. The private sector is agile and can act more quickly, while government agencies tend to be hamstrung by lengthy deliberative processes surrounding innovation and procurement.
Yet these processes are necessary to ensure democratic control over defence matters is maintained. Efforts to increase transparency over military spending foster democratic accountability over decision-making, while corporate accountability has to be upheld against regulatory requirements and democratic oversight.
These processes are necessary to ensure corporations do not prioritize profit at the expense of compliance with laws, policies and ethical standards. Given the high stakes involved in the deployment of military AI, which can sometimes mean life or death to civilians, corporations practically lose the right to say ‘Oops’.
While the perfect blueprint does not exist, there are certain considerations when seeking corporate accountability in the context of military AI. First, there are a number of exemplary mechanisms in other sectors, including the UN Guiding Principles on Business and Human Rights and environmental, societal and governance frameworks. Incentivisation is key, and security and defence industries would do well in drawing inspiration and the lessons learnt from these existing frameworks.
Second, states, remaining the main clients for military AI solutions, are bound by the social contract – at least, in democracies – to uphold the people’s fundamental rights. As such, not only do industries have a duty to act in the best interest of their shareholders and clients, they are bound by applicable laws, policies and reputational considerations.
Strengthening existing processes to enable democratic oversight on security expenditures is necessary, with the eventual adoption of measures to clarify what this would look like in the context of military AI. Beyond parliamentary processes, there are calls to reform procurement and acquisitions as key enablers for accountability.
Third, risks surrounding profit-driven innovation must not only be identified but also addressed. While, for example, Palantir claims never to have had dealings in Russia, nothing prevents it from doing so in future, unless such matters are expressly stipulated in contractual agreements.
Yet, this one-off, case-by-case solution is neither sustainable nor reliable. Not only are there risks of sales to adversary states – the field of arms sales including the British manufacturers is no stranger to human rights violators, including Libya and Saudi Arabia – but civil society organizations are already raising the alarm on AI use for mass surveillance and control. One example is the Israeli government’s use of facial recognition technology in the occupied West Bank and East Jerusalem, reportedly relying on CCTV cameras provided by Hangzhou Hikvision Digital Technology and TKH Security.
Beyond state clients, nothing prevents companies from supplying military-grade AI technologies to non-state actors such as paramilitary forces and armed groups. This, in turn, can be destabilizing for states, the effects of which can be profound, long-lasting and spill over borders. In the interest of international security, AI capitalism must be kept in check. Technological deregulation carries too high a risk.
AI’s Oppenheimer moment
In a recent article, Karp, Palantir’s chief executive, argued that with the development of military AI, we are living through an ‘Oppenheimer moment’. He believes that in the case of AI, while we must choose ‘whether to proceed with the development of a technology whose power and potential we do not yet fully apprehend’, other countries will not stop pressing forward in developing such technologies.Thus, if free and democratic societies are to prevail, they will have an obligation to build hard power through software.
Yet, Karp fails to mention the deep concerns Robert Oppenheimer, considered the father of the atomic bomb, had about starting a nuclear arms race with the Soviet Union. Not only did the world witness the devastating humanitarian impact of the nuclear weapons dropped on Hiroshima and Nagasaki, but we are left with the long-lasting problem of nuclear arms control and disarmament.