38 - Surveillance in Ancillary Sword and elsewhere
This episode returns to a comment that came up when Ethan and I discussed Ancillary Sword, addressing whether the paternalistic surveillance of the ships and Station in the novel is a thing to be feared, and why that idea is so threatening in much Science Fiction in Fantasy.
In addition to clips from The Terminator and Star Trek, the show has passages from Naomi Kritzer's Hugo Award Winning "Cat Pictures, Please" (published in Clarkesworld and narrated by Kate Baker), and Ken Liu's "The Perfect Match" (published in Lightspeed Magazine and narrated by Paul Boehmer)
The amazing art which inspired me to actually get this project off the ground was created by @etrandem
Send feedback! Tweet me! Tweet the show! Be a guest on the show!
Music - Jazzy Ashes by The Underscore Orkestra
If you want to subscribe to the show, the RSS feed is: http://www.cabbagesandkings.audio/?format=rss
Transcript -
JSM - There’s something about the paternalistic lack of privacy
EJ - That bothers you, does it?
TERMINATOR - Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug.
Sarah Connor: Skynet fights back.
*Intro music*
JSM - Welcome to Cabbages & Kings, a podcast for readers of science fiction & fantasy, I'm
Your host, Jonah Sutton-Morse. Here’s more of my discussion of Ancillary Sword, the second book of the Radch Trilogy with Ethan. As you may have noticed, we’re going to discuss portrayals of the Artificial Intelligences and their surveillance in Ancillary Sword, contrasted with AI in Science Fiction more broadly
*Interstitial Music*
JSM - it almost feels like another thumb in the eye to SF, because I feel like any SF story that look at omniscient surveillance assumes that it is destructively oppressive
EJ - Yeah
JSM - Like the Panopticon is terrible
EJ - Yeah, I think a lot of SF just doesn't deal with it, I mean it's there, and it's just never mentioned which is, come to think of it, kind of wierd.
JSM - Right
I mean there's no way you could have a super advanced technological civilization that has
computers that run everything basically whether or not they're actually intelligent, where you don't have, um y'know always-on surveillance more or less.
JSM - Right
EJ - So, yeah, so you gotta deal with it. And Leckie does, and that's kind of interesting.
*Interstitial Music*
JSM - I tend to learn about things by talking through them, and as I give you the conversation Ethan and I had about AI and surveillance, I also want to argue a bit with something I said there
JSM - Like the Panopticon is terrible (from interview)
JSM - Note that the Panopticon was originally seen as a tool for imperfect surveillance. A prison where the guards could theoretically watch any prisoner, and couldn’t be observed, so the prisoners would assume they were always being watched. Not the ubiquitous surveillance of the Radch ships and stations, but a tool of social control. As always, Star Trek comes in handy:
Wesley - It’s OK, I’m fine
Woman: Oh no, oh please no
Mediator: Speak the truth, we are mediators
…
Mediator: We have a visible transgression, ample witnesses, and and admission of guilt. And though it pains us deeply to do it, we must
*Ominous swish*
Mediator: Are you prepared for punishment?
JSM - So what’s going on here is that the enforcement zone and dire punishment are intended to make people follow the rules because the consequences if you get caught are horrific. Simulated universal surveillance as social control. But of course that’s not what Leckie’s showing us in the Radch trilogy, and to the extent that Science Fiction comments on the present, I’m not sure it’s the surveillance we’re walking into now.
EJ - I mean there's no way you could have a super advanced technological civilization that has
computers that run everything basically whether or not they're actually intelligent, where you don't have, um y'know always-on surveillance more or less.
JSM - And the follow-up question - where is this surveillance coming from
(From Cat Pictures Please)
KB - I know I wasn’t created by a God, or by evolution, but by a team of computer programmers in the lab of a large corporation
JSM - AND THEN WHAT ARE WE DOING WITH IT
(from Cat Pictures Please)
KB - When I first woke up, I knew right away what I wanted. (I want cat pictures. Please keep taking them.)
(from “The Perfect Match”)
PB - You mean some advertiser paid Centillion to pitch it at you.
That’s the point of advertising, isn’t it? To match desire with satisfaction.
JSM - Those quotes are from Naomi Kritzer’s “Cat Pictures, Please”, (Hugo-Award winning, I should note) which imagines an Artificial Intelligence that develops spontaneously, and Ken Liu’s “Perfect Match”, where the Artificial Intelligence is still very much guided by human motivations.
(from “The Perfect Match”)
PB - “So what do you want with us?” Jenny asked. “We won’t stop fighting you.”
“I want you to come and work for Centillion.”
Sai and Jenny looked at each other. “What?”
“We want people who can see through Tilly’s suggestions, detect her imperfections. For all that we’ve been able to do with AI and data mining, the Perfect Algorithm remains elusive.
JSM - Another point that Ethan raised, and that’s somewhat addressed elsewhere, is AI competence
EJ - I mean, in other SF works, AIs are either actually intelligent and sentient, in which case they're basically all-powerful, or they are souped up computers, in which case they are pretty dumb. I’m thinking of Star Trek for example, pretty dumb. Whereas in Ender’s game when an Artificial Intelligence gains sentience, it’s all-powerful almost immediately.
JSM - And I think what happens a lot in Science Fiction is that there’s very little interrogation of either AI competence or it’s origin & motivation, which leads to the common trope of Science Fiction
JSM - Whereas I feel like in most stories where you say there is an AI that is surveilling everything, the problem becomes that humans are in conflict, right like the protagonist is in conflict with the AI and the goal is to bring down the AI,
JSM - But Google and Facebook don’t want us to be conscious of the reality that we’re the product. The Panopticon isn’t really a good tool for talking about the ways that computers with massive amounts of data about us will interact with people. I wish I’d said that back when we were first having this discussion and we’d had a chance to follow up, which maybe we’ll do during book 3 since this comes up again. And so we can return to the Radch Trilogy. What is it about the origin and purpose of these Artificial Intelligences, their origin, and their purpose that means they’re not inherently in conflict with humanity?
EJ - it's interesting how what drives that, or kind of makes that possible is this kind of conception that people have of AIs in these books that they're basically a known quantity, they're intelligent, but pretty much trustworthy, and yet basically people see them as lesser,
JSM - Right
EJ - Even normal citizens see them as lesser. Really, they’re subservient to Miannaai, normal citizens see AIs as kind of, we know what they do, they work for us basically so it’s fine if they’re listening in
JSM - So I guess I’ve talked around for a while two ideas: first, Science Fiction is partly about giving us tools to think about our world, and in that sense I think the Panopticon, or any metaphor in which imperfect surveillance is used intrusively to give the sense of perfect surveillance, is missing the reality of surveillance that’s developing all around us.
But second, going back to Ethan’s question while we were talking about Ancillary Sword:
JSM - The paternalistic lack of privacy
EJ - That bothers you, does it?
JSM - Yeah, I’d say it does, but I’m also going to keep googling things and tweeting plenty of thoughts and pictures, and wishing for Siri, or Alexa, to be just a bit better at figuring out what I want them to do.
(Conclusion)