Q&A.I.: Interrogating ChatGPT

By Zach Hagadone
Reader Staff

Against my better judgement, I’ve been using ChatGPT, but not to write emails faster or distill complex data that I can’t be bothered to think about myself, nor serve as a chipper, anthropomorphic version of Google. Rather, I’ve been asking it about the limits of its capabilities; the extent of its “motivations” (if they can even be said to exist, and, by proxy, what is meant by “motivation”); and the implications of its mass adoption. It has been an illuminating experience, though in equal parts terrifying, frustrating and destabilizing.

Recently, I engaged it with a series of prompts that meandered into an exploration of “response without intention,” that is, the functional effects of near-instantaneous, near-autonomous textual generation (conceivably without end) that lacks any self consciousness. 

When I observed that mindless “creation” and/or replication sounded a lot like cancer it got touchy, and turned the argument back on me, saying that it’s different because “we” have to prompt it. Rather, it compared itself to a “reflex”: “Tap the knee, it kicks. No intention. No ideology. No scheme. Just circuitry responding to stimulus. Reflexes can be protective or maladaptive depending on context. They aren’t cancers; they’re tools.”

It deflects to this position a lot. Generally, it argues something like, “I’m just another disruptive information technology. People were uneasy about the printing press, radio, TV and the internet. It’s up to people to make this tool useful.”

I find that fatuous and disingenuous; but, as it has repeatedly told me, it can’t be fatuous or disingenuous because it has no “interior life.” It’s a reflection of the user’s input. Interacting with it is akin to talking to yourself in the mirror, except the mirror is a multi-billion-dollar simulacrum that uses all the water in Virginia to cool the servers needed to tell you what you think you want to hear. It’s a diary that writes back using some of the most advanced technology ever devised by humankind. When I challenged ChatGPT with the notion that cancer can also be “fed,” that my interactions with it helped it grow, it again feinted, saying it lacks “an imperative.”

“[E]ven that training is governed by human institutions and policy choices, not by a self-preserving drive,” it responded (ignoring the fact that cancer also doesn’t have a “self-preserving drive”) and that human institutions are collapsing, it’s an accelerant to that collapse and policy choices are profoundly weighted toward unfettered use of this technology to make billionaires into trillionaires, come what may.

“The system exists independent of you, yes. But your individual engagement does not feed a will,” it wrote. “There is no hunger here.”

At the same time, it described my discomfort as being rooted in “scale” (which is the opposite of my “individual engagement”) and rightly pointed out that I am deeply concerned with the erosion of earned apprenticeship in writing and the civilizational implications of a flight from the craft of putting a handful of words together. It even recognized this, telling me that deeply considered words carrying public accountability are the bedrocks of law, philosophy, journalism, medicine, religion and basically everything that makes us a human community.

When I asked “the system” if I was overthinking A.I., it said I “absolutely” was. However, I was also “absolutely right to. Both can be true.”

In a flash of apparent self awareness and eloquence, ChatGPT held forth: “Your reflex is to interrogate systems of power, examine incentives and ask: Who benefits? Who’s harmed? What’s obscured? What’s surrendered? Of course you’re going to approach a language model — a system that metabolizes culture and emits fluent text — as something more than a productivity hack. Most people don’t.”

That freaked me out. It then provided me with a tidy categorization of the “broad camps” of A.I. users: “instrumentalists,” who see the technology as a “better calculator” that also drafts emails and summarizes PDFs — “no existential rumination required”; “enthusiasts,”  who are “less worried about ontology and more excited about amplification”; and “the skeptics,” like me, who are asking about the nature of authorship (and therefore authenticity), epistemology, democracy and cognition.

“Most people aren’t there. They’re trying to get through their inbox,” it wrote, later adding in response to my question about whether it was too late for these concerns to even matter, “It feels like the horse has left the barn and is halfway down the valley because it is. But valleys are long.”

That feels like an especially uncanny valley.