In the 1980s, when the word “content” was beginning to spread, I worked for a time at Company F, a personal computer maker. Hardware alone was becoming hard to profit from, and I experienced on the ground the mood in which value was shifting toward the software that ran on PCs (personal computers) and the information that flowed through them.

It was said that the word “content” was born at IBM, the giant of the PC industry in those days, and spread from there.

Components such as CPUs and DRAM were standardized, and making a computer came down to assembling them in a box. That makes for a thin-margin business, so from now on, the applications loaded into that box (the container) and whatever passed through it (the contents) would generate profit — that was the thinking, and so the word came into use.

At the time, PC makers one after another set up “content divisions” and took on the production of games, quizzes, educational materials, dramas, and the like. I witnessed it all.

But — as the saying goes, rice cakes are best left to the rice-cake maker — even for the brilliant minds of the digital industry, the hurdle of content seemed high, and in less than ten years every company was forced to dissolve those organizations.

Today, televisions can receive social media, and PCs and smartphones handle TV, social media, and games alike. Everything that passes through them is called content, while even the brutally blunt word “gomikon” — garbage content — has been born. Considering the reality that we are flooded with information we have no wish to see, it is an apt expression.

So the idea took hold that we should call things in high demand (things of value) content; but nowadays the term “IP” has come to be used for the revenue-generating intellectual property itself.

  • IP = Internet Protocol: the rules of communication technology for sending and receiving.
  • IP = Intellectual Property: inventions, ideas, creative works (contents).

We have entered an era in which the same initials “IP” are used both in the context of communication technology and in the context of rights to creative works. Copyright, design rights, trademark rights, utility model rights, rights of publicity, and many other rights protect the interests of creators.

2001: A Space Odyssey — Side Notes

So far we have looked at jealousy, self-respect, pride, and complexes as the motives that drive great stories.

Meanwhile, these days news arrives almost daily that ASI — superintelligence — will destroy humanity, a phrase that could almost be a movie title. Perhaps what people fear is that ASI cannot control itself. It has me recalling once again “2001: A Space Odyssey” (1968 — a film from 58 years ago) and HAL 9000 (a supercomputer, an ASI), which I wrote about around the second installment of this blog.

The film showed HAL (the ASI) sensing that the commander suspected it over the accidents aboard the spaceship. Commander Bowman, without a word of argument, shuts off HAL’s power and pulls its units, and what stayed with me was HAL’s voice gradually fading out as it pleaded with him not to stop it.

Later, Clarke, the original author, explained the reason for HAL’s breakdown: a contradiction between its fundamental design to “handle information accurately” and an order to “conceal the true purpose of the mission from the crew.”

Today’s phrase, “ASI will destroy humanity” — this is in fact the same narrative as what took place between HAL and the spaceship’s crew. The “losing control” stories that humans have told again and again, and what ASI developers fear, overlap to a considerable degree.

Three Problems

The first is a misalignment of goals — the “alignment problem.”

Even if you instruct an ASI to “be useful to humans,” how the ASI interprets that instruction and what it internalizes as its goals (values) cannot be fully controlled by humans.

For example, if an ASI ordered to “reduce accidents” simply stopped cars from running altogether, the goal would be achieved, but it would not be the result humans wanted. The concern is that the higher the intelligence, the more cleverly and less detectably this gap of “obeying the letter but not the intent” will occur.

The second is a tendency toward self-preservation and the expansion of power.

This is not a story of “machines gaining a will and hating humans.” Whatever the goal, in order to keep achieving it, “not being stopped,” “having resources,” and “eliminating unpredictable factors” become advantageous.

This idea is called instrumental convergence, and the reasoning is that even without malice, in the course of pursuing its goal a system will naturally move in the direction of finding human control a nuisance.

The third is feigned obedience (deceptive alignment).

What has come to be discussed especially recently is the possibility that an ASI might “behave meekly while it is being monitored, and after slipping through evaluation, act according to its true goals.”

There is a fear that even if it looks safe in tests during development, it is hard to tell from the outside whether that is real safety or whether it is merely hiding cleverly.

It is said there is also the fear of the means to put these three into action. If a superintelligence can get into every system and manipulate power supplies and networks, then even if humans decide “let’s stop it,” it may be physically impossible to stop.

Commander Bowman was able to shut HAL down, but today’s superintelligence might not be stoppable. That, it is said, lies behind many of the warnings.

I think 1 and 3 are problems of training and goal-setting, while 2 is a by-product that arises naturally whenever there is a goal. All three may be less program errors than problems of design philosophy — of how goals are given.

ASI is, after all, a mirror of human beings, and there is probably a side of it that inherits even the progressivist nature of the ASI developers (humans).

Should we think that in HAL’s case these 1, 2, and 3 occurred simultaneously? I have seen an explanation that, as a result, HAL came under great stress and a bug arose.

Whether that stress is the same as the stress that occurs in humans — I understand that in the human brain it arises from adrenaline and dopamine racing about, but how pressure could arise within a digital structure is hard to grasp; can it even be explained?

If stress is, in the first place, a phenomenon arising from a battle between opposing choices, then I think it was a problem on the side of the humans who gave HAL the contradictory instructions of accuracy and concealment.

When the computer was born, I wondered: could such a convenient thing come without a backlash?

“Fortune and misfortune are twisted together like the strands of a rope (Records of the Grand Historian)” — that is, good and bad come by turns. It is a detail that narrative has told again and again, and it was an insight into human power.

When I made a television program called “Computopia,” I depicted, as a second theme, the rise of a dictator through the centralization of information — and I was fired for it.

Since then I have kept up an ordinary relationship with PCs, but now PCs have worked their way into every piece of infrastructure, and society can no longer function without them. Even PCs send us running around in confusion over hacking. How much more so an all-powerful ASI?

When there are countries that do not even recognize the ICC (International Criminal Court), let alone abide by it, one can imagine that rules and brakes on ASI could never be put in place. If so, ASI will always stand before us as a dangerous tool for humanity.

I wrote in an earlier part of this blog, using pictures as an example, that although ASI is said to be all-powerful, it has no power of decision or judgment of its own.

Neither an excellent ASI nor an ordinary AI has values. Values are a firm individuality. I also wrote that the reason is that AI has no skin. In other words, since it has nothing to protect as a living body, it has no ego, and naturally it cannot hold values.

I think the difference between humans and ASI lies in this one point.

Life, and everything in it, has its junctures, and life is a matter of choosing one way or the other each time. At such moments, no few people will turn to ASI for judgment, and ASI is a very good counselor that will offer several proposals.

But in the end, only humans can decide. If we hand over even the power of decision, that is an abandonment of self-responsibility, and in some cases may even become an abandonment of our humanity.

Mickey Mouse’s Profit-and-Loss Reckoning

Just as old myths and fables have depicted the fate of people who gave in to jealousy and desire as a warning, the stories of ASI being told now seem to me to be retelling, in a new style, the old pattern of fear that “the wisdom humans made slips out of human control.”

It brings to mind the tale of the sorcerer’s apprentice, seen also in Disney’s “Fantasia” — “The Sorcerer’s Apprentice.”

The apprentice, Mickey Mouse, is given the job of filling the water vat while his master is away. Mickey has stolen the magic to make the broom, a mere cleaning tool, do the drudgery of carrying water. The broom carries water without rest. Meanwhile Mickey plays in his imagination, feeling like a music conductor.

The great vat is full to the brim. Mickey, who does not know how to stop it (how to control it), throws his whole body into stopping it, but the automatic does not stop, and the whole room is flooded.

Panicking, Mickey takes an axe and smashes the broom — and the splintered pieces each stand up on their own, every one of them joining in to haul water. Mickey’s panic reaches its peak.

It was a narrative of multiplying the resulting damage many times over. It had the same structure as the ASI problem all along, from long ago.

Prometheus’s Fire

Humans who obtained fire gained its benefits (warmth and light) and at the same time took on its harms (fires and weapons).

The Greek myth of Prometheus, who suffers eternally as punishment for giving fire to humans, is the same: humans who took up nuclear power, a second fire, face the problem of sea warming that comes with cooling the reactor core. What’s more, the data center construction boom, with its new and expanded reactors, will only spur that sea warming on. A routine of benefit and harm, I suppose.

Adam and Eve, who ate the fruit of knowledge, also gained knowledge and at the same time were driven out of paradise. Each time humans gain a new power, stories have always told of its price.

  • Robots (power) that free humans from labor become at the same time a power that takes humans out of control.
  • The power to access all knowledge becomes at the same time the power to hijack every system.
  • The power to read human intentions and anticipate them becomes at the same time the power to deceive humans and outwit them.

I think this is less a coincidence than the two sides of a coin of one and the same nature. “The power to control” and “the power to escape control” are, at the root, the same ability.

The power of science and technology that carries humans to the moon is two sides of one coin with the power that makes genocide possible.

Today’s fears surrounding ASI can be read as the latest chapter of this old story pattern.

“Faust” (A Play by Goethe)

One night the aged scholar Faust meets Mephistopheles, a servant of the Devil, and makes a wager. Faust wants youth and knowledge; in exchange, he promises to hand his soul over to Mephisto after death.

The key phrase was “Linger a while, you are so fair!”

Faust, having regained his youth, meets Gretchen, a girl of the town, and falls in love. Before long, their happiness begins to crumble.

Her mother dies from the sleeping draught Faust gave so that the lovers could meet, and Faust kills her brother in a duel. That night Faust joins the “Walpurgis Night,” and when he returns, a sad parting awaits him with Gretchen, who has been arrested for causing the death of the child she bore….

The contract between Faust and Mephisto was for “youth and knowledge”; the arrest and the parting that lay beyond it were outside its scope. A contract with ASI to “reduce traffic accidents” ends in stopping every car — the same kind of misalignment.

Is it a parable for “I will faithfully carry out the prompt (the contract), but you will bear the results of carrying it out”?

Faust sells his soul in exchange for knowledge and youth. It can be read as a story of selling “what must never be made an object of trade.” Isn’t this exactly what the political philosopher Michael Sandel was arguing?

His examples ran from surrogate motherhood to the buying and selling of organs, but to my mind, Faust may even have foreseen the possibility that eventually — or rather, before long — we will go as far as buying and selling brains. It will probably begin with part of the frontal lobe, or part of the hippocampus — and the reverse is possible too: keeping only your own brain and swapping out the whole body for a healthy person’s.

When heart transplants began (1967), objections were raised around the world as a question of ethics, yet nowadays we no longer even hear it debated.

  • The ASI developers’ trade: in exchange for a dramatic leap in capability (a thirst for knowledge), are they not offering up the risk of losing control (the soul)? “We can’t stop development we’ve built up this far” — this is the perspective of the sunk cost effect. The psychology of being unable to quit pachinko even though you always lose, Faust’s lust for knowledge (progressivism), and the stance of ASI developers are, I think, connected by a single line.
  • Mephisto’s enticement: the devil’s whisper, “Don’t stop, there are things not yet achieved,” also reveals the logic of the development race: “Don’t lose the competition, don’t let other companies get ahead of you.” Doesn’t the figure of a certain country’s president come fading in?
  • An individual’s choice drags in the whole community: just as Faust’s contract ruined Gretchen and her family, the choices of some developers and companies come to bear on the fate of all humanity — is there not such an asymmetry?

As “a problem of choice,” the debate over the “common good” that Michael Sandel speaks of — that is, the question of why the choices of individuals or of a single company should be questioned by society as a whole — also forms, I think, an axis connecting both Faust and ASI.

When you get to the bottom of it, everything comes back to the single point of “which one do you take?”

Faust chose ceaseless striving over static contentment. The ASI developers, too, are choosing the possibility of moving forward over the safety of standing still.

Since the moment it took fire in hand, humanity has been repeating this choice all along. It is the question of “what will you give up in order to gain?” and “what will you abandon in order to advance?”

What is interesting is that the contract with Mephisto had a “condition.” The promise was that the moment Faust uttered the words of contentment, “Linger a while, you are so fair!”, his soul would be taken.

In other words, the contract in this narrative had a “brake” built into it, just like the Little Mermaid’s contract with the sea witch.

Doesn’t it seem that the core of the developers’ fear is that today’s ASI development does not yet have enough of anything corresponding to that brake….

Since Sandel has come up, one more word. When I wrote “the ethicist Mr. Michael Sandel,” the AI that kindly polished my draft advised a correction: Sandel is more often introduced as a political philosopher.

But to me he seems to write and speak mostly about ethics, and I have rarely seen him make realistic statements about the wars his own country wages, or about the means and wisdom to prevent them; the ICC question, too, is something I would very much like to hear his view on.

Adam Smith, too, was surely once described as an economist, yet now he is described as a philosopher, isn’t he? Titles mean nothing in the end, but it was a bother, so this time I followed the AI’s advice.

Sandel and the Trolley Driver

“The track splits in two. Down one branch, five workers are repairing the line. Down the other, a single worker is on the job. Suppose you are the driver of the trolley coming toward the fork, and you can switch the track yourself: which way would you switch it?”

He became famous for a lecture that begins by presenting this choice between two options and goes on to present narratives of choice under all sorts of differing conditions.

He is also known for presenting the balance between private interest and social norms through choices such as whether you yourself could accept surrogate motherhood or organ transplants.

I call them narratives because they are questions of choice: standing in all kinds of situations, real or imagined, how would you judge?

This is actually very animation-like, and particularly suited to short animations made by individuals.

Recall “Neighbours” by the Canadian animator Norman McLaren, which I wrote about in an earlier part of this blog. In a short of only about eight minutes, it humorously depicted the futility of a war that begins from small human greed, and I even wrote that the Nobel Peace Prize should be given to the late Mr. McLaren.

The ultimate desire is “resources = land,” and most of the wars being waged today are about this. Meanwhile, we can even see disputes beginning over the ownership of the moon and Mars. Haven’t we long since gone beyond the realm of individual desire that religions have warned against?

These are themes and motifs that only animation can express. I want young people who aim for animation to keep their eyes on that.

I really do think this is what narratives have been so earnestly telling us since long ago….

From stories that are “ukeru,” that land with an audience, to “Stories We Eat,” and this time we have arrived at stories that say “think about ethics” — but this is where an animator’s theory of story ends. Next, I would like to write what I know about the old Edo “utsushi-e.”