----- Forwarded message from Anand Manikutty <> -----
From: Anand Manikutty <>
Date: Sat, 5 Feb 2011 09:45:54 -0800 (PST)
To:
Subject: [silk] Skepticism on Technological Singularity
X-Mailer: YahooMailRC/555 YahooMailWebService/0.8.108.291010
Reply-To:
There has been a lot of interest around the idea of the technological
singularity. There is even an operating system by Microsoft carrying that name.
Anyway, I have been quite skeptical about the whole concept. Anyway, I emailed
Mr. Jasen Murray of the Singularity Institute about some of the issues I had
with the concept of singularity. He suggested for me to read a chapter by one
Eliezer Yudkowsky in a book that is apparently forthcoming. And here is the
chapter : http://singinst.org/upload/artificial-intelligence-risk.pdf.
My thoughts on the chapter are below. I will add that while it may be a
reasonable hypothesis to work with, I am deeply skeptical about the idea of the
technological singularity.
As a public service, I emailed Prof. Noam Chomsky to find out his thoughts on
the concept of the singularity. I was very pleased to note (in his two sentence
reply to me yesterday) that he was similarly "skeptical". The fact that we are
both "skeptical" having arrived at our conclusions entirely independently says
something.
Anand
=+= http://groups.yahoo.com/group/indo-euro-americo-asian_list/message/223
=+= http://groups.yahoo.com/group/indo-euro-americo-asian_list/message/231
================================================================
Hi Jasen:
I have gone through the paper you sent me (I assume this book you mention is a
book of papers, and this is one of the chapter?). I am puzzled by some of the
exposition in the paper. The paper suffers from quite a few problems, in my
opinion. If I were reviewing this paper, I would give it a "Reject" simply
because the author does not seem to appreciate the organizational perspective.
What I would like to note (perhaps it is a new claim, but it is a rather obvious
one) is that businesses are not interested in developing technologies that could
spiral out of control. The potential damage to a business is too great.
Ultimately, we must view technological systems, social systems and economic
organizational systems as acting in conjunction. To be clear, the
organizational perspective is a rather intuitive perspective and one does not
need to have studied organization behavior deeply to understand it. Perhaps,
working in a business or a university for a certain period of time will provide
the same intuitions. (The response paper by John Seely Brown and Paul Duguid
(http://www.aaas.org/spp/rd/ch4.pdf) seems to have none of these problems.) This
intuitive sense is missing in this paper.
I have an extract from the paper in the section below. I recognize that the
author is trying to draw some sort of analogy between communism and technology
developers - creators (authors of???) of catastrophes need not be evil.
Technology developers may be developing something evil without being aware of
it. However, he seems to not be aware of the organizational perspective.
The reasons for the problems with communism from an organizational perspective
is that it is not a very economically efficient way of structuring society.
There were two schools of thought that argued that communism was doomed to
failure. The first was the Austrian school of whom the most famous economist was
Hayek. Hayek argued that price is unable to act as a signal in such economies
(and so you had the situation in Russia that there were huge inefficiencies due
to central planning). The second was a set of maverick economists such as
Stigler and Friedman who argued that it would be best to simply leave the market
unregulated. There were some elegant refutations of communist ideas by Paul
Samuelson which underpin the theoretical response to communism/Marxism.
This part of the paper "The folly of programming an AI to implement communism,
or any other political system, is
that you're programming means instead of ends. You're programming in a fixed
decision,
without that decision being re-evaluable after acquiring improved empirical
knowledge
about the results of communism. " seems quite wrong-headed. Communism is a form
of economic organization. Artificial intelligence is a technology. Any sort of
mix-and-match of economic organization and technology is possible. You have AI
systems in China, a communist nation. It is entirely unclear what it even means
to say that "
The folly of programming an AI to implement communism, or any other political
system, is
that you're programming means instead of ends."
I would reject this paper if it came to my desk.
Anand
==
In the late 19th century, many honest and intelligent people advocated
communism, all in
the best of good intentions. The people who first invented and spread and
swallowed the
communist meme were, in sober historical fact, idealists. The first communists
did not
have the example of Soviet Russia to warn them. At the time, without benefit of
hindsight, it must have sounded like a pretty good idea. After the revolution,
when communists came
into power and were corrupted by it, other motives may have come into play; but
this itself
was not something the first idealists predicted, however predictable it may have
been. It is important to understand that the authors of huge catastrophes need
not be evil, nor even
unusually stupid. If we attribute every tragedy to evil or unusual stupidity,
we will look at ourselves, correctly perceive that we are not evil or unusually
stupid, and say: "But that would never happen to us."
What the first communist revolutionaries thought would happen, as the empirical
consequence of their revolution, was that people's lives would improve: laborers
would no
longer work long hours at backbreaking labor and make little money from it.
This turned
out not to be the case, to put it mildly. But what the first communists thought
would
happen, was not so very different from what advocates of other political systems
thought
would be the empirical consequence of their favorite political systems. They
thought
people would be happy. They were wrong.
Now imagine that someone should attempt to program a "Friendly" AI to implement
communism, or libertarianism, or anarcho-feudalism, or favoritepoliticalsystem,
believing
that this shall bring about utopia. People's favorite political systems inspire
blazing suns of positive affect, so the proposal will sound like a really good
idea to the proposer.
We could view the programmer's failure on a moral or ethical level - say that it
is the result of someone trusting themselves too highly, failing to take into
account their own fallibility, refusing to consider the possibility that
communism might be mistaken after all. But in the language of Bayesian decision
theory, there's a complementary technical view of the problem. From the
perspective of decision theory, the choice for communism stems from combining an
empirical belief with a value judgment. The empirical belief is that communism,
when implemented, results in a specific outcome or class of outcomes: people
will be happier, work fewer hours, and possess greater material wealth. This is
ultimately
an empirical prediction; even the part about happiness is a real property of
brain states,
though hard to measure. If you implement communism, either this outcome
eventuates or it
does not. The value judgment is that this outcome satisfices or is preferable
to current
conditions. Given a different empirical belief about the actual real-world
consequences of
a communist system, the decision may undergo a corresponding change.
We would expect a true AI, an Artificial General Intelligence, to be capable of
changing its
empirical beliefs. (Or its probabilistic world-model, etc.) If somehow Charles
Babbage
had lived before Nicolaus Copernicus, and somehow computers had been invented
before
telescopes, and somehow the programmers of that day and age successfully created
an
Artificial General Intelligence, it would not follow that the AI would believe
forever after
that the Sun orbited the Earth. The AI might transcend the factual error of its
programmers, provided that the programmers understood inference rather better
than they understood astronomy. To build an AI that discovers the orbits of the
planets, the programmers need not know the math of Newtonian mechanics, only the
math of Bayesian probability theory.
The folly of programming an AI to implement communism, or any other political
system, is
that you're programming means instead of ends. You're programming in a fixed
decision,
without that decision being re-evaluable after acquiring improved empirical
knowledge
about the results of communism. You are giving the AI a fixed decision without
telling the
AI how to re-evaluate, at a higher level of intelligence, the fallible process
which produced
that decision.
================================================================
----- End forwarded message -----
--
Eugen* Leitl
leitl http://leitl.org
______________________________________________________________
___________________________________________________
Posted on the Tt mailing list. Go to http://postbiota.org/mailman/listinfo/tt to subscribe.
