Down with Upvalues

[ramsdell@... (John D. Ramsdell)]

Some of the postings on lua-user.org demonstrate that upvalues are
very confusing.  They confuse some people so much that they think the
current version of Lua is lexically scoped!  Let's get some basics
straight. 

Lua is a statically scoped language in which each variable has either
local or global scope.  Functions can be defined within functions,
however, a nested function does not have access to variables defined
in any of its enclosing functions.  Therefore, Lua is not lexically
scoped.  Thus, the following Lua code is illegal:

function addn(x)
  function sum(y)
    return x+y
  end
  return sum
end
print((addn(3))(1))

This example is illegal because variable x defined in addn is
inaccessible to the sum nested function.

Nested functions can get access to a copy of a variable defined in its
immediately enclosing function.  An upvalue reference to a variable
within a function extracts this copy when the function is evaluated to
produce a closure.  A variable reference preceded by a percent sign
signifies an upvalue reference.  Placing a percent sign in front of
the x reference in sum makes the above example legal Lua code.

This poor substitute for lexical scoping has been adopted because it
makes it easy to produce an implementation in which all non-global
variables are stack allocated, however, one need not give up lexical
scoping to have stack allocated non-globals.

The Python language previous to version 2.2 had scoping rules similar
to Lua's current rules--each variable had either local or global scope.
As of version 2.2, Python is lexically scoped, or in their terms,
non-global variables have static nested scope.

In Python's implementation, non-global variables referenced from a
nested function are immutable.  With this extra assumption, stack
allocated non-globals is easily implemented.  

Python's implementation demonstrates the validity of this approach.
They have a fast implementation as a result of using flat closures, as
described in 1984 by Luca Cardelli in a paper called "Compiling a
Functional Language".

  http://citeseer.nj.nec.com/cardelli84compiling.html

The relevant section is 4 on "Fetching variables".

For those of us who hope that future versions of Lua will be lexically
scoped, I suggest we can help by finding implementation techniques
that can be used by the core Lua implementors.  I think we should
carefully study the relevant sections of the Python 2.2 implementation
for ideas, and make them available.  Suggesting Lua is lexically
scoped when it is not, is counter productive.

I believe programs written in a good very high-level language should
be easily read by people only casually familiar with the language.
With the exception of upvalues, I think Lua excels in this respect as
it adopts Pascal-like syntax for control structures whose meaning is
just what one would expect.  Programs that employ upvalues are
unlikely to be understood by casual users.  One needs a manual to
understand when they get their value.  Most people intuitively
understand lexical scoping.  Making variables immutable when they are
referenced from within a nested function puts a burden on the authors
of a Lua program, but not on readers of the program.  I think Lua
should be designed to meet the needs of readers foremost.

John