Extension Proposal

wiki

Description

A recent (Jan 2006) discussion on the mailing list has prompted me to attempt to design an extended API which extends the Lua API by adding functions to the os and io namespaces.

This is not a proposal to modify the Lua core, but a design proposal for an API which extends the Lua core. This API is meant to provide a more complete programming environment for stand-alone Lua programs on today's popular operating systems (Windows, MacOSX and POSIX platforms).

Source

There are [implementations for POSIX and Windows] hosted on LuaForge. These are highly usable implementations, but should be considered only for testing purposes while the API is still being standardized.

ex API

Note that all these functions return the standard (nil,"error message") on failure and that, unless otherwise specified, they return (true) on success.

Initialization

marks a Lua program which uses this API

Environment

os.getenv(name)

get an environment value

os.setenv(name, value)

set/modify an environment value

os.setenv(name, nil)

remove an environment value

os.environ()

returns a copy of the environment (a simple Lua table)

File system (mostly borrowing from LuaFilesystem?)

os.chdir(pathname)

change working directory

os.mkdir(pathname)

create a directory

os.remove(pathname)

remove a file or directory

pathname = os.currentdir()

get working directory path

for entry in os.dir(pathname) do ; end

iterates over the entries in a directory. The pathname argument is optional; if missing the current directory is used.

Special directory entries such as "." and ".." are not returned.

entry = os.dirent(pathname)
entry = os.dirent(file)

gets information about a directory entry via pathname or open file

Both the iterator function returned by os.dir() and the os.dirent() function return an 'entry' table. This table contains at least the following fields:

entry.name= the filename (Note that os.dirent() does need to set this field)

entry.type= "file" or "directory" (or an implementation-defined string)

entry.size= the file size in bytes

Implementations may add other fields or even methods.

I/O (locking and pipes)

file:lock(mode, offset, length)
io.lock(file, mode, offset, length)

lock or unlock a file or a portion of a file; 'mode' is either "r" or "w" or "u"; 'offset' and 'length' are optional

A mode of "r" requests a read lock, "w" requests a write lock, and "u" releases the lock. Note that the locks may be either advisory or mandatory.

file:unlock(offset, length)
io.unlock(file, offset, length)

equivalent to file:lock("u", offset, length) or io.lock(file, "u", offset, length)

Note that both file:lock() and file:unlock() extend the metatable for Lua file objects.

rd, wr = io.pipe()

create a pipe; 'rd' and 'wr' are Lua file objects

Process control

os.sleep(seconds)
os.sleep(interval, unit)

suspends program execution for interval/unit seconds; 'unit' defaults to 1 and either argument can be floating point. The particular sub-second precision is implementation-defined.

os.sleep(3.8) -- sleep for 3.8 seconds
local microseconds = 1e6
os.sleep(3800000, microseconds) -- sleep for 3800000 µs
local ticks = 100
os.sleep(380, ticks) -- sleep for 380 ticks

proc = os.spawn(filename)
proc = os.spawn{filename, [args-opts]}
proc = os.spawn{command=filename, [args-opts]} 

create a child process

'filename' names a program. If the (implementation-defined) pathname is not absolute, the program is found through an implementation-defined search method (the PATH environment variable on most systems).

If specified, [args-opts] is one or more of the following keys:

[1]..[n]= the command line arguments

args= an array of command line arguments

env= a table of environment variables

stdin= stdout= stderr= io.file objects for standard input, output, and error respectively

It is an error if both integer keys and an 'args' key are specified.

An implementation may provide special behavior if a zeroth argument (options.args[0] or options[0]) is provided.

The returned 'proc' userdatum has the following method:

exitcode = proc:wait()

wait for child process termination; 'exitcode' is the code returned by the child process

Summary

All functions are also available under the ex namespace:

ex.getenv(name)
ex.setenv(name, value)
ex.environ()
ex.chdir(pathname)
ex.mkdir(pathname)
ex.currentdir()
ex.dir(pathname)
ex.dirent(pathname)
ex.lock(file, mode, offset, length)
ex.unlock(file, offset, length)
ex.pipe()
ex.sleep(interval, [unit])
ex.spawn(...)
ex.wait(proc)

Note that ex.getenv is here mostly for parallelism, but also because under Windows, using the SetEnvironmentVariable?() API requires overriding the standard os.getenv implementation which uses getenv() to use GetEnvironmentVariable?() instead.

Examples

require "ex"

-- run the echo command
proc = os.spawn"/bin/echo"
proc = os.spawn{"/bin/echo", "hello", "world"}
proc = os.spawn{command="/bin/echo", "hello", "world"}

-- run the id command
vars = { LANG="fr_FR" }
proc = os.spawn{"/bin/id", "-un", env=vars}
proc = os.spawn{command="/bin/id", "-un", env=vars)

-- Useless use of cat
local rd, wr = assert(io.pipe())
local proc = assert(os.spawn("/bin/cat", {stdin=rd}))
rd:close()
wr:write("Hello world\n")
wr:close()
proc:wait()

-- Run a program with a modified environment
local env = os.environ()
env.LUA_PATH = "/usr/share/lib/lua/?.lua"
env.LUA_CPATH = "/usr/share/lib/lua/?.so"
local proc = assert(os.spawn("lua", {args = {"-e", 'print"Hello world\n"'}, env=env }))
proc:wait()

-- popen2()
function popen2(...)
  local in_rd, in_wr = io.pipe()
  local out_rd, out_wr = io.pipe()
  local proc, err = os.spawn{stdin = in_rd, stdout = out_wr, ...}
  in_rd:close(); out_wr:close()
  if not proc then
    in_wr:close(); out_rd:close()
    return proc, err
  end
  return proc, out_rd, in_wr
end
-- usage:
local p, i, o = assert(popen2("wc", "-w"))
o:write("Hello world"); o:close()
print(i:read"*l"); i:close()
p:wait()

Comments

Mark, rather than insert functions into the standard library namespaces, I suggest that you consider your work to be a utility module, pick a name for it, and place the functions under the new module name. If your functions catch on as they are now, it's going to cause confusion, especially to Lua beginners who might read some example code and not understand why certain functions in the standard modules are not in the upstream docs. --JohnBelmonte

I agree with John. Perhaps an ex.install() or similar could be used to install it into the os library. But I'd rather have it be a separate library. --Doug Rogers

I agree with the previous comments. I'd would prefer not inject the funcs to "os", but rather have separate libs.'

env.getenv(name)
env.setenv(name, value)
env.environ()
--no unsetenv. It's meaningless.
--function env.unsetenv(name)
--    env.setenv(name, nil)
--end

fs.chdir(pathname)
fs.mkdir(pathname)
fs.currentdir()
fs.dir(pathname)
fs.dirent(pathname)
fs.lock(file, mode, offset, length)
fs.unlock(file, offset, length)

proc.pipe()
proc.sleep(seconds)
proc.spawn(...)
proc.wait(proc)

Mark, thanks for this implementation. I'll add to the chorus here of avoiding "polluting" the os and io modules, at least in the current implementation, so that people can use the modules now in production code without danger of future conflict if and when the interface extending os and io namespaces are standardized (why "considered only for testing purposes"?). As mentioned, your implementation (not your proposed interface) might have an ex.install() or similar function that will update os and io according to the interface proposal. That gives users a choice for now.--DavidManura

Taking this a step further, if you were going to break this out into its own table (such as filesys) copying the file related entries from os might be useful. This keeps new code cleaner and sort of 'obsoletes' the os table when this extension is called.

(1) In the default conf.in, change "-llua51" to "-llua5.1" to be consistent with LuaBinaries?

This is consistent with the Lua distribution, but I can add a note to conf.in. -Mark

I've wondered why there is this inconsistency between Lua and LuaBinaries.

(2) os.sleep(math.huge) returns immediately. should it never return?

I'll need to investigate why this happens ... -Mark

os.sleep(1e100) also returns immediately. Win32 Sleep(INFINITE) or any huge value doesn't seem that useful though, but who knows (INFINTE seems useful in SleepEx?). [1] Sleep(0) means "relinquish the timeslice".

(3) os.setenv and os.mkdir(1) - implementation automatically converts number parameter to a string. intended?

Probably -- they expect a string. -Mark

I see this behavior also in string.lower(123) == "123" (is true).

(4) Intended?

> =os.setenv("a", {})
nil     203 (0xCB): The system could not find the environment option that was entered.
> =os.setenv("a", nil)
nil     203 (0xCB): The system could not find the environment option that was entered.

The API doesn't say what happens if a non-string is passed as the second argument, or if an attempt is made to remove a non-existent variable. Should it?

Don't know, but the error message was odd.

(5) Change os.dir(), with no parameter, to use the current working directory?

This is a good idea; it will need to be part of the API then. -Mark

(6) entry.name was missing when I called os.dirent.

Yes, this is intended and the API is incorrect as specified. Unless there's a reason it should be different? -Mark

(7) Intended?

> f = io.open("123")
> = io.lock(f, "w")
nil     6 (0x6): The handle is invalid.
> =f
file (0080F060)

Yes, ex doesn't say what happens if you try to write-lock a read-only file. Should it? -Mark

Maybe not. What about this under Win32?

Lua 5.1.1  Copyright (C) 1994-2006 Lua.org, PUC-Rio
> require "ex"
> f = assert(io.open("234", "w"))
> = f:lock("w")
nil     6 (0x6): The handle is invalid.

This was a bug; I think it is now fixed. -Mark

(8)

$ make mingw
make -C w32api ex.dll
make[1]: Entering directory `.../ex/w32api'
cc -W -Wall -Wno-missing-braces -DWIN32_LEAN_AND_MEAN -DNOGDI -...  -mno-cygwin -c -o ex.o ex.c
cc -W -Wall -Wno-missing-braces -DWIN32_LEAN_AND_MEAN -DNOGDI -...  -mno-cygwin -c -o spawn.o spawn.c
spawn.c: In function `spawn_param_init':
spawn.c:35: warning: missing initializer
spawn.c:35: warning: (near initialization for `si.lpReserved')
cc -W -Wall -Wno-missing-braces -DWIN32_LEAN_AND_MEAN -DNOGDI -I...  -mno-cygwin -c -o pusherror.o pusherror.c
cc -W -Wall -Wno-missing-braces -DWIN32_LEAN_AND_MEAN -DNOGDI -I...  -mno-cygwin -c -o dirent.o dirent.c
cc -mno-cygwin -shared -o ex.dll ex.o spawn.o pusherror.o dirent.o -L... -llua5.1
make[1]: Leaving directory `.../ex/w32api'

See http://msdn2.microsoft.com/en-us/library/ms686331.aspx Might want to initialize structure via memset instead to avoid warning.

The warning is an unfortunate feature of GCC -- C guarantees that the other struct members are default-initialized to zero, which is the intention here. -Mark

Warning could be avoided with "static const STARTUPINFO si = {sizeof si, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};"

(9)

$ make cygwin
make -C posix T=ex.dll ex.dll
make[1]: Entering directory `...ex/posix'
cc -W -Wall -std=c89 -D_XOPEN_SOURCE=600 -DMISSING_POSIX_SPAWN -DENVIRON_DECL="e
xtern char **environ;" -I...   -c -o ex.o ex.c
ex.c: In function `ex_setenv':
ex.c:53: warning: implicit declaration of function `setenv'
ex.c:53: warning: implicit declaration of function `unsetenv'
ex.c: In function `new_file':
ex.c:131: warning: implicit declaration of function `fdopen'
ex.c:131: warning: assignment makes pointer from integer without a cast
ex.c: In function `ex_dirent':
ex.c:151: warning: implicit declaration of function `fileno'
cc -W -Wall -std=c89 -D_XOPEN_SOURCE=600 -DMISSING_POSIX_SPAWN -DENVIRON_DECL="e
xtern char **environ;" -I...   -c -o spawn.o spawn.c
cc -W -Wall -std=c89 -D_XOPEN_SOURCE=600 -DMISSING_POSIX_SPAWN -DENVIRON_DECL="e
xtern char **environ;" -I...   -c -o posix_spawn.o po
six_spawn.c
posix_spawn.c:49: warning: unused parameter 'act'
cc -shared -o ex.dll ex.o spawn.o posix_spawn.o -L...
make[1]: Leaving directory `...ex/posix'

Note cygwin stdlib.h:

#ifndef __STRICT_ANSI__
long    _EXFUN(a64l,(const char *__input));
char *  _EXFUN(l64a,(long __input));
char *  _EXFUN(_l64a_r,(struct _reent *,long __input));
int	_EXFUN(on_exit,(_VOID (*__func)(int, _PTR),_PTR __arg));
_VOID	_EXFUN(_Exit,(int __status) _ATTRIBUTE ((noreturn)));
int	_EXFUN(putenv,(char *__string));
int	_EXFUN(_putenv_r,(struct _reent *, char *__string));
int	_EXFUN(setenv,(const char *__string, const char *__value, int __overwrite));
int	_EXFUN(_setenv_r,(struct _reent *, const char *__string, const char *__value, int __overwrite))
#ifndef __STRICT_ANSI__
#ifndef _REENT_ONLY
FILE *	_EXFUN(fdopen, (int, const char *));
#endif
int	_EXFUN(fileno, (FILE *));

The purpose of ExtensionProposal is to provide non-ANSI functions, so why compile under ANSI?

Right, -std=c89 is incorrect -- I'll change conf.in. -Mark

--DavidManura

(10) It seems that ex.dir opens a file but does not close it. Thus this code:

while true do for f in os.dir("./") do print(f.name) end end

fails with error:

stdin:2: attempt to call a nil value stack traceback:

stdin:2: in main chunk

[C]: ?

after some cycles. Other calls to os.dir give:

print(os.dir("./")) -> nil Too many open files

lsof shows that multiple (1020) copies of that directory are open.

I understand that the open file is something that must live accross multiple calls to the ex API, but the possibility to call os.dir with an open file would provide this temporary workaround:

while true do local d = io.open("./") for f in os.dir(d) do print(f.name) end d:close() end

btw, this ex is very useful. Great work -- m.i.

After a quick search over POSIX manuals it seems not easy or impossible. Instead changing a few lines in posix/ex.c (at line 240, in ex_dir) from:

if (!d) return push_error(L);

to:

if (!d) { diriter_close(L); return push_error(L); }

seems to solve the problem (and not create others, but that will need test).

sorry for not using diff, I've never learnt...

Again, great work -- m.i.

Yes, file handles run out long before memory, so yours is the correct solution. BTW, this is perhaps a better idiom:

for f in assert(os.dir".") do print f.name; end

-Mark

Yes, this is a known bug and I have a fix, I just need to post it, sorry. -Mark

Nice to know... but known to whom? If you knew about these bugs perhaps sharing this knowledge with the wider world would have been a good idea? For instance on this very page? Well... <shakes head> -- ThomasLauer

void spawn_param_args(struct spawn_params *p)
{
	 lua_State *L = p->L;
	 debug("spawn_param_args:"); debug_stack(L);
	 luaL_Buffer args;
	 luaL_buffinit(L, &args);
	 size_t i, n = lua_objlen(L, -1);
	 /* concatenate the arg array to a string */
	 for (i = 1; i <= n; i++) {
	 	 int quote;
	 	 lua_rawgeti(L, -1, i);     /* ... argtab arg */
	 	 /* XXX checkstring is confusing here */
	 	 quote = needs_quoting(luaL_checkstring(L, -1));
	 	 luaL_putchar(&args, ' ');
	 	 if (quote) luaL_putchar(&args, '"');
	 	 luaL_addvalue(&args);
	 	 if (quote) luaL_putchar(&args, '"');
--->	 lua_pop(L, 1);             /* ... argtab */
	 }
	 luaL_pushresult(&args);        /* ... argtab argstr */
	 lua_pushvalue(L, 1);           /* cmd opts ... argtab argstr cmd */
	 lua_insert(L, -2);             /* cmd opts ... argtab cmd argstr */
	 lua_concat(L, 2);              /* cmd opts ... argtab cmdline */
	 lua_replace(L, -2);            /* cmd opts ... cmdline */
	 p->cmdline = lua_tostring(L, -1);
}

require 'ex'
ex.spawn{'ls'}

in the lua 5.1.1 interpreter (on gentoo, amd64, gcc-4.1.2). Gdb backtrace

#0  0x00002adc9351a885 in raise () from /lib/libc.so.6
#1  0x00002adc9351bb3e in abort () from /lib/libc.so.6
#2  0x00002adc93550a27 in ?? () from /lib/libc.so.6
#3  0x00002adc93555b1d in ?? () from /lib/libc.so.6
#4  0x00002adc935579b6 in ?? () from /lib/libc.so.6
#5  0x00002adc9355950d in malloc () from /lib/libc.so.6
#6  0x00002adc93177351 in ?? () from /usr/lib64/liblua.so.5
#7  0x00002adc9317add9 in ?? () from /usr/lib64/liblua.so.5
#8  0x00002adc93170237 in lua_getfield () from /usr/lib64/liblua.so.5
#9  0x00002adc938867e9 in ex_spawn (L=0x503010) at ex.c:412
#10 0x00002adc93173fc1 in ?? () from /usr/lib64/liblua.so.5
#11 0x00002adc9317d50e in ?? () from /usr/lib64/liblua.so.5
#12 0x00002adc9317440d in ?? () from /usr/lib64/liblua.so.5
#13 0x00002adc93173b77 in ?? () from /usr/lib64/liblua.so.5
#14 0x00002adc93173bf4 in ?? () from /usr/lib64/liblua.so.5
#15 0x00002adc9316fc75 in lua_pcall () from /usr/lib64/liblua.so.5
#16 0x0000000000401746 in ?? ()
#17 0x0000000000401b54 in ?? ()
#18 0x0000000000402137 in ?? ()
#19 0x00002adc93173fc1 in ?? () from /usr/lib64/liblua.so.5
#20 0x00002adc931743bd in ?? () from /usr/lib64/liblua.so.5
#21 0x00002adc93173b77 in ?? () from /usr/lib64/liblua.so.5
#22 0x00002adc93173bf4 in ?? () from /usr/lib64/liblua.so.5
#23 0x00002adc9316fc17 in lua_cpcall () from /usr/lib64/liblua.so.5
#24 0x000000000040167d in main ()

This is with standard (Gentoo distributed) lua 5.1.1.

I compiled lua 5.1.2 and get the same (more symbols)

Compiling lua with -O0 holds the same error. Just more information:

#0  0x00002b8a1dc7a885 in raise () from /lib/libc.so.6
#1  0x00002b8a1dc7bb3e in abort () from /lib/libc.so.6
#2  0x00002b8a1dcb0a27 in ?? () from /lib/libc.so.6
#3  0x00002b8a1dcb5b1d in ?? () from /lib/libc.so.6
#4  0x00002b8a1dcb79b6 in ?? () from /lib/libc.so.6
#5  0x00002b8a1dcb950d in malloc () from /lib/libc.so.6
#6  0x0000000000419c8d in l_alloc (ud=0x0, ptr=0x0, osize=0, nsize=29) at lauxlib.c:636
#7  0x000000000040d519 in luaM_realloc_ (L=0x533010, block=0x0, osize=0, nsize=29) at lmem.c:79
#8  0x0000000000411e3b in newlstr (L=0x533010, str=0x2b8a1de8c235 "args", l=4, h=7976507) at lstring.c:56
#9  0x00000000004120ab in luaS_newlstr (L=0x533010, str=0x2b8a1de8c235 "args", l=4) at lstring.c:92
#10 0x00000000004060f4 in lua_getfield (L=0x533010, idx=2, k=0x2b8a1de8c235 "args") at lapi.c:546
#11 0x00002b8a1de8b4be in ex_spawn (L=0x533010) at ex.c:379
#12 0x0000000000409e92 in luaD_precall (L=0x533010, func=0x533410, nresults=0) at ldo.c:319
#13 0x00000000004175b9 in luaV_execute (L=0x533010, nexeccalls=1) at lvm.c:589
#14 0x000000000040a106 in luaD_call (L=0x533010, func=0x533400, nResults=-1) at ldo.c:377
#15 0x0000000000406a30 in f_call (L=0x533010, ud=0x7fff8d573bb0) at lapi.c:796
#16 0x0000000000409192 in luaD_rawrunprotected (L=0x533010, f=0x406a01 <f_call>, ud=0x7fff8d573bb0) at ldo.c:116
#17 0x000000000040a49d in luaD_pcall (L=0x533010, func=0x406a01 <f_call>, u=0x7fff8d573bb0, old_top=48, ef=32) at ldo.c:461
#18 0x0000000000406ad6 in lua_pcall (L=0x533010, nargs=0, nresults=-1, errfunc=1) at lapi.c:817
#19 0x0000000000403dfa in docall (L=0x533010, narg=0, clear=0) at lua.c:100
#20 0x00000000004043d0 in dotty (L=0x533010) at lua.c:219
#21 0x0000000000404b3f in pmain (L=0x533010) at lua.c:367
#22 0x0000000000409e92 in luaD_precall (L=0x533010, func=0x5333e0, nresults=0) at ldo.c:319
#23 0x000000000040a0f4 in luaD_call (L=0x533010, func=0x5333e0, nResults=0) at ldo.c:376
#24 0x0000000000406be0 in f_Ccall (L=0x533010, ud=0x7fff8d573f50) at lapi.c:842
#25 0x0000000000409192 in luaD_rawrunprotected (L=0x533010, f=0x406b11 <f_Ccall>, ud=0x7fff8d573f50) at ldo.c:116
#26 0x000000000040a49d in luaD_pcall (L=0x533010, func=0x406b11 <f_Ccall>, u=0x7fff8d573f50, old_top=16, ef=0) at ldo.c:461
#27 0x0000000000406c37 in lua_cpcall (L=0x533010, func=0x404973 <pmain>, ud=0x7fff8d573fa0) at lapi.c:852
#28 0x0000000000404bb4 in main (argc=1, argv=0x7fff8d5740b8) at lua.c:385

The ex.spawn'ls' syntax fails too, but in spawn_param_execute, instead.

I can't understand why.

Parameters passed to the realloc would be correct (gdb sees them as ptr=0x0, nsz=48), but from debug output it seems that it calls malloc instead (is it normal?). I don't experience this kind of failures in normal Lua use, so I assume it is not its fault.

--m.i.

This is a known bug which is now fixed but not yet available. Soon! I promise. - MarkEdgar

Not knowing this I reimplemented most of the functions. It's sort of duplicating code, so I'll simply stick to your implementation from now on. Still I made some adjustments to the API, that I think could be useful. I post the notes in my page as they are quite long -- MauroIazzi

I think os.isatty would be useful to allow avoiding the explicit "-" in "cat myfile | lua myapp.lua -". Lua itself calls isatty, though its in luaconf.h --DavidManura

Perhaps a glob function would be useful here. See also FileGlob. --DavidManura