Re: Lua without IO

[Stojan Dimitrovski <sdimitrovski@...>]

Just to clarify a few points from the discussion:

Lua without IO actually means without system-provided IO. In my
current implementation, much of that IO is abstracted away into
non-syscall ways of doing IO. (Basically reading and writing to
special places in memory.) The problem is that Lua depends on the IO
provided by the kernel, which is not a problem most of the time.
However, in some restricted environments like Intel SGX kernel-based
IO is not at all possible to do. I imagine it's quite similar in other
applications as well.

So the request is not to abstract away the kernel IO use of Lua, as
that would not really be beneficial in general, but just define a few
more macros (and make existing more useful) so that when someone needs
to use Lua in a syscall-less environment, they don't have to do it
manually (and possibly with errors).

I personally would not expect such a Lua compile-time configuration to
implement the OS, IO and some base library functions. Those would be
up to the embedder to do correctly -- something I'm doing in my own
project.

On Wed, Feb 27, 2019 at 9:40 PM Philippe Verdy <verdy_p@wanadoo.fr> wrote:
>
> frexp() is not so hard, even if it is much faster if you have access to the native assembly instructions. In pure C, this could be still implemented with some platform-testing function (using tests on known constants), but you have to use unsafe pointer conversion and be aware of the byte order (endianness).
> In practice you can still go faster if you know the CPU and FPU architecture and some of them are now used everywhere : x86, x64/amd64, and arm64 (all the others are older processors or processors for niche markets on embedded systems, many of them using now some arm32 or arm64; there's almost no more use for other x80, x88, 65xx, 68k, Sparc and Mips processors, but there remains now a growing niche with GPUs, mostly from nVdidia and ATI, and for some high-end very costly processors used in supercomputers, and for "virtual processors" running on top of a VM with its own "portable" ABI, including Java, Perl, Lua, or Javascript).
> abort() may seem complex but can be implemented on top of longjmp (and more complete implemention if adding support for signal and raise).
> If you compile Lua against a C++ compiler, you can still use the C++ exception handlers to implement setjmp/longjmp in an additional C++ compiled module (but of course you need a C++ compiler like GCC or clang) to declare the needed function with 'extern "C"...' linkage . I bet there are today a C++ compiler available on almost all platforms and they already come with the machinery library to link the few C++ modules with the other C modules or libraries needed by Lua (this is difficult only when you don't have access to the native instruction set and are bound only to a virtual processor, in which case you need to use the VM API, but most VMs have basic support for exception handling or error handling).
> The really complex part is if you need to make Lua exist within limits of a multithreading environment, and make manage these limits (notably in memory allocation, time slot, time precision, and permissions).
> If you don't have the standard C I/O the program in Lua will still perform I/O with another library (it could be a network API or an IPC mechanism with named channels, using some specific resource allocators, or a database or "registry" service).
> One thing that is still missing in Lua is a safe way to integrate it with true (Posix) threads instead of being limited to the "cooperative" model (by which all Lua threads used in a program exist within the same system thread, using the very complex setjmp/longjmp mechanism): we still cannot safely and simply declare to Lua to exist the cooperative model and allow threads to run concurrently, because Lua has no way to coordinate them (it would require the addition of synchronization and serialization, and a definition of event queues, and resource managers: in standard Lua there's a single queue and a single resource manager for everything, the rest is part of the internal integration library but inacessible to Lua programs themselves; this caues huge difficulties to implement efficient networking, notably for implementing services without excessive blocking and without huge variation in response time in a very busy service replying to thousands of queries per second or more, or for interfacing with external database services).
> And unlike Java or Javascript, Lua still has no model for implementing security domains (with efficient resource sharing and data exchanges), and still no clear definition of a basic set of datatypes built for portability (including with known and reproduceable limits): for that you need to add other Lua packages, the basic library is too much limited and too much oriented for basic POSIX C (not much more than the old K&R model but still not enough to even support the POSIX standard). This causes many Lua programs to be non-portable (not even between x86 and x64 architectures, or between ARM32 and ARM64, on the same type of OS like Linux, Windows, or major branches MacOS/iOS).
> So Lua is basically a language defined by its syntax, it still has no formal API (let's put aside the ABI or its implementing VM): it works, yes, but the results are not reproductible, so Lua-written software is difficult to test: you need to develop Lua programs as well as test programs at the same time to target each platform and check the dependencies. There's no test framework like in Java, Perl or Javascript, not even for basic conformance level: you cannot develop on a system and deploy to others easily: you need to test/debug against each platform and there's nothing in Lua that allows asserting that a Lua module is compatible with some known platform type, no versioning system; each Lua implementation is very tightly linked to its initial platform (including the current "standard" Lua), and this also applies to its documentation: you need to tweak also this documentation to specify the limits and target platforms on which it was tested.
>
> Lua is still now is in the same "infant" stage as what was "K&R" C in the 1980's (unfortunately it remained too long in that stage and caused a proliferation of ABI's and later huge complexity to write portable C programs; the same risk exists now for Lua). May be there will be a change later when there will be a version of the language sponsored by a good standard body (ISO, IEEE, ECMA, ANSI...) and its development will be open (and Lua will not just be "opensource" as of today). Or when its development will be open to more members (e.g. the Mozilla, Linux, GNU foundations) and with formal process (but it will introduce bureaucracy and costs and possibly tensions between members like those for Java or C++ today). The costs would be supported by large service providers and sellers offering services in Lua but they may influence the future design of the language.
>
> Le mer. 27 févr. 2019 à 12:47, Luiz Henrique de Figueiredo <lhf@tecgraf.puc-rio.br> a écrit :
>>
>> > I've started work on a (not yet public) project that requires Lua to
>> > run without any IO / syscall dependencies.
>>
>> The Lua 5.3 core depends only on these standard C functions:
>>
>> abort abs floor fmod frexp localeconv longjmp memcmp memcpy pow setjmp
>> snprintf strchr strcmp strcoll strcpy strlen strpbrk strspn strtod
>> time
>>
>> Most of these can be written directly in C if needed. Some are hard:
>> frexp. Some are impossible: longjmp, setjmp. Some are controlled via
>> luaconf.h and l_mathop: floor, food frexp, localeconv, pow, snprintf.
>>
>> In other words, changing the dependency of the Lua core on libc can be
>> mostly done by editing  luaconf.h, as expected.
>>


-- 
Stojan Dimitrovski
http://stojan.me