Re: New Batteries? (Was: Dead Batteries)

[Dael Muñiz <daelvn@...>]

> There are probably many "invisible" Lua users that are some kiddos that
> mod their favourite game and sooner or later want to create "real"
> applications outside of the host game.
> But they aren't full-blown programmers with an elaborate toolchain and a
> stack of reference books on their desks (yet ^^).
> So they download some things from the internet, fiddle around a bit and
> then give up on Lua...

[Another wall of text, I will try to sum stuff up by the end of it]

This was probably one of the most relatable parts of the previous
response for me.

I personally started getting into programming by wondering how
Minecraft worked. Quite the intro, I know. I somehow started with Java
and modding, and descarted it as a feasible idea and moved on to other
languages to do my usual script kiddie stuff thati uploaded onto
Microsoft CodePlex. It was all quite weird and sketchy. I went through
Python, Batch, C#, and almost Ruby, until I found Lua.

By the time I found Lua it wasn't so far from my original intentions
when I started programming. I learned it from the hand of a Minecraft
mod that implements fantasy computers that run Lua, since it is an
easily embeddable language. (There are actually two different mods
that allow doing this). I started fading off from the mod, moving onto
making practical stuff, leaving all of that phase behind. I don't know
exactly what made me stay, but I think it was the simplicity of the
language, not necessarily in syntax, but I just didn't like what
Python did, and I still suffered from some "Not Invented Here
Syndrome". It's true that I found myself working on libraries most of
the time because while there are many availiable, I felt like many of
them wouldn't work for me.

Now I know that many of these I looked at could have actually worked,
however, I'm still working on libraries, why?
While the community has made really important contributions and useful
libraries, I still feels like there are many holes to fill, many
libraries that Lua is lacking. That isn't necessarily a bad thing and
I understand the nature of the Lua language. Lua *could* be a good
GPL, but I feel like a parallel project to Lua should fill that hole
instead of Lua. I will expand more on that later.

I still find myself around the community of that Minecraft Lua mod,
and I see new people coming in every month with the same excitement I
had. The problems they ask about are usually about opening a door or
solving some logic problem, but the experience is not much different
from what a kid could do with Scratch and moving blocks. Truth is,
many of these people, as you said, move on from Lua onto other
languages. That is, of course, only if they do end up falling in love
with programming as I did.

You can see Lua in many ways, and depending on the way you see it,
your answer to "Lua needs a standard library" will vary. If someone
thinks of Lua as a mostly embedded language, they probably are not
going to be very fond of a standard library, because excluding it or
including it at compilation would not be as easy for the unexperienced
end user, say. In the end, you could end up not knowing the
implementation you got if you are working anywhere that uses embedded
Lua. Or, in the case of the Minecraft mods, they may end up being
completely irrelevant, as for example it implements its own "API" for
"filesystem" handling like creating folders, listing files and the
such. If you think of it as a General Purpose Language, of course you
are going to want a standard library, it will certainly make many
people stay and increase the "market share" that Lua currently has by
making it a far more powerful language out of the box. If you think of
it as a passing language, one that you use merely to learn to later
move on to a better language, chances are that it doesn't matter too
much.

Now, regardless of your view of Lua, I think it's undeniable that if
it did get a standard library, included batteries, whatever; it would
change the way people see and use Lua. It is not a solution that works
for everyone, however.

Now, expanding on the previous idea of the parallel project. I don't
feel like Lua as a project, with its maintainers and current purpose,
is ready to take on the responsibility of maintaining a standard
library and turning into a GPL. Besides, many people will not like
that solution. These are not necessarily bad things, merely opinions.
This is why I don't think that Lua should try to go for a "majority
vote" and find the solution that works for *most*, but rather keep the
Lua project as it currently is, and have a certain group of volunteers
maintain a parallel project to Lua, forked from it and fairly
up-to-date, that does implement these standard libraries and has the
focus on becoming a GPL rather than being embedded.

I personally believe that it would only work if everything is open
standard, and publicly hosted. By open standard I mean all decisions
being discussed and agreed for the most part, and while Lua is
publicly hosted, people usually check out the GitHub repository. It
would rather be helpful to have it hosted on a platform where it can
get a lot of attention.

(Now, I know there's many parts about the last e-mail I am missing,
I'm not well-versed on thread and don't really have a stance on it)

[Conclusion]
I think a good conclusion would be:
- Many people use Lua as a passing language that they use to learn.
- Many more people would stay on Lua given the right included batteries.
- There is disagreement as to what people think of Lua and whether it
should have a stdlib or not.
- I believe that a project with included batteries would have to work
parallel to Lua and more open than it currently is.


On Mon, 6 Jan 2020 at 19:02, Stefan <ste@evelance.de> wrote:
>
> Am 05.01.2020 um 23:00 schrieb Dibyendu Majumdar:
> >
> > Well Python is arguably a better general purpose language and has a
> > very powerful set of libraries. Why does the world need another one?
> >
> > Regards
> >
>
> [Warning: wall of text - jump to "What about Lua?" or "Conclusion" for a
> shortcut]
>
> Python's insufficient thread support is a *big* disadvantage.
> The GIL (Global Interpreter Lock) basically combines the
> unpredictability of preemptive multithreading with the low performance
> of cooperative multithreading and is only useful for some I/O tasks[1].
>
> Making Python and all of its libraries ready for "real" multithreading
> is a lot of work and the Python community hasn't done this yet.
>
> But soon this will become very important!
>
> Fast CPUs are a "solved problem" nowadays and we will probably only get
> more of them, more efficient and cheaper ones and specialized hardware
> but no longer faster cores[2].
> This trend has already started - AMD produces mobile CPUs with 8 cores
> at 15W[3] and Intel will certainly not sit still.
> Maybe 8 cores for laptops and 16 cores for PCs will be the default in
> less than five years...
>
> OTOH data production/transport/storage technology is *still* improving
> at an unbelievable pace[4][5] (excerpt: new technology will help 500
> million Indians to join the internet).
> With most users possessing only dual-core machines (today), not having
> actual multicore support wasn't that bad. But with 8/16/32/special
> cores? And a lot more data to process?
>
> Even the CPU architects are concerned about this[6, page 35/36].
>
> Unfortunately weak typing and multithreading doesn't mix well and
> multithreading in general has many pitfalls.
> Jython/JRuby use the JVM to take advantage of its sophisticated runtime.
> But a threaded garbage collector and per-object locking seem to be
> anything but trivial and the JVM is a memory-hogging behemoth.
> JavaScript doesn't allow shared context at all and WebWorkers basically
> provide processes with bolted-on message passing.
> PHP seems to allow threading without actually being thread-safe...
> solves the problem by ignoring it ;)
> About others I'm not sure.
>
> What about Lua?
>
> Making the language itself multithreaded is obviously not going to
> happen. But creating many independent Lua states that are executed by
> multiple threads is easy!
>
> While starting hundreds or thousands of threads is not a good idea,
> having more independent program contexts than cores is very useful:
> - map one network connection to one Lua state, i.e. application servers
> - one state per user for multiuser applications
> - one state per view for GUI programs
> - process a directory recursively and have a state per directory
> - for divide-and-conquer strategies in general
> - partitioning of complex programs
>
> Compared to various other scripting languanges Lua has by far the
> fastest startup time by at least an order of magnitude. Btw check out
> `strace ruby -e "puts 'hello world'" 2>&1 | wc -l`, Ruby 2.5 needs
> 1790 syscalls for "hello world" (yes I know this example is unfair :))
>
> For Lua, one of my i5230M cores can create 100K states + load a
> precompiled script in *one second*. On x86_64 they use up ~700MB RAM
> afterwards.
>
> Unfortunately the reason why this is so fast is that the lua_States are
> basically empty. Loading just the standard library slows it down by a
> factor of 10.
>
> So, a theoretical future feature-rich standard library should support:
> - lazy binding/loading so that adding more functions doesn't impede
> state creation
> - thread safety or awareness of its functions (one C function could be
> called by multiple Lua states in parallel)
> - communication/synchronization between states
> - offloading of blocking calls into separate threads
> - some kind of compatibility header/subset of the C API so that minor
> changes in the language don't break existing library sources
>
> Conclusion:
> - It's 2020, a useful standard library must support real threading
> - And Lua states are handy since they provide separate contexts without
> giving up the user-friendly cooperative model
>
> It has always bummed me out that there is no agreed-on bigger standard
> library for Lua that is not directly meant for embedding. I mean it is
> also the perfect glue language, why isn't there anything ready to glue
> together?
>
> There are probably many "invisible" Lua users that are some kiddos that
> mod their favourite game and sooner or later want to create "real"
> applications outside of the host game.
>
> But they aren't full-blown programmers with an elaborate toolchain and a
> stack of reference books on their desks (yet ^^).
> So they download some things from the internet, fiddle around a bit and
> then give up on Lua...
>
> Providing them with a powerful and extensible, pre-compiled library
> would go a long way!
>
> I think due to its simple and divisible nature Lua *could* be a general
> purpose scripting language of the future!
>
> Best regards and a happy new year :)
>
> [1] "Inside the Python GIL"
> https://www.dabeaz.com/python/GIL.pdf
> [2] "Trends in Processor Architecture"
> https://arxiv.org/ftp/arxiv/papers/1801/1801.05215.pdf
> [3] "Ryzen 7 4700U: AMD's First Ever 8-Core APU"
> https://www.tomshardware.com/news/ryzen-7-4700u-amds-first-ever-8-core-apu
> [4] "Ericsson Mobility Report November 2019"
> https://www.ericsson.com/4acd7e/assets/local/mobility-report/documents/2019/emr-november-2019.pdf
> [5] "Cisco Visual Networking Index: Forecast and Trends, 2017–2022"
> https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-741490.pdf
> [6] "A New Golden Age for Computer Architecture"
> https://californiaconsultants.org/wp-content/uploads/2018/04/CNSV-1806-Patterson.pdf
>