Re: reply:Re: reply:Using fixed string buffers to avoid extensive allocs for small strings

[Flyer31 Test <flyer31@...>]

Hi Wendal,
I got __newindex working now for _G nicely.

But unfortunately ( I could have thought about this earlier I must
admit), __newindex will be invoked only if a NEW global is defined ...
so only first time, not later for reassign.

For my strbuf buffers __newindex nicely works, because I do not use
this userdata as table, so __newindex is just coming every time when I
write buf[i] or buf[1] or buf[anything] ... just NOT unfortunately if
I write buf=... (it would be super-nice, if then it would come with
key=nil... but as nil is a valid key, maybe also a bit misguiding /
not possible...).

On Thu, Oct 14, 2021 at 12:13 PM Wendal@合宙 <wendal@openluat.com> wrote:
>
> it can't be done.
>
> buff = "xxx"
>
> this "buff" variable will be replace by the new string value,not userdata anymore. it is lua's role.
>
> --------------原始邮件--------------
> 发件人："Flyer31 Test "<flyer31@googlemail.com>;
> 发送时间：2021年10月14日(星期四) 晚上6:06
> 收件人："lua mailing list" <lua-l@lists.lua.org>;
> 主题：Re: reply:Using fixed string buffers to avoid extensive allocs for small strings
> -----------------------------------
> Thank you - this looks nice.
>
> What does your lib do if the user writes buff='hallo'?
>
> On Thu, Oct 14, 2021 at 11:41 AM Wendal@合宙 <wendal@openluat.com> wrote:
> >
> > it look like "zbuff" , in our "LuatOS" repo.
> >
> > local buff = zbuff.create(1024)
> > buff[0] = 0x5A
> > buff[1] = 0xBA
> > buff:write(xxxdata) -- no gc ,no malloc at all
> >
> > https://github.com/openLuat/LuatOS/blob/master/luat/modules/luat_lib_zbuff.c
> >
> > --------------原始邮件--------------
> > 发件人："Flyer31 Test "<flyer31@googlemail.com>;
> > 发送时间：2021年10月14日(星期四) 下午5:30
> > 收件人："Lua mailing list" <lua-l@lists.lua.org>;
> > 主题：Using fixed string buffers to avoid extensive allocs for small strings
> > -----------------------------------
> > Hi,
> > I am currently designing a Lua32 application on a very
> > memory-restricted system with ca. 256kB ROM and 128kB RAM for some IoT
> > application.
> >
> > ROM space is very well sufficient for me for Lua, I need only about 100kB.
> >
> > But RAM alloc space should be limited to something in the range 30kB,
> > and this for me somehow seems to be impossible if I allow Lua its
> > "normal" extensive string allocation, which it does extensively even
> > for smallest strings. E. g. when I want to re-program my Lua user
> > program, the complete Lua ASCII program file will split into small
> > segments of 4-8 bytes, and if I do this with strings (just concat ..
> > and string.sub), then quite fast the 30kB memory limit will be hit...
> > . I tried to invoke the garbage collector in regular intervals (I use
> > yielding, so e. g. after every yield cycle...), then the garbage
> > collector invocation typically will fail after the 2nd invocation with
> > the some "memory problem" error message... (my alloc function refuses
> > any alloc attempt above 50 or 60kB).
> >
> > So now I wanted to do a cute small library with fixed string buffers,
> > which I call strbuf. This buffers use a limited buffer size, in case
> > of any overflow they will mark the string with a final "~" sign, this
> > should be fine ... . Typically the strings in my programs anyway are
> > limited to 100 chars / typical line length, but this already would be
> > extreme ... most strings are much shorter (e. g. "myriads of strings"
> > with 4-8 chars as described above).
> >
> > I want to use the following programming style for this in Lua (e. g.
> > defining two such buffers Tmp and InBuf, and then playing around with
> > them a bit):
> >
> > Tmp= strbuf.new(100)
> > InBuf= strbuf.new( 100)
> >
> >     Tmp[1]= 'hallo'
> >     InBuf[1]= Tmp
> >     InBuf[1]= Tmp[1]
> >
> >
> > This works all very nicely with a very simple strbuf lib which uses a
> > metatable and supports __index / __newindex:
> >
> >     Tmp[1]= 'hallo' invokes __newindex for Tmp and then fills the Tmp-Buffer
> >     InBuf[1]= Tmp invokes __newindex for InBuf and copies Tmp-Buffer
> > to InBuf-Buffer
> >     InBuf[1]= Tmp[1] invokes __index for Tmp, copies to a
> > (user-hidden) global strbuf element
> >                                               __STRBUF, and returns
> > this __STRBUF, then it will inoke
> >                                               __newindex for InBuf and
> > gets __STRBUF (it is important here to
> >                                               use __STRBUF, otherwise
> > __index of Tmp would have to return a
> >                                               string, but this then is
> > again will result in stupid allocs...).
> >
> > I can extend this quite easily also to further cute applications, e. g.
> > Tmp[n]= InBuf[m]      (to overwrite char n... of Tmp with char m... of InBuf)
> >                                   (also negative n/m allowed, then
> > counting from string end...)
> > index_of_f = Tmp['f']     to get the index of some substring (e. g. 'f')
> > 1== Tmp['f']                   to check whether Tmp starts with 'f'
> > Tmp[n]= mul                  to "multiply" the n'th char of Tmp (or
> > delete if mul negative)
> >
> > ... and later also of course things like Tmp==InBuf, or Tmp[5]==InBuf,
> > or Tmp..Inbuf, or Tmp[m]..Inbuf[n] ... (or some additional helper
> > functions like strbuf.format or strbuf.scan...)
> >
> > ... this really looks very cute to me, I am happy with this, and I was
> > very impressed how fast this could be accomplished with this metatable
> > functionality of Lua, thank you for this...
> >
> > Just there is one very dangerous and nerving error source for the user now:
> >
> > If the user by some accident writes
> > Tmp='hallo'
> >
> > instead of
> > Tmp[1]='hallo'
> >
> > Because in this case then __newindex will NOT invoke, but instead by
> > Tmp varialble will somehow "nervingly" will be converted to a
> > string... .
> >
> > Do you have some smart idea do avoid this, or block this type
> > conversion of my nice strbuf variables by Lua error?
> >
> > Nice would be, that any such strbuf element (generated by strbuf.new,
> > and thus having this metatable strbuf...) should NOT be allowed to
> > "change type" any more, orto be re-assigned to anything else (but of
> > course Tmp[1]=... should still be possible and invoke __index).
> >
> > Super-great of course would be, if my c software could be notified by
> > Tmp='hallo' (e. g. if there would be some meta function "__new" or
> > "__reassign" or so ... but this I did not find unfortunately... I am
> > quite sure the Lua machine will do this Tmp='hallo' typically extremly
> > fast without checking any metatables of Tmp?).