Re: reply:Using fixed string buffers to avoid extensive allocs for small strings

[Flyer31 Test <flyer31@...>]

Thank you - this looks nice.

What does your lib do if the user writes buff='hallo'?

On Thu, Oct 14, 2021 at 11:41 AM Wendal@合宙 <wendal@openluat.com> wrote:
>
> it look like "zbuff" , in our "LuatOS" repo.
>
> local buff = zbuff.create(1024)
> buff[0] = 0x5A
> buff[1] = 0xBA
> buff:write(xxxdata) -- no gc ,no malloc at all
>
> https://github.com/openLuat/LuatOS/blob/master/luat/modules/luat_lib_zbuff.c
>
> --------------原始邮件--------------
> 发件人："Flyer31 Test "<flyer31@googlemail.com>;
> 发送时间：2021年10月14日(星期四) 下午5:30
> 收件人："Lua mailing list" <lua-l@lists.lua.org>;
> 主题：Using fixed string buffers to avoid extensive allocs for small strings
> -----------------------------------
> Hi,
> I am currently designing a Lua32 application on a very
> memory-restricted system with ca. 256kB ROM and 128kB RAM for some IoT
> application.
>
> ROM space is very well sufficient for me for Lua, I need only about 100kB.
>
> But RAM alloc space should be limited to something in the range 30kB,
> and this for me somehow seems to be impossible if I allow Lua its
> "normal" extensive string allocation, which it does extensively even
> for smallest strings. E. g. when I want to re-program my Lua user
> program, the complete Lua ASCII program file will split into small
> segments of 4-8 bytes, and if I do this with strings (just concat ..
> and string.sub), then quite fast the 30kB memory limit will be hit...
> . I tried to invoke the garbage collector in regular intervals (I use
> yielding, so e. g. after every yield cycle...), then the garbage
> collector invocation typically will fail after the 2nd invocation with
> the some "memory problem" error message... (my alloc function refuses
> any alloc attempt above 50 or 60kB).
>
> So now I wanted to do a cute small library with fixed string buffers,
> which I call strbuf. This buffers use a limited buffer size, in case
> of any overflow they will mark the string with a final "~" sign, this
> should be fine ... . Typically the strings in my programs anyway are
> limited to 100 chars / typical line length, but this already would be
> extreme ... most strings are much shorter (e. g. "myriads of strings"
> with 4-8 chars as described above).
>
> I want to use the following programming style for this in Lua (e. g.
> defining two such buffers Tmp and InBuf, and then playing around with
> them a bit):
>
> Tmp= strbuf.new(100)
> InBuf= strbuf.new( 100)
>
>     Tmp[1]= 'hallo'
>     InBuf[1]= Tmp
>     InBuf[1]= Tmp[1]
>
>
> This works all very nicely with a very simple strbuf lib which uses a
> metatable and supports __index / __newindex:
>
>     Tmp[1]= 'hallo' invokes __newindex for Tmp and then fills the Tmp-Buffer
>     InBuf[1]= Tmp invokes __newindex for InBuf and copies Tmp-Buffer
> to InBuf-Buffer
>     InBuf[1]= Tmp[1] invokes __index for Tmp, copies to a
> (user-hidden) global strbuf element
>                                               __STRBUF, and returns
> this __STRBUF, then it will inoke
>                                               __newindex for InBuf and
> gets __STRBUF (it is important here to
>                                               use __STRBUF, otherwise
> __index of Tmp would have to return a
>                                               string, but this then is
> again will result in stupid allocs...).
>
> I can extend this quite easily also to further cute applications, e. g.
> Tmp[n]= InBuf[m]      (to overwrite char n... of Tmp with char m... of InBuf)
>                                   (also negative n/m allowed, then
> counting from string end...)
> index_of_f = Tmp['f']     to get the index of some substring (e. g. 'f')
> 1== Tmp['f']                   to check whether Tmp starts with 'f'
> Tmp[n]= mul                  to "multiply" the n'th char of Tmp (or
> delete if mul negative)
>
> ... and later also of course things like Tmp==InBuf, or Tmp[5]==InBuf,
> or Tmp..Inbuf, or Tmp[m]..Inbuf[n] ... (or some additional helper
> functions like strbuf.format or strbuf.scan...)
>
> ... this really looks very cute to me, I am happy with this, and I was
> very impressed how fast this could be accomplished with this metatable
> functionality of Lua, thank you for this...
>
> Just there is one very dangerous and nerving error source for the user now:
>
> If the user by some accident writes
> Tmp='hallo'
>
> instead of
> Tmp[1]='hallo'
>
> Because in this case then __newindex will NOT invoke, but instead by
> Tmp varialble will somehow "nervingly" will be converted to a
> string... .
>
> Do you have some smart idea do avoid this, or block this type
> conversion of my nice strbuf variables by Lua error?
>
> Nice would be, that any such strbuf element (generated by strbuf.new,
> and thus having this metatable strbuf...) should NOT be allowed to
> "change type" any more, orto be re-assigned to anything else (but of
> course Tmp[1]=... should still be possible and invoke __index).
>
> Super-great of course would be, if my c software could be notified by
> Tmp='hallo' (e. g. if there would be some meta function "__new" or
> "__reassign" or so ... but this I did not find unfortunately... I am
> quite sure the Lua machine will do this Tmp='hallo' typically extremly
> fast without checking any metatables of Tmp?).