lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index] 


On 22-Nov-06, at 9:06 PM, Glenn Maynard wrote:


On Wed, Nov 22, 2006 at 05:18:42PM -0800, Sam Roberts wrote:
On the other hand, maybe it doesn't matter that the heap is large, the 
pages will get swapped out, and if you never touch them, who cares?


You're going to touch them the next time you allocate more memory, and
the swapper--not knowing that memory was stale--will swap the old data
back in; whatever the user is doing that needs that memory will be
blocked while that happens.  I care if applications are wasting my
system's performance by increasing swap file activity.
Some Unix systems have a way of telling the kernel that memory is stale, through the madvise or posix_madvise interface. However, the only Unix I work with a lot that really implements this feature in a way that malloc() can make use of it is FreeBSD, which provides the MADV_FREE advice option to madvise. Linux, as I understand it, will interpret MADV_DONTNEED to mean that a page of zeros can be provided instead of swapping back in, if the page was originally allocated through mmap with anonymous backing store.
The FreeBSD standard malloc() implementation can be configured to use this feature but it is rarely a win; although it does reduce swapping, it also increases overall overhead because of increased bookkeeping on page tables.
In my experience, swapping out is more expensive than swapping in; in the case where large regions can be marked stale, the FreeBSD facility can speed things up quite a bit, but that is not very common.
glibc malloc()'s can be configured to return memory to the operating system by providing a negative argument to sbrk(); this can only be done if the memory to be returned is at the end of the heap. See mallopt() for details (there's no manpage for it in the Linux system I have access to right now, but it is documented on the glibc info pages at http://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable- Parameters.html 



By the way, it might be possible to improve Lua's memory use by having
the allocation function use a C++ allocator.  Knowing the size on
deallocation can give better characteristics than malloc/free can do,
especially for small allocations (eg. 4-byte allocations that really use 
4 bytes).  I don't know if it would have an appreciable impact on total
memory use by Lua.
FreeBSD's standard malloc does not store allocation sizes in the allocated block; all allocations on a given page are the same size. (That's slightly imprecise, the details are available in the documentation.) This can be a big win, but it also can be a memory hog; it means that 40-byte allocations occupy 64 bytes of memory, instead of 48 bytes.