Here's what I'm seeing if I install the whole thing from scratch for lua 5.3 specifically. The include directories for the lua headers are correctly versioned to 5.3, but I'm not seeing it linked explicitly with any particular liblua at build time, so it should use the default for the interpreter that loads it (5.3).
What I would guess is happening, based on the -L and -rpath options that get set, is it's finding the system default /usr/lib/libreadline.dylib before the one under /opt/local/lib, and it doesn't match what libhistory is expecting (which isn't installed at all under /usr). But what I don't understand is why it works under 5.2, which uses the same -L and -rpath directives. Thoughts?
$ sudo luarocks-5.3 install luaprompt HISTORY_DIR=/opt/local
export MACOSX_DEPLOYMENT_TARGET=10.5; gcc -O2 -fPIC -I/usr/local/include/lua5.3 -c prompt.c -o prompt.o -DHAVE_LIBREADLINE -DHAVE_READLINE_HISTORY -DHAVE_IOCTL -DCOMPLETE_KEYWORDS -DCOMPLETE_TABLE_KEYS -DCOMPLETE_FILE_NAMES -DCOMPLETE_MODULES -DSAVE_RESULTS -DRESULTS_TABLE_NAME="_" -I/usr/include -I/opt/local/include
export MACOSX_DEPLOYMENT_TARGET=10.5; gcc -O2 -fPIC -I/usr/local/include/lua5.3 -c module.c -o module.o -DHAVE_LIBREADLINE -DHAVE_READLINE_HISTORY -DHAVE_IOCTL -DCOMPLETE_KEYWORDS -DCOMPLETE_TABLE_KEYS -DCOMPLETE_FILE_NAMES -DCOMPLETE_MODULES -DSAVE_RESULTS -DRESULTS_TABLE_NAME="_" -I/usr/include -I/opt/local/include
export MACOSX_DEPLOYMENT_TARGET=10.5; gcc -bundle -undefined dynamic_lookup -all_load -o prompt.so -L/usr/local/lib prompt.o module.o -L/usr/lib -L/opt/local/lib -Wl,-rpath,/usr/lib: -Wl,-rpath,/opt/local/lib: -lreadline -lhistory
Updating manifest for /usr/local/lib/luarocks/rocks-5.3
luaprompt 0.6-1 is now built and installed in /usr/local (license: MIT/X11)
$ cat (which luap)
#!/bin/sh
exec '/usr/local/bin/lua5.3' -e 'package.path="/Users/prime/.luarocks/share/lua/5.3/?.lua;/Users/prime/.luarocks/share/lua/5.3/?/init.lua;/usr/local/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?/init.lua;"..package.path; package.cpath="/Users/prime/.luarocks/lib/lua/5.3/?.so;/usr/local/lib/lua/5.3/?.so;"..package.cpath' -e 'local k,l,_=pcall(require,"luarocks.loader") _=k and l.add_context("luaprompt","0.6-1")' '/usr/local/lib/luarocks/rocks-5.3/luaprompt/0.6-1/bin/luap' "$@"
$ luap
/usr/local/bin/lua5.3: /usr/local/share/lua/5.3/luarocks/loader.lua:117: error loading module 'prompt' from file '/usr/local/lib/lua/5.3/prompt.so':
dlopen(/usr/local/lib/lua/5.3/prompt.so, 6): Symbol not found: _rl_completion_suppress_append
Referenced from: /usr/local/lib/lua/5.3/prompt.so
Expected in: dynamic lookup
stack traceback:
[C]: in local 'a_loader'
/usr/local/share/lua/5.3/luarocks/loader.lua:117: in function </usr/local/share/lua/5.3/luarocks/loader.lua:114>
(...tail calls...)
[C]: in function 'require'
/usr/local/lib/luarocks/rocks-5.3/luaprompt/0.6-1/bin/luap:29: in main chunk
[C]: in ?