Re: Shared libraries

[Jay Carlson <nop@...>]

On May 13, 2014, at 4:48 PM, Coda Highland <chighland@gmail.com> wrote:

> On Tue, May 13, 2014 at 1:27 PM, hasufell <hasufell@gentoo.org> wrote:
>> I'm confused why this thread gets ignored.
>> 
>> You probably don't realize that distros randomly fix this and cause more
>> of a mess for lua users, because there is no consistency whatsoever.

An advance apology: please consider "upstream" to be jargon talking about a role from the perspective of a different role. I've used this terminology before on this list, and have later regretted the tone; I think it over-personalizes the situation, especially here. I don't know how else to describe the relationships, though. Also, I have never done any official Debian anything, so I can't be speaking for them.

My self-centric recap of the situation for people not wanting to read through decades of mailing list posts:

In Lua 5.0--remember, this is 2003--the upstream toplevel Lua Makefile produced a shared library pwhich set off automated QA warnings in many package systems and in many paranoid Unix build engineers from the early 1990s.[1]

Debian half-fixed this. First, "-fPIC" isn't just a good idea for shared libraries on some architectures--it's THE LAW. Without "-fPIC", the package would not build on many architectures. On i386, you still needed a good excuse for why you weren't using -fPIC code, and it was easier to just the -fPIC flag there too. Debian followed upstream in exporting symbols from each executable and leaving fully unresolved symbols in dynamically loaded modules.
air 
In Lua 5.1, upstream removed shared library builds, no doubt tired of people like me offering multiple vaguely overlapping, vaguely contradictory suggestions. Debian's lua5.1 then treats the shared library as any other library, with a little dance about how you should still continue the "depend on symbols you don't have a path for" eccentricity encouraged by upstream. Things are cleverly engineered so 

Debian suggests that anybody linking Lua inside the distro do so dynamically. IIRC this is more bendable than ABI violations, which need a good excuse. (There were some i386 libraries which had non-PIC assembly code.) But linking static libraries is prone to repeated security NMUs in stable, which annoys the wrong people. 

If your distro or organization can't decide or enforce a process like "pick one: link dynamic, give me source, or update monthly," you probably have more pressing QA problems. Sometimes, cleaning up after the repeated remote break-ins. Lua can't fix your management problems.

It Would Be Nice If the upstream 5.0 Makefile had not contained a red herring--I would call it a bad example. I am not sure I helped by requesting the Makefile compile with "-fpic" instead of "-fPIC", because this is precisely the kind of arcane build engineer nit-picking which makes reasonable people decide that build engineers cannot ever be satisfied. I got out of this business before pthreads became an issue, and just last month pthreads ABI compatibility popped up on *Solaris*. Which, mea culpa, I should have noticed before on general principles.

Anyway, there is a *lot* of software where upstream has a different opinion about packaging than build engineers. Upstream can work on what it wants to work on, which usually does not involve a lot of obsession about rigid rules like "-fpic" vs "-fPIC"--which has a small effect on performance for 0.05% of users.

>> Not fixing this also breaks compatibility with C# programs under linux
>> that cannot link to the static lib (you don't want to link it statically
>> into mono).

Which distro packages Mono and doesn't package Lua at least as well or better? Wait, I don't care. :-)

> On Windows and OSX, the behavior of a static library is exactly what
> you DO want. You don't want to trust that the shared libraries are
> going to be compatible, because you don't have any promises -- there's
> no package tree maintainer to put trust in.

I thought the situation on Windows was that there was essentially only one DLL per link-compatible compiler. 

> If you're worried about hardening your application by sandboxing Lua,
> the behavior of a static library is desirable, to prevent bypassing
> the sandbox by hijacking the dynamic loader or by replacing the
> library without the application's knowledge.

"It sucks to use a dynamic linker that sucks." 

I dunno. If I have an executable in my hand which is misbehaving, not having LD_PRELOAD is not going to stop me from fixing it.[2] The question is more like, what will stop me from fixing your server's executable. If you're not linking libc and everything else static, you're still going to have all those juicy GOT indirections lying around, and you gave up ASLR to get it. I haven't been paying attention to the ABI-level warfare lately though.

Jay

[1]: The Makefile is not quite right when it says "END OF USER SETTINGS -- NO NEED TO CHANGE ANYTHING BELOW THIS LINE". But if you're building shared libraries from source on Unix boxes, you don't pay any attention to what *any* Makefile says if you survived the 1990s as a build engineer. After "-fPIC" the next most important thing to do in shared libraries is link with your current edition of "-Wl,-z,defs" and no, that's not portable, nor was it well-documented outside places like http://www.debian.org/doc/debian-policy/ch-files.html#s-libraries .

[2]: As far as misbehaving executables go, it *is* more pleasant to deal with LD_PRELOAD than to write your own ld.so or to short out system calls. I was going to mention a vendor here, but why antagonize? It's not worth blogging about unless it escalates.