that only works for byte blobs. ... anything further has to have some semantic agreements to work
No, it works for arbitrary multidimensional arrays. Semantic agreement for how to represent multidimensional numerical arrays has existed for over half a century: a 1D typed arrays together with a concrete representation of collections of bounds, offsets, and strides. BLAS, PIL, LINPACK, NumPy, OpenCV, ImageMagick, and thousands of other libraries and tools use this representation. And because they all use the same representation in memory, they can all easily talk to each other.
This discussion makes me think that perhaps a proposal for a standard descriptor like "Py_buffer" might be better after all than a proposal for a typed 1D array, because people might understand better what they are supposed to use it for.
Tom