Daniel Rossberg (Mar 06 2026 at 21:25): Daniel Rossberg (Mar 06 2026 at 21:25):Hi @Divyansh Khatri, you should look for an appropriate topic on #Google Summer of Code. E.g., this one for discussion about https://github.com/opencax/GSoC/issues/108.
Divyansh Khatri (Mar 08 2026 at 07:39):Hi @Daniel Rossberg @Sean
I did some digging in the codebase and here's what I understood what's happening and how I plan to fix it for https://github.com/opencax/GSoC/issues/108.
The issues mostly come down to ABI mismatches, memory ownership across DLLs, and a few threading and OSL compatibility problems.
src/liboptical/CMakeLists.txt)What’s happening now:
The current build setup strips RTTI pretty aggressively and defaults to static linking for Boost.
Why that’s a problem:
Modern versions of OSL (1.10 and up) actually depend on RTTI for dynamic casting inside their execution contexts. On top of that, statically linking Boost across different DLL boundaries (like the host executable and the optical library) can trigger ODR violations and even lead to MSVC heap fragmentation.
What we should do instead:
Turn RTTI back on and switch Boost to dynamic linking globally. That way everything shares the same C-Runtime (CRT) and type information across the BRL-CAD ↔ Appleseed boundary.
src/art/brlcadplugin.h & .cpp)What’s happening now:
The BRL-CAD geometry proxy allocates standard library containers on the heap and passes those pointers across the boundary between the host executable and the Appleseed plugin.
Why that’s risky:
On Windows, MSVC keeps heap allocators isolated per module. If the host executable allocates memory and the plugin tries to free it, things can go south fast—usually ending in heap corruption during shutdown.
The fix:
Refactor the proxy class to use value semantics instead of heap pointers. That means embedding the data directly in the object instead of allocating it dynamically. This keeps the entire memory lifecycle inside the same module and avoids cross-DLL heap issues.
src/art/brlcadplugin.cpp)What’s happening now:
The BRL-CAD hit callback sends ray intersection data (like distance and normal) back to Appleseed through a global thread_local structure.
Why that’s a problem:
Appleseed runs its own internal thread pool, which is highly concurrent and mostly opaque. If TLS variables were initialized outside that lifecycle, you can get desynchronized state or even memory access violations during rendering.
The fix:
Get rid of the global TLS entirely. Instead, allocate the intersection data struct on the stack and pass its address through BRL-CAD’s application user-data pointer. Since each call gets its own stack frame, you automatically get safe thread isolation.
src/liboptical/render_svc.* & liboslrend.h)What’s happening now:
The BRL-CAD shading backend classes don’t use Windows export macros, and their virtual function overrides are written against older OSL function signatures (for example, passing render state as void*).
Why that causes trouble:
Without export macros, the MSVC linker can’t see the symbols at all.
Newer OSL versions expect strongly typed render-state pointers. If the function signatures don’t match exactly, the C++ vtable layout breaks, which usually ends in a crash during virtual dispatch.
The fix:
Add proper DLL export macros so the classes are visible to the Windows linker.
Add compile-time checks for the OSL version and switch the virtual function signatures accordingly.
This ensures the signatures match perfectly and keeps the vtable layout intact.
Last updated: Mar 20 2026 at 01:12 UTC