starseeker (Mar 03 2023 at 15:08): starseeker (Mar 03 2023 at 15:08):Not for immediate action, but thought I'd start a thread to accumulate various thoughts and resources about git submodules and other potential solutions to moving src/other/ext out of the primary tree. Prompted by coming across this: https://diziet.dreamwidth.org/14666.html
starseeker (Mar 03 2023 at 15:14):https://github.com/ingydotnet/git-subrepo
Erik (Mar 09 2023 at 21:09):if you don't check out the initial version with submodules enabled, it can be a bit of a pita from what I recall
Sean (Mar 14 2023 at 05:56):Instead of submodules, my thought was either bit the bullet to get Conan working or more easily, use CMake itself:
1) have our deps live in another BRL-CAD repo where we manually sync/pull from upstream as they're verified,
2) write a .cmake module using FetchContent to pull the other repo(s),
3) add logic to the module so it can optionally report when a dep is out-of-date (if not already a feature of FetchContent),
4) modify our CMakeLists.txt to conditionally fetch if it's not been fetched before, and lastly
5) make sure we have a cmake incantation, script, or other 1-liner to prefetch for offline use.
Source tarballs would of course include prefetched and ideally have the out-of-date check disabled/advisory so they just work. Will also probably want some safeguard from having git blather and avoid accidental check-ins (probably fetch into build tree). On the upside, that approach should extend nicely and doesn't inject configuration code outside of cmake (error prone, build system modularity weakness). Think it avoids the downsides of submodules and conan, but will have to see it in practice to know for sure.
If that's something you think we could try, might be a good idea if someone else took a swing at it if only to increase familiarity with the build system. Happy to give it a go.
starseeker (Mar 14 2023 at 10:35):Definitely sounds like something to try. I'm certainly good with you having a crack at it @Sean - I agree expanding familiarity with the build system is a good idea.
starseeker (Jul 26 2023 at 00:32):Looking at the Qt6 build and how find_package(Qt6) works, I'm almost certain I won't be able to spoof find_package(Qt6) the way I do for the other src/other/ext packages in order to integrate them into the main BRL-CAD build. That brings me back to the question of how we want to go about integrating an extracted src/other/ext into the main BRL-CAD build - with Qt and anything using a similar target-defining approach to find_package, we're going to basically have to have the compiled, installed 3rd party dep to point to before our configure calls find_package.
starseeker (Jul 26 2023 at 00:34):That's what's sometimes referred to as a "superbuild" pattern - a meta build manages the order of project building, and then the "main" project build is the last one triggered by the superbuild.
starseeker (Jul 26 2023 at 00:51):I explored that a little bit early on in the last round of build system fun, but if we go with that route we lose a number of things our current approach is supposed to guarantee:
Being buildable and runnable with no access to the targeted install directory. The third party builds will have to complete their final install step before we even being configuring.
Having our build automatically bundle up the relevant elements of the third party dependencies into our CPack outputs. Essentially, the problem I'm currently looking at of how to bundle the required elements of Qt for Qt executables would be the same issue we would have with all ext third party deps.
Our build system would cease to have explicit origin knowledge of every generated output file that ended up in the final build if we glob in the 3rd party build outputs - that may have implications we want to consider.
Relocatability of the binary install tree - our current approach does some work to make sure external libraries are relocatable the way our own are (TLDR - rpath related, can discuss more in detail if necessary.) I'm not 100% sure what we'd do about creating binary BRL-CAD installers in the "ext build dumped 3rd party outputs into the install tree" scenario, so I can't say yet for certain what the options would be, but it'd most likely be a challenge.
I'm not necessarily against embracing some loss of relocability in order to achieve a different (and hopefully simpler) build setup where the BRL-CAD 'main' build just does find_package(*** REQUIRED) for everything, but before doing any extensive explorations I'd like to have some sense of what constraints we're going to try and satisfy...
starseeker (Aug 02 2023 at 14:27):On a more mundane level, I have a question about system vs. bundled libs and a separate "pre-built" externals repository. If we (for example) build the bundled libpng in the externals repo using the bundled zlib, it may be impractical for the BRL-CAD build to subsequently want to use a system zlib and the "bundled" libpng, as the latter will be built against the bundled version of zlib.
starseeker (Aug 02 2023 at 14:28):Right now this is handled at BRL-CAD's CMake configure time by specifying what we want to do before both zlib and libpng are built, but if we are pre-building an external repository we will lose the ability to mix and match like that as part of the BRL-CAD configure process - all those decisions will have to be made at the time the external repository is built.
starseeker (Aug 02 2023 at 14:29):If we go that route, it seems to me like the BRL-CAD logic should first check the targeted bundled directory to see if any given find_package target is present, and always use it if found?
starseeker (Aug 02 2023 at 14:30):That way the system/bundled decision shifts entirely to the ext repository's configure/build cycle, and BRL-CAD either uses what's in bundled or requires the system version, based only on a detection check.
Last updated: Oct 09 2024 at 00:44 UTC