An MCP Server for BRLCAD (Natural Language CAD) · brlcad

I've been playing around with BRL-CAD lately on my setup and started building something I thought you guys might find interesting.

I have put together a working local prototype of an MCP (Model Context Protocol) server for BRL-CAD. Right now, it uses a non-blocking Tcl socket bridge to pipe geometry commands into a live MGED session. I hooked it up to an LLM via LangGraph, so I can literally just type something like "make a 15mm cylinder and subtract it from that sphere," and the BRL-CAD GUI updates live with the boolean math. (Right now I've been playing with boolean operations and spheres, but could definitely index more tools with time).

I know one of the long-term goals for BRL-CAD is improving the UX and lowering the learning curve. Letting users interact with the CSG engine using natural language feels like a really fun way to tackle that. Also, I noticed that tools like FreeCAD and OpenSCAD already have active MCP integrations out there, but BRL-CAD doesn't seem to have one yet. It feels like a missed opportunity to get BRL-CAD plugged into the new AI agent ecosystem.

I'm really interested in applying for GSoC this year. I know an MCP integration isn't currently listed on the official problem statements, but would the mentors be open to considering a proposal around this? My goal for the summer would be to turn this Tcl socket prototype into a production-grade interface, perhaps integrated with the main GUI as a model-agnostic MCP interface with a BYOK system.

Sean (Feb 24 2026 at 20:27):

Hi @Raghav Sharma and thanks for the outstanding introduction. Short answer is yes.

Longer answer is that is something that I’ve been actively working and thinking about, albeit with manual scaffolding to explore how practical and effective it can be. So yes you should definitely submit your ideas and we should continue to discuss. The potential is tremendous for UX but also as a means for discovery and productivity. The details of how and what skills/tools/features and data integrity will need to be developed as well.

Raghav Sharma (Feb 24 2026 at 20:32):

I'd love to discuss the tools and processes with a mentor while I continue to build upon this scaffold and before I start drafting a proposal. Things like integrations with the mged CLI/GUI window, memory persistence and a priority list for tool integration. Would it be possible to perhaps have a call some time?

Raghav Sharma (Feb 27 2026 at 19:58):

I've been building upon this in the meanwhile. I have added support for Sphere, Cylinder and Box creation along with basic boolean operations

I have also added a soft fallback to the user in case of missing parameters (If i ask the AI to make a cylinder of radius 5, without specifying the height or the location, it would give me a message asking me to specify the missing parameters before sending it to the socket bridge)
image.png

Raghav Sharma (Feb 27 2026 at 20:01):

I am also handling command formatting in the MCP layer itself so that there are no weird hallucinated commands going to the editor. (though I need to add some sort of try-catch validation here for invalid values coming from the LLM before they hit the editor, or to evaluate errors raised by the editor after the command goes through)

Sean (Mar 01 2026 at 22:23):

@Raghav Sharma I would love to set up a call to talk in more depth. We have held open meetings with all contributors before and may do that again this year as well. It really though often boils down to motivation, having a plan, and determination. If you’re excited about what you’re doing others often will be excited also.

Sean (Mar 01 2026 at 22:25):

For mcp integration, I wonder if you could make a tool robust enough to discover how it needs to issue commands, maybe by querying the help system or running commands like “in” interactively where it describes each.

Sean (Mar 01 2026 at 22:25):

Sean (Mar 01 2026 at 22:26):

Having an error handler sounds like a great idea regardless so it can hopefully recover given expected situations.

Raghav Sharma (Mar 02 2026 at 00:07):

@Sean That sounds great! I really am excited to build this and I am in the process of drafting the technical architecture for this. Please let me know whenever you'd be available for it, I'd love to hop on some time to discuss the details and make sure it aligns with BRLCAD's goals.

Raghav Sharma (Mar 02 2026 at 00:17):

I was initially thinking of building a standard MCP for this (ones the likes of slack or postgres currently use, with static defined tools) but something like that might not work with BRLCAD as it has a large number (around 400+) commands that would have to be indexed that way (each having different parameters too) which would be a pain to index manually and would absolutely blow up the context window of the LLMs

So I was in the process of redefining it as a Dynamic MCP with a discovery and error handling API to actively fetch context from terminal responses for tool selection, query formatting and errors being thrown.

MCPs with requirements like this rely on a ReAct (Reason + Act) sequence which is actually pretty achievable (with our Agent ingesting man pages on the fly to understand commands in runtime, or using ls to query existing elements in the database). I'll try to get a prototype of this approach before our call.

Raghav Sharma (Mar 02 2026 at 00:24):

We could also do some visual self-correction stuff with VLMs (Visual LLM Models) too, with the Agent actually getting visual context. The agent could just run rt commands to output PNGs in different camera angles to visually verify things like boolean ops and then self-correct the coordinates if needed

Though it would be more of an experimental thing we could try after building the base MCP

Raghav Sharma (Mar 04 2026 at 16:05):

successfully built a prototype where the agent finds the commands it needs to execute via a resource sheet and can automatically query man pages for their usage

the agent first checks the query, goes through a bunch of predefined tools (that I had to define for some tougher syntactic or sequential operations like boolean ops)

Then if the agent does not find them in predefined tools, it loads up a resource (imagine a cheatsheet) with a list of available commands with 1 liner descriptions

if the agent sees something relevant, it opens up the man page for that command, learns how to execute it, and then runs it in the CLI

If there is any error, it goes back to the agent via the socket, the agent reads the error and then corrects the method to run the operation (Done in an incremental loop upto a set number of executions, in this case the limit is set to 5, just so we don't run into a recursive infinite loop)

(In this case, an error occured, with the LLM seeing that a sphere of the name sphere.s already exists, so it went back and changed the name and re-executed the command)

Raghav Sharma (Mar 04 2026 at 16:06):

Its pretty rough around the edges right now, but I could certainly refine it across the GSoC developement period

Raghav Sharma (Mar 04 2026 at 16:33):

You can also ask about any commands
It will automatically query the help pages and tell you about it

Raghav Sharma (Mar 04 2026 at 17:01):

An example where the agent finds the necessary command, reads its documentation, and executes it autonomously while also answering a user question about the file location

Sean (Mar 09 2026 at 03:55):

@Raghav Sharma Definitely showing some promise in a quick span of time. What would be good to articulate in your proposal is just how far would you plan on taking it? What sorts of user stories do you envision achieving? What about possible RAGification of the tutorials or curated examples of the mcp tools being defined so it has various patterns to leverage? Would it be able to be an instructive agent that helps teach as it works? What sort of and how much base knowledge would be good to have defined in advance? What about making it work offline usably well enough with a local model? Or via some online service, but without consuming credits? Just a few questions like that to think about and help scope what it would and would not attempt to achieve under GSoC. Obviously can't do everything but could probably get a lot done at the pace you're demonstrating. Think about how to make it deployable to users. Think about what tech stack changes are assumed. Think about multiple platforms. Think about maintainability/extensibility, etc.

Володимир (Mar 09 2026 at 11:23):

Raghav Sharma (Mar 21 2026 at 13:30):

Thank you so much for the detailed feedback. The questions you raised around RAG, offline usability, user stories, and deployability have been incredibly helpful for shaping the scope of the proposal. I've taken all of it into account while drafting.

Would really appreciate it if you could give it a read and let me know if the direction and scope feel aligned with BRL-CAD's goals, and if there's anything I should expand on or cut down before submission. Also while submitting it on the GSoC portal, should I put it as a medium or large project?

Sean (Mar 22 2026 at 14:59):

@Raghav Sharma This looks good, but obviously heavily planned out by AI which isn't a problem per se but calls into question several aspects that need to be considered. One is the scope is very broad and extensive. While this may be possible, there is lots of baked in assumptions about how things are going to work. For example, I'm unclear on the uplevel tcl injections being feasible or even functional across all command modalities. That said, it's an implementation detail that didn't have to be mentioned and the actual result could be discovered. The risk is that the summer ends with a broad but shallow prototype. A stronger proposal should be described incrementally or narrow success criteria sharply. That doesn't mean simply proposing less, it's about the software engineering being intentional and having room to expect integration problems, bugs, development iterations, etc.

What I would like to see is a few examples in your own words on what types of interesting prompts that are BRL-CAD-specific are going to be the goal. You mention golden tools, but that's so general and vague that it leaves much to the imagination. Basically, show a few specific user stories of something they're attempting to achieve -- and not discuss the solution or the "how" but identify the "what" as a target set of examples. The paper goes into great detail on how but very little is specific to BRL-CAD and what it might mean for users and workflows. If this is integration in MGED, what is the specific nature of the integration (from a user perspective) -- i.e., where is this cursor agent going to reside? Another panel? A command? A menu tool?

The weakest use case is the ambitious but general “generate/manipulate/analyze complex 3D geometry from natural language” promise. That is where ambiguity, hallucination, partial failure, and semantic mismatch are high risk. Your proposal leans heavily on that vision without evidence or measurement activity baked in to ensure it'll be or can be made trustworthy within scope/time.

Architecturally, a big one is you're proposing introducing Python as a new requirement to BRL-CAD. That's a major change and needs to be discussed and justified. How's that integration going to look from a software engineering perspective? Are you going to address the necessary changes to "bext" to support it? You mention json parsing several times -- what's doing that parsing and how is that dependency managed? I'll note we have a couple options already, but you don't mention them.

Lastly, would be very helpful to hear your thoughts (not AIs) on MGED vs Archer vs Arbalest vs future Qt deployments. You can't and shouldn't target them all, but should have done enough research to be aware of them and how your work will transition into a bigger picture or be reusable as we continue to evolve our UX.

Raghav Sharma (Mar 22 2026 at 15:51):

Hi @Sean
Yes some of the details were based on assumptions and some were a result of experimentations I had experienced while working on the Tcl socket based prototype. I didn't run into any problems while building it using Tcl uplevel injections, but I wasn't aware that there would be any functional limitations with modalities, maybe we could explore some other way to interface the MCP to it, perhaps a C++ socket, closer to the geometry engine

Also if the "generate/manipulate/analyse complex 3D geometry" thing seems a little too ambitious for this summer, would a more analytics/ repetition/ database analysis tool (eg: finding objects with unassigned materials, or batch automation of some repetitive commands) be aligned with the organisation? It would be a lot less prone to hallucinate and less risky, though I'd need some user input from people using the software quite often to actually know what parts they'd need automated. I was also thinking of having transactional actions for this to reduce the risk of partial failure (Someone in GSoC 2021 had apparently implemented transactions and undo features in BRL-CAD but the wiki seemed to be down so I couldn't read about it in depth)

For the python thing, I wasn't planning on having the MCP coming strapped in along with the BRL-CAD codebase (as mentioned in the proposal). Instead I thought of it as a decoupled extension the user could optionally clone and integrate locally with their own agent and use with BRL-CAD. So the parsing stuff would actually be stuff managed between the agent and the MCP. Because Python handles the JSON, BRL-CAD doesn't need a new C-dependency.

For the MGED vs Archer vs Arbalest vs future Qt stuff I had read about them and the direction BRL-CAD is going in with MOOSE and Qt. For this summer I had initially planned for going with Tcl uplevel injections (compatible with MGED and Archer) as they were fairly straight forward places to test the core AI logic and database interactions. As the heavy lifting for the MCP is done in the decoupled Python extension for now (the LLM routing, tool discovery, error recovery etc) and that architecture would inherently stay future proof. When MOOSE does become mature though we'd probably need to switch away from the Tcl thing.

Now that I think about it, a C++ socket listener hooking in closer to the geometry engine would be a better idea than Tcl overall as it would be even more future proof. I'll do some reading into this, thanks a lot for the feedback again Sean

Raghav Sharma (Mar 23 2026 at 16:39):

Just a follow up, I explored how commands could be piped through both MOOSE and the current MGED/archer exec pipelines and did some more reading into how I could use existing libraries and make this more future proof.

I am thinking of going for a C++ based socket for piping in commands so its more future proof for MOOSE (and has minimal friction to port once it matures)

I am thinking of having a C++ socket that receives strings coming in from the Python FastMCP server (flat commands like 'in sph1 sph 0 0 0 5' with no json). This could be caught by the C++ listener and then broken down by bu_argv_from_string from libbu to break it into argc/argv strings safely

This above part can be the same for both MGED/Archer and the upcoming arbalest as both of them have the ability to run argc/argv commands for backwards compatibility with the rest of the BRLCAD libraries. (for example if i am running a libged command, in MGED/Archer it could be using ged_exec and in MOOSE it could be using CommandString::Parse)

To avoid complex mutex locking and race conditions, the socket itself could be watched by the main thread's event loop. For Phase 1 (MGED/Archer), this means registering the socket file descriptor with Tcl_CreateFileHandler. For Arbalest/MOOSE later, some one can just swap that Tcl hook for a Qt event loop hook. (This synchronous approach feels a lot safer than attempting background thread execution, which would require heavy file-locking, though I'm not sure if there would be any limitations with any particular command modalities in this case as there was with Tcl uplevel injections so please let me know if there are).

The pipeline would be:
Event Loop Trigger -> Read Socket -> Parse (libbu) -> Execute

Also on a side note, I think that proposing a C++ socket connection that could actually enable the user to safely pipe commands over an exposed port could be very impactful even without an MCP directly attached in the particular proposal, as it could actually lead to users building many different web based or local extensions to BRL-CAD that could be able to execute commands safely. While building the MCP I kind of wished there was one, where I could just run a script and be able to send commands to the software via a port. Would it be a good idea to propose that socket connection itself as a proposal considering the future possibilities in extensions/plugins and MCPs it could unlock?

I remember neovim had a bunch of similar projects last year, focusing more on building the functionality to support more AI integrations for the future in a couple of projects (especially the one that had built primitives to streamline AI plugin development)
https://summerofcode.withgoogle.com/archive/2025/organizations/neovim

I am thinking of making the initial decoupled MCP extension more specific with lighter automation commands, but more focus towards actually building an extension agnostic C++ socket pipeline to run those commands. It would be focused more building the infrastructure required to support future extensions and MCPs in BRLCAD and demonstrating its capabilities with a lighter more specific MCP.

Raghav Sharma (Mar 29 2026 at 19:20):

@Sean I had gone back and made some architectural changes along with the inclusion of more relevant user stories. The focus being more on the establishment of a secure pipeline to support future extensions for BRL-CAD via a C++ socket API along with the MCP Server.

Would really appreciate it if you could give it a read and let me know if the direction and scope feel aligned with BRL-CAD's goals

Sean (Mar 30 2026 at 05:46):

@Raghav Sharma just a caution, a tcp/ip socket is critically problematic for our corporate uses as it will get blocked by security policies or trigger anti-virus flagging or simply be unavailable. pipes (stdio) are typically okay, along with other non-ip based IPC.

That's not to say a tcp socket can't be made to work, but then you'll definitely need to carve out a couple weeks to ensure you're registered correctly (cross-platform) and that it's not triggered/blocked, etc. It may be sufficient to target configuration settings that avoid the potential issues (like limiting to ports >= 1024 if that's enough, don't know).

Raghav Sharma (Mar 30 2026 at 15:13):

I will update the proposal to use a stdio based approach for the primary transport instead of TCP sockets. Perhaps we could experiment with the target configuration settings for TCP later on after the stdio thing is done.

Raghav Sharma (May 01 2026 at 03:42):

@Sean Thanks a lot for considering this proposal into GSOC'26. I am very excited to work on this.

Could you please let me know when we could perhaps have a meeting and discuss a plan for this summer? I really look forward to working on this

Sean (May 01 2026 at 16:37):

@Raghav Sharma we’re excited to be working with you and our other contributors also! There was a lot of tough decisions made this GSoC and a couple more we would have accepted if we had been able to get more slots. You however did and continue to do a great job with community engagement and communication so please do keep it up!

Lots to discuss but we have a little bit of time to get set up. As far meeting synchronous, just need to touch base with the other mentors but likely not in the next couple days, sometime next week/weekend possibly unless the chips align quickly beforehand. Will get back to you on that.

Raghav Sharma (May 01 2026 at 22:52):

Raghav Sharma (May 04 2026 at 10:03):

While doing some dives into the stdio pipeline and libbu for Phase 1, I caught a potential architectural roadblock regarding cross-platform IPC that I wanted to get your thoughts on.

While the POSIX implementation for Linux is straightforward, it looks like Windows handles anonymous pipes very differently. Specifically, it seems the Windows implementation of Tcl's event loop could struggle to monitor standard anonymous pipes natively without blocking. (Seems that Tcl_CreateFileHandler is only supported on Unix. this could be from how windows handles system calls differently from POSIX. Windows is basically blind to pipe data but can receive via TCP)

To ensure I hit the Week 3 and 4 milestones reliably, would you be open to me strictly targeting the Linux/POSIX pipeline first to prove the core C++ listener and the event loop hook? once the architecture is proven stable on Linux, we could use the buffer weeks to explore the Windows port (perhaps using libbu macros or revisiting a local-only TCP configuration if corporate security allows port > 1024).

Or would it be a better idea to start with local-only TCP configuration with some experimental validation on the allowed port numbers? If that does end up being viable for corporate users we could have a fairly reliable crossplatform solution

Sean (May 05 2026 at 04:04):

@Raghav Sharma there was some new cross-platform IPC API added to libbu just a few days ago. I believe it's in a branch, but potentially helpful for what you described.

Sean (May 05 2026 at 04:05):

I think IPC is definitely the way to go and preferred over TCP since that will trigger security watch tools.

Sean (May 05 2026 at 04:07):

look for a branch related to remrt/fbserv refactoring to see the interface and how it was wired into existing tools. that should give a good direction for how to integrate for an mcp server (or editor gui).

Raghav Sharma (May 05 2026 at 06:11):

Found it
it seems to be on the rtwizard branch, commit 072efe3. Just going through pkg_transport.cpp right now. Thank you for the direction!

Is the rtwizard branch intended to merge to main before GSoC starts, or will I be developing against it directly? Also would the BRL-CAD-side IPC wiring for the MCP server target main or a feature branch during the summer?

starseeker (May 06 2026 at 11:25):

I merged the rtwizard work into main a couple days back. The IPC part is primarily in libpkg, rather than libbu.

Raghav Sharma (May 07 2026 at 05:11):

Hi again
so while digging into the spawn architecture for the MCP listener I had a question about bu_process_create.

My use case needs the Python subprocess to inherit the pkg pipe fds from pkg_pair() but I noticed the child process has a close loop for (int i = 3; i < 20; i++) close(i) before execvp, and on Windows it uses STARTUPINFO rather than STARTUPINFOEX with PROC_THREAD_ATTRIBUTE_HANDLE_LIST.

Would those prevent the pkg fds from surviving into the Python process? Or is there an existing pattern in the codebase I should be following for this kind of fd inheritance? Trying to figure out whether I need a custom spawn wrapper scoped to the MCP extension or if there's already a clean way to handle this.

Raghav Sharma (May 07 2026 at 05:19):

existing uses of bu_process_create in the codebase communicate via the stdio redirects (fd 0/1/2), so the close loop is fine for them. My case is different since I need extra pkg fds to survive beyond stdio as I want my child process to be talking to both the LLM client (over stdio) and then BRL-CAD (an additional pkg pair)

Raghav Sharma (May 08 2026 at 02:26):

@Sean @starseeker
would there be any sort of unix domains/named pipes solution that I could go for? libpkg in the current state doesn't seem to support them. Would that be a viable option to go for to build the listener?

There seems to be an issue with anonymous pipes right now. I could start an MCP server with BRL-CAD as a child process and that child could have bidirection communication with BRL-CAD (and an LLM client, considering we solve the problem with an additional fd mentioned above)

The problem would be that VS code or Cursor or any other existing IDE clients would not be able to connect to the application (connections to existing MCP servers is an HTTP only thing, and we're avoiding TCP now)

So the architecture would flip a little bit to accommodate this, by having the LLM client start the MCP and they talk to eachother via stdio (which is fairly standard).

Now this MCP server would need an address to talk to my BRL-CAD listener, and that could be possible via named pipes or Unix Domains (I could essentially pass it to my MCP as an API endpoint and they could communicate)

I can't seem to find anything in libpkg for named pipes or unix pipes but there does seem to be implementations with some TCP stuff going on.

Would there be any existing solution within the libraries for this sort of peer-to-peer communication without TCP

Sean (May 14 2026 at 20:44):

@Raghav Sharma that close loop doesn't close stdin/out/err -- it closes higher level sockets if any. That said, even that is not terribly well handled and can be changed if it's causing a problem. Same can be said of STARUPINFO and other implementation details in libpkg -- they're not immutable libraries. We change them as requirements evolve and changes are justified.

That said, I don't think the fds being closed will affect a python subprocess simply because the standard ones are still there (one could talk on out, the other on err, no?). If that's not the case, then lets consider changing libbu's guts.

Sean (May 14 2026 at 20:47):

pkg now supports sockets with his latest changes, so that shouldn't be a problem. they're not named pipes but you shouldn't need that I'd think.. If you really want to go the named socket route, I'd look into making a libpkg modification to support it cleanly that has API that works identically as the tcp path and socket ipc path.

Raghav Sharma (May 17 2026 at 06:18):

Got it. I'll try experimenting with the current inventory with a dummy server once and then decide upon the direction.

Also in the case that there does seem to be a need for named sockets, honestly speaking, I have no Windows dev experience, would it be okay to start with the Unix socket implementation on Linux first and treat Windows named pipe / TCP fallback as a later milestone? Just ant to make sure I'm not blocked on platform setup before coding period starts.

Raghav Sharma (May 17 2026 at 06:21):

Getting this deep into IPC was not something I had initially expected while submitting this idea initially (as an Agentic AI proposal) but it does seem to be quite interesting. Thanks a lot for being this patient with me

Sean (May 18 2026 at 15:00):

@Raghav Sharma I'd want to get a lot more specific into your code about why specifically named pipes are needed versus the other IPC mechanisms since that would introduce an entire new platform-specific portability concern. that said, libpkg is fair game to only use/test on one platform and not worry about whether it's behaving on other environments. similarly, we could consider some existing wrapper library like asio, libasync, or maybe even libuv, but that would really be after ruling out using or modding pkg as a viable path. It's such a simple library that it should be able to do what you need, I'd think. Still, can be and should be a discussion.

Sean (May 18 2026 at 15:01):

you can also always start with tcp, but not lose track that something more performant ipc is needed before the end.

Raghav Sharma (May 24 2026 at 07:49):

The reason why the IPC stuff is a concern here because when we're trying to integrate our BRL-CAD listener with an MCP, where usually the MCP is instantiated by the client (could be VS code, cursor or a custom client)

So its an independent process to process communication (which means that the current IPC mechanisms within libpkg, which rely on inheriting passed fds from child processes) might not suffice from my understanding (Though I could be missing something here)

I'd basically need the capability of 2 independent processes (BRLCAD and the MCP) finding each other on a system.
Thats currenly not possible with libpkg (except for the TCP part in it) as it does not have any spawn independent IPC mechanisms so something like a unix domain socket would solve it (right now theres a handler in libpkg but that also expects a passed fd to talk to a child process)

I am currently scaffolding the adapter in a source build for the libbu argc/argv splitter and the ged execution part first with a basic TCP based approach first and then exploring some other ipc before the end of summer (only the IPC would need to be swapped out, with the rest of the chain being the same)

Perhaps we could even provide both the TCP based and a local IPC based solution as alternatives/options for future extensions

Raghav Sharma (May 24 2026 at 07:53):

Where would the adapter for the MCP go? Shall I make a new library called libmcp or something? or is there a spot where auxiliary extensions like this usually go to? Also could I get a feature branch for this where I could send my PRs? (considering only the listener/parser is a part of the actual BRL-CAD source code, and the actual python MCP is a decoupled repository)

Sean (May 30 2026 at 15:28):

@Raghav Sharma I think we're getting on the same page of understanding. I'm expecting the MCP would be instantiated by MGED (i.e., it's the client). This means really any IPC can be viable including libpkg's mechanism. Another easy option available to you is QProcess since Qt is also fair game as an already-existing dependency we're moving towards -- that is very similar to libpkg and works well cross-platform with named pipes if that's easier that figuring out libpkg. That said, I do think both can be made to work.

For now, go with what is familiar to you and we can look into adapting it later -- sounds like that's the basic TCP-based approach you mentioned, which I think is also fine (for now). TCP will 100% work of course, but it just introduces a whole class of security implications and triggers additional requirements if we want to install this in corporate environments. For example, our research lab use cases require all TCP communication to be authenticated and encrypted, which would mean establishing some sort of PKI handshake and integrating encryption for on-the-wire packets. We could do all that -- it's just a major distraction and pain to get right, imho. I'd be more in favor of spending time adding named pipe support directly to libpkg, or even just directly embedding named pipes + QProcess as the main IPC method before doing all the auth/encrypt stuff.

Sean (May 30 2026 at 15:44):

Great question. I think a library sounds best givne the intent to allow mged (and other apps) to become an mcp or if even to allow apps like it to communicate with mcp servers. That means something like libmcp but I think that's way too generic -- maybe libmcpcad, libcadai, libgedai, ... something like that. Then it'd be src/libgedai and include/gedai.h (and subheaders if any in include/gedai/) for public API.

If the intent is complete standalone server (which I don't think is our goal here?), then it would live in something like src/mcpserv to mirror src/fbserv/.

Pick a name you like and I can stub in a feature branch for it. (doesn't it do that automatically if you just submit a branch PR??)

Raghav Sharma (Jun 03 2026 at 11:35):

got it
Will be sending my first PR soon (with the implementation and necessary unit tests for the parser and executor with TCP for now, will be switching to IPC at later stages)

For the decoupled python MCP server I'll transfer the repository ownership over to BRL-CAD
For that could I be added as a member to the BRL-CAD org on github? Or could you give me an empty repo and I could push to that?

Raghav Sharma (Jun 04 2026 at 06:21):

Raghav Sharma (Jun 04 2026 at 06:24):

@Sean Following up on this, turns out the PR base dropdown only lists branches that already exist in BRL-CAD/brlcad, so a fork PR can't create the feature branch automatically; only maintainers can stub one in.

Meanwhile I've opened a draft PR against main so the work is visible: https://github.com/BRL-CAD/brlcad/pull/238 . It currently has the libmcpcad skeleton + the command parser (libbu-based, CTest'd), with the executor and IPC framing arriving as I commit them.

Whenever you get a chance, could you stub an upstream libmcpcad branch? I'll retarget the PR to it in two clicks. If you'd rather just track the draft against main, that works for me too.

Raghav Sharma (Jun 18 2026 at 08:49):

@Sean I've hit something worth your input while testing the libged exec path through the listener.

Geometry/database commands (in, ls, comb, mv, kill, l, tops, etc.) all work cleanly. They write to ged_result_str, which I capture and send back over the socket. But the help/man commands return a success status with empty output. Traced it: src/libged/help/help.cpp emits via bu_log() rather than ged_result_str, so the actual man-page text goes to the terminal/stdout, not into the result buffer I can capture.

This matters because the discovery/tutor workflow (agent reads a man page to learn a command's syntax before running it) is one of my core user stories, and right now that content never makes it back to the client. It's also the same bu_log vs return-buffer issue I flagged as exploratory for librt diagnostics in the proposal (turns out it affects the help system too, not just librt)

my plan is to add a scoped bu_log capture hook (bu_log_add_hook / bu_log_delete_hook) around ged_exec in the libmcpcad exec layer. Keeps it in the library so any transport benefits, and it's unit-testable.

Raghav Sharma (Jun 28 2026 at 16:34):

Another quick demo. This one shows the analytics path running through the libmcpcad listener (PR #238) via TCP.

In the video, I ask for the spheres in the database, along with their analytics and bounding boxes. The agent handles this by:

As discussed, it's still running on TCP for now. Because the parse/exec/framing layer is transport-agnostic, the next step is to swap the transport to a libpkg named-pipe/QProcess path where MGED spawns the MCP. This won't disrupt the rest of the chain, and the executor and framing updates are being pushed to the PR incrementally.

(On a side note, I'm also waiting on the bu_log/help-capture issue. I'm just waiting on your call for the man-page/tutor path before I start building that out.)

Raghav Sharma (Jun 28 2026 at 16:35):

Raghav Sharma (Jun 28 2026 at 16:51):

also just a quick check-in on planning. Since I'm gonna be back in college next week, I'd want to pace the second half of the project realistically rather than trying to ship everything at once.

Which of these would you consider the highest priority right now? I’ll focus my energy there first and let the rest follow. also whats the procedure for the mid-term review? Should I prepare anything specifically for that?

Sean (Jul 02 2026 at 15:43):

@Raghav Sharma My apologies on the quiet unresponsiveness of late, but looking forward to catching back up with you on your project. Am/Was getting pulled in multiple directions on several fronts (personally and professionally). Definitely appreciate seeing this latest update, though, and think we should fully sync at your next availability (chat or video).

Sean (Jul 02 2026 at 15:47):

Catching up on the questions that haven't been answered, lets deal with the repo ownership later but can definitely add you as a member regardless. I made and pushed that libmcpcad branch for you to target that should be up-to-date.

Sean (Jul 02 2026 at 15:49):

for the bu_log issue, I think your best bet is to register a bu_log callback handler so you get that as a secondary stream in addition to the ged_result_str, and can then deal with output from it however you like. If you look at bu_cmd() API doc in include/bu/cmd.h, there is a little snippet in the comment that shows how to register the callback to catch the output.

Sean (Jul 02 2026 at 15:52):

which is to say, what you suggested is acceptable in this instance -- you'll just hook, stash a copy, and pass it along.

whether it's the long-term direction probably isn't terribly relevant as we're not able to integrate MOOSE/Qt anytime soon for other architecture reasons.

we could fix help/man to go through results, but honestly any command can log and that's potentially useful output to the mcp.

Sean (Jul 02 2026 at 15:57):

great questions. at this point, I'd say lets defer the IPC swap since that doesn't buy us anything new. I'm not entirely sure what you mean by the other two but what I think we need is a showcase demonstration. which is perhaps a combo of discovery and a specific user story. like make a skill/workflow that does one thing that is hard for the human to do but definitely automatable. couple ideas are identifying and fixing glint issues systematically, or creating a showcase rendering with advanced lighting, or routing a pipe from point A to B without introducing any overlaps.

Sean (Jul 02 2026 at 15:59):

for the midterm, I think a current demo write-up is worthwhile. 1-page with a couple images that summarizes everything done so far and an example. your video is close, but super abstract. something concrete on a real model like moving an antenna on the m35 truck model from one side to the other based on the bounding box, or something similar. something that's actually possible but is a real assisted action.

Raghav Sharma (Jul 02 2026 at 19:35):

Yes that does make sense. A lot of the librt commands are also bu logging, so the output catching could actually unlock a lot of things.

Sure I'll look into some possible workflows.
A pretty low hanging fruit could be chore tasks like mass renaming upto some organizational standard or some analytical calculations using BRLCAD's internal tools that could be piped into some documentation. I'll do some more problem discovery on this and get back.

Sean (Jul 02 2026 at 20:44):

mass renaming and analytical calcs aren't typically chore tasks. any of those three I mentioned are real tasks, and a chore to do. resolving overlaps well is a chore. all the checks the V&V tool performs that @Kanchan Borole is working on, those are a chore. extracting an exterior mesh is a huge chore (many methods and strategies, lots of commands involved). or the actual chore of modeling, lean into the agentic side and have it generate modeling recipes based on english descriptions.

Raghav Sharma (Jul 03 2026 at 16:36):

I took a shot at one of the chores you mentioned about resolving overlaps and built it into a working agent capability. I wanted to show you where it landed and get your read on whether this is the showcase direction worth developing.

The agent basically finds interferences using gqa and resolves them non-destructively by moving the misplaced part. It avoids subtracting one from the other, since subtraction carves out geometry that should stay intact, which isn't how you'd actually fix a positioning error in practice. It calculates the minimal move to clear the interference, picks a sensible exit direction, and then verifies with gqa that the overlap is completely gone.

I've been testing it on real models and will bring rendered before and after shots to show you. As a quick example, I dropped a ground slab so it intersected a toy jeep's wheels. The agent slid the slab down just enough to clear them and verified zero overlaps afterward. It also handles things like a plate full of bolts sunk to different depths by backing each one out its own minimal amount in one pass. It even handles cascades, so if fixing one overlap creates a new one, it rescans and resolves that too until the scene is clean.

I went with a few specific design choices here. By default, it moves the smaller part, like a bolt instead of the plate, and there is an option to anchor structural parts so they never move. It also moves the whole assembly instead of just a leaf. Since gqa reports leaf regions but otranslate moves a subtree, moving just the leaf tears the part, so it resolves the meaningful assembly instead. Lastly, it fails gracefully. If clearing a part would require sliding it indefinitely, it just refuses and explains the issue rather than mangling the geometry.

The main catch right now is that it resolves overlaps greedily, pair by pair, rather than acting as a global optimizer. In a tightly packed cascade, it can converge to an ugly configuration. Doing this perfectly is essentially simultaneous constraint solving, which is a physics-engine-scale problem that I don't think fits the scope. Because of that, I've framed it as an assisted tool that is reliable for common cases, rather than a full solver.

Let me know if this feels like the kind of showcase you had in mind. I'm also curious to hear your thoughts on the move-vs-subtract default and whether the greedy limitation is the right place to draw the line for our scope.

Raghav Sharma (Jul 03 2026 at 16:36):

Raghav Sharma (Jul 03 2026 at 16:38):

Example with some bolts evenly sunk into a plate (this was a headless rt render)

Raghav Sharma (Jul 03 2026 at 16:48):

So as a developer I've been using a lot of headless rendering piped into a temp folder read by claude code while building a little non-deterministic unit test suite for myself while debugging the MCP.

Making me wonder if adding a png render piped into a VLM pipeline could be a nice addition to the MCP, basically giving the agent eyes into the database, helping the user perform actions like ask about components visually (eg: "Move the grey colored pipe to the right")

A certain limitation worth documenting would be that the MCP is heavily dependant upon the user's data hygiene practices while naming certain components in text (eg: the agent would be able to identify a component named wheel.r1 better than the same component named r2121. in the second case the agent would be very confused if I asked it to move the right wheel, as it would have no context about what's constituting a wheel in that scene)

A vision capability could possibly help address that, though that again would be fairly experimental

Sean (Jul 08 2026 at 18:19):

@Raghav Sharma I'm liking this direction, lots of food for thought. the trick I think is going to be defining the tools typically used to resolve overlaps, and then describing a detailed skill/workflow with how to go about it. like with your bolt example, sure it fixed the overlap but the proper fix was probably to subtract them from the plate given how they were embedded, so some user-hinting would probably come into play.

Raghav Sharma (Jul 09 2026 at 20:48):

Hi @Sean
the bolt case you're right, subtract was probably the correct fix there since the bolts are embedded in the plate. That kind of decision is one I could leave on the user's side rather than the tool's (something like options offered to the user in a TUI)

A quick note on how the resolution actually works, since it's real tool usage rather than a heuristic guess. For each overlapping pair, the agent:

So it's basically like Kanchan's V&V + an auto overlap resolver
(now as this is a turn based greedy collision solver it has limitations so it can never solve very nested collisions completely, for something like that we'd have to write our own physics engine which might be overkill for this)

Raghav Sharma (Jul 09 2026 at 20:58):

Apart from this, I am also exploring the other chore that you mentioned of creating showcase renderings with advanced lighting. It's all headless through rt, so it slots into the same MCP pipeline as another tool

So far I've got a three-point studio light rig working (each light is a light-shader region with bright and shadows set), rendered with rt's full lighting model (-l0, which gives shadows plus reflection and refraction) and converted to PNG with pix-png. Hypersampling (-H) handles anti-aliasing, and I place the lights relative to the camera per view so the same model looks consistently lit from any angle.

A bunch of isometric views I've made with curl calls into the TCP socket with claude code (using the claude code agent harness for quick iterations, then solidifying the findings into the formalized MCP tool)

(also managed to segment the ktank's outer surface and made it transparent for the showcase, all done by the MCP)

Raghav Sharma (Jul 09 2026 at 20:59):

Raghav Sharma (Jul 09 2026 at 21:08):

An amusing finding: we can actually use claude code now for ops within BRL-CAD after the TCP socket implementation (I could also try building claude skills and shell scripts for it, though its a little more heuristic and probabilistic than proper MCP tools. The MCP tools however can be interfaced to claude code, tempted to explore that too as we could benchmark some SOTA models like that, though the IPC switch later on won't let us have this, perhaps we could have both IPC and TCP in the library side by side for the different use cases later on)

Raghav Sharma (Jul 10 2026 at 15:41):

Sean (Jul 13 2026 at 02:39):

Our main use case is to hook into SOTA, not do local LLMs -- BYOK. I've done several experiments and they work quite powerfully well already without any mcp or rigorous skills. But still, rather unpredicatable and then depends very heavily on how capable the model is you're hooking into. I'd rather have well defined workflows that knowingly work pretty well regardless of the model (to an extent of course). I think we can assume, however, at least something as capable as opus 4.6 or gpt 5.5, and if it's newer it should just do that much better.

Sean (Jul 13 2026 at 02:47):

Those renders look pretty good but where's the transparency coming from? Did you do that or is that an MCP skill doing that or what? If it's a skill, what's it look like?

Sean (Jul 13 2026 at 02:53):

Some more tricks that make renders look even better (typically) are to increase the amount of ambient lighting -- notice how dark some of them are. The -A option will increase the brightness and a value of -A1.0 to -A1.5 is typically desirable depending what all is going on and what custom lights there are and what they're doing. If there isn't transparency involved, then ambient occlusion greatly improves the shot with ambient samples -- fewer is gritty (-c "set ambSamples=32"), more is smooth (-c "set ambSamples=200"), both with good use cases but greatly increasing render time. Same can be said with the number of shadow rays if you want soft shadows (but that one is a shader parameter, so you have to tweak it differently -- currently no way to override at runtime without db edit).

Sean (Jul 13 2026 at 02:55):

Curious why they're so dark if you have a three-point studio light rig working. Sounds like you need to scale their intensity based on model size maybe? or there's some bug in the light rig setup?

Sean (Jul 16 2026 at 13:43):

@Raghav Sharma any update? The midpoint request for a render or other work flow wasn’t just to see pictures … it’s to see the recipe/skill input and any thinking response. The output is not the goal or request.

Raghav Sharma (Jul 16 2026 at 14:17):

Hi @Sean , apologies for the delay here, I wanted to work through your notes properly before replying.

Raghav Sharma (Jul 16 2026 at 14:18):

On the transparency: it's BRL-CAD's own glass shader. I set it with mater <shell> "glass" R G B 1, and the inherit flag (that trailing 1) pushes it down onto all the child regions, so a single command turns a whole subassembly transparent while the internals keep their opaque materials.

For the cutaways that meant glassing just the outer shell combs (hull and turret on the tank, cab and bed on the m35) and rendering the internal parts in the same scene.

It isn't an MCP tool yet, it's a documented rendering recipe I've been iterating over: the material choices, the light rig, the two-pass framing trick, and the exceptions. I'll send that over so you can see the input directly. Though it would be better if I could get my hands on some more vehicular models apart from the ones provided in the share db (could help me figure out possible edge cases in the whole loop)

Raghav Sharma (Jul 16 2026 at 14:19):

Also you're right about the darkness, it was two bugs on my end. Ambient was far too low (-A 0.3), and the light rig used fixed intensities that didn't scale with model size, so anything large came out underlit. I've moved ambient into the -A 1.0–1.5 range like you suggested and scaled the light intensity to the model's bounding box, which sorted it out. I'm also folding in the ambient-occlusion tip (ambSamples) for the opaque shots.

Raghav Sharma (Jul 16 2026 at 14:22):

got it, I'll send the recipes and thinking trace for the glint and render workflows shortly after fixing some stuff

Raghav Sharma (Jul 17 2026 at 19:04):

Update on this:
Interestingly enough there doesn't seem to be any lighting attenuation in the way BRL-CAD processes lighting
Can't be the intensity of the lighting itself, but the scale/positioning perhaps

along with that I'm experimenting with model centric lighting vs camera centric lighting
Will send that too shortly

Sean (Jul 18 2026 at 03:12):

@Raghav Sharma you need to dial back on the agentic replies, if it continues we will need to switch to face-to-face -- you can use AI to understand my questions if unclear (or ask me), but you serving as the middle-man in a conversation with an LLM is not something I'd like to continue and you've been using it a lot in responses (too much in my humble opinion) to the point that I feel it's starting to waste too much time. too many mistakes, distractions, and tedium.

Example case in point, I know that's rt's glass shader -- you didn't need to explain it to me, especially how it works. that should have been obvious in any other context. that's the type of non-constructive use of AI that nobody likes. before that, it was the platitude "quick note on how the resolution actually works" that is so embellished it's a chore..

I'd asked for a showcase demonstration and you said "it slots into the same MCP pipeline as another tool". I basically said I want to see the instructions for that render tool, and you explained how rt works. Then you said it's not an MCP tool which contradicts what you said earlier. That's exactly the kind of conversation mistake I'd expect from an LLM, so again please dial back or cut out the llm replies. your own voice is more appreciated.

Sean (Jul 18 2026 at 03:20):

There most certainly is lighting attenuation. It depends on several factors including the size of the scene, placement of lights, size of lights, type of lights, other lighting params, etc. Too much to get into really.

The basic knobs to be aware of are ambient (-A), ambient occlusion (-c "set ambSamples=32"), light shader fraction settings (fract), light shader intensity settings (inten), and shadow settings (hard vs soft). There are more settings that matter like direction, but that's the primary ones that influence the most.

Sean (Jul 18 2026 at 03:24):

Circling back to this, I think this is definitely something worth considering and testing, but only after we have a least one demonstration recipe working well -- a VLM would greatly improve the richness of the interaction and UX.

Raghav Sharma (Jul 18 2026 at 10:15):

I'm really sorry about this, was just having a little trouble with understanding the librt stuff lately. I'll be a little more thorough and reply myself for now.

Raghav Sharma (Jul 18 2026 at 10:21):

Sorry for the confusion back then
Actually it wasn't an MCP tool as of 10 July, that was my end goal with the experiment

I was just experimenting with raw LLM calls into TCP to first solidify a method and then building a recipe around it (and then that recipe into an MCP)

I'm sorry if my communication at the time had not been clear about this. I was just in the process of iteration (and my LLM replies may have convoluted that further)

Raghav Sharma (Jul 18 2026 at 11:27):

Right now I am working on packaging this render workflow into an MCP tool
So there are 6 stages the tool takes to render an image:

(this workflow is the only time we're actually using a subprocesses within our MCP tooling)

Raghav Sharma (Jul 18 2026 at 11:30):

This is a trace of the rendering workflow. At first it tried to draw ktank, but then realised its not a drawable object and ended up resolving via tops to draw tank.
I'll probably refine this step so that the agent runs tops before running the tool so that this error goes too

Raghav Sharma (Jul 18 2026 at 11:37):

I'm currently facing this one error with the opendb step that breaks on the MGED GUI but works fine over my libmcpcad test server (which is essentially like headless libged)

Raghav Sharma (Jul 18 2026 at 12:16):

Demo video with a slight stage 2 change (agent asking for the db path manually)
I'm gonna check whats the issue with opendb under gdb and get back on that. Once that's working the user wouldn't need to provide the path manually

Raghav Sharma (Jul 18 2026 at 12:17):

Renders made via the agent in the video (these are in the default ambient lighting)

Also please let me know if there's a better way to do this. I'm also looking at using ged_rt to avoid the opendb issue (which seems to be because of some callback specific to the mged session, will get back on that in detail)

ged_rt could also eliminate the need for rt as a subprocess started from the tool as it goes over the socket as a command into ged_exec though I am yet to test it (but pix-png would remain to be a sub process)

Raghav Sharma (Jul 18 2026 at 12:56):

Another trace, this one on studio lighting with ambient occlusion requested by the user

Raghav Sharma (Jul 19 2026 at 23:59):

Good news!
I got this workflow working with ged_rt (so we don't need opendb for file path, as ged_rt handles that internally)
There is a slight difference in the workflow now, with polling being involved for the .pix files coming from rt (as the process for rendering the .pix files now lies within the rt subprocess at the listener end rather than being a subprocess at the MCP)

Raghav Sharma (Jul 20 2026 at 00:21):

opendb seemed to have some problems when being injected into the event loop. As the commands run through the MGED command table before they hit ged_exec theres some commands like opendb and closedb which run through some wrappers (https://github.com/BRL-CAD/brlcad/blob/main/src/mged/f_db.c for opendb, which has a callback which is causing MGED to segfault whenever opendb reaches ged_exec directly instead of running through the wrapper)

the ctx passed into _post_opendb_failed seems to arrive with a null argv (only when directly injected by the MCP while MGED is running) which causes a null deref on line 50 which ends up crashing the application

I've been tracing the issue with the help of AI (I'm still in the process of understanding it, to be honest)

Should I put effort on exploring and solving this immediately? Would the MCP require access to file path of the current db quite often for workflows, or can I leave it for later? Most tools do seem to run fine on the socket and none of the other experimental workflows have required this yet

Raghav Sharma (Jul 20 2026 at 00:54):

Also, is there a specific format for the one pager in which I should document the workflows in? (The render one and the gqa one)

Raghav Sharma (Jul 20 2026 at 15:44):

okayyy we have the studio rendering also working via ged_rt (with the 3 point directional lighting)
I've brightened the ambient lighting to 1

Is there anything else I could try experimenting with in the rendering side to go deeper or to improve render quality?

Sean (Jul 21 2026 at 05:42):

@Raghav Sharma appreciate the updates and the videos are great. I'd really like to see the technical details -- what is the skill/workflow/prompt write-up that it's using? what's the recipe? it definitely needs adjustment as there are actions performed by users every time when doing production renders that it's not doing that will save incredible amounts of time. also like to hear more about how exactly it's all working. you have a skill/workflow description written up somewhere -- is it loaded on startup, hard-coded in the code, dynamically loaded at runtime? what about the individual piece-parts?

Sean (Jul 21 2026 at 05:43):

few things -- get rid of pix unless you're using a command that requires it. you can render directly to png.

Sean (Jul 21 2026 at 05:45):

and another reason I want to see the workflow -- it should not be rendering full/final rendering until it confirms all the settings. for example, it should render a tiny stamp first to make sure layout is optimal and lighting is balanced. then it renders a stamp with A and ao dialed in. One then should it render the full image with AO, lights, and such enabled.

Sean (Jul 21 2026 at 05:50):

you could create a collage of images (there's a pix tool for compositing automatically), but I think just the requested highlight image is adequate. the workflow can be a "make this look really great automatically" and the skill figures out how best to achieve that along with any other constraints or requests the user specifies (e.g., random colors or specific colors, include a background environment, different lighting setups, ground plane, etc)

Raghav Sharma (Jul 21 2026 at 06:30):

theres one system prompt for the agent (which is hardcoded in code, maybe i could modularise it as a loadable skill?) and docstrings defined for each MCP tool

I am using the fastMCP framework in python that basically lets me define tools under decorators that get imported at the start time of my LLM client. in my current workflow I have 12 tools defined for different purposes and the docstrings for those tools are loaded in every turn along with the system prompt of the agent. Based on my queries the LLM client can pick the tool for execution and then pass the parameters.

You can have a look at the system prompt here, though the codebase is a bit scrappy right now (a messy folder structure that I am gonna fix along with a slightly outdated readme.md)

Raghav Sharma (Jul 21 2026 at 06:34):

Ah sure I can do that. Might have to scaffold the VLM integration a little early then. Would it be better to display the tiny stamp to the user for approval or have a VLM take a call on it?

Raghav Sharma (Jul 21 2026 at 19:26):

Is this a better flow with that into account? (with a preview and manual approval)

Also should I try improving the LLM client itself with maybe a TUI to preview images (and maybe slash commands with docstring previews) or would that be scope creep?

Raghav Sharma (Jul 21 2026 at 22:40):

Update: rendering directly to PNG now without pix, also taking media input for the VLM via absolute disk path

I found this random lego brick spec with dimensions and thought of seeing if our agent could accurately recreate it just with VLM inference of the spatial context and dimensions

and it did end up doing a fairly accurate job on the proportions given to it in the image (though the long brick is rotated the other way, I could possibly explore a method to infer dimensions, then draw, then render it for the VLM via rt, check if the drawing matches and then correct if it doesn't)

I could probably go deeper into this workflow after completing the rendering one

Raghav Sharma (Jul 21 2026 at 22:59):

I just decided to experiment with a hand drawn spec of a birthday cake and hand it over to the agent

And it came out fairly well
there was a minor error in the agent's first inference of dimensions. I corrected it in the following prompt (basically 0.5 mm diameter of the candles instead of 2 mm which it misread) and it created the exact same object in CAD

Raghav Sharma (Jul 22 2026 at 00:37):

Another round of experiment (this time I'm making the agent review 3 stamp views of the model it has constructed against the original spec sheet)

Raghav Sharma (Jul 23 2026 at 01:10):

An end to end modeling + rendering workflow of an L-bracket with a reference image. The full flow involves the model ingesting the reference image and prompt, deriving the dimensions and confirming that with the user. Once user gives a go, it first writes its own temporary spec sheet in json (something inspired from Openscad) and then draws that with MCP commands.

Once the drawing is done, it checks the model by rendering it from multiple angles and comparing that to the original reference. If there is no change, it stops. Otherwise it makes edits to the spec sheet and then that entire spec sheet is re-rendered (this flow only happens for models which are spec driven, otherwise it uses commands directly). The spec workflow is easier on the LLM and also allows for backups and transactions at a much simpler level.

After the model is finalised, the user asks for a render. The agent first generated a small 400x400 stamp for the user to view and confirm lighting, after which the user confirms and the final studio render is completed.

The modeling part seems to work fairly reliably for small, clearly defined parts like L-brackets, bearing housing, legos and stuff. I got a little too ambitious in the middle and tried to model a MiG-21 directly but it struggled with it (maybe because of its tendencies to go for primitive shapes first) though its quite open to experimentation and tuning in this state

I have these left to complete, could possibly extend the current rendering tool with these options directly or incorporate this a different workflow extending the render tool (something like inheritance perhaps?).

Raghav Sharma (Jul 23 2026 at 01:15):

Also would there be a clean way to render a wireframe to PNG using some tool? right now we're sending solid renders back to the VLM for reviewing which might hide mistakes in the wireframes and not let the AI observe them properly. Could just be an enhancement for more complex modeling workflows

I saw the rtedge renders but those seemed to be outlines without the actual see-through wireframes

Sean (Jul 23 2026 at 07:19):

The system prompt in here is exactly what I was referring to earlier -- thank you. I wasn't entirely expecting you would have baked several mini-skill/workflows directly into the system prompt! That's why I was asking to see the specific workflow -- there are a dozen or so I see baked in.

That prompt is really great for a first revision, and good to finally see it! I think that it will very likely need to evolve from a giant operational though to do even one workflow robustly.

That prompt currently has elements of agent identiy, mged command ref, safety cautions / guardrails, tool selection, workflows, exception/recovery handling, memory, modeling+rendering concepts, and more all baked into 4k tokens. There's already at least one big contradiction where you simultaneously tell it to stop after a successful tool while also dictating several multi-tool operational definitions (which obviously can work but unpredictably). The guardrails are also probably going to increasingly be a problem as most beg for tooling

I'd propose breaking this up into a set of thin agents, each with a specific defined role. First is the conversational agent (which you have) as the primary / initial agent that takes the input and determines if the users's input is asking for work. Then some sort of router and/or planner agent that takes that raw work input and elaborates into a more formal plan or workflow course of action, and that's picked up by a worker with access to actual tools. Story doesn't end there though, you still need a thin verification agent that evaluates whether the worker did the job or if the task needs to get kicked back to the planner for follow-work or pass it forward to the final formatting agent that knows has knowledge of the output constraints and user's request (they may ask for a file or a database change or some text printed or ...

Sean (Jul 23 2026 at 07:23):

Given the current status, I think we should hold off on VLM integration. That's just going to add a lot of complexity and there are some fundamentals to sort out better just on the tooling and workflow side of things first. We could even just focus the rest of GSoC on just perfecting one or two demonstration workflows.

Sean (Jul 23 2026 at 07:26):

full sequence of agents would probably be something like: understand -> inspect -> plan -> authorize/schedule/iterate -> execute -> verify -> recover/rollback -> report. that's off the hip, though, and could use some more careful thinking.

Sean (Jul 23 2026 at 07:30):

If you do that, then you can define proper skills that spell out in a standard format (not prose) the I/O params, preconditions, examples, cautions, deps, steps to take, success/abort criteria, recovery actions, and effects.

Sean (Jul 23 2026 at 07:35):

of course your VLM demonstration is incredibly compelling. it's just still a bit concerning that it's layered on top of some instability that I suspect it'l be hit or miss. That is an outstanding use case however that we could focus on as the target demonstration -- modeling given a reference image or description.

Sean (Jul 23 2026 at 07:36):

there are hundreds of commands and many ways to do that. easiest is probably "screengrab".

Sean (Jul 23 2026 at 07:48):

would be something like this mixture of composable skills, tools, and workflows:

model_from_dimensioned_sketch
  depends on:
    ingest_drawing
    identify_drawing_views
    extract_dimension_constraints
    resolve_feature_topology
    audit_constraints
    plan_csg_features
    build_model_spec
    verify_model_dimensions
    render_orthographic_checks
    revise_model_spec
    promote_draft

of course that was also the motivation for making the definitions get read dynamically so we can actually debug and change them on the fly at runtime -- the underlying agentic setup would remain the same structure.

Raghav Sharma (Jul 23 2026 at 18:41):

Got it. I'll break this into a multi-agent workflow and iterate upon that. Will get back on this!

Raghav Sharma (Jul 23 2026 at 18:44):

Ah I did end up scaffolding the VLM pipeline after the message. Though perhaps after solidifying the tooling/workflow side of things could we finalise the rendering and the spec image to model workflows for demonstration refinement?

Raghav Sharma (Jul 23 2026 at 18:49):

Sean (Jul 24 2026 at 17:22):

I did see that and the demonstration is very cool to see somewhat working. One of my concerns is without the underlying foundation being reliable, there's going to be lots of "sand upon sand" building to fix things like the lego part being rotated incorrectly.

Fixing any one of them will probably be pretty easy, but the cases are pretty endless and where there should be more attention is on the underlying skills and workflow definitions that got it into that situation.

Not saying you're there, but we currently have zero metrics so this is entirely subjective ad hoc discovery at the moment. That tends to result in really cool demos, but minimal value in production user environments due to unreliability.

Sean (Jul 24 2026 at 17:22):

I think at this point it's a given you've proven that both vlm can/will work well as well skill-based resolutions. We need to nail one of them down into a really robust feature from the ground up that always does a predictably and consistently good job.

Raghav Sharma (Jul 24 2026 at 18:15):

got it
I'll iteratively strip this agent into smaller predictable workflows and get back on this

Stream: brlcad

Topic: An MCP Server for BRLCAD (Natural Language CAD)

Raghav Sharma (Feb 24 2026 at 15:38):

Sean (Feb 24 2026 at 20:27):

Raghav Sharma (Feb 24 2026 at 20:32):

Raghav Sharma (Feb 27 2026 at 19:58):

Raghav Sharma (Feb 27 2026 at 20:01):

Sean (Mar 01 2026 at 22:23):

Sean (Mar 01 2026 at 22:25):

Sean (Mar 01 2026 at 22:25):

Sean (Mar 01 2026 at 22:26):

Raghav Sharma (Mar 02 2026 at 00:07):

Raghav Sharma (Mar 02 2026 at 00:17):

Raghav Sharma (Mar 02 2026 at 00:24):

Raghav Sharma (Mar 04 2026 at 16:05):

Raghav Sharma (Mar 04 2026 at 16:06):

Raghav Sharma (Mar 04 2026 at 16:33):

Raghav Sharma (Mar 04 2026 at 17:01):

Sean (Mar 09 2026 at 03:55):

Володимир (Mar 09 2026 at 11:23):

Raghav Sharma (Mar 21 2026 at 13:30):

Sean (Mar 22 2026 at 14:59):

Raghav Sharma (Mar 22 2026 at 15:51):

Raghav Sharma (Mar 23 2026 at 16:39):

Raghav Sharma (Mar 29 2026 at 19:20):

Sean (Mar 30 2026 at 05:46):

Raghav Sharma (Mar 30 2026 at 15:13):

Raghav Sharma (May 01 2026 at 03:42):

Sean (May 01 2026 at 16:37):

Raghav Sharma (May 01 2026 at 22:52):

Raghav Sharma (May 04 2026 at 10:03):

Sean (May 05 2026 at 04:04):

Sean (May 05 2026 at 04:05):

Sean (May 05 2026 at 04:07):

Raghav Sharma (May 05 2026 at 06:11):

starseeker (May 06 2026 at 11:25):

Raghav Sharma (May 07 2026 at 05:11):

Raghav Sharma (May 07 2026 at 05:19):

Raghav Sharma (May 08 2026 at 02:26):

Sean (May 14 2026 at 20:44):

Sean (May 14 2026 at 20:47):

Raghav Sharma (May 17 2026 at 06:18):

Raghav Sharma (May 17 2026 at 06:21):

Sean (May 18 2026 at 15:00):

Sean (May 18 2026 at 15:01):

Raghav Sharma (May 24 2026 at 07:49):

Raghav Sharma (May 24 2026 at 07:53):

Sean (May 30 2026 at 15:28):

Sean (May 30 2026 at 15:44):

Raghav Sharma (Jun 03 2026 at 11:35):

Raghav Sharma (Jun 04 2026 at 06:21):

Raghav Sharma (Jun 04 2026 at 06:24):

Raghav Sharma (Jun 18 2026 at 08:49):

Raghav Sharma (Jun 28 2026 at 16:34):

Raghav Sharma (Jun 28 2026 at 16:35):

Raghav Sharma (Jun 28 2026 at 16:51):

Sean (Jul 02 2026 at 15:43):

Sean (Jul 02 2026 at 15:47):

Sean (Jul 02 2026 at 15:49):

Sean (Jul 02 2026 at 15:52):

Sean (Jul 02 2026 at 15:57):

Sean (Jul 02 2026 at 15:59):

Raghav Sharma (Jul 02 2026 at 19:35):

Sean (Jul 02 2026 at 20:44):

Raghav Sharma (Jul 03 2026 at 16:36):

Raghav Sharma (Jul 03 2026 at 16:36):

Raghav Sharma (Jul 03 2026 at 16:38):

Raghav Sharma (Jul 03 2026 at 16:48):

Sean (Jul 08 2026 at 18:19):

Raghav Sharma (Jul 09 2026 at 20:48):

Raghav Sharma (Jul 09 2026 at 20:58):

Raghav Sharma (Jul 09 2026 at 20:59):

Raghav Sharma (Jul 09 2026 at 21:08):

Raghav Sharma (Jul 10 2026 at 15:41):

Sean (Jul 13 2026 at 02:39):

Sean (Jul 13 2026 at 02:47):

Sean (Jul 13 2026 at 02:53):

Sean (Jul 13 2026 at 02:55):

Sean (Jul 16 2026 at 13:43):

Raghav Sharma (Jul 16 2026 at 14:17):