A good place to talk about all things related to STEP support and STEPcode.
things I'm working on:
simplified pre-build process
appveyor powershell based build
MSVC build compatibilty
libexpress
things I have in mind
exp2cxx
part 28
tcl
in retrospect, not convinced about p21 re2c - looking at ~150MB step files, a preparse into a sqlite database might be a better solution... testing!
Um. Not sure sqlite will scale well to something really large - see, for example, https://papers.freebsd.org/2018/bsdcan/schwarze-new_lessons_from_mandoc_development/
I'll look in a moment - was my first thought because it's available with Python and TCL
now watching
okay - that's an interesting talk, sqlite is probably overkill - but dbm is not quite enough (I envisage I need 4 columns [id (pk), type_name (ix), args (parsed on demand), lineno].
that said, similar stuff already exists in STEPcode built on top of the libexpress hashing functionality
sqlite should handle <GB no problem. it's an interesting approach. Do you actually need query or just storage?
primarily query
the reason to move towards a two step parse for Part 21 is to allow on-demand reading of types
testing shows the lexing is not that significant a bottleneck
and the time is spent creating objects... so only create the objects when you need them is my thinking... and has worked with other tools I've written
wrote preliminary code for lexer, parser and database interface... should be able to push a branch tomorrow after checking performance on real data
Would be really interesting to see how it affects our importer. It can churn for several minutes on some large step files currently.
I verified functional correctness (all stp / p21 files in the stepcode repo), and then on my very large file - it completed, but took longer than expected. I'm now profiling to try and determine where the bottleneck is in the new code, and the original code. (both Python)
performance bug fixes to the original parser, and a prototype of an alternative parser are in this branch:
https://github.com/cshorler/stepcode/tree/python_p21_bugfixes
next step, profile again on large data and see where the remaining bottle necks are - there could be no point doing anything with re2c, we will see
bug fixes gave a performance increase of around 50x! (there was some bad quadratic behaviour), then profiling new and old and further improving. End result - the two Parsers do slightly different things, they both spend collectively about 40% of the time lexing - but drilling into the detail makes me think you'd get no more than a 10% gain by implementing an re2c python lexing module. Relatively, one runs around 3x faster than the other.
tested with real world data (100+ MB STEP files)
Aweomse @Chris Horler ! Does that only affect parsing from python-bound sdai or does it affect other languages too?
currently only the Python part 21 parser, which is not connected to the SDAI. The classes which exp2py generates today, can be registered with one of the parsers - but today there is still a lot of work to do on exp2py - this was why I've previously rewrote the libexpress part 11 parser / lexer - as the framework for generating those (python) classes needs improvement.
ah, gotcha
I noticed the string regex is slightly wrong versus the grammar spec, so I fixed it (not yet pushed). In the tests, one of the files seems to have an error:
INFO:__main__:processing /home/chorler/projects/src/stepcode/data/ap214e3/io1-cm-214.stp
INFO:__main__:db_path: :memory:
INFO:__main__:db_path: :memory:
ERROR:__main__:Lexer failure: io1-cm-214.stp
Traceback (most recent call last):
File "cPart21.py", line 521, in test
parse_check(p)
File "cPart21.py", line 513, in parse_check
parser.parse(s)
File "cPart21.py", line 265, in parse
result = self.parser.parse(lexer=self.lexer, **kwargs)
File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 333, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 1063, in parseopt_notrack
lookahead = get_token() # Get the next token
File "cPart21.py", line 105, in token
return self.lexer.token()
File "/usr/lib/python3.6/site-packages/ply/lex.py", line 396, in token
raise LexError("Illegal character '%s' at index %d" % (lexdata[lexpos], lexpos), lexdata[lexpos:])
ply.lex.LexError: Illegal character ''' at index 36830
looking at the line in question, it looks incorrect to me:
#8350=TEXT_LITERAL('','\X2\30D630EC30F330C9\X0\ R1',#8250,
'baseline left',.RIGHT.,#8340);
\X2\30 ....\X0\R1
\ is not allowed according to the spec
anyway, looks like a hex string with the wrong quotes to me - with junk either end
I had it slightly wrong \X2\30 ...\X0\ R1 (slash space)
ah....
my mistake
\X2\ is a control directive
Did you figure out what the truth was @Chris Horler ? spec or parser was wrong? We have found a number of errors in a few of the APs, some that could be fixed, some that could not. I can get you in touch or help submit a bug report back upstream if you're sure it's an error in the standard.
Parser was incomplete
On the branch above - I'm still experimenting on the alternate implementation though, so I haven't pushed it yet.
I didn't check the c/c++ code yet
(I've fixed it)
I've started to improve performance further as well, and written a database plug in... The approach is quite effective for what I'm testing with
I'm now also thinking about compression, something like zpaq
And partial relational rebuilding of files
Continuing on this topic, I tested roughly the same output requirements with re2c to sqlite (except the parser is considerably more capable). And I was able improve performance by more than 10 times (again)
That is awesome @Chris Horler .. how far off do you think it'd be to do everything going on in the parser?
I'm refactoring towards a generic parser accepting an arbitrary set of actions. In C this is working, now checking about what interfaces to expose to python and how to maintain this separation
@Sean @starseeker
HNY, can we discuss a master and develop and topic branch policy for stepcode?
I've been working with it the way we work with BRL-CAD - active master, branching at need for things like stabilizing for release...
We do it that way to (among other things) try and avoid long running forks in branches that are difficult to review and merge
Did you have a different approach in mind?
The reason I brought it up was last year I did some work on a branch multiple times before it was deemed mature enough to merge to master. I also had a topic branch (long running) that I'm making more deep changes some not yet shared. I was pushing the robust bits of this to develop branch, with a view to that being the place to integrate any changes that may break things. The idea to intermittently merge develop to master, rebase any topic branches.
Then I went back and looked last week and saw you committing on master, I'm a bit concerned we may be duplicating work from the description - I can't now recall if I made cmake changes on develop or master, but there was some modernisation.
I'd like Master to only allow PR / merges and two people (author + 1) to review
@Sean what are your thoughts?
@Chris Horler My concerns would be that stepcode doesn't have a lot of active development, and if folks don't know where to look for current development they may think nothing is happening. I initially tried master and thought upstream stepcode was broken for BRL-CAD's purposes, and only later (by accident, more or less) discovered fixes in develop.
I think we can work with any approach for our purposes. I think the bigger issue is communication and how to go about doing that effectively. Having one committing to devel and the other to master is definitely a disconnect that should be resolved.
@Chris Horler I'm game to try things a different way, if that will work better for you - typically the only reason I'm working on stepcode is to try to fix something to do with BRL-CAD integration, and that's not a focus that should dictate development style.
Personally I’m fine doing pull requests for stepcode so long as they don’t sit for weeks. I’m also fine with trunk based development for stepcode if that results in faster collaboration. How about a middle ground where devel branch could be the de-facto “trunk” with master reserved for PRs. That way we can continue to directly commit to devel ( or merge to devel from branches ) for day to day and let master have more rigor?
That could work, if we can make develop the "default" github branch
Right, that’s also resolve the activity issue.
@Chris Horler thoughts?
Fine for me, that's kind of what I was proposing anyway. So master via PR and develop direct commit.
It doesn't look like I have the permissions to change the default github branch - @Sean are you able to?
I think @Sean has a good point though about the risk of pull requests lingering too long - we should probably have a rule that if a pull request lingers longer than some time without comment or review, and passes CI tests, it can then be merged.
Ideally we'll figure out some way to do a full-up "BRL-CAD integration test" as part of the process... that should alleviate most of the need I would have to change anything quickly in stepcode anyway...
I think I can do it, I'll check this afternoon. It will also be necessary to rebase develop just this once. Within a week should normally be okay
So we'll have initially old_develop (to be deleted not rebased) , develop (rebased) and master
There are 94k diff lines?!
And you've changed the linkage of some tests, which I wrote and were intentionally not linked to libexpress
I had the (evidently mistaken) impression there wasn't much activity in the stepcode project, so I was focused on shifting the working version of stepcode into a form that would work cross platform as a CAD subbuild and be somewhat simpler for me to keep working with minimal effort.
The diff line count is mostly driven by my switching the astyle used to format the code - I'm not strongly attached to keeping it that way if you wish to simplify matters by reverting to the older style. I switched it because it is easier for me to read, and I was anticipating having to make more serious adjustments to the codebase than I ended up having to make to fix a Windows issue. If you want to reduce the diff or prefer the other style, it can be switched back.
The other option would be to simply revert back to the last common point with your branches and proceed to redo the necessary changes in a more acceptable form.
If it's simpler for you to revert-and-reapply, we can do that.
I have a number of specific needs I have to meet with the stepcode codebase for BRL-CAD. Hopefully it can be done with upstream stepcode, so I don't have to maintain a fork, but some of these issues are challenging to test:
1) Compilation and execution (step-g converter in BRL-CAD) needs to work with our ExternalProject_Add subbuild (current work on that is in the extbuild branch of BRL-CAD's svn tree, but is slated to move to trunk in the next few months - if you are interested in testing that I can provide more detailed instructions).
2) The above setup has to work cleanly on Windows with MSVC, Linux, *BSD systems and MacOSX. The version simplification (not incorporating the git hash) and a few other similar changes were driven by obscure failures on one of the platforms (Windows, IIRC).
The challenging part is that some of the failures aren't easy to reproduce - builds that succeed in a graphical Visual Studio, but fail with a Ninja command line build, or working locally on laptops but failing when tested on Github CI runners. The OpenBSD platform was another fun case - it has very specific library naming requirements that I didn't hit until I actually tried to find and use installed stepcode libs with a FindSTEPCODE.cmake configure test.
In the end I looked at each commit. I don't have a problem with most, but anything which touches libexpress or exp2py will give me issues. I'll see what I can do to satisfy both of us on a separate branch, if we can agree I'll rename it develop
@Chris Horler did you decide how you wanted to proceed?
Branch, rebase and incorporate all of my changes you removed. It proves not to be so simple last time I checked
Would it be easier to back up and reapply my changes from the previous starting point?
I'm willing to attempt that if it's the best way to get where we need to go.
What's the sha1 of the last commit where things were "OK" from your standpoint?
fwiw, running astyle --options=misc/astyle.cfg --recursive "src/*.c" "src/*.cc" "include/*.h" "src/*.h" "test/*.cc" "test/*.h"
with the previous style file does seem to pretty much restore the formatting and reduce the diff a lot in my testing.
(previous astyle.cfg being:)
suffix=none #don't create backup files
style=java #compact bracket style
indent=spaces=4
indent-classes
indent-switches
indent-namespaces
pad-oper #pad (space) around operators
pad-paren-in #pad inside parenthesis
unpad-paren #remove parenthesis padding other than requested above
add-brackets #add brackets on one-line conditionals
convert-tabs #convert all tabs to spaces
align-pointer=middle #char * foo
lineend=linux #lines end with LF (linux), not CRLF (windows)
@Chris Horler if you think it will help we can go ahead and commit the reversed formatting change to the github repo.
I have a more general question, but I'm going to ask here just because the STEP stuff is mostly what I'm chasing.
What's the relationship between the svn and github repositories?
svn is the legacy repository, now read only. New development is happening on the github repo
Our step support is still at the AP203 importer stage, and even there we don't have 100% coverage. There has been some early exploration into other standards, but nothing that can really be called functional.
If you're new to programming, I'd suggest some basic C++ tutorials to get started - the generated code from stepcode would be a pretty steep hill to climb if you're also not familiar with C++.
The list_elements.cpp program is a simple(r) example of working with STEP structures.
Generally speaking AP203 is a subset of AP242, so improvements to AP203 support will also, as a rule, be progress towards AP242.
I did some C++, mostly windows MFC stuff like ... 20 years ago. Then I stopped programming ( and got in to machining ) until about 2 years ago. I remember enough C++ to not be spooked. Just browsing through github makes it appear that it's more complete than what you are saying.
It looks like this is mostly generated from stepcode? I'll take a look at list_elements.cpp. Thanks for the hint.
There are a number of stub programs that hook up the machinery of generating executables from the schemas, but for the most part they're not actually wired up to produce conversions.
There's not really a way for it to produce a default conversion without knowing/picking/providing a default target format. Since entities are typically intimately intertwined, it does as much as it can giving you the in-memory data (via SDAI) as an object, and then you implement the converter to handle some set of entities.
probably could provide some sort of default text dump diagnostic output, but I'm not sure how generally useful that'd be other than as stubs for the real thing
@Chris Horler did you want me to go ahead and revert the astyle update, or restore the repo back to before my changes started? I'm OK with either at this point - we need to start moving forward.
Sorry I missed all of the recent messages, I only got an email notification for the last
I would suggest this...
The last common point between develop branch and master is the good point for master
Master should be reverted
Develop should have PR submitted for all your changes not impacting express and exp2py... Where I have other branches with prototype work I want to begin to integrate
@starseeker
I also have a part21 parser I've written which is very zippy I'd like to integrate
Oh, and more part 11 improvements to support exp2py evolution
All right, I'll revert master and we can give it a try...
@Chris Horler Looks like that would be commit 548fd82ad7ec37301fde7d889fbd359c86f7f0dc?
git merge-base master develop
Tells me this is 548fd82...
OK - do you prefer a hard reset or should I just make a commit putting master back into that state?
Good question
Best practice for a shared project is a commit to revert
So, I guess that would work better for everyone
Does that get you where you need to be?
Yes
Thanks
i.e. git revert
Sure. We'll need to move out on getting the other changes merged from develop though - master needs to be in a working state, and from what I remember 548fd82a is not.
I thought it was, at least I tested with gcc, msvc 2008,2012,2019 and one other
Happy to apply fixes to develop and help to review / test
Let me figure out the revert and I'll double check, but if I'm remembering correctly when I integrated that version into BRL-CAD's step-g it failed.
Okay, the reason for failure would be useful
@Chris Horler All right, I think that's got master in shape
Let me see if I can put develop back where it was
I guess you only need to split / rebase your changes on develop? And make PR of your branch?
I was figuring to start over - put develop back to 548fd82ad7ec373 as well, and then submit pull requests individually for the various pieces.
From what you said earlier, some of my changes were problematic f or you - we'll need to figure out which
Actually, first can you confirm master is in an OK state for you? That revert might not actually be enough - that might have the effect of merging develop up until the point I started working into master.
@Chris Horler I don't think git retains enough info to know for certain what the last state of master was (commits don't store an 'origin' branch - it's caused me a fair bit of annoyance in other contexts)
Let me see if the CAD repo happened to store it...
Alright, develop is reverted. Let me see if I can figure out where master was...
@Chris Horler If you're good with that starting point I am as well - the older state of master was the one that didn't work, and I merged develop because it actually worked better....
Figuring out where master was at the point of the original develop->master merge could be tricky, and I'd rather not bother unless you feel strongly about recovering it.
I think that's okay, looking at the revert commit
OK, if that unbreaks things for you we'll proceed from there. I'll refork and submit a series of pull requests to develop
We'll start with develop, but we also need some criteria for when master gets updated.
I still don't have the permissions to change the default displayed branch.
Thanks, I will rebase my other branches onto it tomorrow evening
I can do that
Change default displayed
And, I will introduce rule that master is only updated via PR
OK, thanks - that's important to update - having the wrong one displayed is part of what landed us in this mess...
Yep
Now for pull requests - how does it work with multiple independent changes? Do I make local branches in my repo for each one and submit them as develop pull requests, or does that introduce conflicts in merging?
This is a foreign workflow for me, so any advice on how not to muck it up any more than necessary is appreciated.
Okay I changed default branch to develop
Confirmed, looks good.
Branch per topic, e.g. compile fixes, another for astyle
Assuming you have commits on a local branch, probably you want interactive rebase or cherry-pick
OK. Would you prefer to skip the astyle change? I only did that because I thought it was going to be just me in the repo - since that's not the case, we can skip it unless other people also want to change.
You can also create branch rebase and select only topic commits
Yes, I think we should keep a style changes on release branches before we merge to master
I mean, did you just want to keep the old style and not change at all?
I'll format to match the old style - that's not a big deal.
I didn't review what you changed in the styles, for me same process PR, review and separate commits for rules changes and formatting updates
I made branch rules for master - PR now required
No style changes on develop
We also need to go through the existing pr list, many are redundant / already fixed or, now to be reevaluated against develop, which we need to configure for build check. Some time ago I created a new appveyor project, need to check on both new and old, also appveyor can also build Mac now so perhaps we can have one less test tool
@Chris Horler Do you recall which tests I'd linked against libs you were intending not to link against? I'm preparing the CMake update pull request, and I'd like to fix those first if I can pinpoint which ones they were.
It was the new tests I made in libexpress folder which use fff.h header
If you have changes to libexpress can you keep them in a separate PR?
Chris Horler has marked this topic as resolved.
Chris Horler has marked this topic as unresolved.
Sure. The only changes in the source files for the CMake pull request should be to handle the switch in how versioning is done - hopefully they shouldn't break anything (let me know if that's not the case). Otherwise, I'll try to keep libexpress pull requests separate.
@Chris Horler We should be able to set up tests (using Github Actions at least, I don't know about the other options) to check whatever breakage areas you're concerned about on a per-pull-request basis, if that would be helpful.
Did you see the changes I made to re2c version detection, and lemon ? I'm preparing to replace the parser in libexpress - the reason for introducing tests is to reduce to risk when I remove most of resolve.c which will ultimately no longer be required in that form.
My concerns are ensuring I don't break anything else in stepcode
So I take my parser branch, make some prep changes in develop, test, rebase the parser branch, repeat
This all came about because I was improving exp2py and hit a brick wall due to the parser. Now I have a 2 pass lexical scan and a completely unambiguous grammar that successfully parses everything in the stepcode repo
I reviewed your pr on github, I wondered if it's possible for you to combine two commits into it - I added them in the discussion
I started to work on a version of the PR to split it into multiple commits, I'm about half way through
Chris Horler said:
Did you see the changes I made to re2c version detection, and lemon ? I'm preparing to replace the parser in libexpress - the reason for introducing tests is to reduce to risk when I remove most of resolve.c which will ultimately no longer be required in that form.
@Chris Horler Is the parser still re2c+lemon or something else?
Re2c and lemon, as previously requested
@starseeker I've pushed the start of reworking your pr 412 as rework_pr412. Not finished yet - I didn't compile anything yet, there's a couple rebase to do yet, I've tried to first start by splitting each commit by topic, this makes it easier for me. Also, it's allowed me to see where I still have some questions, there's probably another two commits to go. Then we should review
@Chris Horler hey - sorry, I had some personal issues come up and haven't been able to look at coding for a few days. I'll try to work on it some more this afternoon and go over your additions.
@Chris Horler I'd like to set up a test where we build and run the BRL-CAD step converter using stepcode develop - that's the key piece that needs to work from my perspective. I would normally use Github Actions to do that - did you prefer to use another system?
Github actions - I guess that's fine
Do you want me to submit the changes you requested as commits to the rework_pr412 branch?
I propose you commit on pr412 directly, then I'll rebase it again to clean it up and check it works for me. Then we'll either update your original pr or create a new one
Sorry, on pr412_rework
I'm starting to look at the lemon simplifications - if we do some of them, I don't think we can use Debian versions of lemon - they relocate lempar.c so it is not in the bin directory with the lemon executable (or at least they did)
I need the perplex_stage target check for the BRL-CAD build - that's a component of our third party management subsystem...
It should be harmless in any other context - that's why its guarded by the if check.
Maybe I can pre-set it ahead of time... hmm...
@Chris Horler Did you also want me to incorporate the lexsupport.c changes?
OK, I think I got it sorted - see if the rework_pr412 commits look like what you have in mind.
I've not tried any of this with the BRL-CAD build yet, btw - I'll wait a bit until these initial changes settle to see what breaks there. I'll try to undo the obj lib removal in my fork to get things closer to the current rework_pr412 state.
I had a quick look, I think we're aligned on intent
No to lexsupport.c on develop for the moment,
I'll try to build the rework branch later today or tomorrow
Objlibs and linker scripts might be a way to build dynamic and static libs for only a link time cost as well, it's another thing I'm wondering about cmake
I'll make a separate branch to explore that this weekend
BRL-CAD uses obj libraries, but my recollection is there are some difficult aspects with Windows builds doing that.
I think you have to be really careful about the dll import/export logic
You said you had a few more commits to go to get the initial CMake rework split finished?
They were mostly what you've added, I'll review against what I stashed tomorrow
@Chris Horler do you want me to try to add back in the obj libraries in the rework_pr412 branch, or separately?
In the pr412 rework branch
Please
And you want it just for libexpress?
Can I scrap the MD5 checking logic? That will simplify doing some of the other work.
Also, do we want one OBJECT library per source file (the current state) or one OBJECT library for the various sources that is in turn used by libexpress's shared and static libs?
@Chris Horler Here's what I'll do - I'll make a series of (possibly not working in incremental state) changes to try and shift this where I'm thinking it should go - if we need to revert and try again that's fine, but I can't justify trying to make a lot of incremental changes work when the end state is going to end up being significantly different.
btw, do we need re2c 1.0.3 or greater for features for your new work? If so, I'll probably throw in the towel and just make sure newer re2c can build using pre-generated inputs without bison present - BRL-CAD's version of re2c builds with lemon not bison, and is thus self contained, but we never successfully upstreamed that work and it seems they've moved on in the years since.
Phew. OK, @Chris Horler I think I've gotten things fairly close - at any rate, ready for you to take a look. We'll have to hammer on the libexpress build cross platform in particular to make sure that obj logic is robust enough.
On a related point, have you guys seen this bug report? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256166
Looks like there's a few issues to unpack in that report, potential null derefs (some of which @starseeker fixed), copies of the complexSupport.h header, unknown source for the nullity (if it even was a null deref -- looked like maybe a 0x8 value to me) converting NIST_MBE_PMI_11.stp, potential fix needing to get pushed upstream, etc..
I figured to revisit that once the fixes from the BRL-CAD tree are successfully merged.
I'm hopeful that once @Chris Horler is OK with the CMake setup, the other bug fixes from those commits can go in as quick, individual pull requests.
Since Rob is interested in a release with those cleanups too now, there's some fairly strong motivation to get this tied off and get a release out the door...
@starseeker Okay, most of your additional commits are aligned with my thinking. I built the branch. There's one issue I found, my cmake interface is not showing the build only express option - which is important when you need to build libexpress without the schemascanner - as you do when rewriting the parser and the library.
@Sean no, I hadn't seen that yet
We also have quite a few different versions of cmake_minimum_version
@starseeker
Ah, OK - I didn't know that option was in active use. I'll restore it.
Thsnks
We probably need to check for const correctness in the code base at some point, the typedef are sometimes concealing these
@Chris Horler Does 92bfdecc4a29235 put back what you need?
Yes, it has to be specified on the command line, but I think that's how it worked before
I can not mark it as advanced if you want it to show up by default.
Let me scrub out some of the unused cmake quick files for clarity...
No, I think it's okay as advanced
When you're done we should reorder / squash etc with rebase
Then suit new pr and merge
s/suit/submit/
Let me do a check against my clone - while I'm doing that, do you see something else that's concerning/needs/adjustment/etc.?
I'll rebase my parser branch and see how close it is
I'd guess there are no more problems
OK - I won't know until we're ready to Windows stress test if the every-file-as-obj-lib will be an issue there, and we've got to get some of the other build fixes in before that's practical, so for the moment we'll assume it's workable. I don't recall for sure, but I think I just smashed it down into one lib for simplicity at the time rather than functional necessity.
OK, I think that's got it - most of the remaining diffs are related to keeping the lib-per-file build and the build-express-only options.
It's okay for me
As far as the CMake minimum version - we have a hard lower limit of 3.12 because of some of the features needed for our 3rd party build system - does that work for you?
If so I'll just change the others to match.
I shouldn't have too much difficulty rebasing my parser branch - only the new parser bits fail now
Yes
OK, let me update the other minimums. You say I need to rebase the branch at that point to prepare a pull request from it to develop?
You or I - I was going to squash a few of the commits which were fix up to previous commits, and check if anything was in the wrong place for the commit history / diff view
I can preserve author I think there's an option for that with rebase
@Chris Horler OK, why don't you go ahead - I just pushed the version fix, so that should be the last bit for this stage on my end.
Okay
Look out for it later today or tomorrow - got to do dinner now!
I'll start going over my other commits to being extracting the warning fixes - did you want one-branch-per-fix, or how did you want to handle it?
One commit per topic, linked topics in a branch (e.g. common parent commit, sublib, schéma etc)
OK. This will take some getting used to - BRL-CAD's SVN history made branches very, very expensive and they were something we only did for really massive, invasive changes - I'll probably get a branch heavy workflow wrong a few times, so please bear with me.
Sure
Is it better to do the branches in a fork on my own account and submit the pull request from there, or do a branch in the main repository like rework_pr412? (Or does it matter?)
Doesn't matter, all topic branches will eventually disappear.
I've a question about 1 commit, "grab a couple of trivial cmake list. Txt updates". There's one fix up for nullptr, so I split the commit and fixed up the nullptr commit. What remains is removal of the core target. I notice in an earlier commit you reworked this - so is that a mistake? If so I'll drop what remains of that commit
I don't use the core target any more - unless you want it, I'd say go ahead and remove it. I think it was originally planned in case someone wanted to work just with exp2cxx and check-express, but at this point I doubt the added build system complexity is justified.
Okay, amend the log message
@Chris Horler You want me to amend that commit's log message and push the results up to github? (sorry, just trying to make sure I understand - I've gotten in trouble a number of times rewriting git history on branches pushed to a shared repo...)
(we almost never rewrite history in the BRL-CAD project - a philosophical position - so I don't really know how to do it for this case...)
I've done it
New pr414 created
Please can you check it? (see pr414) If you need to add fixups, I pushed the branch too - just add to it
I think we should be able to merge this today
And, yes never rewrite history of master or develop. Topic branches are another matter
@Chris Horler I (think) I approved the pull request - do I merge it or do you?
I'll do it
Done, develop is updated
@Chris Horler For the moment I used the existing code formatting style in the repo for the astyle sweep, rather than trying to change it - if we do want to shfit the style, presumably we should do it after your parser work is merged. I wanted to homogenize the style as much as possible to make subsequent comparisons simpler while preparing the other pull requests.
@starseeker I commented on the pr as it might be introducing a bug, but what motivated the de-encapsulation of GetAttribute()?
That was back when I was doing the ext work with stepcode - I updated our code to use the access methods, and you indicated a preference for keeping direct access.
It's unusual to de-encapsulate, so looking to understand the context..
ahh really? You added Raw() and GetAttr() and such?
I do not recall that discussion...
No, upstream did. I was updating our code to be forward compatible with that encapsulation approach, and you indicated (IIRC) you preferred the direct approach.
Hang on, I'm trying to find the history...
I mean when all getters/setters do is literally insert a layer of indirection via a function call, they're not good.
Discussion was "step converter" around Dec. 8, 2020
found it, reading, thx!
I'm not seeing your comment - is it on the pull request in github?
Yeah
Ah, I see it now - the summary hadn't updated yet.
It's quite possible I misinterpreted what you were saying - my focus at that point was hammering stepcode flat enough to work with the ext build logic, so I was looking for the shortest path to functional...
now that discussion is coming back to me. jeesh! amazing how quick that was shifted off my context plate.
<snort> my context stack these days could be mistaken for a mound of spaghetti
So yeah, lots to unpack going on. The original issue was primarily having a commit that changed API under the banner of ext restructuring, but then also calling out Raw() that was a pass through.
Honestly didn't look at Raw() in the PR as what jumped out was the set attributes call
so if I have this right.. the attribute was private, someone made it protected, you added an accessor, then you removed the accessor and made it public? I probably have something wrong in that timeline..
That sounds about right.
It may have been public a while back though - if it came up it was because our converter was already using it somehow.
I wasn't adding anything to step-g at that point, so it would have been whatever logic was already there in the original converter work.
I know STEPWrapper has a GetAttribute() that returns the SDAI STEPattribute iirc... is that the same entity?
Probably? I'd have to dig.
I'm going through my stepcode commit history to make those pull requests, so I don't have all the BRL-CAD context handy...
iirc, we use that to get trimming curves, so might be the connection but it is a completely different class so maybe not
The modus operandi was to remove our fork of stepcode, drop in whatever vanilla upstream was at that particular point, and see what broke.
oh my bad, is is the same class
I looked back through the history and deleted/updated my comment on the PR.
Knowing that it was previously direct access means the copy constructor issue wouldn't be a new issue, if there is an issue.
still think Raw() is not necessary, but can go either way on the STEPattribute getter/setter. Looks like attribute access was at least partially incomplete when step-g was implemented. So may still be a memory management issue but it would have been an issue independent of the setter. I suppose we need to put the converter through a good valgrinding to be sure.
I ran it through valgrind a couple months back trying to debug a conversion issue (which I think was ultimately a 214 file issue), but it crashed valgrind on me (mac support is dodgy).
/me nods - stepcode is rather difficult to deal with in that respect - if the issue is in generated code, even finding where it came from can be tricky and when you do often times the generator is handling lots of other code as well and you can't just change it for the one case without causing wonky results somewhere else.
@Chris Horler I've tried to separate out various topics into pull requests - that's probably most but not quite all of the changes. I wanted to get these resolved before I go too much further, since after this point I may be doing things slightly differently than in my first iteration. Once we get those handled in some fashion, I'll start trying in earnest to get the Github Actions builds going and address whatever remaining platform specific fixes I've not yet gotten moved over.
starseeker said:
/me nods - stepcode is rather difficult to deal with in that respect - if the issue is in generated code, even finding where it came from can be tricky and when you do often times the generator is handling lots of other code as well and you can't just change it for the one case without causing wonky results somewhere else.
Yep, and to be expected in general really. That's why the question is always "who created it", "who owns it" , and "who destroys it". When in doubt, one can make a copy at the expense of runtime (minor) but that can also have unintended side-effect on hierarchical data as copies are typically shallow (but not always). That's part what made me leery of a function that was implicitly invoking a copy constructor getting replaced with a call that might not copy (turns out it did too).
It can turn into a rats nest fast if it's shotgunned. I don't like casting shade on c++ unnecessarily, though - the rules are pretty straightforward, just a bit more complicated than we typically deal with in c due to operator overloading, implicit initialization, and copy construction.
By the way, commented on another -- commit mentioned line numbers.
the line numbers were wrong..
I am wondering about STEP converter history, how did BRL start to develop :))
@scorp08 We had a temporary lapse of sanity...
@scorp08 I started looking into STEP in the early 2000's before we went open source. It took a number of years to get going, but eventually was able to get the ISO specifications sponsored so we could have access to the specs and begin working on development. We came across the NIST STEP Class Libraries (SCL) which was the only open source implementation available. We took over SCL development and after years putting in effort to modernize, clean it up, and make it useful to others, STEPcode was born.
STEPcode forms the basis for BRL-CAD's step-g importer (which currently is AP203 only).
@Sean updated the line number messages.
@Sean @Chris Horler On my local copy of stepcode, with all the various changes applied, I just got a fully clean test cycle with the Github Actions setup from https://github.com/stepcode/stepcode/blob/93e7a135183cce0d9d985f5fe2ea4af5538ecaea/.githhub/workflows/build.yml
I was also able to rebase my parser branch successfully with minimal changes, although I didn't test building it yet.
Good to hear tests are go, we should deploy that
Well, they're a go with the various pending pull requests applied - not sure what will happen as is...
Should we go ahead and put it in anyway? I kinda figured to wait until the other bits were done so we could start off "clean" as it were...
Does anybody know if the cdash related files in the toplevel are still working/workable? I'm thinking we should scrub them out unless they're in active use (we can always put them back later if we end up enabling something else...)
@Chris Horler You may need to disable the CI webhooks for the travis and appveyor systems - I don't think I have those permissions for the github project.
I'll have a look later today
Two things!
There are 4 configured webhooks, in addition to the two mentioned... circleci.com and scalar.vector.im
W.r.t. Travis and appveyor, I'm a little concerned with deleting things, we can try instead to stop them - I've deactivated them
Now...
How to test your PR before accepting it? I couldn't figure this out - I'd assumed by going to the branch we'd be able to do that
But no!
W.r.t. The PR appveyor messages, they're caused by the organisations appveyor integration... I think. @Sean can deactivate it, I don't have access
@Chris Horler The pull request is on develop, not master - if it's not easily testing in the PR, my vote would be to just add it and move forward, fixing as necessary.
Although again my advise is to get the other PRs in first, since I only tested with those applied...
Okay, let's focus on the other pr first
@Chris Horler oh, bty - am I supposed to add review requests for the PRs?
Normally yes
Shall I do that for the remaining outstanding PRs?
No, I'll try to look them over this weekend
Hi, I would like to contribute to the Project as a Volunteer, is there any scope for me to join the team
@Chris Horler @Sean We're coming up on the 1 week mark for the pull requests - I'll go ahead and merge most of them tomorrow, if there are no objections. (I can't merge the Github Actions PR myself because I can't disable the other systems.)
I've approved most of them -
I don't see any obvious problems
Maybe I missed a pr or two... I'm checking now
Just the deprecated copy PRs?
416, 417, 418, 421, 422, 423, 425
There's a bunch of old pr that should be closed / rejected
Yep, think that's got it. For the Github Actions PR, did you want to experiment with it in a fork in your own account if we can't properly test it in the PR itself?
And a few slightly newer that should be reviewed (e.g. 408, I've assigned myself to do this)
Yes, I think that's probably a good idea
I did notice when reviewing the prs, there's another tab "checks" I think this is for the actions
Although, I'm still waiting on it loading the page... Perhaps it's a catch-22 - you can't see it until it's configured the first time
I'll test in my fork
Hmm. Yeah, I'm getting the same thing here...
Okay, I'm done for tonight - I'll try to look at actions tomorrow
Sounds good, thanks! Once we've got Actions merged and functional, I'd appreciate it if you could add logic to test the python side of things - I'm not familiar with that aspect of the system.
I also rebased my parser branch, but I've got failing tests to check - although they were broken before rebase
Okay will do
It may be as simple as making sure the correct options are enabled for the test targets, but if it needs more we should turn it on as well... it's looking as if none of the stepcode tests are big enough (at least, at the moment) to overload the Actions CI runners.
That being the case, more is better :-)
@Rob McDonald If you've got an equivalent to the step-g test for OpenVSP we could toss in as well, that'd be of interest.
Not at this time.
OpenVSP does have the ability to do a non-graphical build (greatly simplifying dependencies and speeding up the whole process) and pretty much all functionality is available through a scripting language that can be triggered from the command line.
Note that OpenVSP does not have any ability to read STEP files -- we are strictly write-only (and only AP203).
We can write out three 'styles' of files -- the first, un-intersected surfaces as SdaiB_spline_surface_with_knots. We also perform surface-surface intersection procedures and can then write out a ManifoldShell or BREPSolid.
So, in addition to testing OpenVSP / STEPCode integration and to a limited extent the functionality of STEPCode, any test we would write would largely be testing OpenVSP behavior -- generation of surfaces and potentially intersection between surfaces.
We could build tests from simple objects whose code rarely changes, but it is likely that we will continue to work on our surface-surface intersection and BREP exporting capabilities -- so any test based on OpenVSP is going to have to be tolerant of the VSP side of the code changing results... From a testing standpoint, it isn't quite as tidy as confirming proper import of a standardized test file...
@Rob McDonald So in that scenario what would be tested is not necessarily the output per say, but the functionality of the integration with stepcode - i.e. does it compile, link and run. Even if the output is not checked, those tests will catch certain categories of error. (That's what the BRL-CAD test is currently checking for, in fact - I'm not comparing the .g output of the conversion to a known file, for much the same reasons - if we change our logic the output may change independent of stepcode.)
From that standpoint, I should be able to construct a test using the AP203Min code that simulates OpenVSP's integration and build system -- with much less complexity and cost to the CI system.
It can't / won't be a perfect test -- but it also greatly reduces the likelihood that some unrelated problem with OpenVSP's build triggers a false positive for STEPCode.
Fareha Nousheen said:
Hi, I would like to contribute to the Project as a Volunteer, is there any scope for me to join the team
@Fareha Nousheen With open source, there's always scope for people to join the team. What works best is to just jump in, start learning/doing something, and ask questions. People are more inclined to respond when you are clearly trying to be productive, have read docs, are trying to compile something, etc. LOTs of ways you can get involved and figuring that out depends primarily on your interests and experience, not on others'. Welcome! :)
Thank you for the response. I would definitely love to be a part of open source projects, however, at this point of time, I wanted to join the GSoD project for which the organization has pitched in for this year. Is there any scope I can contribute for the selected GSoD project? Kindly guide me
Also, it would be helpful if you can guide me, how do I begin with, when going forward in the coming months, I want to join any of the open-source projects. I'm completely new to this and would need guidance, to begin with. Kindly help me out.
@Chris Horler any update on enabling the CI with Github Actions for stepcode?
Sorry I had no time to complete yet, I'll have more spare time this weekend
Testing now, first issue (perhaps on purpose?) is subdir was named githhub (extra h). Anyway, my fork is now building it...
@starseeker
So far all the workflows failed, looks like they're trying to rebuild the parser
Anyway, it's a starting point. I don't see à branch name mentioned in the workflow... I guess I've some reading to do so I can understand exactly what it's attempting to build
In fact the failure is due to md5 checksum failure, which I vaguely recall was removed in a previous pr...
Although there are 101 warnings about dll linkage too
Quite odd really
I'd like to see preprocessor output for some of the dll defines
I think the extra h was probably a typo...
The BRL-CAD tests as written aren't targeting the main stepcode repo yet, since at the time it was written we didn't have the updates merged - they're checking out my local stepcode develop branch . If we want to run the BRL-CAD tests on arbitrary branches, we'll need to adjust those checkout lines to use a variable of some sort for the -b option.
I don't know why the "main" tests would fail on md5 issues, if you've merged the latest develop updates into the branch...
as you say, that whole logic block should have been removed. Can you tell where it's getting it from?
Didn't look yet, I checked what commit was being built - appears to be correct. Next thing will be to see how to debug the action
I checked now, it's fairly obvious - the branch needs rebasing on develop as is now
I'll do it and test
Yep, it's building now
OK, success
I don't think rebasing is correct though - github can do things to preserve the original history on the PR page... I'm checking now
Well I tried to do it the correct way and have the PR tested against the latest, but it refuses - so I'm going to push the rebased to the topic branch. You may see an error due to this and need to disassociate your remote on the branch, rename the local branch before pulling the topic (if you need to). Develop will remain stable, so no issue if you're only interested in the result
Sounds good, whatever works.
The PR is now building, assuming it's successful I'll merge it
Assuming the python bits are also working correctly, we should probably start thinking about what criteria and methods to use for the next release. Any thoughts?
For 0.9 release?
I've à couple things I would like to integrate - I have a c python extension for reading part21 files, I'd like to add that
Sounds good. I'd just like to have an "official" stable release that distros might pick up for packaging which also supports BRL-CAD.
I guess you're also talking about things like astyle
Not particularly - if you prefer not to mess with that I'm fine.
Okay, I prefer not to mess with astyle for the moment
The only remaining major(ish) change I'd like to consider at some point is a series of changes I prototyped in a BRL-CAD branch a while back which altered the #include statements in the headers so a client code can include just one stepcode directory rather than needing multiples.
IIRC I was also able to eliminate the base library.
I'd also like to have a bit of a review over what are public interfaces and macro functions
Let me see if I can find a link to that work - it's not in a form that can be applied to the current tree, it will have to be redone, but it may give you a sense of what I was looking at.
(It would be post 0.9 though)
I was thinking to make the base library static internally linked, or remove it. In the parser branch I build auxlib as an internal library with bstrlib
Ok
I think with the C++11 standard most of base can either be removed or put specifically with the isolated parts of the code that use particular features.
So, we only have appveyor failures - but these are expected for the first work on github actions. It's completing now
Ok
Are you wanting to leave appveyor on?
@Chris Horler so don't panic about this stepcode tree - I'm not proposing to strip things down this far in the main repo (we'll keep python support, for example - this was a specific testing scenario on Windows where I was removing all the complexity I could.) Also, after discussions with @Sean I'll leave exppp alone as a library. The thing to focus on is the include/stepcode header relocation and related changes.
In addition to the header shuffle, this also eliminated base. It's based on a much older stepcode than the current tree though, so if it's a go i'll need to redo the changes using develop as a starting point.
Oh, probably relevant for generalizing the BRL-CAD stepcode integration tests to work with more than just develop branch: https://stackoverflow.com/questions/58033366/how-to-get-current-branch-within-github-actions
I'll have a look, got to stop now - early start tomorrow... Work!
hmm, pull requests may complicate life - need to check out https://github.com/marketplace/actions/github-environment-variables-action
I've merged it
Looks like the appveyor hook is still active/failing, but otherwise awesome.
Yeah, that needs disabling to org level - I don't have access @Sean
I contacted Mark - I think I've got it straight. Testing now.
:thumbs_up:
Well, I removed the webhooks but there's still some kind of weird zombie continuous-integration check listed. Fortunately it seems to be a no-op that "succeeds", so it's not causing breakage reports, but I don't see so far how to get rid of it.
When I looked yesterday pm I didn't see any issue, where are you looking?
Looks like it only appears on old pr? I just created #426 to test, and it doesn't appear there
Chris Horler said:
Yeah, that needs disabling to org level - I don't have access Sean
Oof, yeah -- thanks for the ping.. I'll look at that asap. Please remind me if I haven't fixed it by next week, but thx for the reminder! I'll fix the perms while I'm in there.
Thanks Sean
@Chris Horler How goes the "needed for 0.9 stepcode release" work?
@starseeker I didn't have time in recent weeks to do anything - work and life took precedence. Hopefully more free time next weekend. Sorry for not responding more promptly.
I reviewed the Python PR this evening, there's some rework to do. Also some additional work required. I'll start on it one hour each evening this week.
Just to confirm, we're all agreed to drop Python 2 support?
@Sean @starseeker
I'm not a user of the Python version, so my vote is relatively low weight, but I'm fine with Python 3 only.
I created a branch pr408_rework, and rebased the branch of the PR in it (last night). I'll do that again to address the points raised on the PR, then push it
@Chris Horler any news on the stepcode front? Also, were you still planning to redo the lexer/parser setup to avoid the need for re2c/lemon et. al.?
I was busy for a few weeks with other stuff, hope to do some catching up tomorrow
Still working on lexer / parser, yes I have some new re2c stuff to add for Python
And ofc the longer term work on the part11 parser
I finished working my way through the python pr, I've adjusted some of it. Also there are three key points not addressed, I should get to these tomorrow. (we're now inconsistant in use of object for inheritance, there is a legacy use of apply to remove, and some other stuff)
@starseeker There's apparently some regression in the step-g converter. This model works in an old release, but the latest appears to run infinitely: Cup.STEP
I had trouble narrowing in on when the regression happened. Old versions gave me a heck of a time compiling; I couldn't bisect commits easily. Hoping you have better luck.
@Chris Horler any news on the stepcode front?
@starseeker I reworked / fixed the two pr for python, there's still an outstanding python bug which I didn't get to yet (I've spent recent weekends hacking my dvd rig in emacs) ill update again on Sunday.
Idea would be to merge 3 pr, and assuming they pass test make a release.
Sounds like a plan. Then post release would the next step be the parser work?
Also, I've been working on the BRL-CAD side to clear some static analyzer warnings - would those be of interest pre-release, or would you prefer to wait?
They'd be of interest for prerelease as well
And yes, afterwards parser work
OK, I'll prepare some pull requests
I'd like to keep the express only build target, it's useful when working on the parser - to be able to build without the schema scanner
I added a couple commits I worked on this morning, whilst looking into bugs. There were several bugs in exp2py scopeprint function. I made the code behave a little more like it's intended to behave (Bugfixes!). There is still some more to do, indentation issues in the output point to incorrect tracking of indentation level in the output functions, I've noticed this in the past too.
@Chris Horler OK, restored.
I'm working on the initialization and string buffer cleanups, but it looks like somehow there's still formatting differences between the BRL-CAD stepcode copy and the upstream, so I'm having to go through a lot of them by hand. I'll try to have something in a few hours.
Okay, note you git diff -w to make this a little easier
I fixed more of the exp2py, now only one schéma failing... lifecycle_integration_schema
I think I've got most of my local changes staged - there's quite a bit more cleanup to do, but it will take a while to get through it all so I figure to just get what I can in, and then proceed further after the release using that version as a new starting point.
Pylint shows many many issues, some are relevant... We should probably start linting all generated python
Found the final schéma problem - entity property, output as class, silently overriding python property decorator. Most confusing reading the trace output. We should probably increase the list of reserved words as well!
Tomorrow I'll rebase and cleanup where necessary
I used pylint and fixed a few more issues, there are big issues with where and unique on entities. And inheritance unitary schema gives mro errors. Of course it's known that exp2py doesn't generate good code at the moment, so won't worry about this until new parser is in use
Just taking a bit of a détour to look if we can adopt mypy and python3 type hints
Also realised there's some generated code in one of my commits which shouldn't be there
I fixed some or all of the attribute issues I noticed in where rules... As we have a number of pr I'll start to merge some stuff this weekend, or early next week
Still need to rebase and remove what shouldn't be included
Rebase done, now building pr 429 and 430. There was a brlcad failure, perhaps something is configured wrong? @starseeker
Checkout failed for brlcad main
Maybe the git protocol is not allowed, also I guess it could be done with checkout action with repo option
Yep, looks like they turned off the git:// support: https://github.blog/2021-09-01-improving-git-protocol-security-github/
@Chris Horler just switching git:// to https:// should work - how do you want to handle it? Shall I just update main, and you can merge in the change? Or some other approach?
PR please - then we'll know it fixes the issue
https://github.com/stepcode/stepcode/pull/441
@Chris Horler Let me know if there are any issues with #441 - I tested the command with https in isolation, but this is one of those things where only success in the full runner environment proves it out fully.
I was reading up on this, I'm testing a slight change to how we launch actions. If it works I'll rebase your branch to have pr results before merge
@starseeker please can you submit it again? I think this time it should trigger a workflow
@Chris Horler I pulled the updates - looks like that kicked off the test in 441, which I think is what you were after?
Yes
It shows a failure linking on linux
Appears to be a macro or multiple definition or type issue.... I didn't read the source only the error
@starseeker
@Chris Horler Ooo, that's a useful catch. Adjusted in BRL-CAD main, re-running stepcode action
The Actions now reports clean. The badge in the readme still shows, failing, but I'm not sure why... maybe we need to re-run all the jobs?
@Chris Horler let me know if there's anything else
Thanks, I'll look over the weekend
@starseeker
To fix the checks on the other PR of yours - it's simple. 1. On your own fork where you have the branches for each PR, checkout develop and git pull the upstream changes.
@Chris Horler done, tests running
@Chris Horler Are we closing in on a 0.9 release?
I think a Bugfix release, the only new feature we have is Py3, the usefulness of the Python code is not production ready though. For 0.9 I'd like to integrate parser changes and python meta bases. Also explore the simplification of macros and correction of type masking in some parts of the libraries.
I'd be happy to release 0.8.x - with the view we also need to manage our branch logic for releases
I'd like to start looking at reorganizing the headers into include and reorganizing so calling codes only need to include one directory for the stepcode headers. How do you want to handle that?
Also, with newer C/C++ standards, I think we can eliminate base
I don't know if those things complicate the python side of the house...
I was also thinking we can eliminate libbase
Agreed about the headers
I'd like it so if we build a package the user only needs to pass -I/usr/include/stepcode to the compiler.
Exactly. I did that once in a minimalist fork - it's a bit involved, but doable. Will it break the python side of the house?
I'd also like to make sure we have all the other pull requests merged into develop before doing that, since the moving will likely complicate merges...
The python part will probably be minimally impacted
I intend to merge everything before proceeding further - I noticed I had one un pushed cmake fix, perhaps there are others too
It's looking like the cmake adjustment to express/CMakeLists.txt may not be agreeing with Windows... let me see if a re-run confirms it. Nice job setting up the workflows to run on the pull requests, by the way...
@Chris Horler OK, forget the CMake PR. More obnoxious Windows DLL import/export shenanigans, and I don't have time to mess with it now.
If you're still planning to eliminate perplex and friends, that and the header reorg might end up simplifying the DLL business.
I think the other PRs are OK now
@Chris Horler Let me know if you see any remaining problems. I should probably hold any additional PRs til after the 0.8.x release tag.
After merging stuff, presumably in the wrong order (I was following the PR number sequence) there's a minimal conflict on the remaining pr for memory management. Please can you rebase it again and resolve the conflicts in two files? (they look trivial, probably best you do it though - I'm assuming some of these changes are valgrind prompted and others just reading the code and seeing issues!)
@starseeker
I think that's got it - let's see how the tests do
Ah, right - need an adjustment in upstream BRL-CAD for the header renames. One sec...
@Chris Horler I think that's got it - tests should be finishing up fairly soon
Well, soon-ish - the Mac test seems to take quite a long time
@Chris Horler Finally. Tests completed successfully
Once that gets merged I'll take a look at eliminating base, but that'll be post release.
Okay, I'll do it after dinner
Or before dinner, computer was still on and only a cursory review required!
@Chris Horler Awesome - thanks! Anything else you need from me for a 0.8.1 release?
No, that's all. Thanks @starseeker
@Chris Horler any progress towards the stepcode release?
No, I took a bit of a détour to work on a TCL Debugger for vscode, I also had two weeks holiday... Backpacking, so no computer
Hi @Chris Horler - any word on stepcode release plans?
@starseeker no, but you're right it's been too long. So I'll try to make it happen on Sunday.
I reviewed the 3 outstanding PR
Trivial changes needed, if not done by sometime tomorrow I'll do it myself
I probably should update the contribution guidelines as branches or repo are problems we can avoid when I want to rebase before accepting
@starseeker
Let me take a look (sorry focused on another problem...)
@Chris Horler Oh, gotcha - I figured to hold off on the base removal until after the release. Did you want me to do it now for this one?
What do you need me to rebase on? The latest stepcode develop?
When I look it says I'm not behind the latest upstream develop...
@Chris Horler If you have specific guidelines for how you want changes prepared I'd definitely spell them out in detail
Okay, I'll work up guidelines
@Chris Horler was there still stuff you wanted to merge, or are you OK to wait on base elimination until after the release?
@starseeker I'm packing for holiday, I doubt I'll do anything before I get back now.
@Chris Horler any update?
I got back from my backpacking last week, I had some chores to do this last weekend. Prospects are better this weekend - although I had my 29byears employment party this Friday
Made two line change to contribution guidelines
@starseeker please can you ack and merge if no issue
I plan to merge all 3 remaining PR after rebase / tests. We should perhaps add a note in the release notes about removing base it could cause some people minor linking issues
@Chris Horler merged. If you want to go ahead and merge the base removal, I agree it probably warrants a release notes mention
Thanks
I rebased your branch, and forced pushed it - the build process / checks have triggered. It's now occurred to me though, this will invalidate your local branch nobase. I guess that's not a problem after it's merged.
I'll also add some maintainer guidelines to the contributing document
Github stuff
Sounds good. Yeah, don't worry about my branches - I'll sort it out.
Merged some more, a few more things to do before release. Probably on Wednesday as I have to do food shopping tomorrow evening
@Chris Horler I think I've got a fix for the BRL-CAD integration test failure in place - tests are re-running now. If that didn't do it I'll go another round this evening
OK, great!
Looks like everything passed - I think you're good to merge now?
Yep, done
I'll check the outstanding PR from me is okay tomorrow or later today
Okay, I'm going to attempt to draft a release
I believe the process is something like this.
Idea from here:
https://stackoverflow.com/a/14168817/1162349
And indeed the first step shows a number of conflicts
@Chris Horler Is there any reason we shouldn't be defaulting to develop content for this process?
develop has changed significantly from master, so I'm not surprised that merge would be an issue this go around
You might want to use the -Xours option if merging from master to develop, and -Xtheirs when merging develop back into master.
My recommendation for this particular case would be to just merge develop into master with the -Xtheirs option, and then make sure any conflicts are resolved such that master ends up looking like develop. Going forward we might be able to do the proposed process, but after the earlier mess in master my guess is this first time it's probably not worthwhile to go master->develop.
There's irony, I just found -X ours, this makes it much easier
Let me test it out a little bit
The reason I wanted a merge commit on develop was to show a point easily in the history where the release was made... And testing it also allows directly no-FF update of master (tested with a locally created aux_master branch). Next step I think is I push the merge develop commit - and manually run the workflow to check nothing was broken
I pushed a new branch with test_master_release, and manually triggered the workflows on this - something broke... Good job by me! I guess it pays to be extra cautious like I was!
I'm going to guess like you suggested one of my merge commits was faulty
In fact it was the second merge I think... Checking now
The first merge seems to have ignored the strategy when resolving inside a file
The first merge should be -s ours reading the Man page... Now testing again
5/6 okay, waiting on macos
OK, all were successful. Locally I've prepared everything now. One final question before I push it. The version number in develop is 0.9.1 in the CMakeLists.txt. When I've merged develop into master locally I've amended the merge commit to set the version number to 0.8.1. This prevents 0.9.1 ever appearing on master, and avoids odd version bump / unbump on develop. Any objections to this?
None - that's fine.
Ok
I've pushed Tags and branches
I'll draft the release now and launch a workflow on master
Any ideas for a release title?
I guess we can collectively edit the draft after I save it
Okay, if you go to the releases page you can edit
https://github.com/stepcode/stepcode/releases
3/6 actions are complete
No reason to think they won't all be successful
Heh - BRL-CAD doesn't title releases - just versions them, so I'm afraid I don't have any clever ideas.
Let me take a quick look at the change set since 2014
Are we still supporting Python 2, or is it 3 only now?
@Chris Horler I took a quick run at it - feel free to adjust.
Once tagged, did you want to upload any binaries? Personally I'm fine with a source only release, unless linux distros indicate they want binaries...
Looks good, I'm going to press the button!
Done, landmark moment
I'll get that version into BRL-CAD's upstream src/other - quite possibly the first time we'll be using a fully vanilla upstream of stepcode
Looking forward to what comes next!
Great!
Good timing too - I'm working in Toulouse this week and next weekend I'm doing my possibly last backpacking trip of the year, before the ski season
Now that we're tagged, I'll put the pull request in for the header shuffle - hoping it will be fairly minimally impacting, but that will depend in part on how it might collide with your parser update work
It's main practical impact is that for client codes, they won't need to specify all the individual library includes just to make the stepcode headers work. If they use prefixes like "clstepcore/sdai.h" in their includes, they'll be able to just include the top level stepcode header directory and have it "just work"
I'll try to rebase my parser work on top of it and see what type of issue it gives. I note there are some failing tests in the PR, I didn't look deeper yet. I imagine it's trivial
Looks like a missing header and a problem on the BRL-CAD end with hard coding the stepcode version. I think I've got those sorted - testing now.
@Chris Horler Is the header adjustment pull request ready for merge on stepcode? Also, curious if there's any update on the parser rewrite to get off of perplex/lemon?
Also, would it be a good time to do a PR to get rid of libbase?
@Chris Horler Unless there's a reason not to, I'd like to do a 0.8.2 patch release of stepcode so our latest release has the adjustments for newer compilers and the parallel build robustness fix.
I'll create an annotated tag tonight, then the release notes can be drafted in GitHub
@Chris Horler If you haven't tagged yet, I've got one more change I'd like to get in - I realized the FindPERPLEX/LEMON/RE2C scripts in stepcode are a fair bit older than the ones in the BRL-CAD repo, so I went ahead and shifted in the new ones.
As it happens, I didn't do it yet.
So all ok
Thanks - I tried the update but hit an issue on Windows, so I've got more digging to do... I closed the first pull request, I'll update if/when I figure out what the issue is.
Okay
We can tag on Wednesday or Thursday
Sounds good
@Chris Horler if I don't have a pull request by Thursday, go ahead and tag
Will do
I forgot, and now I'm away until the 18th backpacking
I'm back now! (after work!)
I went ahead and added the tag, although I haven't set up the release yet. Feel free to undo and redo if you prefer.
okay, what you've done is not quite correct - here's why:
chorler@localhost:~/projects/src/stepcode> git for-each-ref refs/tags
df87e021276b4d28d792a89aba529d02b4018e4c commit refs/tags/v0.1
9c2018e1cf1def5ee268459cdb42ac96151ad337 commit refs/tags/v0.2
35cd701e5200020b21e078a027cbf00ad5bd82ed tag refs/tags/v0.3
88a1229a23e844b65d36b2a854a1079f27929a10 commit refs/tags/v0.4
4d68ec0b19203960ca6df7df86301b906e89e0e9 commit refs/tags/v0.5
698b9e0371d12978761cdc81289c6fb0ae67d276 commit refs/tags/v0.6
21fd1ed1e4f158fe185820bb95271508051bc73f tag refs/tags/v0.7
953021ec9d3a2ea71cae9ff6bdf5d726e00bb35c commit refs/tags/v0.8
e6b496412d72a0a1af1cbe7ec11f9c2d28044f8d tag refs/tags/v0.8.1
5cf8fc1a8907983365b233280f128566956713fa commit refs/tags/v0.8.2
chorler@localhost:~/projects/src/stepcode> git describe
v0.8.1-27-g5cf8fc1a
chorler@localhost:~/projects/src/stepcode>
you created a light-weight tag (a private or temporary tag)
this is why git describe doesn't consider it
I'll see if I can replace it
okay I've fixed it
if you git fetch, git pull, git describe - hopefully you should now see this in your local copy
chorler@localhost:~/projects/src/stepcode> git describe
v0.8.2
chorler@localhost:~/projects/src/stepcode>
I added initial draft release notes on github. @starseeker it would be good if you could add a minimal example how compiler include flags are likely to differ -e.g. -I/usr/include/stepcode
(release notes are draft - so feel free to edit)
@Chris Horler I added a brief explanation
Okay, I'll publish after work today
done - we have a new release 0.8.2!
Last updated: Jan 09 2025 at 00:46 UTC