Sumagna Das (Jul 23 2020 at 16:25): Sumagna Das (Jul 23 2020 at 16:25):hey @starseeker , Sean said that i can help the migration out by checking if theres anything missing from the github nonotes repo that is present in the svn repo
Sumagna Das (Jul 23 2020 at 16:25):so should i still try that?
Sumagna Das (Jul 23 2020 at 16:25):and on the recent migrated repo brlcad_conv5?
starseeker (Jul 23 2020 at 16:29):Sure. The diff line that I was using can be found in this file:
https://sourceforge.net/p/brlcad/code/HEAD/tree/brlcad/trunk/misc/repoconv/common_structs.h#l152
starseeker (Jul 23 2020 at 16:29):That filters out known differences to look for anything unexpected.
starseeker (Jul 23 2020 at 16:30):And yes, brlcad_conv5 is the latest
Sumagna Das (Jul 23 2020 at 16:30):i was thinking about using a script
starseeker (Jul 23 2020 at 16:31):Sure, whatever works for you - the diff line just tells you what differences you should expect to see.
Sumagna Das (Jul 23 2020 at 16:31):ok
Sumagna Das (Jul 23 2020 at 16:31):i will use that
starseeker (Jul 23 2020 at 16:34):If you want to iterate through revisions, you'll need to parse the svn revision and branch out of the git commit message, construct the correct svn checkout line, and then diff the trees.
Sumagna Das (Jul 23 2020 at 16:35):i was thinking about using python for this. will it be possible in python?
starseeker (Jul 23 2020 at 16:36):Don't see why not - anything that can invoke the correct commands will do fine. The trick is to construct them in the first place - that's what I've got encoded in the C++ logic I linked to.
Sumagna Das (Jul 23 2020 at 16:36):ok
Sumagna Das (Jul 23 2020 at 16:37):the conv5 repo is going to take some time with my internet speed
Sumagna Das (Jul 23 2020 at 16:37):so bye
starseeker (Jul 23 2020 at 16:37):good luck!
Sumagna Das (Jul 23 2020 at 17:29):do you have any idea why the git clone is taking so much time?
Sumagna Das (Jul 23 2020 at 17:30):it didnt take so much time for the svn pull
starseeker (Jul 23 2020 at 17:35):It's a full copy of the entire history - that's how Git works.
starseeker (Jul 23 2020 at 17:36):Locally here it's about 340 megabytes.
Sumagna Das (Jul 23 2020 at 17:36):no not that
starseeker (Jul 23 2020 at 17:36):Oh, you mean a local checkout?
Sumagna Das (Jul 23 2020 at 17:36):(deleted)
Sumagna Das (Jul 23 2020 at 17:37):i am cloning the thing and it hasnt completed yet
Sumagna Das (Jul 23 2020 at 17:37):its at 14%
starseeker (Jul 23 2020 at 17:38):Yeah, that's what I was talking about. WIth git there is a "clone" - which pulls a copy of the entire history - and "checkout" which retrieves a version of that history from the local copy.
starseeker (Jul 23 2020 at 17:38):"git clone" is downloading more data than an "svn checkout". The equivalent (more or less) for svn is rsync copying the entire history.
Sumagna Das (Jul 23 2020 at 17:39):oh i got confused with the terms
starseeker (Jul 23 2020 at 17:39):The upside (especially for repository verification) is that you'll find local checkout "git checkout" is much faster, since you won't have to go over the network.
Sumagna Das (Jul 23 2020 at 17:39):so the whole repo might be how much big?
starseeker (Jul 23 2020 at 17:39):My local copy here is on the order of 340 megs.
Sumagna Das (Jul 23 2020 at 17:40):the total repo?
starseeker (Jul 23 2020 at 17:40):yes.
Sumagna Das (Jul 23 2020 at 17:40):mine is at 125.4 MB
starseeker (Jul 23 2020 at 17:40):It's actually quite a bit more compact than the SVN rsync copy. Of course it's just the BRL-CAD history, without the secondary repositories.
starseeker (Jul 23 2020 at 17:41):Probably about 1/3 of the way there then.
Sumagna Das (Jul 23 2020 at 17:42):but the number of files received is 90560 out of 596546
starseeker (Jul 23 2020 at 17:42):@Sumagna Das if you want to do a lot of checkouts of SVN, you may want to consider doing the rsync copy of the full SVN history so you can do local checkouts with SVN as well - otherwise, every time you want a new revision from SVN you'll have to pull data over the network.
starseeker (Jul 23 2020 at 17:43):That's quite a bit larger than the Git checkout though... closer to 2 gigabytes, if I recall correctly.
Sumagna Das (Jul 23 2020 at 17:43):which one is faster if i have 3MBps speed - pull or rsync?
starseeker (Jul 23 2020 at 17:44):for git, "git clone" is the correct way to get a copy. rsync is what you will need to use if you want a local copy of the full SVN repository
Sumagna Das (Jul 23 2020 at 17:44):i might use pull every time and delete the revision i had
starseeker (Jul 23 2020 at 17:45):Um - once you have the clone, "git checkout" should work fine.
Sumagna Das (Jul 23 2020 at 17:45):that might be faster?
Sumagna Das (Jul 23 2020 at 17:45):i am talking about the svn repo
starseeker (Jul 23 2020 at 17:45):"git pull" means bring changes into your own repository from upstream - that's probably not what you're going to need to do here, unless I push updates to brlcad_conv5
starseeker (Jul 23 2020 at 17:46):The fastest way I know of to do a lot of SVN checkouts is to first rsync the full repository to your local disk, and then interact with that. It will take more space locally, but will be much faster for each checkout from SVN.
Sumagna Das (Jul 23 2020 at 17:47):then let me start it
starseeker (Jul 23 2020 at 17:47):I'd wait for the git clone to finish
Sumagna Das (Jul 23 2020 at 17:47):will that slow down the speed of the git clone?
starseeker (Jul 23 2020 at 17:48):If your connection is saturated, yes.
Sumagna Das (Jul 23 2020 at 17:48):saturated?
starseeker (Jul 23 2020 at 17:48):Already downloading as much data as possible
Sumagna Das (Jul 23 2020 at 17:49):i dont think so
starseeker (Jul 23 2020 at 17:49):Well, you can try it - but it sounds like there is a bottleneck somewhere between you and github.
starseeker (Jul 23 2020 at 17:50):rsync -av svn.code.sf.net::p/brlcad/code .
Sumagna Das (Jul 23 2020 at 17:50):it once downloaded 4gb in 1 hour without me knowing it.(i started Epic games launcher and didnt knew that i had set fortnite to auto update and it updated in the background until i started watching youtube videos and it was buffering)
starseeker (Jul 23 2020 at 17:50):That will copy the SVN repository to the directory "code" locally.
Sumagna Das (Jul 23 2020 at 17:50):thanks
starseeker (Jul 23 2020 at 17:52):To checkout a particular revision from SVN using that local copy, it will be something like:
svn co -q -r29886 file:///your/local/directory/code brlcad_svn-r29886
Sumagna Das (Jul 23 2020 at 17:52):starseeker said:
Well, you can try it - but it sounds like there is a bottleneck somewhere between you and github.
i dont know but i am having problems with the github home page. My feed is not loading. is you home page feed loading well?
starseeker (Jul 23 2020 at 17:53):seems to be
Sumagna Das (Jul 23 2020 at 17:53):starseeker said:
rsync -av svn.code.sf.net::p/brlcad/code .
this is the correct command?
starseeker (Jul 23 2020 at 17:56):That's the one I use - is it causing problems?
Sumagna Das (Jul 23 2020 at 17:57):i forgot to include the . at the end of it
Sumagna Das (Jul 23 2020 at 17:58):thats why it caused some problems
starseeker (Jul 23 2020 at 17:59):ah, yeah - that would do it. Remember, that's going to be quite a bit of data - more than the git clone
Sumagna Das (Jul 23 2020 at 17:59):ok
Sumagna Das (Jul 23 2020 at 17:59):i dont have much problem with that
starseeker (Jul 23 2020 at 17:59):how is the git clone coming - still slow?
Sumagna Das (Jul 23 2020 at 18:00):it is at 147 mb right now
starseeker (Jul 23 2020 at 18:00):ouch. Yeah, that'll take a while then.
Sumagna Das (Jul 23 2020 at 18:00):hmm
starseeker (Jul 23 2020 at 18:01):On the plus side, I really like the view gitk offers into the repository history - makes it way easier to go hunting for something.
Sumagna Das (Jul 23 2020 at 18:48):i reported the issue and waiting for the reply. Until then, it is still trying to download 340 mb and out of that it has downloaded 214 mb so it might take some time
starseeker (Jul 23 2020 at 19:06):How about the rsync of SVN?
Sumagna Das (Jul 23 2020 at 19:25):nearly r50000
Sumagna Das (Jul 23 2020 at 19:26):and the speed is back for github
Sumagna Das (Jul 23 2020 at 19:34):github repo done
Sumagna Das (Jul 23 2020 at 19:34):only rsync is left
Sumagna Das (Jul 23 2020 at 20:35):both the repos are done
Sumagna Das (Jul 23 2020 at 20:35):i will take you help during the checkout part
Sumagna Das (Jul 23 2020 at 20:35):of the svn repo
starseeker (Jul 23 2020 at 21:58):It's the line I posted earlier:
svn co -q -r29886 file:///your/local/directory/code brlcad_svn-r29886
Sumagna Das (Jul 24 2020 at 07:53):i tried absolute and relative paths to the code directory but it doesnt work
Sumagna Das (Jul 24 2020 at 07:54):gives me something like this
svn: E170013: Unable to connect to a repository at URL 'file://mnt/sda2/brlcad_migr/brlcad_svn/code'
svn: E170000: Local URL 'file://mnt/sda2/brlcad_migr/brlcad_svn/code' contains unsupported hostname
fixed the problem
Sumagna Das (Jul 24 2020 at 08:09):was missing a slash
starseeker (Jul 24 2020 at 10:01):Right - that syntax drives me nuts, can't count how many times I've missed that slash.
Sumagna Das (Jul 24 2020 at 10:07):it took some time to do the checkout
Sumagna Das (Jul 24 2020 at 10:08):went to eat something while the checkout
Sumagna Das (Jul 24 2020 at 10:09):going to let the laptop rest for a while. its hot as the sun right now due to the things i was running in parallel
starseeker (Jul 24 2020 at 10:10):Heh. Yeah, this kind of testing is a stress test, especially for hard disks. The main conversion really hammers my machine for over a week when I run it.
Sumagna Das (Jul 24 2020 at 10:11):the checking will take some hours or more for my machine
starseeker (Jul 24 2020 at 10:14):Make sure you only check out (say) trunk or a given branch - the way the SVN repository is set up by default, a root checkout will create every tag and branch of every repository in the entire repo
starseeker (Jul 24 2020 at 10:14):That's a LOT of data, and not particularly useful
Sumagna Das (Jul 24 2020 at 10:18):I saw that
Sumagna Das (Jul 24 2020 at 10:19):Will try to checkout only trunk but I dont know how to checkout a specific branch from local repo
starseeker (Jul 24 2020 at 10:25):It's the same as checking out from subversion on sourceforge
starseeker (Jul 24 2020 at 10:25):just uses the file:// location instead of the http location
Sumagna Das (Jul 24 2020 at 10:26):Oh ok
Sumagna Das (Jul 24 2020 at 10:27):Then it will not be a problem if it's the same mechanics
starseeker (Jul 24 2020 at 10:27):svn checkout https://svn.code.sf.net/p/brlcad/code/brlcad/trunk
becomes
svn checkout file:///home/user/code/brlcad/trunk
starseeker (Jul 24 2020 at 10:27):where "/home/user/code" is replaced by your local path
starseeker (Jul 24 2020 at 10:28):(rather different from Git)
Sumagna Das (Jul 24 2020 at 10:32):Hmmm
Sumagna Das (Jul 24 2020 at 21:23):Hey @Sean do u know of a way to scroll through svn log just like we can do in case of git log?
starseeker (Jul 24 2020 at 21:39):I pipe it to a file and then use an editor or less on it...
Sumagna Das (Jul 24 2020 at 21:40):i was piping to less directly :yum:
starseeker (Jul 24 2020 at 21:40):That works too.
Sumagna Das (Jul 24 2020 at 21:41):creating a script right now to compare every commit in the github repo with its svn counterpart
starseeker (Jul 24 2020 at 21:43):/me suspects you'll be able to cook with your laptop before that's done...
Sumagna Das (Jul 24 2020 at 21:43):hmmm
Sumagna Das (Jul 24 2020 at 21:43):only on the trunk branch
starseeker (Jul 24 2020 at 21:44):Slightly easier - that's still most of the commits though.
Sumagna Das (Jul 24 2020 at 21:44):so what will e much easier?
Sumagna Das (Jul 24 2020 at 21:44):*be
starseeker (Jul 24 2020 at 21:45):checking only trunk instead of all the branches + trunk
Sumagna Das (Jul 24 2020 at 21:45):i was going to do only on trunk
starseeker (Jul 24 2020 at 21:46):/me nods.
Sumagna Das (Jul 24 2020 at 22:32):My computer crashed :disappointed:
Sumagna Das (Jul 24 2020 at 22:34):Probably due to a bug In the which cause the ram usage to go up
Sean (Jul 25 2020 at 01:13):Sumagna Das said:
Hey Sean do u know of a way to scroll through
svn logjust like we can do in case ofgit log?
"git log" is nearly equivalent to "svn log | less -S"
Sumagna Das (Jul 25 2020 at 21:46):@Sean just set up the basic part of the script
Sumagna Das (Jul 25 2020 at 21:47):so what should it check - files, commit description or something else?
Sumagna Das (Jul 25 2020 at 22:06):hey @starseeker, should i also check for the commit descriptions?
Sumagna Das (Jul 25 2020 at 22:06):along with the files?
starseeker (Jul 25 2020 at 22:07):the commit messages will be different, since we have merged the notes - you'll have to script in the removal of the extra information or the commits won't match. We also line-wrapped most of them, so you'd have to mash both messages down into newline free strings.
starseeker (Jul 25 2020 at 22:08):Although your copy might not have the line wrapping, actually... don't remember
Sumagna Das (Jul 25 2020 at 22:08):i have scripted in the removal of the extra info
Sumagna Das (Jul 25 2020 at 22:08):line wrapping?
Sumagna Das (Jul 25 2020 at 22:10):(deleted)
starseeker (Jul 25 2020 at 22:10):Most of the time, Sean and I will just type out a message on the command prompt rather than opening an editor, so we don't add newlines to the commit messages. That's rather contrary to how the Git world at large works.
Sumagna Das (Jul 25 2020 at 22:10):ok
Sumagna Das (Jul 25 2020 at 22:10):i will have to see to that then
starseeker (Jul 25 2020 at 22:11):https://github.com/starseeker/brlcad_conv8 should have it
Sumagna Das (Jul 25 2020 at 22:11):the line wrapping?
starseeker (Jul 25 2020 at 22:11):yes
Sumagna Das (Jul 25 2020 at 22:11):then i have to clone that one
starseeker (Jul 25 2020 at 22:12):No big deal - your copy should be fine for an initial check
Sumagna Das (Jul 25 2020 at 22:17):the diff line you provided - if it returns something, then something is off?
starseeker (Jul 25 2020 at 22:28):Possibly - what is the result?
starseeker (Jul 26 2020 at 17:14):@Sumagna Das Actually, a difference is not guaranteed to be an issue - the SVN repository was pre-processed prior to conversion to remove expanded RCS tags from its internal content, so differences might manifest between a non-preprocessed checkout and what was used for the conversion. the misc/repoconv/CONVERT.sh script gives you an idea of what was done to produce the initial git repo.
Sumagna Das (Jul 26 2020 at 19:29):(deleted)
Sumagna Das (Jul 26 2020 at 19:40):@starseeker the final script has been set up with the diff line you provided and commit message checker(if they are the same in every one of the commits
Sumagna Das (Jul 26 2020 at 19:45):should i check all of the commits(all of them from the github repo to svn)?
starseeker (Jul 26 2020 at 20:03):I would start with some of them - there will be differences, so you'll want to make sure you can understand them
Sumagna Das (Jul 26 2020 at 20:03):commit c97468e924 /rev 76457:
Only in svn/r76457/misc/repoconv: cvs2git
Only in svn/r76457/src/other/gdal/ogr/ogrsf_frmts/dgn: web
commit adb54fde73 /rev 76456:
Only in svn/r76456/misc/repoconv: cvs2git
Only in svn/r76456/src/other/gdal/ogr/ogrsf_frmts/dgn: web
these are the differences from two of the commits and their svn counterpart
starseeker (Jul 26 2020 at 20:04):Hmm. OK - so if you look at the two checkouts, can you confirm those differences?
Sumagna Das (Jul 26 2020 at 20:04):i think so
starseeker (Jul 26 2020 at 20:05):OK. Now, the next question - why would they not be in git?
Sumagna Das (Jul 26 2020 at 20:05):thats something i dont know
Sumagna Das (Jul 26 2020 at 20:05):i was going to ask you that
Sumagna Das (Jul 26 2020 at 20:06):btw i am storing them in a file
starseeker (Jul 26 2020 at 20:06):The first one is because cvs2git is an empty directory - SVN allows the checking of empty directories, git does not
starseeker (Jul 26 2020 at 20:06):The second is more puzzling
starseeker (Jul 26 2020 at 20:07):https://github.com/starseeker/brlcad_conv8/tree/master/src/other/gdal/ogr/ogrsf_frmts/dgn is there...
Sumagna Das (Jul 26 2020 at 20:08):so...
starseeker (Jul 26 2020 at 20:08):what is your git checkout line?
Sumagna Das (Jul 26 2020 at 20:09):git checkout <sha1 of the commit>
Sumagna Das (Jul 26 2020 at 20:12):is that suspicious?
starseeker (Jul 26 2020 at 20:12):No, that's correct
Sumagna Das (Jul 26 2020 at 20:12):ok
Sumagna Das (Jul 26 2020 at 20:13):is that on another branch or something or is it not that much required?
starseeker (Jul 26 2020 at 20:13):https://sourceforge.net/p/brlcad/code/76456/tree/brlcad/trunk/src/other/gdal/ogr/ogrsf_frmts/ also doesn't show a "web" directory
starseeker (Jul 26 2020 at 20:13):Ah! I'm wrong - missed one level
starseeker (Jul 26 2020 at 20:14): starseeker (Jul 26 2020 at 20:14):And what's in that directory?
Sumagna Das (Jul 26 2020 at 20:14):empty
Sumagna Das (Jul 26 2020 at 20:14):thats why
starseeker (Jul 26 2020 at 20:14):bingo.
starseeker (Jul 26 2020 at 20:14):So my recommendation would be to script the removal of all empty directories from the SVN checkout before doing the comparison.
Sumagna Das (Jul 26 2020 at 20:15):hmm
Sumagna Das (Jul 26 2020 at 21:10):starseeker said:
So my recommendation would be to script the removal of all empty directories from the SVN checkout before doing the comparison.
completed that part
Sumagna Das (Jul 26 2020 at 21:10):should i add any more things to check or anythin?
Sumagna Das (Jul 26 2020 at 21:20):@starseeker i am going to let the script run the whole night and i will tell if it reported anything (after working correctly)
starseeker (Jul 26 2020 at 21:24):Not offhand, but there are a lot of potential issues (that's why I've got all this scripting and C++ logic in place...)
starseeker (Jul 26 2020 at 21:24):sounds good
Sumagna Das (Jul 26 2020 at 21:25):starseeker said:
Not offhand, but there are a lot of potential issues (that's why I've got all this scripting and C++ logic in place...)
potential issues?
starseeker (Jul 26 2020 at 21:25):reasons for differences in files. RCS tag expansion is a classic - I've tried to filter it with the diff script, but it could use more testing to make sure I've got the expressions right
Sumagna Das (Jul 26 2020 at 21:27):commit c97468e924 /rev 76457(commit messages match? True):
no differences
commit adb54fde73 /rev 76456(commit messages match? True):
no differences
commit 992d64fadf /rev 76454(commit messages match? True):
no differences
this is the text file which is being outputted to
starseeker (Jul 26 2020 at 21:28):Looks good. My expectation is that the SVN era commits will be in pretty good shape - it's the CVS era where I know there are still issues.
Sumagna Das (Jul 26 2020 at 21:28):if there are differences, then it will be outputted in there
Sumagna Das (Jul 26 2020 at 21:30):i am going to ignore the commit checker because as you said earlier, there might some differences in commit messages.
Sumagna Das (Jul 26 2020 at 21:31):i also added the empty directory remover part
Sumagna Das (Jul 26 2020 at 21:34):its working till now and i expect to work all the night
Sumagna Das (Jul 26 2020 at 21:34):so good night
Sumagna Das (Jul 26 2020 at 21:39):why is it sometimes jumping revisions?
Sumagna Das (Jul 26 2020 at 21:39):sometimes upto 4 revisions at a time?
starseeker (Jul 26 2020 at 21:49):Not all SVN revisions map into git commits.
starseeker (Jul 26 2020 at 21:50):Then, in older commits, some commits will be in other repositories rather than brlcad - the SVN repo holds multiple projects.
Sumagna Das (Jul 26 2020 at 22:45):@starseeker the differences it found until it aborted due to error during parsing this commit's (2bb78b9d04781650bf9226393281d738e7622824) description
difference.txt
Sumagna Das (Jul 26 2020 at 22:46):the commits were in the format
<desc>
svn:revision:<rev>
svn:author:<author>
svn:branch:<branch>
until the aforesaid commit
Sumagna Das (Jul 26 2020 at 22:47):commit desc -> (add view subdirectory (preliminary file move commit))
Sumagna Das (Jul 26 2020 at 22:47):will fix that tomorrow
starseeker (Jul 26 2020 at 22:49):Ah, right - preliminary file move commits are not SVN commits - they're autogenerated to make it easier for git log --follow to track back along file moves.
Sumagna Das (Jul 26 2020 at 22:50):I know
Sumagna Das (Jul 26 2020 at 22:51):It was parsing them until it encountered the mentioned commit (I can quote because quoting is not available on the mobile app)
Sean (Jul 27 2020 at 15:16):starseeker said:
The first one is because cvs2git is an empty directory - SVN allows the checking of empty directories, git does not
What about dropping a .gitignore into empty folders? Slight concern that leaving them out might cause breakage that did not exist before (e.g., something references the dir). Speaking of which, we should add compiling a couple of the tagged releases to the validation list... maybe 7.12 and 7.22 or similar, just to make sure nothing was introduced that's not a result of tool modernization.
Sean (Jul 27 2020 at 15:58):Sumagna Das said:
starseeker the differences it found until it aborted due to error during parsing this commit's (2bb78b9d04781650bf9226393281d738e7622824) description
difference.txt
This is cool @Sumagna Das and helpful. This is one of the validation tasks I had on the to-do list.
Sumagna Das (Jul 27 2020 at 16:45):I added the part to skip these type of commits
starseeker (Jul 27 2020 at 16:50):@Sean I think avoiding them is the best available option - adding them would be tricky, and even if we could get them in any such files would be guaranteed to break distcheck for those checkouts...
Sean (Jul 27 2020 at 17:17):I was thinking them not being there might break the build for the same reason -- some file somewhere in the checkout listing that dir, and it not existing
Sean (Jul 27 2020 at 17:17):there's a couple manual compilation verfications on our list, so that should present itself then if it's an issue
Sean (Jul 27 2020 at 17:19):there's only a concern if their absence adds to build breakage
starseeker (Jul 27 2020 at 17:20):Let's wait and see if we have to then - new files in empty dirs would be difficult to add and after the CMake based file tracking was added, virtually guaranteed to break any distcheck tests.
Sean (Jul 27 2020 at 17:26):sounds good. if it comes to a choice, breaking build should override breaking distcheck but I'll be a little surprised if it does break the build.
Sean (Jul 27 2020 at 17:26):I don't think we ever made empty dirs (other than as process of creating the dir in svn) that stayed empty, so it's mainly our 3rd party codes
Sumagna Das (Jul 27 2020 at 17:43):@Sean difference.txt this file contains all the differences it found in any of the commits
Sumagna Das (Jul 27 2020 at 17:51):i changed the part where it prints "no differences" to the file for every commit to just print the differences for the commits it finds in
Sumagna Das (Jul 27 2020 at 18:14):so my script had a bug in it because of which it found those so called differences
starseeker (Jul 27 2020 at 18:31):That's my life in a nutshell these last weeks...
Sumagna Das (Jul 27 2020 at 18:31):hmmm
Sumagna Das (Jul 27 2020 at 18:32):it didnt change branches according to the commit desc
Sumagna Das (Jul 27 2020 at 18:32):because of which some changes were not found on trunk
Sumagna Das (Jul 27 2020 at 18:34):on the github repo, master == trunk or is it the branch which contains all the branches' stuff?
starseeker (Jul 27 2020 at 18:44):master == trunk
Sumagna Das (Jul 27 2020 at 18:46):because in some of the commits, there were svn:branch:bioh or something like that which stated that it was from other branch and this created some differences with the trunk
Sumagna Das (Jul 27 2020 at 18:47):so should i ignore them or let it checkout the correct branch in the svn part only and find any differences?
starseeker (Jul 27 2020 at 18:58):that notation indicates the commit in svn was made to the bioh branch, not to trunk.
Sumagna Das (Jul 27 2020 at 18:59):so should i ignore commits from other svn branches?
starseeker (Jul 27 2020 at 18:59):The best thing to do would be to check out the branch and check it against the branch...
Sumagna Das (Jul 27 2020 at 18:59):ok done
starseeker (Jul 27 2020 at 19:01):@Sumagna Das that's a question, actually - if you check out bioh and master from git at that point, what do you get?
starseeker (Jul 27 2020 at 19:01):Git's branching model is quite different from SVN and I'm still working my way through all the implications.
Sumagna Das (Jul 27 2020 at 19:01):mismatching files
starseeker (Jul 27 2020 at 19:02):I mean if you check a git checkout of bioh at that point against SVN's bioh, and git's master vs SVN's trunk at the same point, does everything line up? The bioh I would expect to, since there was a commit to the branch at that point, but post-merge the bioh commit is also on master's history so I'm wondering what Git does vs SVN in trunk at the same point.
Sumagna Das (Jul 27 2020 at 19:04):so should i ignore reference to other svn branches in master for now?
starseeker (Jul 27 2020 at 19:05):Probably, yes.
starseeker (Jul 27 2020 at 19:06):The commits where svn:branch:trunk is noted are where master and trunk should line up - I'm less certain what to expect if you pull master at the merged bioh commit...
Sumagna Das (Jul 27 2020 at 19:06):later i will check for other branches
Sumagna Das (Jul 27 2020 at 19:10):i am going to make it ignore commits for other branches onwards because checking about 360+ revisions again is going to take some time for now atleast
Sean (Jul 27 2020 at 20:56):So that's a curious discussion -- so what does it mean to have a commit on trunk that says svn:branch:bioh ? or at least, what's that supposed to mean?
Sean (Jul 27 2020 at 20:57):I wouldn't have expected to find any/many branch commits on master unless there was a branch-to-trunk merge that it's tracking and it just pulled the commits from the branch
Sean (Jul 27 2020 at 20:57):meaning the work originated on the branch, but got merged to trunk
starseeker (Jul 27 2020 at 21:52):Right - work originating in branches, but merged to trunk. The way git history works, the branch commits then become part of master's history (if I'm understanding this correctly.)
starseeker (Jul 27 2020 at 22:25):That's in fact the main reason I added the svn:branch labels, and what's making my life complicated with CVS right now - in Git, once that merge happens, the commit that happened on bioh shows up as part of the history of both bioh and master. I guess a checkout would reference the rev-list relationships, if I'm understanding the proposal for a "date based" checkout here: https://stackoverflow.com/a/6990682
git rev-list --first-parent bullet --pretty=format:"%B" actually looks like it might do what I would expect for following an SVN branch's history (although it doesn't stop when the branch was created) - I'll have to compare that output to what the svn:branch based output produces. It certainly looks closer than anything I've found yet... Still doesn't tell me when the branch was originally created though.
starseeker (Jul 27 2020 at 23:46):@Sean Checked a little over 4700 CVS commits thus far, looks like about a dozen in there that have the appearance of legit differences between the CVS checkout and the git checkout. (getting some empty CVS checkouts as well trying to generate checkout lines from git info, but not 100% sure why - could be my fault, and I'm thinking this is starting to hit diminishing returns...)
Based on that rate of processing, it'll take at least a couple days to work through all the checks, maybe a bit more, when the final conversion run rolls around - is it worth spending more time on to try and make the fixes, or should we just accept the CVS conversion as-is?
Sean (Jul 28 2020 at 01:52):o.O
Sean (Jul 28 2020 at 01:55):Not much info to work with there other than ... "a dozen legitimate differences" sounds like an outright verification failure. We should inspect and what matters is whether the commit really is there or orphaned or elsewhere or what. Can you point me at a couple of the commits in question?
starseeker (Jul 28 2020 at 01:56):r29322 I think is one of them
starseeker (Jul 28 2020 at 01:57):29839
starseeker (Jul 28 2020 at 01:58):CVS checkout line for r29839:
cvs -d /home/user/brlcad_cvs -Q co -ko -D "2007-12-20 20:34:31 +0000" -P brlcad
starseeker (Jul 28 2020 at 02:00):For r29322:
cvs -d /home/user/brlcad_cvs -Q co -ko -D "2007-11-13 17:32:36 +0000" -P brlcad
starseeker (Jul 28 2020 at 02:03):The logic I'm using is in misc/repoconv/verify - although for this part I'm not using an SVN repository, just the CVS.
starseeker (Jul 28 2020 at 02:10):I'm not sure I'm right to be comparing against master in some of these cases, but I'm also having a heck of a time figuring out which branch I should be using instead...
starseeker (Jul 28 2020 at 02:20):29011, for example, looks like it was a merge onto a branch, but I can't seem to tell from git which one it was merged onto
starseeker (Jul 28 2020 at 03:18):/me is beginning to think he can only reasonably use the -D option on cvs against trunk/master... can't find anything about applying it to branches... grr.
Sumagna Das (Jul 28 2020 at 03:20):i have found some commits which dont have any information for svn or are for another branch
starseeker (Jul 28 2020 at 03:21):Unexpectedly so?
Sumagna Das (Jul 28 2020 at 03:22):i will give you the list i found until now
starseeker (Jul 28 2020 at 03:24):@Sumagna Das it's getting pretty late here - go ahead and post it, and I'll take a look tomorrow.
Sumagna Das (Jul 28 2020 at 03:24):ok
Sumagna Das (Jul 28 2020 at 03:24):i am in class right now
Sumagna Das (Jul 28 2020 at 03:48):so......
Sumagna Das (Jul 28 2020 at 03:49):i had to force shutdown my computer and i lost the commit sha1 list
Sumagna Das (Jul 28 2020 at 03:49):i can only say that i had encountered 4-6 of them in total
Sumagna Das (Jul 28 2020 at 04:45):i found this one just now
de92ecbe1a
Sumagna Das (Jul 28 2020 at 04:45):it is also skipped
starseeker (Jul 28 2020 at 12:23):Gah. @Sean r19102 is representative of a general issue I'm seeing starting somewhere in the 19k range - the git checkout has libitcl, libtcl and libtk present in the checkout but CVS apparently does not.
I'll have to use the better cvs branch mappings I generated this morning to try and create better checkout commands - see if I can rule that out as a potential trouble source...
Sumagna Das (Jul 28 2020 at 14:29): Sumagna Das (Jul 28 2020 at 14:29):these are the commits i found after the previous wipe out
starseeker (Jul 28 2020 at 14:30):Any surprising/unexpected?
Sumagna Das (Jul 28 2020 at 14:31):i changed the script to print to a file the list of commits which were skipped from one of two reasons
Sumagna Das (Jul 28 2020 at 14:31):starseeker said:
Any surprising/unexpected?
i cant check while the script is running(i run it whenever i come online on my laptop else i dont have time)
Sumagna Das (Jul 28 2020 at 14:32):it is checking r75780 right now
starseeker (Jul 28 2020 at 14:34):Which github checkout are you using?
Sumagna Das (Jul 28 2020 at 14:37):Repo you mean?
Sumagna Das (Jul 28 2020 at 14:39):If yes, then github.com/starseeker/brlcad_conv8
starseeker (Jul 28 2020 at 14:42):OK, good (that tells us how to decode the sha1 hashes). Unfortunately, they change each time I change the history, so it matters which iteration is being checked.
Sumagna Das (Jul 28 2020 at 17:44):ok
starseeker (Jul 29 2020 at 00:56):@Sean I'm really not sure what to make of this. If I check out in an early date range, for example:
cvs -d /home/user/brlcad_cvs -Q co -ko -D "1999-04-21 17:23:51 +0000" -P brlcad
I'm getting a CVS checkout that does not contain libtcl or libtk. (The equivalent git clone does contain them.) However, if I look at doc/install.doc it does seem to refer to libtcl, as does gen.sh) Is it possible modern CVS is misinterpreting the old CVS repository somehow? I'm more surprised those directories are absent in the CVS checkout than that they are present in the git clone...
starseeker (Jul 29 2020 at 01:06):I do admit, though, that the conversion produced by https://github.com/rcls/crap does seem to skip including those directories when I check the equivalent commit...
starseeker (Jul 29 2020 at 01:29):Back on the bandwagon at r15263 from the looks of things...
starseeker (Jul 29 2020 at 01:34):So the SVN checkout at r15264 seems to agree better with the cvs-fast-export result, in that it has libtcl and libtk in the checkout.
starseeker (Jul 29 2020 at 01:49):CVS checkout for r15624:
cvs -d /home/user/brlcad_cvs -Q co -ko -D "1998-06-02 19:54:25 +0000" -P brlcad
starseeker (Jul 29 2020 at 12:05):@Sean If it's of interest, here were my CVS comparison results:
verify_cvs.log
starseeker (Jul 29 2020 at 12:32):Just shy of 500 checkout attempts generated an empty CVS tree, a lot of which I think are due to my not having a correct CVS checkout command for historical commits on branches. Since an empty CVS tree most likely isn't the goal there, I'm not generating 'eliminate the tree' diffs.
4948 commits generated a non-empty difference file, of which the majority (though certainly not all) are in that series in the 15k-19k range where CVS doesn't check out libtcl/libtk and friends.
At this point I see 3 options:
1) Just accept the cvs-fast-export conversion for what it is. Spot checking r15624, CVS, SVN and Git come to three different conclusions about what's there, so we never had perfect agreement with CVS to begin with, and we do seem to be able to trace back file history successfully (I've checked src/libnmg/bool.c and src/librt/primitives/ell/ell.c.)
2) Cherrypick those commit diffs that aren't part of the libtcl/libtk/etc. wrangle and apply just those, to match the CVS checkout when we don't appear to be caught up in that tangle. (That one's the most work.)
3) Try to force all the commits for which we have the non-empty delta to match the CVS checkout. Defensible as it gets us closest of the available options to what a CVS checkout would yield, but we may very well end up losing some of the libtcl/etc. folder contents in the garbage collect if CVS would never check them out.
Thoughts?
starseeker (Jul 29 2020 at 12:47):@Sumagna Das Of those commits, the two interesting ones are ecae1b1ed1 and 29b15aa2c4 - what differences are you seeing? Or is it reporting skipping those because they are on a branch?
starseeker (Jul 29 2020 at 12:48):@Sumagna Das oh, nevermind, I see you're printing skipped commits not commits where a difference was found.
Sean (Jul 29 2020 at 13:46):starseeker said:
Sean I'm really not sure what to make of this. If I check out in an early date range, for example:
cvs -d /home/user/brlcad_cvs -Q co -ko -D "1999-04-21 17:23:51 +0000" -P brlcad
I'm getting a CVS checkout that does not contain libtcl or libtk. (The equivalent git clone does contain them.) However, if I look at doc/install.doc it does seem to refer to libtcl, as does gen.sh) Is it possible modern CVS is misinterpreting the old CVS repository somehow? I'm more surprised those directories are absent in the CVS checkout than that they are present in the git clone...
No, that's not likely. We did mess with those directories a lot back then. CVS doesn't track directories in any capacity, so if a directory is renamed for example, CVS does not know about it. It's just a tree of directories with RCS files.
Sean (Jul 29 2020 at 13:47):What probably happened is those directories were manually renamed in the repo, so you probably have a libtcl8.3 or something checked out, whatever the last name was before we moved to SVN and they started getting tracked.
Sean (Jul 29 2020 at 13:48):With the dir moved in the repo, it makes it that moved name back through time so even if you check out and older rev, you'd get the new dir name.
Sean (Jul 29 2020 at 13:49):I think that renaming nonsense was isolated to the various tcl dirs.
starseeker (Jul 29 2020 at 13:49):Seems to be - I haven't scanned all of the diffs manually, but that's the only place I noticed it
Sean (Jul 29 2020 at 13:50):if you check the checkout, you almost certainly have the dirs that are "missing", they just have a version number on the dir name
Sean (Jul 29 2020 at 13:51):the fix is as simple as renaming them back to whatever the build is looking for, so I wouldn't be terribly worried about it so long as they're there
starseeker (Jul 29 2020 at 13:52):I've not tested it at anything like this scale, but in principle I can actually force the conversion to match the checked out CVS tree.
Sean (Jul 29 2020 at 13:52):in this case, it's debatable which is right
Sean (Jul 29 2020 at 13:53):basically, a backwards-incompatible change was made to the CVS repo at various points in time
starseeker (Jul 29 2020 at 13:53):Ah. Hmm.
Sean (Jul 29 2020 at 13:53):such that checking out an old version as it actually existed is no longer possible without knowing what directories were renamed from-to
starseeker (Jul 29 2020 at 13:54):/me is inclined to go with the cvs-fast-export results, in that case... don't see much benefit to fighting with this, and on the whole the diff results were actually pretty good.
Sean (Jul 29 2020 at 13:55):CVS repo had src/libtcl in r123, it was renamed to src/libtcl8 in r234 by renaming the backend directory -- thus checking out r123 now will also be src/libtcl8
starseeker (Jul 29 2020 at 13:55):ew.
starseeker (Jul 29 2020 at 13:56):Thank you for moving us to SVN just as I came onboard - much appreciated!
Sean (Jul 29 2020 at 13:56):like I said, CVS does not know about or track directories in any capacity
Sean (Jul 29 2020 at 13:57):and I think there was a reason for changing the backend directory.. i think if one created a src/libtcl8 and added all the files from src/libtcl and then deleted all of the files in libtcl, that has some undesirable effect like giving everyone an empty src/libtcl on future checkouts, so you had to scan and prune empty dirs
starseeker (Jul 29 2020 at 13:58):/me nods
Sean (Jul 29 2020 at 13:58):and not just an empty src/libtcl, but a hierarchy of all the src/libtcl dirs .. all empty. I could be wrong, but it was something fugly
Sean (Jul 29 2020 at 14:01):Sumagna Das said:
Sean difference.txt this file contains all the differences it found in any of the commits
This is awesome @Sumagna Das thank you for all your help with this! It's great to get some independent verification and validation.
Sumagna Das (Jul 29 2020 at 20:00):oh i have the new updated skipped_commits.txt
Sumagna Das (Jul 29 2020 at 20:01):and due to some bug in my code, there are commits which have been printed twice or multiple times.
Sumagna Das (Jul 29 2020 at 20:01): starseeker (Jul 29 2020 at 20:04):If you can, it would be helpful to translate those sha1 values into log messages when you generate the skipped_commits.txt file:
git log -n1 <sha1> --pretty=format:"%h:%n%B"
Sumagna Das (Jul 29 2020 at 20:05):to see why it has been skipped?
starseeker (Jul 29 2020 at 20:05):yes
Sumagna Das (Jul 29 2020 at 20:05):ok
Sumagna Das (Jul 29 2020 at 20:05):wll do that the next time i run the script
starseeker (Jul 29 2020 at 20:06):Skipped is less critical than finding a difference though - the latter indicates at least a possibility of a problem in the conversion.
Sumagna Das (Jul 29 2020 at 20:06):ok
Sumagna Das (Jul 29 2020 at 20:06):until now, no new difference has been found
starseeker (Jul 29 2020 at 20:07):that's good :-)
Sumagna Das (Aug 02 2020 at 14:45):hey @starseeker, i used git svn to clone the full history of the svn repo
Sumagna Das (Aug 02 2020 at 14:45):because pulling it everytime was taking much time
Sumagna Das (Aug 02 2020 at 15:10):if thats going to change anything please inform me
Sumagna Das (Aug 02 2020 at 15:33):i am not going to run the script before you tell me anything about any issues with this method if present
Sumagna Das (Aug 02 2020 at 15:34):as if i run the script, its going to go very fast through checking out commits, diff'ing them and repeat the process
starseeker (Aug 02 2020 at 16:23):I've never tried that, but in principle it should work - go ahead and give it a shot.
Sumagna Das (Aug 02 2020 at 16:24):it will give the script a speed boost, a major speed boost
starseeker (Aug 02 2020 at 16:24):that'll be faster even than an svn checkout from a local rsync copy of the SVN repo?
Sumagna Das (Aug 02 2020 at 16:24):yea
starseeker (Aug 02 2020 at 16:25):OK, sure - give it a try. If you find a difference we'll need to double check it with the regular checkout, but see what it can do
Sumagna Das (Aug 02 2020 at 16:25):ok
starseeker (Aug 02 2020 at 16:27):Remember, once you get below 29887 there are likely to be some differences, as you'll be into the CVS portion of the history. (I posted my results of a direct CVS/git comparison a couple days back...)
Sumagna Das (Aug 02 2020 at 16:27):ok
starseeker (Aug 02 2020 at 16:27):Anything that shows up 29887 or later as a difference is definitely a concern.
Sumagna Das (Aug 02 2020 at 16:28):ok
starseeker (Aug 02 2020 at 16:29):Before that... it depends, but my inclination at this point barring some sort of catastrophic issue is to accept the cvs-fast-export results - from what Sean was saying there was at least one change made in the CVS days that would royally complicate the reconstruction challenge for converters and probably explains some odd results in the current CVS checkout in the 15k-19k range.
starseeker (Aug 02 2020 at 16:31):However, a final decision must wait on the results - that's why we need the testing in the first place. :wink:
Sumagna Das (Aug 02 2020 at 16:35):ok
Sumagna Das (Aug 02 2020 at 16:37):and it will not be similar to "import repo" in github because the email doesnt have to be mapped to any github account
Sumagna Das (Aug 02 2020 at 16:37):so each revision/commit will be like the svn one
Sumagna Das (Aug 02 2020 at 16:37):i asked someone in git's irc channel and he said that it wont change anything about the svn repo
Sumagna Das (Aug 02 2020 at 17:30):now the overall time taken is the time for the diff command
starseeker (Aug 02 2020 at 17:31):nice!
Sumagna Das (Aug 02 2020 at 17:32):earlier it took about 10mins to check a revision and now it takes about (maximum) 2-3 minutes if the diff command doesnt take too long
Sumagna Das (Aug 02 2020 at 17:32):else about 1 minute
starseeker (Aug 02 2020 at 17:33):/me winces. Yeah, that's pretty rough.
Sumagna Das (Aug 02 2020 at 17:33):it took about 3-4 nights (not days because i could turn it on for the whole night only as my father worked during the day in the windows part) which was the time i had to compromise
Sumagna Das (Aug 02 2020 at 17:33):and it paid off
starseeker (Aug 02 2020 at 17:37):@Sean Do you think it might be better (given those performance constraints) to do a random scattering of revision checks instead of marching through all of them? I know that's less thorough, but at 2-3min per commit that'd be more than a month to run just back to the beginning of the SVN history...
Sumagna Das (Aug 02 2020 at 17:38):until now, there is no changes in the difference.txt file but because of a porting bug(i forgot to change the rmdir empty directories part according to the recent change) it is filled with "web folder only in svn repo" lines
starseeker (Aug 02 2020 at 17:38):@Sumagna Das that happens - as long as we know to ignore them, should be OK.
Sumagna Das (Aug 02 2020 at 17:38):(deleted)
Sumagna Das (Aug 02 2020 at 17:39):starseeker said:
Sumagna Das that happens - as long as we know to ignore them, should be OK.
i found out the bug and fixed it
Sumagna Das (Aug 02 2020 at 17:39):now it is deleting them
starseeker (Aug 02 2020 at 17:39):@Sumagna Das can you print the log msgs?
Sumagna Das (Aug 02 2020 at 17:39):log?
starseeker (Aug 02 2020 at 17:40):git log -n1 <sha1> --pretty=format:"%h:%n%B"
Sumagna Das (Aug 02 2020 at 17:40):ohh
starseeker (Aug 02 2020 at 17:40):instead of just the sha1 (so we know which commits we're seeing)
Sumagna Das (Aug 02 2020 at 17:40):for each of them?
starseeker (Aug 02 2020 at 17:40):yes
Sumagna Das (Aug 02 2020 at 17:41):in the difference.txt?
Sumagna Das (Aug 02 2020 at 17:41):or a new file?
starseeker (Aug 02 2020 at 17:41):in skipped_commits.txt was what I was thinking
Sumagna Das (Aug 02 2020 at 17:41):ok
Sumagna Das (Aug 02 2020 at 17:41):i will do that
starseeker (Aug 02 2020 at 17:42):But essentially, whenever you're reporting a commit it will be more useful/informative to have the log message. The sha1 itself is useful only as a lookup key - if the log has already done the lookup, we can understand what we are seeing.
Sumagna Das (Aug 02 2020 at 17:42):i have to clear the skipped_commit.txt
starseeker (Aug 02 2020 at 17:42):sure
Sumagna Das (Aug 02 2020 at 17:42):it still has some commits from the earlier conv8 repo
starseeker (Aug 02 2020 at 17:43):Ah. I think we already checked and cleared those, so don't worry about them.
Sumagna Das (Aug 02 2020 at 17:49):thanks for the git command
Sumagna Das (Aug 02 2020 at 20:37):at the speed the script is doing work....
Sumagna Das (Aug 02 2020 at 20:39):as per calculations, it would take 56-60 hours for my script to check all the commits at max(i did not take in consideration the fact that revisions might be skipped)
starseeker (Aug 03 2020 at 01:05):wow, that is quite a speed-up. I hadn't realized SVN checkout was that slow.
Sumagna Das (Aug 03 2020 at 01:05)::grinning_face_with_smiling_eyes:
Sumagna Das (Aug 03 2020 at 01:06):couldnt sleep last night..... fixing the script time to time :sleepy:
starseeker (Aug 03 2020 at 01:07):heh - don't need to go sleep-deprived, time pressure isn't that critical. Just didn't want you to have to have the computer thrashing on commits for a month.
Sumagna Das (Aug 03 2020 at 01:07):and take this
Sumagna Das (Aug 03 2020 at 01:07): Sumagna Das (Aug 03 2020 at 01:08):the script was one of the reasons. the other reason was my sleep cycle because of which i am awake at night and asleep at day
starseeker (Aug 03 2020 at 01:08):excellent!
starseeker (Aug 03 2020 at 01:09):So, you can see the pattern - skipping file move commits and branches, as expected.
Sumagna Das (Aug 03 2020 at 01:09):yea
Sumagna Das (Aug 03 2020 at 01:10):all of them are either file move commits or branch merge ones
Sumagna Das (Aug 03 2020 at 01:10):right now it is at 61k
starseeker (Aug 03 2020 at 01:11):which is good. Have any differences been found yet?
Sumagna Das (Aug 03 2020 at 01:11):and when i started it 6 hours ago it was around 73k
Sumagna Das (Aug 03 2020 at 01:11):lemme check
Sumagna Das (Aug 03 2020 at 01:11):actually yea
Sumagna Das (Aug 03 2020 at 01:11):i dont know if its because of my script or not
Sumagna Das (Aug 03 2020 at 01:12): Sumagna Das (Aug 03 2020 at 01:12):heres the file
starseeker (Aug 03 2020 at 01:13):Hmm. So you'll want to pick a couple of those files from one of the checkouts and inspect the differences to see what it is finding.
Sumagna Das (Aug 03 2020 at 01:13):see if the differences are actually present or just a bug in my script
Sumagna Das (Aug 03 2020 at 01:13):will do that later
Sumagna Das (Aug 03 2020 at 01:14):i was telling you to check because the script is constantly checking out a new commit from the github repo
starseeker (Aug 03 2020 at 01:14):One sec... I'll try 70527
starseeker (Aug 03 2020 at 01:17):r70527 looks like it was a commit to the brep-debug branch
starseeker (Aug 03 2020 at 01:18):so if you're checking a commit to brep-debug against trunk at that revision, that's not going to match (nor would it be expected to)
Sumagna Das (Aug 03 2020 at 01:21):that was probably present in the trunk/master branch in github repo
Sumagna Das (Aug 03 2020 at 01:24):so it was checked
starseeker (Aug 03 2020 at 01:50):If you mean in git log, it is, but that doesn't mean there will be a trunk checkout that corresponds to it - that's one of the reasons for the svn:branch: labels. The way git reports its history is rather different than SVN, and git log in any given branch won't line up with what svn log would produce without a fair bit of fiddling.
starseeker (Aug 03 2020 at 01:52):If you don't want to check out the branches, you'll want to filter out any commit that doesn't have svn:branch:trunk in its message
Sean (Aug 03 2020 at 06:43):starseeker said:
Sean Do you think it might be better (given those performance constraints) to do a random scattering of revision checks instead of marching through all of them? I know that's less thorough, but at 2-3min per commit that'd be more than a month to run just back to the beginning of the SVN history...
4-way parallel, they're done in a week
starseeker (Aug 03 2020 at 12:06):Well, if his machine can handle the I/O load...
Sumagna Das (Aug 03 2020 at 14:48):starseeker said:
Well, if his machine can handle the I/O load...
what am i supposed to do?
Sumagna Das (Aug 03 2020 at 15:41): Sumagna Das (Aug 03 2020 at 15:41):take this
starseeker (Aug 03 2020 at 17:19):Looks good
Sumagna Das (Aug 03 2020 at 17:19):Sumagna Das said:
starseeker said:
Well, if his machine can handle the I/O load...
what am i supposed to do?
what were you talking about?
starseeker (Aug 03 2020 at 17:20):Was concerned about the stress on your computer when you were doing the checkouts from SVN directly - it sounds like you solved the problem with git-svn though, so no longer a concern.
Sumagna Das (Aug 03 2020 at 17:20):oh ok
Sumagna Das (Aug 04 2020 at 03:44):the script is at 30k-40k range right now
Sumagna Das (Aug 04 2020 at 03:44): Sumagna Das (Aug 04 2020 at 03:45):no differences dound after r69289 until now
Sumagna Das (Aug 04 2020 at 12:25):Sumagna Das said:
no differences dound after r69289 until now
which range should i expect differences?
starseeker (Aug 04 2020 at 14:07):You'll probably seem some differences appearing below r29886
starseeker (Aug 04 2020 at 14:09):At that point, you're comparing how cvs2svn and cvs-fast-export (the git conversion) interpreted the CVS history - ours was/is convoluted enough that the interpretation is likely to have ambiguities.
Sumagna Das (Aug 04 2020 at 14:13):my script is right now at 35859
Sumagna Das (Aug 04 2020 at 20:09):currently at 30646.... woohoo !!!! :celebration: :tada:
Sumagna Das (Aug 04 2020 at 20:10):it will probably check upto 27k or 28k if left tonight
Sumagna Das (Aug 04 2020 at 20:58):29k reached
Sumagna Das (Aug 04 2020 at 20:58):so some differences may start appearing now?
starseeker (Aug 04 2020 at 21:20):I wouldn't be surprised.
starseeker (Aug 04 2020 at 21:20):So no differences found?
starseeker (Aug 04 2020 at 21:20):(no differences found so far, rather?)
Sumagna Das (Aug 05 2020 at 02:59):its actually skipping all the cvs commit with "other branch" skipping functionality
Sumagna Das (Aug 05 2020 at 03:01):so should i remove the "if not trunk then skip" skipping functionality?
Sumagna Das (Aug 05 2020 at 03:33):hey @Sean what should i do? should i comment the skipping part?
Sumagna Das (Aug 05 2020 at 03:51):i have to clone the repo again because my local copy of the repo somehow broke and i cannot fix t
starseeker (Aug 05 2020 at 12:42):I would adjust the skipping to try cvs:branch:trunk commits as well
Sean (Aug 05 2020 at 15:59):Sumagna Das said:
so should i remove the "if not trunk then skip" skipping functionality?
yeah, I'm not as concerned with branch commits unless they have some impact on trunk. they're already at risk of being orphaned in git, which sucks but is the git way.
Sean (Aug 05 2020 at 16:00):that said, cvs wasn't "on a branch" so that's a separate beast to tangle with
Sean (Aug 05 2020 at 16:01):Sumagna Das said:
i have to clone the repo again because my local copy of the repo somehow broke and i cannot fix t
that's super concerning ... do you know how? is all you were doing checkouts? did you manually modify files in .git dirs?
Sean (Aug 05 2020 at 16:03):there is always the possibility of a sleeper in the repo that corrupts when accessed, so it would be good to know how you corrupted a checkout. if you cause it by editing .git files, for example, then there's no concern whatsoever. if git did it simply by reading the repo, that would be concerning and worth investigating more deeply.
starseeker (Aug 05 2020 at 17:27):I've done a few odd things to repos, but it's usually when I try something fairly dangerous - I agree checkouts shouldn't do anything weird.
What were the symptoms of the corruption? What error messages did you see?
Sumagna Das (Aug 05 2020 at 17:34):they were like
please stash or commit the following changes:
misc/......
..........
Ah - that means something changed the contents of the git archive between one checkout and the next.
Sumagna Das (Aug 05 2020 at 17:37):Sean said:
Sumagna Das said:
i have to clone the repo again because my local copy of the repo somehow broke and i cannot fix t
that's super concerning ... do you know how? is all you were doing checkouts? did you manually modify files in .git dirs?
didnt do anything. i just do a checkout in svn repo, use diff checkout the next commit in git repo and repeat
starseeker (Aug 05 2020 at 17:37):Git will try to prevent you wiping out changes you have been making - for example, if I'm editing file A.txt on branch A and want to checkout branch B, Git wants to know what to do about the modified A.txt.
starseeker (Aug 05 2020 at 17:37):Right - so the question is why something was seen as being modified.
Sumagna Das (Aug 05 2020 at 17:38):i just workaround this error using git checkout -q -f <sha1>
starseeker (Aug 05 2020 at 17:39):One possibility might be if it's assigning line endings on checkout - that could change line endings and produce locally modified files - but I'd have expected something like that to show up earlier in the process...
Sumagna Das (Aug 05 2020 at 17:40):its showing up more often recently
starseeker (Aug 05 2020 at 17:41):My suggestion is to do a git checkout somewhere of one of the sha1 commits that is showing this problem, then see what git diff thinks after it is checked out.
Sumagna Das (Aug 05 2020 at 17:42):now difference actually
starseeker (Aug 05 2020 at 17:42):The converted history doesn't have a .gitattributes file in it (we'll need to add that once we transition, I've got a template staged in misc/repoconv) so that's one possibility.
Sumagna Das (Aug 05 2020 at 17:43):error: unable to unlink old 'src/mged/dm_old/dm-hp.c': Permission denied
error: unable to unlink old 'src/mged/dm_old/dm-oglX.c': Permission denied
error: unable to unlink old 'src/mged/dm_old/dm-tek4109.c': Permission denied
this is one of the things that showed up
starseeker (Aug 05 2020 at 17:44):Um. That's interesting, actually. What sha1 is that?
Sumagna Das (Aug 05 2020 at 17:45):i tried to checkout this -> b74655fa52 from this -> 78553fdee7
starseeker (Aug 05 2020 at 17:46):Are you on Linux or Windows doing this?
Sumagna Das (Aug 05 2020 at 17:46):linux
starseeker (Aug 05 2020 at 17:46):What's your umask?
Sumagna Das (Aug 05 2020 at 17:46):umask?
starseeker (Aug 05 2020 at 17:47):Run the command "umask" - it will report some numbers
starseeker (Aug 05 2020 at 17:47):they relate to default file permissions on your filesystem
Sumagna Das (Aug 05 2020 at 17:47):0002
starseeker (Aug 05 2020 at 17:47):OK, that looks right - what version of git are you using?
Sumagna Das (Aug 05 2020 at 17:47):2.25.1
starseeker (Aug 05 2020 at 17:48):OK, that matches. Are you still checked out at commit 78553fdee7?
starseeker (Aug 05 2020 at 17:48):if so, what does git status report?
Sumagna Das (Aug 05 2020 at 17:48):that was my previous head position
Sumagna Das (Aug 05 2020 at 17:49):if u want me to checkout, then thats no problem
starseeker (Aug 05 2020 at 17:49):I'm wondering if you can reproduce that particular failure.
starseeker (Aug 05 2020 at 17:50):Checkout 78553fdee7, see what git status says, then try to checkout b74655fa52 and see what happens
Sumagna Das (Aug 05 2020 at 17:50):actually the modified files wouldnt let me checkout the 78553fdee7 commit
Sumagna Das (Aug 05 2020 at 17:50):HEAD detached at b74655fa52
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: src/mged/dm_old/dm-hp.c
modified: src/mged/dm_old/dm-oglX.c
modified: src/mged/dm_old/dm-tek4109.c
no changes added to commit (use "git add" and/or "git commit -a")
this git status report right now before switching to 78553fdee7 commit
starseeker (Aug 05 2020 at 17:51):what does git log say about src/mged/dm_old/dm-tek4109.c
starseeker (Aug 05 2020 at 17:52):and git diff, for that matter
Sumagna Das (Aug 05 2020 at 17:52):git log -1?
Sumagna Das (Aug 05 2020 at 17:53):commit f81cd07759396ef3706069575bb415ef4739eca9
Author: Christopher Sean Morrison <brlcad@gmail.com>
Date: Tue Feb 20 08:19:51 2007 +0000
update all usages of fgets() to instead use john's swanktastic bu_fgets() that behaves as one would generally want regardless of the line ending type of the compilation platform or of the input files. bu_fgets() responds to input files that use CR (usually old mac), LF (usually unix, new mac), or CR/LF (usually windows) for the line ending so now these file do too effectivley squashing buggish/bad behavior.
svn:revision:27628
cvs:account:brlcad
cvs:branch:trunk
git log -1 src/mged/dm_old/dm-tek4109.c :up_button:
Sumagna Das (Aug 05 2020 at 17:54):git diff src/mged/dm_old/dm-tek4109.c :down_button:
diff --git a/src/mged/dm_old/dm-tek4109.c b/src/mged/dm_old/dm-tek4109.c
index 53106cc4ac..d8bb9d1078 100644
--- a/src/mged/dm_old/dm-tek4109.c
+++ b/src/mged/dm_old/dm-tek4109.c
@@ -161,7 +161,7 @@ T49_open()
char line[64], line2[64];
bu_log("Output tty [stdout]? ");
- (void)bu_fgets( line, sizeof(line), stdin ); /* \n, null terminated */
+ (void)fgets( line, sizeof(line), stdin ); /* \n, null terminated */
line[strlen(line)-1] = '\0'; /* remove newline */
if( feof(stdin) )
quit();
So if you do git checkout f81cd07759396ef3706069575bb415ef4739eca9 what happens?
Sumagna Das (Aug 05 2020 at 17:58):it reports no issue
starseeker (Aug 05 2020 at 17:58):what does git status say from that checkout?
Sumagna Das (Aug 05 2020 at 17:58):but the changes are still there
Sumagna Das (Aug 05 2020 at 17:59):git status still says the files (which were stated before) were changed
starseeker (Aug 05 2020 at 17:59):What does git diff report?
Sumagna Das (Aug 05 2020 at 18:00):same change
Sumagna Das (Aug 05 2020 at 18:00):Sumagna Das said:
git diff src/mged/dm_old/dm-tek4109.c :down_button:
diff --git a/src/mged/dm_old/dm-tek4109.c b/src/mged/dm_old/dm-tek4109.c index 53106cc4ac..d8bb9d1078 100644 --- a/src/mged/dm_old/dm-tek4109.c +++ b/src/mged/dm_old/dm-tek4109.c @@ -161,7 +161,7 @@ T49_open() char line[64], line2[64]; bu_log("Output tty [stdout]? "); - (void)bu_fgets( line, sizeof(line), stdin ); /* \n, null terminated */ + (void)fgets( line, sizeof(line), stdin ); /* \n, null terminated */ line[strlen(line)-1] = '\0'; /* remove newline */ if( feof(stdin) ) quit();
this change :up:
starseeker (Aug 05 2020 at 18:03):So it's checking out that revision, but it has the older files somehow
Sumagna Das (Aug 05 2020 at 18:04):think so
starseeker (Aug 05 2020 at 18:04):I can't seem to reproduce it here - do you have any steps that will? (A series of checkouts?)
Sumagna Das (Aug 05 2020 at 18:05):it just happened suddenly
Sumagna Das (Aug 05 2020 at 18:05):and this is a new clone of the repo
Sumagna Das (Aug 05 2020 at 18:05):should i try cloning the repo again and see if it happens there?
starseeker (Aug 05 2020 at 18:08):yeah
Sumagna Das (Aug 05 2020 at 18:10):my internet speed is down again (for github) :sad:
starseeker (Aug 05 2020 at 18:11):What if you clone from your local checkout to another local checkout?
Sumagna Das (Aug 05 2020 at 18:11):??
Sumagna Das (Aug 05 2020 at 18:12):i will try to clone after 20 minutes ( at 12:00 am midnight). the speed will be back at that time
Sumagna Das (Aug 05 2020 at 18:12):starseeker said:
What if you clone from your local checkout to another local checkout?
is that possible?
starseeker (Aug 05 2020 at 18:13):if you have a repo in:
/home/user/brlcad_conv10
try
mkdir /home/user/checkout2 && cd /home/user/checkout2 && git clone /home/user/brlcad_conv10
Sumagna Das (Aug 05 2020 at 18:13):started it already
Sumagna Das (Aug 05 2020 at 18:14):and its fast (19 MB/s)
starseeker (Aug 05 2020 at 18:14):You should end up with a checkout at /home/user/checkout2/brlcad_conv10
Sumagna Das (Aug 05 2020 at 18:16):starseeker said:
Checkout 78553fdee7, see what git status says, then try to checkout b74655fa52 and see what happens
i did this in the local checkout and it worked properly without any issues or errors
Sumagna Das (Aug 05 2020 at 18:16):ohhhhhhhhhhhh
Sumagna Das (Aug 05 2020 at 18:17):now i remember why this permission issue might be happening. can you tell me if this is the reason?
Sumagna Das (Aug 05 2020 at 18:19):i set up the "empty directory remover" but for some reason it did not work giving permission issues. i added a part in the script which elevates the script to superuser(which is pretty much dangerous)
Sumagna Das (Aug 05 2020 at 18:19):so it uses the sudo part even during git checkouts i think
Sumagna Das (Aug 05 2020 at 18:19):can that be the reason?
Sumagna Das (Aug 05 2020 at 18:28):Sean said:
that said, cvs wasn't "on a branch" so that's a separate beast to tangle with
but in the commit description it shows cvs:branch:trunk?
starseeker (Aug 05 2020 at 20:06):@Sumagna Das that certainly could be related, if you're mixing (or trying to mix) sudo and non-sudo operations.
In any case, you shouldn't need sudo for any of this - I'm not sure why you were getting permission errors on the empty dir removal, but sudo almost certainly wasn't what you wanted to do. (Also, make sure none of your script activities are operating inside the .git directory.)
Sumagna Das (Aug 05 2020 at 20:08):ok
starseeker (Aug 05 2020 at 20:08):The cvs:branch:trunk label indicates a CVS commit that was not in one of the branch chains in the CVS conversion. "cvs" denotes a version control system usage, rather than a branch - so "trunk" in this case is the conceptual branch used by CVS for a given commit, according to the cvs-fast-export analysis of the repo.
Sumagna Das (Aug 05 2020 at 20:09):so there might not be another cvs:branch:<branch_name> where branh_name is something other than trunk?
starseeker (Aug 05 2020 at 20:10):yes
starseeker (Aug 05 2020 at 20:10):in fact, there are
Sumagna Das (Aug 05 2020 at 20:10):thats done
Sumagna Das (Aug 05 2020 at 20:11):starseeker said:
in fact, there are
so i might wanna skip commits which are not for trunk?
starseeker (Aug 05 2020 at 20:11):yes
Sumagna Das (Aug 05 2020 at 20:11):i will review the script tomorrow and rerun the script again
Sumagna Das (Aug 05 2020 at 20:12):the laptop will get some rest for tonight
starseeker (Aug 05 2020 at 20:12):/me chuckles
Sumagna Das (Aug 06 2020 at 10:04):will most of the commits after 29886 differ from the github ones?
Sumagna Das (Aug 06 2020 at 10:04):because until 27637, no differences have been found
starseeker (Aug 06 2020 at 12:40):Most of them should be the same, based on my testing - I saw differences regularly starting in the 19k range with a CVS checkout, but it's not actually certain that will happen with GIT vs SVN - the SVN history is a cvs2svn reconstruction, so it may be closer to what the git history says than the vanilla CVS checkout.
Sumagna Das (Aug 06 2020 at 15:55):i dont know if this is weird but ....
check the last few lines of the skipped_commits.txt
starseeker (Aug 06 2020 at 15:56):what about it?
Sumagna Das (Aug 06 2020 at 15:57):why doesnt it have any svn:revision:<rev>?
starseeker (Aug 06 2020 at 15:58):Because it doesn't have an equivalent SVN commit. cvs2svn and cvs-fast-export broke down the CVS history in slightly different ways. That means we don't have an exact match for every commit in the new Git conversion when comparing to the old SVN conversion.
Sumagna Das (Aug 06 2020 at 15:58):oh ok
Sumagna Das (Aug 06 2020 at 15:58):that was weird for me because i didnt know about that
Sumagna Das (Aug 06 2020 at 15:59):btw no differences found after 69289
Sumagna Das (Aug 06 2020 at 15:59):right now at 26k
starseeker (Aug 06 2020 at 15:59):/me nods - nice!
Sumagna Das (Aug 06 2020 at 16:00):oh actually it found some differences :mischievous:
Sumagna Das (Aug 06 2020 at 16:03): starseeker (Aug 06 2020 at 16:39):Do you know why it's reporting differences?
starseeker (Aug 06 2020 at 16:44):I'll give you a hint - check the svn:branch: labels of the commits newer than r29886
starseeker (Aug 06 2020 at 16:47):For the older ones, you'll want to take a look and see what the difference is in those two text files.
Sumagna Das (Aug 06 2020 at 20:51):(deleted)
Sumagna Das (Aug 06 2020 at 20:52):@starseeker
sumagnadas@hp-laptop:/mnt/sda2/brlcad_migr$ diff github/libtcl/library/reg1.0/pkgIndex.tcl brlcad/libtcl/library/reg1.0/pkgIndex.tcl --color=auto
1c1,2
< package ifneeded registry 1.0 "load [list [file join $dir tclreg82.dll]] registry"
---
> package ifneeded registry 1.0 "load [list [file join $dir tclreg82.dll]] registry"
>
why is this even a difference?
Sumagna Das (Aug 07 2020 at 03:16):finally at 2k range :tada: :celebration:
Sumagna Das (Aug 07 2020 at 04:12):the check is complete
Sumagna Das (Aug 07 2020 at 04:12):(deleted)
Sumagna Das (Aug 07 2020 at 04:12): Sumagna Das (Aug 07 2020 at 04:12):here are the two files
Sumagna Das (Aug 07 2020 at 04:14): Sumagna Das (Aug 07 2020 at 04:17):the difference.txt is a bit big and long because it found many differences after 19k just as you stated @starseeker
starseeker (Aug 07 2020 at 12:41):That's probably a difference due to line endings, or possibly trailing whitespace - you can use a graphical diff tool like meld or kompare to take a look...
starseeker (Aug 07 2020 at 12:44):@Sumagna Das nice job!
starseeker (Aug 07 2020 at 12:44):@Sean what are the next steps?
Sumagna Das (Aug 07 2020 at 13:36):mention me if i can help in the next steps
Sumagna Das (Aug 07 2020 at 13:40):but i will not be much online from 14th august-5th september due to exams
Sumagna Das (Aug 07 2020 at 13:50):starseeker said:
Sumagna Das nice job!
thanks by the way
Last updated: Jan 09 2025 at 00:46 UTC