Vikram Atreya (Feb 05 2021 at 19:55): Vikram Atreya (Feb 05 2021 at 19:55):I went through GSoC project plans of 2018,’19,’20
So these were the projects I found most interesting :
Vikram Atreya (Feb 05 2021 at 19:55):1)Improving ray tracing looked very interesting as it involved a lot of algorithms and increasing efficiency sort of thing, but it was marked hard on github, so I had second thoughts on this.
Vikram Atreya (Feb 05 2021 at 19:55):2) Expanding on Rishabh’s work for other primitives, and I’ll also get to learn OpenCL for this. I’m expecting it to be easier as similar work has already been done.
Vikram Atreya (Feb 05 2021 at 19:55):3)Adding annotation, GSoC ‘19, this also looked very useful to expand upon. I was thinking if we could add automatic dimensioning, like instead of selecting 2 points and then getting a dimension, if we directly select a circle and its radius is dimensioned (more params could be given to position better), etc
Vikram Atreya (Feb 05 2021 at 19:56):I haven't researched too deeply about them, but these were the ones that looked best to me at first glance
Vikram Atreya (Feb 05 2021 at 19:57):I would love to hear your comments and which would be best suited to work upon
Sean (Feb 08 2021 at 15:26):@Vikram Atreya those are all interesting project ideas. You'll just want to make sure you scope your proposal appropriately. If you've not done performance profiling and/or ray tracing before, I wouldn't recommend #1; if you've not done OpenCL or Cuda before, I wouldn't be too excited about you doing #2 either at least until/unless I saw a useful patch from you involving OpenCL. The timeframe of GSoC is such that there's not going to be much time to learn -- so you should propose something that you have skills for already. That way, the time you spent learning is on learning BRL-CAD's code. No matter which you intend to propose, you should be prepared to submit a patch specifically related to that topic (it can be minor or not, but it should be related).
Vikram Atreya (Feb 08 2021 at 18:26):Okay got it.
So could you suggest me some bug for raytracing or related to openCL?
Sean (Feb 10 2021 at 05:48):@Vikram Atreya You'll first want to compile with OpenCL enabled and with OpenCL disabled, so you can compare the two side-by-side. See CMake for the flag on turning opencl on. After compiling, run rt with the -z option on to do an opencl render.
Sean (Feb 10 2021 at 05:51):There's a number of potential opencl issues you could tackle, but getting set up and showing basic competency first is important. Next, you could tackle something related to either of the two OpenCL tasks listed in the top-level TODO file (search for "OpenCL").
Sean (Feb 10 2021 at 05:55):For gsoc application purposes, the second one (clFFT) is probably more achievable as it's more isolated and easier to explain, but anything you can do related to the first one (boolean weaving) is more important even if it doesn't involve OpenCL (e.g., write an isolated unit test for it).
Vikram Atreya (Feb 10 2021 at 06:01):Sean said:
Vikram Atreya You'll first want to compile with OpenCL enabled and with OpenCL disabled, so you can compare the two side-by-side. See CMake for the flag on turning opencl on. After compiling, run rt with the -z option on to do an opencl render.
Okay I will start with this
Vikram Atreya (Feb 11 2021 at 19:05):I installed OpenCL using apt install ocl-icd-opencl-dev and also downloaded the Nvidia SDK
Next I did to cmake .. -DBRLCAD_BUNDLED_LIBS=ON -DCMAKE_BUILD_TYPE=Release -DBRLCAD_ENABLE_OPENCL=ON
This gave
-- Could not find OpenCl (missing: PENCL_ROOT_DIR OPENCL_INCLUDE_DIRS OPENCL_VERSION)
I tinkered a lot to find out where I had to specify them
But I couldn't get what to do, so could you tell me how to proceed?
Vikram Atreya (Feb 14 2021 at 10:23):If anyone is free could they guide me on compiling with OpenCL enabled
Sean (Feb 18 2021 at 07:05):@Vikram Atreya make sure you delete your CMakeCache.txt file after running apt install, so it doesn't use a cached result when looking for whether it's available.
Sean (Feb 18 2021 at 07:07): Sean (Feb 18 2021 at 07:08):So it sounds like you need to set CUDA_PATH if you intend to use Nvidia's opencl toolkit
Sean (Feb 18 2021 at 07:09):alternatively you can set OPENCL_ROOT_DIR and OPENCL_INCLUDE_DIRS and OPENCL_VERSION directly, but you shouldn't need to.
Vikram Atreya (Feb 18 2021 at 07:25):Sean said:
While locating the root directory, the module will try to detect OpenCL
implementations provided by AMD's Accelerated Parallel Processing SDK,
NVIDIA's GPU Computing Toolkit and Intel's OpenCL SDK by examining the
AMDAPPSDKROOT, CUDA_PATH and INTELOCLSDKROOT environment variables,
respectively.
So I installed Intel's OpenCL SDK and did set INTELOCSDKROOT to the correct folder
Vikram Atreya (Feb 18 2021 at 07:26):Sean said:
Vikram Atreya make sure you delete your CMakeCache.txt file after running apt install, so it doesn't use a cached result when looking for whether it's available.
Even did this
Vikram Atreya (Feb 18 2021 at 07:28):But still it says
-- Could NOT find OpenCL (missing: OPENCL_ROOT_DIR OPENCL_INCLUDE_DIRS OPENCL_VERSION)
Vikram Atreya (Feb 18 2021 at 07:35):Vikram Atreya said:
Sean said:
While locating the root directory, the module will try to detect OpenCL
implementations provided by AMD's Accelerated Parallel Processing SDK,
NVIDIA's GPU Computing Toolkit and Intel's OpenCL SDK by examining the
AMDAPPSDKROOT, CUDA_PATH and INTELOCLSDKROOT environment variables,
respectively.
So I installed Intel's OpenCL SDK and did set INTELOCSDKROOT to the correct folder
Okay..... I made a typo here, INTELOCSDKROOT ,missed the 'L' that's why it wasn't recognizing OpenCL all this while
Vikram Atreya (Feb 18 2021 at 07:36):Its working now
Vikram Atreya (Feb 18 2021 at 10:30):Sean said:
Vikram Atreya You'll first want to compile with OpenCL enabled and with OpenCL disabled, so you can compare the two side-by-side. See CMake for the flag on turning opencl on. After compiling, run rt with the -z option on to do an opencl render.
Done :)
Sean (Feb 19 2021 at 08:13):Woohoo! So next see if you can observe the performance difference on a specific object. Pick any of the primitives in src/librt/primitives/OBJdirs that have an opencl implementation, create one (or lots of that type) in mged, then ray trace it with ocl and without (helps to keep two compiles on hand) using "time rt" outside of mged.
Sean (Feb 19 2021 at 08:18):you'll want a render that takes at least a few seconds to complete, which you can achieve by adding the -H flag (e.g., -H10 will increase work by 10x, -H100 by 100x) or -s flag (makes a bigger image, e.g., -s10000 for a 10000x10000 image).
Sean (Feb 19 2021 at 08:32):@Vikram Atreya once you have that demonstrated, the next step would be to focus on related coding. If you're set on submitting OpenCL for gsoc, I suggest focusing on learning about boolean weaving -- rt_boolweave() and rt_boolfinal() being critical functions. I'd suggest trying to write a tiny little main() program that calls either function. Try to the bare minimum to set up proper inputs, call the function, and show that it did something. A simplistic unit test of sorts.
Vikram Atreya (Feb 19 2021 at 08:59):Okay will start working on this :+1:
Vikram Atreya (Feb 19 2021 at 19:32):When I tried to run time rt -z 1 sphere.g sph1.s
it gave me an error, failed to set OpenCL kernel arguments
I'm attaching the terminal output and the log it generated.
output.txt
rt-22345-bomb.log
Vikram Atreya (Feb 19 2021 at 20:14):I tried applying this patch - https://sourceforge.net/p/brlcad/patches/551/
but it doesn't seem to solve the problem
Vikram Atreya (Feb 19 2021 at 20:41):The patch dint get applied properly before, problem solved
Vikram Atreya (Feb 19 2021 at 20:58):Sean said:
Woohoo! So next see if you can observe the performance difference on a specific object.
I tried it for a sphere with -H1000, the OpenCL enabled compilation did it in 15s wile the other one took 24s. So I think I can consider this done :)
Sean (Feb 19 2021 at 22:08):Excellent @Vikram Atreya . I mean, that's not a huge performance gain oddly enough, but good to see different numbers. Have you ever run a profile before? It would be informative to know where the time was spent for both of those. If you're on Windows, would be good to set up Intel VTune. If you're on Linux, Perf is really easy to set up. On Mac, Instruments is king.
Sean (Feb 19 2021 at 22:09):See if you can figure out what top ten functions they're spending their time in for ocl vs non-ocl runs.
Vikram Atreya (Feb 20 2021 at 12:24):No, I haven't run a profile before. I setup perf and got the top CPU consuming functions
with ocl: with_ocl.png
without ocl: without_ocl.png
Erik (Feb 20 2021 at 17:06):nifty! (it might also be interesting to see how -J impacts performance differences?). Does RoCM's cuda backend change openCL's place in the world?
Sean (Feb 20 2021 at 19:13):That's a good thing to check - @Vikram Atreya see what performance looks like with rt -P1 for ocl vs non. That requests a single CPU core. Also notice in that non-opencl profile rt_boolweave() and rt_boolfinal() ... that's the two functions mentioned earlier.
Vikram Atreya (Feb 21 2021 at 19:08):I tried to observe the differences with the previous values, when applying the -J and -P 1 flags separately , with -J CPU usage went up in both cases but while using the -P 1 flag I did not observe any major change in values
Vikram Atreya (Feb 21 2021 at 20:32):Meanwhile, I generated the executable for rtexample.c, but when I try to run it using ./rtexample, it says:
error while loading shared libraries: libbu.so.20: cannot open shared file: No such file or directory
Vikram Atreya (Feb 21 2021 at 20:33):The directory I linked while compiling does have the libbu.so.20 file
Vikram Atreya (Feb 22 2021 at 18:25):Vikram Atreya said:
Meanwhile, I generated the executable for rtexample.c, but when I try to run it using ./rtexample, it says:
error while loading shared libraries: libbu.so.20: cannot open shared file: No such file or directory
I solved this. I had to to put an env variable to the dir i had linked
Vikram Atreya (Feb 22 2021 at 18:37):Sean said:
Vikram Atreya once you have that demonstrated, the next step would be to focus on related coding. If you're set on submitting OpenCL for gsoc, I suggest focusing on learning about boolean weaving -- rt_boolweave() and rt_boolfinal() being critical functions. I'd suggest trying to write a tiny little main() program that calls either function. Try to the bare minimum to set up proper inputs, call the function, and show that it did something. A simplistic unit test of sorts.
I understood how rtexample.c works more or less. I will start working on this.
Vikram Atreya (Feb 25 2021 at 20:30):Is there some place where I get previous year GSoC proposals that were accepted/rejected?
Daniel Rossberg (Feb 26 2021 at 12:26):Did you had a look at https://brlcad.org/wiki/Google_Summer_of_Code ? Some of the abstracts are still accessible.
Vikram Atreya (Feb 26 2021 at 16:15):Yeah I did
Most of the devlogs and project plans are accessible but abstracts aren't
Vikram Atreya (Feb 26 2021 at 16:15):And the project plans looked very brief
Daniel Rossberg (Feb 26 2021 at 17:33):Maybe, this is a better one: https://brlcad.org/wiki/User:Clouddrift/GSoC2014
For the BRL-CAD project I can assure you that we will ask for more details if they are missing. I.e., you won't fail because of a too brief first version of your proposal, provided that it shows a honest intent. However, the final version should address all our requests ;)
Sean (Feb 26 2021 at 18:54):@Vikram Atreya We're also totally cool with this being a collaborative process if you want to start on a proposal and share progress as you go along for awareness/feedback. It doesn't/shouldn't be a "throw it over the wall" when you think you're done. Happy to discuss it with you and help you figure out how to scope it appropriately.
Vikram Atreya (Feb 26 2021 at 18:58):Thank you!
I was planning on starting with the proposal parallelly as I explore the code base. I'll start once my exams get done on Monday.
Btw is there some quick way to find out what all primitives have successfully been parallelized using OpenCL?
Vikram Atreya (Feb 26 2021 at 19:36):Sean said:
Vikram Atreya once you have that demonstrated, the next step would be to focus on related coding. If you're set on submitting OpenCL for gsoc, I suggest focusing on learning about boolean weaving -- rt_boolweave() and rt_boolfinal() being critical functions. I'd suggest trying to write a tiny little main() program that calls either function. Try to the bare minimum to set up proper inputs, call the function, and show that it did something. A simplistic unit test of sorts.
I have made a small program that calls rt_boolweave() with minimum inputs. hello_boolweave.c
Vikram Atreya (Feb 26 2021 at 19:38):
Sean (Feb 26 2021 at 19:52):Vikram Atreya said:
Btw is there some quick way to find out what all primitives have successfully been parallelized using OpenCL?
Sure, you can just run: grep -r *.cl .
That'll report all the opencl files in the source tree.
Sean (Feb 26 2021 at 19:53):the primitives are in subdirs under src/librt/primitives, so that'll be all the current committed ones. There are a couple more sitting in pending patches I believe.
Sean (Feb 26 2021 at 19:59):Cool progress on rt_boolweave() .. you may be the first to actually write a standalone that directly calls it, even if it's not yet doing anything useful. Do you think you can whittle that down to what is strictly necessary? More than half that code is for calling rt_shootray() which you don't / won't need to do.
Vikram Atreya (Feb 26 2021 at 19:59):Sure, will do
Vikram Atreya (Feb 26 2021 at 20:03):Should I remove the checks also like RT_CK_AP?
Vikram Atreya (Feb 26 2021 at 20:22):hello_boolweave.c Removed everything that I felt unnecessary and commented the checks.
Sean (Feb 26 2021 at 20:46):I think more than half of what is still in there is unnecessary... you're not ray tracing.
Sean (Feb 26 2021 at 20:47):How does rt_boolweave() use the application struct? You'll need to read the implementation to see how/where it's used. Depending how it's used, it may just be for book-keeping and can be empty.
Sean (Feb 26 2021 at 20:50):Vikram Atreya said:
Should I remove the checks also like RT_CK_AP?
So to answer this it's the same for removing any other lines. What's that line doing and how does it pertain to rt_boolweave()? If you don't know, then the next step will be to find it's implementation (or ask).
Sean (Feb 26 2021 at 20:50):All the _CK_() and _CHECK_() macros do simple validity testing to make sure that memory didn't get corrupted and/or that structures passed from elsewhere were initialized correctly.
Sean (Feb 26 2021 at 20:52):In this case, that 'ap' application struct is being created and initialized by you, so there's no point in calling CHECK/CK on it as there's no potential for corruption, just maybe a mistake in initialization like if you forgot to call RT_APPLICATION_INIT().
Vikram Atreya (Feb 26 2021 at 20:53):Sean said:
In this case, that 'ap' application struct is being created and initialized by you, so there's no point in calling CHECK/CK on it as there's no potential for corruption, just maybe a mistake in initialization like if you forgot to call RT_APPLICATION_INIT().
Yeah I knew this, just wanted to confirm
Sean (Feb 26 2021 at 20:55):if it's not needed, then it boils down to somelike like...
int main(int argc, char **argv) {
struct seg waiting_segs;
struct seg finished_segs;
BU_LIST_INIT(&waiting_segs.l);
BU_LIST_INIT(&finished_segs.l);
/* Weave these segments into partition list */
(void)rt_boolweave(&finished_segs, &waiting_segs, NULL, NULL);
return 0;
}
But then that still doesn't do anything useful.
Sean (Feb 26 2021 at 20:57):So the trick is figuring out what those params mean, which are needed, how to set up the inputs (segs especially) manually, etc
Sean (Feb 26 2021 at 20:58):Reduced the example even more... I don't know if it really needs a pointer to the initial partition.
Sean (Feb 26 2021 at 20:58):I mean notionally, do you understand what the function is supposed to do?
Vikram Atreya (Feb 26 2021 at 21:00):Sean said:
I mean notionally, do you understand what the function is supposed to do?
Yeah I know the basics, you had explained it to me earlier
Sean (Feb 26 2021 at 21:04):Well here's a quick recap anyways.. :) Say you have this expression "A u B" where we're combining two objects with a union operation. Imagine we shot a ray through that scene. The ray is going to hit A and B, say that looks something like this:
______________
| |
| A |-----|--------
| | | |
-------|------ B |
| |
---------------
______________
| |
ray | A |-----|--------
---> | | | |
-------|------ B |
| |
---------------
results in two segments:
|----A.seg1--|
|----B.seg1---|
So given a union, the boolean weaving is going to end up with:
|------weaved.seg----|
In creating a unit test, you aren't going to shoot a ray, you don't have geometry. You're starting at "results in two segments" where all you have is a list of partitions and segments, and they're literally just sets of numbers.
Sean (Feb 26 2021 at 21:11):I suggest trying to describe that exact scenario for starters. One segment going from 0.0 to 10.0, another going from 5.0 to 15.0, and try to weave them into a 0.0 to 15.0 segment.
Vikram Atreya (Feb 26 2021 at 21:11):Okay, will start working on it
Sean (Feb 26 2021 at 21:13):If you get it right, then you should be able to change one line that makes it do a subtraction to weave to a 0.0 to 5.0 segment (A-B) or 10.0 to 15.0 segment (B-A), or an intersection that gives a 5.0 to 10.0 segment.
Sean (Feb 26 2021 at 21:14):Now big caveat is I don't remember the exact interplay between rt_boolweave() and rt_boolfinal()... one or the other might be responsible for parts of the boolean eval.
Vikram Atreya (Feb 26 2021 at 21:19):I will look into it
Also, Thank you so much for taking out time to guide me :big_smile:
Vikram Atreya (Feb 28 2021 at 10:27):hello_boolweave_final.c
This is the least code that is required afaik, I tried to compile without one line at a time, and removing even 1 line gave a seg fault. I got ERROR: NULL struct partition list head pointer if I passed a NULL pointer in place of the initial part.
Vikram Atreya (Feb 28 2021 at 10:28):Maybe the initialization of rtip can be done in a better way but I couldn't figure out one.
Vikram Atreya (Feb 28 2021 at 10:29):Sean said:
I suggest trying to describe that exact scenario for starters. One segment going from 0.0 to 10.0, another going from 5.0 to 15.0, and try to weave them into a 0.0 to 15.0 segment.
Ill work on this now
Vikram Atreya (Feb 28 2021 at 20:22): struct seg s1;
BU_LIST_INIT(&s1.l);
s1.seg_in.hit_dist =5;
s1.seg_out.hit_dist =15;
(s1.seg_in.hit_rayp) = (s1.seg_out.hit_rayp) = &(ap.a_ray);
BU_LIST_INSERT(&waiting_segs.l, &s1.l);
I have tried to initialize a seg like this, but when I run the program, RT_CK_SEG inside boolweave gives an error as follow:
ERROR: bad pointer 0x7ffde9fe2fa0: s/b struct seg(x98bcdef1), was bu_list(x1016580), file /home/vikram/brlcad/src/librt/bool.c, line 174
Could someone tell me how to solve this?
Vikram Atreya (Feb 28 2021 at 20:23):I assume this initialization is done by stp->st_meth->ft_shot in shootray, but imnt able to find the code of ft_shot
Vikram Atreya (Feb 28 2021 at 20:24):Attaching the entire program just in case UBC.c
Sean (Mar 01 2021 at 16:36):@Vikram Atreya what's triggering that ERROR is the BU_LIST_INIT. Librt uses an old-school C technique called aliasing in the libbu containers. What you're missing is something like BU_LIST_MAGIC_SET(&s1.l, RT_SEG_MAGIC); before you do BU_LIST_INSERT.
Sean (Mar 01 2021 at 16:37):db_alloc.c is where that initialization happens during regular ray tracing.. search for RT_SEG_MAGIC to see where.
Sean (Mar 01 2021 at 16:37):it's set directly there, but basically the same as the BU_LIST_MAGIC_SET call.
Vikram Atreya (Mar 01 2021 at 16:38):Sean said:
What you're missing is something like BU_LIST_MAGIC_SET(&s1.l, RT_SEG_MAGIC); before you do BU_LIST_INSERT.
Ohhhh! I knew I had to set the magic variable for seg but did not find a macro for it, I searched for RT_SEG_MAGIC and similar macros.
Got it now
Sean (Mar 01 2021 at 16:55):note that you don't need an application struct either potentially ... at least the seg doesn't need it. It has a ray field but I dont know if boolweave uses/checks it.
Sean (Mar 01 2021 at 16:56):If it does check/use it, then you can create a struct ray on the stack and set that instead of &(ap.a_ray), which is just a pointer to a ray struct.
Vikram Atreya (Mar 01 2021 at 16:56):yeah it does check ap->a_rt_i, so ap is required for that
Sean (Mar 01 2021 at 16:56):do you know why?
Vikram Atreya (Mar 01 2021 at 16:56):Sean said:
If it does check/use it, then you can create a struct ray on the stack and set that instead of &(ap.a_ray), which is just a pointer to a ray struct.
Okayy
Vikram Atreya (Mar 01 2021 at 16:56):Sean said:
do you know why?
Dont know rn, but I could try finding out
Vikram Atreya (Mar 01 2021 at 16:58):Superficially checking the code, ap->a_rt_i is assigned to rtip and that is used at a lot of places
Sean (Mar 01 2021 at 16:59):I see, it uses the app struct to get at the cpu resources for memory pooled allocations...
Sean (Mar 01 2021 at 16:59):rtip isn't actually used
Sean (Mar 01 2021 at 17:00):well, so it's technically used but not in significant logic
Sean (Mar 01 2021 at 17:00):the tolerance is pulled from there, the number of solids
some debugging
Sean (Mar 01 2021 at 17:01):but, for example, it's passed to GET_PT_INIT() and that macro does diddly squat nothing with it
Sean (Mar 01 2021 at 17:01):a vestige from long ago
Sean (Mar 01 2021 at 17:02):so that's important from a testing perspective. the resource structure is the only thing significant in there and it isn't necessary. that's an optimization for memory management.
Sean (Mar 01 2021 at 17:02):I mean the code currently requires it so in setting up a rt_boolweave() unit test, you'll have to provide it or modify the implementation
Sean (Mar 01 2021 at 17:03):but for an openCL version, which is what we'd want to end up with, it's not necessary at all since opencl would be handling memory differently
Vikram Atreya (Mar 01 2021 at 17:06):Okay
Vikram Atreya (Mar 01 2021 at 17:11):Sean said:
I mean the code currently requires it so in setting up a rt_boolweave() unit test, you'll have to provide it or modify the implementation
So for now, I will just provide it to boolweave
Sean (Mar 01 2021 at 17:27):You'll still want to provide as minimal as possible. Like try passing an rtip that is not really initialized, only has rti_tol.dist set.
Vikram Atreya (Mar 01 2021 at 17:27):Okay, first I will make it functional then, I will reduce the inputs
Sean (Mar 01 2021 at 17:28):similarly application struct that just has the rtip set to that dummy rtip and resources
Vikram Atreya (Mar 01 2021 at 19:14):In bool.c, rt_pr_seg() is called which requires segp->seg_stp->st_bit & segp->seg_stp->st_dp->d_namep to be printed. I set segp->seg_stp (which is a soltab pointer ) to NULL but the prog. throws a seg fault when run. How can I bypass this without actually setting segp->seg_stp ? or am I doing smthng wrong?
Vikram Atreya (Mar 01 2021 at 19:15): Sean (Mar 02 2021 at 06:35):it dereferences seg_stp to st_bit and st_dp... so it will need a valid "soltab" struct if rt_pr_seg() is called, which is a structure for a solid primitive object (e.g., an 'ell' ellipsoid or a 'tor' torus).
Sean (Mar 02 2021 at 06:35):that said, I only see rt_pr_seg() in a debug printing block in rt_boolweave(), which shouldn't be getting called. are you sure it's crashing there? I suspect it's crashing elsewhere.
Vikram Atreya (Mar 02 2021 at 06:38)::grimacing: My bad, I had changed the code before so that it prints the debug block, Ill try without that
Vikram Atreya (Mar 02 2021 at 07:42):Slightly unrelated but....
What does stp->st_meth->ft_shot() do? , I tried a grep in src but couldnt find the function
Vikram Atreya (Mar 02 2021 at 07:48):Or you could just tell me where the code for ft_shot lives
Sean (Mar 02 2021 at 07:57):That is a function pointer. It's set in src/librt/primitives/table.cpp
Sean (Mar 02 2021 at 07:58):function pointers are somewhat advanced concepts, but you can think of it as just a pointer to another function.
Sean (Mar 02 2021 at 07:58):so I could have a function like int foo() {printf("hello");}
Sean (Mar 02 2021 at 07:59):and I can create a variable like (int)(*bar)() = foo;
Sean (Mar 02 2021 at 07:59):and with that, I can now call bar() .. and it'll invoke foo()
Sean (Mar 02 2021 at 08:01):so ft_shot() is set to a different function for each object type, so for example there is ft_shot == rt_ell_shot which is defined in src/librt/primitives/ell/ell.c
so when I call ft_shot() it calls that rt_ell_shot() function if it was an ell object
Vikram Atreya (Mar 02 2021 at 08:01):Got it
Sean (Mar 02 2021 at 08:02):that table.cpp file is basically a big massive listing of functions that are getting set to function pointers
Sean (Mar 02 2021 at 08:02):as a simple way of registering what callback to call when a given entity id is encountered
Erik (Mar 02 2021 at 14:18):where is BoT on the opencl agenda? "advanced topic"?
Sean (Mar 02 2021 at 15:05):haven't prioritized any of the primitives yet because there's a few bits of the pipeline (boolean weaving) that still need to be done first.
I would probably put nurbs before bot myself, but anything getting converted is good. we still have to sort out how to manage keeping multiple copies of implementation, or how to eliminate down to one.
Vikram Atreya (Mar 02 2021 at 16:22):Just asking, can I reimplement an already converted primitive as well?
Erik (Mar 02 2021 at 16:27):I think you'd have to make a fairly compelling argument?
Vikram Atreya (Mar 02 2021 at 16:30):Ohh okay
Vikram Atreya (Mar 03 2021 at 04:06):Along with the unit test for boolweave could I work on something else parallelly, I want to take a break from that code :P
Sean (Mar 03 2021 at 05:22):@Vikram Atreya haha, yeah, no I completely understand. That's some really tough code. Note you don't have to feel compelled to propose that task if there's another that ends up being more interesting. Other projects that are also interesting are Qt GUI coding, Appleseed render integration, and support for annotations+labels+dimensions.
Sean (Mar 03 2021 at 05:33):you could try a task for one of them to see if they're more appealing, or I could just throw a task your way and you could see how you manage. for example, we've had a user ask for an option to the dbconcat command (which lets you merge geometry files together) to not use a prefix or a suffix during import, but to instead have a flag that overwrites existing objects. implementation is entirely in src/libged/concat/concat.cpp
Vikram Atreya (Mar 03 2021 at 07:27):Sean said:
That's some really tough code.
I feel better :laughing:
Vikram Atreya (Mar 03 2021 at 07:29):This is quite interesting actually, but surely I would like to try out the other tasks as well, Though I feel working on the GUi is very important Im somehow not drawn to it that much, I would like to try a task for appleseed or annotation, which one is easier?
Vikram Atreya (Mar 03 2021 at 07:32):Sean said:
you could try a task for one of them to see if they're more appealing, or I could just throw a task your way and you could see how you manage. for example, we've had a user ask for an option to the dbconcat command (which lets you merge geometry files together) to not use a prefix or a suffix during import, but to instead have a flag that overwrites existing objects. implementation is entirely in src/libged/concat/concat.cpp
Interesting, let me go through concat.cpp and let you know if I find it enthusing......Also does it help me go forward in GSoC prep because that's the main priority for now :P
Sean (Mar 03 2021 at 14:59):Annotation is probably easier overall and is completely unrelated to the performance work. appleseed is related in that it involves shooting rays at geometry, but is definitely a lot more involved concepts to understand and an entirely separate API involved (appleseed's).
Sean (Mar 03 2021 at 15:00):Vikram Atreya said:
Interesting, let me go through concat.cpp and let you know if I find it enthusing......Also does it help me go forward in GSoC prep because that's the main priority for now :P
I'm not sure what you mean by prep ... working on code and demonstrating you can be productive is the best prep you can do for gsoc.
Vikram Atreya (Mar 03 2021 at 17:29):Sean said:
working on code and demonstrating you can be productive is the best prep you can do for gsoc.
Okay
Vikram Atreya (Mar 03 2021 at 17:30):I checked out the code in concat.cpp and I feel this task is doable.
Sean (Mar 04 2021 at 07:11):@Vikram Atreya it should just take you an hour or two but feel free to ask questions about our API for the bits you can't decipher what's going on. There's a manual page for the command (run "man dbconcat" in mged or archer) to understand what the command does, but it basically lets you import another .g into the currently open .g. Presently, it has options to apply a suffix or prefix when the object being imported already exists. The goal is an option that overwrites the existing option.
Alternative patch needed on that same command is a dry run option that doesn't actually import anything, but says what it would do without the dry run option (i.e., import as-is, import with prefix/suffix, overwrite, skip, etc).
Vikram Atreya (Mar 04 2021 at 07:18):I understood most of the code, I have an exam today so haven't really worked on it yet. Yeah, I will surely ask if I get stuck
Sean (Mar 04 2021 at 07:27):Another fun little project that might be of interest is writing a procedural geometry generator (i.e., a little app) that makes a shaderball. You'd probably have to model one by hand first, then figure out how to repeat the modeling instructions in C/C++, letting the caller specify the shader and shader options from what's available.
Thusal Ranawaka (Mar 04 2021 at 07:31):(deleted)
Thusal Ranawaka (Mar 04 2021 at 07:31):(deleted)
Thusal Ranawaka (Mar 04 2021 at 07:32):(deleted)
Sean (Mar 04 2021 at 07:35):Heh, oops, that wasn't meant for you, @Thusal Ranawaka sorry about that. haha
Thusal Ranawaka (Mar 04 2021 at 07:36)::sweat_smile:
Sean (Mar 04 2021 at 07:38):@Rishabh Suthar Hey bud, you around? There's a potential GSoC student (@Vikram Atreya) possibly interested in picking up where you left off. Any chance you can share/post your proposal? The wiki only has your log and a now dead link to the abstract on Google's site (they clear the data every year).
Vikram Atreya (Mar 04 2021 at 18:55):Am I restricted to use any letters for the flag? Could I assign "-o" for override?
Vikram Atreya (Mar 04 2021 at 20:59):@Sean So I have a plan to do this but it involves putting a function call, that accomplishes the override, inside get_new_name() would that be okay?
Or is it compulsory that the get_new_name() is restricted to giving a new name , because then it would make it harder for me to implement
Sean (Mar 05 2021 at 01:58):@Vikram Atreya that's a big "it depends". I would say that "get_new_name() has a very specific implication and you should not violate that if it's not going to get a new name. That said, there's no problem renaming that function to "get_import_name()" or even simply "get_name()" or something similar that encompasses the logic for determining the name of the object.
Sean (Mar 05 2021 at 02:04):As for letters, -o is pretty universally 'output' so I wouldn't choose that first.
Sean (Mar 05 2021 at 02:07):The cp command has relevant options (-n and -i and -f) so you could pattern after that, like making default to be overwrite, but then have a -n option to not overwrite. Either way, it's a little tricky because it should definitely not write some objects and then halt mid import.
Sean (Mar 05 2021 at 02:09):A -r replace flag could work.
Vikram Atreya (Mar 05 2021 at 10:25):When I'm overriding the object present in the file (which we are concat-ing to), I should kill the object right?
Vikram Atreya (Mar 05 2021 at 10:26):And to kill an object can I call ged_kill or some other already written function to do it, or should I write the code to kill the object?
Sean (Mar 05 2021 at 21:03):@Vikram Atreya I believe you can just export it an it'll overwrite the object.
Sean (Mar 05 2021 at 21:05):but you'll want to test that theory
if that doesn't work or if it creates a duplicate and there's not a way aroun dit, you'll need to add the new object with a temp name, move old object to a temp name, rename the new object to the old object name, and THEN kill the old object.
Vikram Atreya (Mar 06 2021 at 04:11):Okay got that
Im going on a 2 day trip, will continue working on it from monday (for me)
Sean (Mar 06 2021 at 18:17):cool, have a good trip! be careful of busses ;)
Vikram Atreya (Mar 08 2021 at 18:37):Im back :)
Vikram Atreya (Mar 08 2021 at 18:37):Sean said:
Vikram Atreya I believe you can just export it an it'll overwrite the object.
How do i do this?
Vikram Atreya (Mar 08 2021 at 18:37):Im not very clear on how to export
Sean (Mar 08 2021 at 21:29):@Vikram Atreya grep -r _export src/libged
Vikram Atreya (Mar 09 2021 at 07:47):Sean said:
Vikram Atreya I believe you can just export it an it'll overwrite the object.
Nope, this is not the case, when I export it with the same name db_dircheck adds a prefix and exports it
Vikram Atreya (Mar 09 2021 at 07:49):Sean said:
but you'll want to test that theory
if that doesn't work or if it creates a duplicate and there's not a way aroun dit, you'll need to add the new object with a temp name, move old object to a temp name, rename the new object to the old object name, and THEN kill the old object.
Just a doubt, why cant I kill the old object first and then directly add the new object?
Vikram Atreya (Mar 09 2021 at 17:33):Vikram Atreya said:
And to kill an object can I call ged_kill or some other already written function to do it, or should I write the code to kill the object?
Any hints on this please
Daniel Rossberg (Mar 09 2021 at 18:30):If there is already a function which removes an object, you should use it. Maybe, not ged_kill() but the functions used there, like db_delete() and db_dirdelete().
However, I'm not sure if it is necessary to explicitly remove the object, wdb_export() seems to make this job implicitly. I created a database test.g with an arb8 test.s. Then, I run the following program on it:
#include <iostream>
#include <brlcad/Database/FileDatabase.h>
#include <brlcad/Database/Sphere.h>
int main
(
int argc,
char* argv[]
){
int ret = 0;
BRLCAD::FileDatabase database;
if (database.Load("test.g")) {
BRLCAD::Sphere sphere;
sphere.SetName("test.s");
database.Add(sphere);
} else {
std::cout << "Could not load file: " << argv[1] << std::endl;
ret = 1;
}
return ret;
}
This doesn't mean that you have to use the C++ interface, but the method used there using wdb_export() could eventually do the job. You could write a program to test this out.
Vikram Atreya (Mar 10 2021 at 18:35):Feels done, tried with few basic examples
Sean (Mar 10 2021 at 18:38):@Vikram Atreya the issue with just killing and adding is leaving the tree irrecoverably corrupted if the task is interrupted (whether a bug or an abort signal or power loss, etc)
Vikram Atreya (Mar 10 2021 at 18:39):Okay
Vikram Atreya (Mar 10 2021 at 18:39):I did both db_delete() and db_dirdelete()
Vikram Atreya (Mar 10 2021 at 18:39):Just saying
Vikram Atreya (Mar 10 2021 at 18:40):Sean said:
but you'll want to test that theory
if that doesn't work or if it creates a duplicate and there's not a way aroun dit, you'll need to add the new object with a temp name, move old object to a temp name, rename the new object to the old object name, and THEN kill the old object.
Will do this
Sean (Mar 10 2021 at 18:42):This doesn't mean that you have to use the C++ interface, but the method used there using wdb_export() could eventually do the job. You could write a program to test this out.
This is something we really need to figure out. I'd love to use the C++ interface, but this is a libged function we're talking about. Seems like we'd be introducing a potential cyclic dependency, or at least a deferred one (e.g., can't build some libged plugins until a second pass later).
Vikram Atreya (Mar 10 2021 at 18:42):Also is it compulsory to do it in the order you have mentioned, like first if I rename the old object, add the new one with the correct name and then delete the old object. Does this work?
Sean (Mar 10 2021 at 18:43):We could maybe pull MOOSE in as a sub-project repository on 'brlcad' checkout and have it trigger after core libs, treating it like a 3rd party dependency...
Sean (Mar 10 2021 at 18:45):Vikram Atreya said:
Also is it compulsory to do it in the order you have mentioned, like first if I rename the old object, add the new one with the correct name and then delete the old object. Does this work?
Yeah, that should work well enough for single object operations I think.
Sean (Mar 10 2021 at 18:46):what we really need is a transaction system where instead of doing the work, it just records a transcript of what is going to happen, then a second pass iterates over the transcript to do it all in a temp working space, and swaps the trees if it completely worked.
Sean (Mar 10 2021 at 18:47):that way, if there's a failure in processing or an I/O error or other interruption, we don't end up with corruption and it's either all there or none of it is there.
Sean (Mar 10 2021 at 18:47):could then have an incremental option to apply objects or changes one at a time instead of transactionally if there were some reason for doing so.
Vikram Atreya (Mar 10 2021 at 18:48):Yeah, I get your point
Vikram Atreya (Mar 10 2021 at 18:51):Sounds a little complex to implement but doable, is this something I should work upon next or do I go back to boolweave/ smthng more related to the OpenCL project?
Sean (Mar 10 2021 at 18:52):working on libged is a completely different scope, would be a different project altogether -- and a good one
Vikram Atreya (Mar 10 2021 at 18:53):Is there a project on libged, I seem to have missed it in the ideas/past projects?
Vikram Atreya (Mar 10 2021 at 18:54):Working on this is quite interesting as well
Sean (Mar 10 2021 at 18:54):no, libged is central to our core development that it's actively changing. it's the most frequently edited place in the code right now.
Sean (Mar 10 2021 at 18:55):that's part why there's not a specific write-up for it, the other is that it begs for a good bit of understanding of the other libs, or at least of modern design patterns
Sean (Mar 10 2021 at 18:55):see src/libged/README (particularly the TODO lines) and src/libged/TODO for some ideas that have been written down on what is needed next
Vikram Atreya (Mar 10 2021 at 18:57):Okay, So I could make a proposal combining few important TODO issues and tackle them as part of my GSoC project?
Sean (Mar 10 2021 at 19:01):You most certainly could.
Sean (Mar 10 2021 at 19:02):you'll have to learn which are 'next' and demonstrate familiarity
Vikram Atreya (Mar 10 2021 at 19:04):Ohkay, this one feels more in-sight rather than the OpenCL project.
Vikram Atreya (Mar 10 2021 at 19:04):I will try to finish the basic override command for dbconcat and look into other TODO issues. Maybe I could propose the transcript + second tree for GSoC as one of the tasks that I will do
Sean (Mar 10 2021 at 19:22):the two biggest needs for libged are proper plugin version management and transactions, with the latter arguably being more important. We really need to be able to run a command, interrupt it with a ctrl-C, and not have bad things happen.
Sean (Mar 10 2021 at 19:22):The ability to interrupt commands in mged and archer become additional challenges.
Vikram Atreya (Mar 10 2021 at 19:26):Dont know what plugins in BRLCAD are as of now, but yeah I surely could add the transactions feature to a some cmds during GSoC :)
Sean (Mar 10 2021 at 20:35):libged commands are intended to be plugins. that's the approach we came up with for them to be fully interruptible, synchronous or asynchronous, and protect the main process from crashes/bugs
Sean (Mar 10 2021 at 20:37):it also lets us support commands that link against GPL functionality without creating a derivative work if we get to on-demand downloading/updating of plugins
Sean (Mar 10 2021 at 20:37):but most important is simply the ability to ctrl-C a command and 1) have it stop the command and 2) have it not corrupt the db or leave trash
Erik (Mar 10 2021 at 22:45):will mged no longer have valid sigint behaviors, just 'pass through if running subcmd'?
Sean (Mar 10 2021 at 22:53):@Erik what do you mean? wtf is a "valid" behavior? heh. mged already does some fairly complicated handling of SIGINT..
Sean (Mar 10 2021 at 22:53):I think sigint is typically ignored presently while a command is running. If of course would be better if sigint stopped the command.
Erik (Mar 10 2021 at 22:56):no idea :D thinking vim and emacs have some mgmt of sigint for subprocs, should probably not be too far off from them? shrug I'm just makin' noise :grinning_face_with_smiling_eyes:
Vikram Atreya (Mar 11 2021 at 18:02):For the temporary names that I will be assigning to the old objects, is there some convention that I will have to follow , if not i was thinking of going with asdfgh_(number)
Vikram Atreya (Mar 11 2021 at 18:26):Done :)
Vikram Atreya (Mar 11 2021 at 18:27):Vikram Atreya said:
For the temporary names that I will be assigning to the old objects, is there some convention that I will have to follow , if not i was thinking of going with asdfgh_(number)
If this needs to be changed can be done within a min
Also I have tested with a basic example as of now
Sean (Mar 12 2021 at 06:10):Vikram Atreya said:
For the temporary names that I will be assigning to the old objects, is there some convention that I will have to follow , if not i was thinking of going with asdfgh_(number)
There isn't anything consistent but there are a number of commands that do something similar to pick a temporary name. Some are as simple (and weak) as adding .tmp suffix. Others had a prefix like ___analyze_cmd_intersect_tmp_obj__ ... if you recursively search on tmpnam in src/libged, you'll find those and others.
Sean (Mar 12 2021 at 06:15):I suggest using something meaningful, descriptive, and unlikely to ever be pre-existing. Even better if you test for pre-existing before doing work. You will want to add some sanity checks at least, like to make sure that the tmp object doesn't already exist, testing that the write succeeded, etc.
Sean (Mar 12 2021 at 06:39):If you really want to make it robust, you can call bu_uuid_create() to create a v4 SHA. That hash string will be random and effectively unique. could couple it within a prefix like you were thinking, like __dbconcat_SHA__.
Vikram Atreya (Mar 14 2021 at 10:00):Thinking about this, if I generate random names for temp objects , I will have to store them as I am deleting them at the end after adding all new objects, I feel the best approach would be to go with __dbconcat_temp_object_no_01__, while adding a check that the name isnt presen't in the database already since there is barely any chance of a name being given as follows
Vikram Atreya (Mar 14 2021 at 19:20):@Sean should I create a patch?
Vikram Atreya (Mar 15 2021 at 04:08):Next could I work on the TODO: automatic help flag recognition ?
Sean (Mar 15 2021 at 20:09):@Vikram Atreya whenever you completed implementing, testing, and documenting (if relevant) a change, a patch should be created. In fact, you can create a patch at intermediate steps as well, just be sure to make it clear that it's a work-in-progress and that you'll be submitting an update.
Vikram Atreya (Mar 16 2021 at 03:50):Creating the patch rn
So what could i work on next?
Sean (Mar 16 2021 at 03:51):@Vikram Atreya you could prove that your flag works
Vikram Atreya (Mar 16 2021 at 03:52):Okay
I'll send screenshots of the tests that I did
Sean (Mar 16 2021 at 03:53):screenshot would be great to see, but I mean you could create a unit/integration test for it that demonstrates it working as it should. the dbupgrade command is in production use, so we'll need to be really sure it does what it's supposed to and doesn't do what it's not supposed to, particularly when inputs are error conditions.
Sean (Mar 16 2021 at 03:57):example test #1 might be to create a db1.g with object A and db2.g with object B, then dbconcat db2.g into db1.g; should result in db1.g containing an A and a B object. second time might be objects A, B in db1.g and repeat the same concat; should result in db1.g still containing A and B, but B from db2.g should have replaced db1's B.
Sean (Mar 16 2021 at 03:58):there are a variety of different types of tests you could implement, but here's an example you could follow easily enough: src/libged/tests/test_tops.c
Sean (Mar 16 2021 at 03:59):if that's too hard but you know how to write a shell script, you could demonstrate it working that way instead
Vikram Atreya (Mar 16 2021 at 04:01):Let me see test_tops.c if I feel its too hard, I can write a shell script
Sean (Mar 16 2021 at 04:02):a relevant shell script example is in regress/bots/bots.sh
Vikram Atreya (Mar 16 2021 at 04:06):Created a patch for now
Vikram Atreya (Mar 16 2021 at 04:06):Will test and post updates here
Sean (Mar 16 2021 at 04:09):ultimately, you want to show how dbconcat behaves with and without the flag as well as without and with name collisions. includes a simple example like the B into A example as well as A into A and a multi-object partial merging like A/B/C into C'/D'/E' -- does it end up with ABCD'E'?
Sean (Mar 16 2021 at 04:10):Vikram Atreya said:
Created a patch for now
That's great. I'll see if I can get you some feedback on it here soon.
Vikram Atreya (Mar 16 2021 at 04:10):Sean said:
ultimately, you want to show how dbconcat behaves with and without the flag as well as without and with name collisions. includes a simple example like the B into A example as well as A into A and a multi-object partial merging like A/B/C into C'/D'/E' -- does it end up with ABCD'E'?
Yeah I have tested all these manually
Sean (Mar 16 2021 at 04:12):More advanced testing would be to combine flags and make sure behavior is correct, like adding a prefix and overwriting.
Sean (Mar 16 2021 at 04:13):like if db1.g has preA and db2 has A and you import with overwrite and prefix 'pre', does it end up with db2's A named as preA?
Vikram Atreya (Mar 16 2021 at 04:18):Sean said:
like if db1.g has preA and db2 has A and you import with overwrite and prefix 'pre', does it end up with db2's A named as preA?
I did not do it this way, you can either add a suffix, prefix or override according to my implementation
Vikram Atreya (Mar 16 2021 at 04:19):When you use the override flag, it override anything that has the same name, all the prefix suffix code is skipped
Vikram Atreya (Mar 16 2021 at 04:20):<deleted>
Vikram Atreya (Mar 16 2021 at 04:21):Maybe as the next step (later on), I could make the flag overwrite only objects that are specified after the flag
Sean (Mar 16 2021 at 15:18):Sure, whatever the behavior is defined to be -- the point was just to make sure it's tested. So if someone tries to add a prefix and overwrite, I would expect it to abort and not modify the database since it'd be bad arguments.
Vikram Atreya (Mar 16 2021 at 16:29):I understood how they have written tests but how do i run that specific test?
Vikram Atreya (Mar 17 2021 at 07:12):In the test script there is a line
MGED ="`ensearch mged`"
What is ensearch here?
Vikram Atreya (Mar 17 2021 at 08:19):@Sean I have written a script where a rhc is overwritten by a sphere. Initally I tried to get the height of the cylinder and then the radius of the same object after dbconcat. So if we get the radius successfully it means the test has passed. Is this a valid test?
dbupgrade_test.sh
Vikram Atreya (Mar 17 2021 at 10:06):Sean said:
A/B/C into C'/D'/E' -- does it end up with ABCD'E'?
Implemeted this example as a test in the following file: dbupgrade_test_2.sh
Sean (Mar 18 2021 at 04:06):@Vikram Atreya see regress/library.sh ... ensearch() is basically a helper function that finds the path to mged, looking in the expected build/install locations.
Sean (Mar 18 2021 at 04:16):Vikram Atreya said:
Implemeted this example as a test in the following file: dbupgrade_test_2.sh
These look really good! If you can clean them up a bit (e.g., file header is still bot.sh) and make them a proper patch in regress/libged/dbconcat or similar subdirectory so they actually run, we could hook this all together.
Can you think of any other useful cases or conditions to test?
Vikram Atreya (Mar 18 2021 at 04:56):Sean said:
Vikram Atreya said:
Implemeted this example as a test in the following file: dbupgrade_test_2.sh
If you can clean them up a bit (e.g., file header is still bot.sh)
:sweat_smile: Will do right away
Vikram Atreya (Mar 18 2021 at 04:58):Sean said:
Can you think of any other useful cases or conditions to test?
One where the -r flag is used but none of the names match
Other than this i cant think of anything specific to the -r flag
Vikram Atreya (Mar 18 2021 at 05:09):Sean said:
regress/libged/dbconcat
There isnt a libged directory in regress, So should i create one or just directly put it in regress?
Sean (Mar 18 2021 at 19:08):Vikram Atreya said:
There isnt a libged directory in regress, So should i create one or just directly put it in regress?
If you were motivated to convert them into C/C++, they really belong in src/libged/tests ... that would ensure the test runs continuously on Windows too. As shell scripts, though, they belong in regress and should be in a subdir somewhere meaningful since there are 2+ of them.
Vikram Atreya (Mar 18 2021 at 19:11):One more option could be to merge them into 1 shell script
Vikram Atreya (Mar 18 2021 at 19:15):I could convert them into C/C++ but I couldn't understand how to run the specific test
Sean (Mar 18 2021 at 19:25):morrison@agua .build % ctest -R ged_test_tops_moss
Test project /Users/morrison/brlcad.RELEASE/.build
Start 874: ged_test_tops_moss
1/1 Test #874: ged_test_tops_moss ............... Passed 0.30 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.40 sec
ged_test_tops_moss is declared in src/libged/tests/CMakeLists.txt
Sean (Mar 18 2021 at 19:26):it creates the test binary and defines a test target -- that's all that's needed
Sean (Mar 18 2021 at 19:27):lots of other examples in other src/lib*/tests/CMakeLists.txt directory unit tests
Vikram Atreya (Mar 18 2021 at 19:40):Will try to implement the test in C/C++
Vikram Atreya (Mar 18 2021 at 19:40):And what could I work on next?
Vikram Atreya (Mar 18 2021 at 20:37):Opened a patch for now with both tests merged into 1 script
Sean (Mar 19 2021 at 03:42):@Vikram Atreya there are limitless possibilities. if you ask me, I'll keep testing your abilities with incrementally more difficult challenges. So far you're doing great ;)
Sean (Mar 19 2021 at 03:44):you could try your hand at something fun and creative -- like creating a model of something (beginner: via mged, advanced: via code)
Sean (Mar 19 2021 at 03:45):or you could see how you do at a technical task like debugging any of the BUGS file issues. here's a recent relatively tractable one:
"running rt -C255/0/0 -o file.pix ... does not result in the background color getting used. It does work with -F file.pix, going through libfb instead of libicv"
Vikram Atreya (Mar 19 2021 at 03:57):Sean said:
or you could see how you do at a technical task like debugging any of the BUGS file issues. here's a recent relatively tractable one:
"running rt -C255/0/0 -o file.pix ... does not result in the background color getting used. It does work with -F file.pix, going through libfb instead of libicv"
Sure I could work on this, but I am thinking of writing a proposal for libged so shud I focus more on coding related to libged as I'll also gain experience to write my proposal?
Vikram Atreya (Mar 19 2021 at 04:01):Coz as of now I feel a that I dont have all the knowledge required to write a good proposal for libged
Vikram Atreya (Mar 19 2021 at 04:07):And regarding that I also wanted to ask if I could propose any of these ideas for GSoC:
1) Transaction for dbupgrade
2) Implementing the undo command at least for 1 time from present state
3) Implement the help flag for all major commands
4) Implement the array command - its like a command for creating patterns right?
If there are more important tasks in libged which are doable within the GSoC timeline please do suggest
Sean (Mar 19 2021 at 07:22):As demonstrated by your recent patch work, I'm sure you will be able to write a good proposal regardless.
Sean (Mar 19 2021 at 07:23):That said, you could certainly propose any of those for GSoC or a combination of them (as a couple aren't that complicated).
Vikram Atreya (Mar 19 2021 at 07:23):Yeah i was planning for a combination of any 2 or 3
Sean (Mar 19 2021 at 07:24):With regard to libged priorities, interruptibility and undo are near the top. Transactions are closely related to both and may be required to implement them properly.
Sean (Mar 19 2021 at 07:26):There are certainly shorter-term solutions to undo that are possible without transactions that might be worth implementing.
Vikram Atreya (Mar 19 2021 at 07:28):I also had this doubt, when applying transactions will it be 1 implementation for all commands or for every command it will have to be done separately?
Sean (Mar 19 2021 at 07:31):I think that's a big "it depends"
Sean (Mar 19 2021 at 07:31):ideally 1 implementation of course
Sean (Mar 19 2021 at 07:31):but it's entirely likely that all commands will have to be adapted or updated
Vikram Atreya (Mar 19 2021 at 07:33):I was thinking maybe at the function call level, we make the transaction, where every cmd entered first affects a copy of the db and then the db is just replaced. By that the code for the cmds never has to get affected
Sean (Mar 19 2021 at 07:34):so for example, the cp command that copies an object. it reads an object, it writes an object. instead, it may need to generate read/write _events_ and let a general parent routine do the actual read/write.
Sean (Mar 19 2021 at 07:35):perhaps a 'copy' event instead of read/write, there is some design discovery needed
Sean (Mar 19 2021 at 07:36):Vikram Atreya said:
I was thinking maybe at the function call level, we make the transaction, where every cmd entered first affects a copy of the db and then the db is just replaced. By that the code for the cmds never has to get affected
that's a good approach and I've had similar thoughts .. BUT
Vikram Atreya (Mar 19 2021 at 07:36):Sean said:
it may need to generate read/write _events_ and let a general parent routine do the actual read/write.
Dint get u from here. Maybe ill have to surf through the code to understand properly
Sean (Mar 19 2021 at 07:37):you have to know / keep in mind that a database can be 10GB in size. single objects can be several GB, and it needs to take that constraint into consideration without introducing non-interactive pauses.
Sean (Mar 19 2021 at 07:38):there's nothing to surf through the code to understand really.. think about it like the unix cp command
Sean (Mar 19 2021 at 07:38):when you copy a file, say you 'cp a b' ... it obviously opens a, reads it, opens b, writes it, yes?
Vikram Atreya (Mar 19 2021 at 07:38):Yeah
Sean (Mar 19 2021 at 07:40):well say we wanted to make 100% sure that neither a nor b are ever corrupted, even if the power plug was pulled out randomly
Sean (Mar 19 2021 at 07:41):for the most part, the operating system and sometimes the file system can help, but assume they didn't...
Sean (Mar 19 2021 at 07:41):what would you do to ensure that the operation is always error-free
Sean (Mar 19 2021 at 07:43):you would probably implement a system like what some file systems use, a transaction log -- where you'd modify cp to create a read event and a write event. the events are written to a ledger, then actually performed, then the ledger is updated.
Vikram Atreya (Mar 19 2021 at 07:44):Hmm yeah
Sean (Mar 19 2021 at 07:45):that way if anything ever goes wrong (say cp crashes or you lose power), you can look at the ledger and fix it automatically (for example) or undo the partial changes etc
Sean (Mar 19 2021 at 07:45):cp is overly simple of course, but the concept extends to much more complicated and longer-running commands or compound commands (commands that call many other commands)
Vikram Atreya (Mar 19 2021 at 07:48):But undoing might not be possible by retracing steps back for many commands right?
Sean (Mar 19 2021 at 07:48):It's worth mentioning that there is an alternative to event-based transactions. We could create an in-memory copy of the .g file, let commands work on it directly so we don't have to modify their code, and then write out the in-memory objects that changed to the on-disk database.
Vikram Atreya (Mar 19 2021 at 07:49):Lets say db -r was half way done, even if we know that overwritten objects cant be restored right?
Vikram Atreya (Mar 19 2021 at 07:49):Sean said:
It's worth mentioning that there is an alternative to event-based transactions. We could create an in-memory copy of the .g file, let commands work on it directly so we don't have to modify their code, and then write out the in-memory objects that changed to the on-disk database.
This was what I was thinking, as undoing complex cmds by retracing steps back is very tough
Sean (Mar 19 2021 at 07:49):I wouldn't mix undo and transactions.. they're different concepts.
Vikram Atreya (Mar 19 2021 at 07:50):Okay
Sean (Mar 19 2021 at 07:50):if a transaction doesn't complete, that's not something a user would "undo". the transaction system can and should automatically ensure that either the whole transaction was successful or none of it was.
Vikram Atreya (Mar 19 2021 at 07:52):Got it
My thoughts got intertwined between undo and transactions for a min
Sean (Mar 19 2021 at 07:53):transactions will likely either need to use the two-file approach (inefficient, slow, but very easy to implement) or a ledger approach (efficient, fast, but a little harder to implement)
Sean (Mar 19 2021 at 07:54):now once a transaction or command completes, that is something that we'll want to be able to undo. that can be achieved by keeping copies of objects in an undo cache.
Sean (Mar 19 2021 at 07:55):and an undo ledger
Vikram Atreya (Mar 19 2021 at 07:56):Yeah, we could undo simple commands by retracing steps back but for complex cmds it would be better to use an undo cache
Sean (Mar 19 2021 at 08:10):yes that's where a ledger is critical because only operations that change object internals would need to be saved as copies. the rest could just save the event like "a was renamed to b" and "applied matrix to c"
Vikram Atreya (Mar 19 2021 at 09:11):Sean said:
"running rt -C255/0/0 -o file.pix ... does not result in the background color getting used. It does work with -F file.pix, going through libfb instead of libicv"
I'll start working on this and parallelly make my proposal, I shud be able to make a first draft by Sun night
Vikram Atreya (Mar 20 2021 at 08:38):I tried rt -C255/0/0 -o file.pix and rt -C255/0/0 -F file2.pix then converted both files to png using the pic-png command
I see that both of them have the right background. Am I missing something here?
Erik (Mar 20 2021 at 12:57):s/ledger/journal/g if ya'll want to stick with the 'normal' words :)
I have vague recollection of a couple "special" input colors that were treated unusually... very well may be misrecollection :) 'color keyed' states... (hopefully removed, may still be lingering bits, if I'm not off in the weeds?)
Vikram Atreya (Mar 20 2021 at 16:17):Pardon me, but I didn't get what you are saying. What/Where is s/ledger/journal/g and where can I find the color keyed states
Mahmoud Hammad (Mar 20 2021 at 18:33):Hello everyone!
I'm interested in Computer graphics, and familiar with Three.js ,WebGL, and Javascript,
I had the pleasure to create these projects https://drive.google.com/drive/folders/1HfHAKOdryzF5XSM4wupNh8IC-2FczkJN
and I am looking to do more by contributing on the OGV project https://github.com/opencax/GSoC/issues/26
Mahmoud Hammad (Mar 20 2021 at 18:38):I found that the last commit on the OGV project was 2 years ago!, so Please let me know if this project has a priority on GSoC21
Vikram Atreya (Mar 20 2021 at 18:45):Vikram Atreya said:
Pardon me, but I didn't get what you are saying. What/Where is
s/ledger/journal/gand where can I find thecolor keyedstates
Nevermind the "What/Where is s/../../g" , havent seen that notation before . Now I know its find and replace :sweat_smile:
Vikram Atreya (Mar 22 2021 at 20:12):For making my proposal I wanted to know how the calls are made from us typing the command in mged till the whatever command 's code is executed. Could someone guide me regarding this or point me to some documentation
Sean (Mar 23 2021 at 20:20):Vikram Atreya said:
I tried
rt -C255/0/0 -o file.pixandrt -C255/0/0 -F file2.pixthen converted both files to png using the pic-png command
I see that both of them have the right background. Am I missing something here?
@Vikram Atreya you're not missing anything. You say those both result in a red background?
Sean (Mar 23 2021 at 20:20):Erik said:
s/ledger/journal/g if ya'll want to stick with the 'normal' words :)
Pfffft. Normalshmormal.
Sean (Mar 23 2021 at 20:21):Mahmoud Hammad said:
I found that the last commit on the OGV project was 2 years ago!, so Please let me know if this project has a priority on GSoC21
Hello @Mahmoud Hammad. OGV and related projects like the Benchmark Database and Materials Database are always excellent projects to propose for someone excited by them and willing to take them to their next level.
Vikram Atreya (Mar 23 2021 at 20:23):Sean said:
Vikram Atreya you're not missing anything. You say those both result in a red background?
yeah , both of them are in red backgrounf
Sean (Mar 23 2021 at 20:24):So either the bug is platform or compilation-specific or got fixed at some point...
Sean (Mar 23 2021 at 20:25):I'll see if I can reproduce it.
Vikram Atreya (Mar 23 2021 at 20:25):Okay
Vikram Atreya (Mar 23 2021 at 20:51):Vikram Atreya said:
For making my proposal I wanted to know how the calls are made from us typing the command in mged till the whatever command 's code is executed. Could someone guide me regarding this or point me to some documentation
Any ping on this plz
Sean (Mar 24 2021 at 02:17):@Vikram Atreya that is a bit hard to follow, so no worries. how calls are made is typically as follows...
Sean (Mar 24 2021 at 02:20):in main() in src/mged/mged.c you'll see mged_setup() is called,
and in there cmd_setup() is called,
and in there you'll see iterates over an mged_cmdtab table.
that table is the mapping of a command's name (e.g., 'ls') and its function (e.g., ged_ls)
Sean (Mar 24 2021 at 02:21):those strings are actually registered with a Tcl interpreter and then cause the function to be automatically called whenever the command's name is provided for evaluation.
Sean (Mar 24 2021 at 02:24):once that happens, nearly the rest of the magic is handled automatically by Tcl. we display a prompt (e.g., "mged> ") and when a user types anything and hits enter, that string is passed to tcl for evaluation (Tcl_Eval()) which causes it to call the function that was registered.
Vikram Atreya (Mar 24 2021 at 04:16):So if I was applying a transaction based on 2 file system, I would add 1 more link in this chain right? Where instead of directly calling ged_cmd, I would add a transaction function which would create a parallel db, apply changes to that(in which it calls ged_cmd) and then finally change pointer (and free old db).
Sean (Mar 24 2021 at 04:38):Hmmm
Vikram Atreya (Mar 24 2021 at 04:38):Im just naming a possibility here, could be done in other ways too ofc
Sean (Mar 24 2021 at 04:42):you're not far off, I just had to refresh my memory on the code
Sean (Mar 24 2021 at 04:42):if you look in setup.c, notice there are three things listed: the name of the command, a wrapper function, and the callback
Sean (Mar 24 2021 at 04:44):that second wrapper value a function that is called when the name of the command is encountered, and it's what calls the callback
Vikram Atreya (Mar 24 2021 at 04:46):Okay
Sean (Mar 24 2021 at 04:46):a simple one to understand is the cmd_ged_edit_wrapper() function that only the "clone" command apparently calls. looking at that cmd_ged_edit_erapper() function, it invokes ged_clone (via ctp->ged_func()), and as long as there wasn't an error or request for help, it draws the cloned object.
Sean (Mar 24 2021 at 04:49):nearly all functions go through cmd_ged_plain_wrapper() which calls the ged function, handles if MORE arguments are needed, and redraws anything on the command line. seems pretty redundant with cmd_ged_edit_wrapper()..
Vikram Atreya (Mar 24 2021 at 04:52):Hmmm so I could add a link from setup.c to the transaction mechanism, that way I wont have to deal with things in between setup.c and MGED
Sean (Mar 24 2021 at 04:54):There's many possibilities. I mean the current state of the code is a work in progress. If you dig in deeper you'll see that there are two command tables, one using a newer style with a few entries and the older one using the big cmdtab.
Sean (Mar 24 2021 at 04:55):The goal is to eventually just have something like ged_exec() that does it instead of having the application have a big list of commands and functions.
Sean (Mar 24 2021 at 04:56):In that scenario, it makes sense for ged_exec() to essentially do all the wrapper setup/teardown necessary
Vikram Atreya (Mar 24 2021 at 04:56):Yeah
Sean (Mar 24 2021 at 05:03):that could be step 1 for a libged gsoc proposal, spend a week migrating the command table from src/mged to src/libged with a ged_exec call, then step 2 could be to eliminate / consolidate the various command table wrappers into just one wrapper, then step 3 demonstrate a working transaction demo outside of libged (just a test.c test case), then step 4 implement a manual transaction command, and step 5 integrate transactions into ged_exec
Vikram Atreya (Mar 24 2021 at 05:08):Also if I plan to implement both UNDO and transactions will that be too much to aim for during GSoC, considering the reduced timeframe from prev GSoCs?
Sean (Mar 24 2021 at 05:35):I would just plan it out as starting with one and transitioning into the other
Vikram Atreya (Mar 24 2021 at 05:38):Okay
Vikram Atreya (Mar 24 2021 at 05:38):But can both of them be done within GSoC?
Vikram Atreya (Mar 24 2021 at 05:38):or is it a risky proposition
Sean (Mar 24 2021 at 05:40):Vikram Atreya said:
But can both of them be done within GSoC?
This isn't something I can answer because it mostly depends on your skill level, available time, ability to stay focused on the task, your ability navigating existing code, etc.
Sean (Mar 24 2021 at 05:41):complexity-wise, it's entirely doable. I'd estimate it taking one of the experienced core devs 2-3 weeks for both so yeah, that is likely very appropriate for gsoc in general.
Vikram Atreya (Mar 24 2021 at 05:43):Fine then I'll include both of them in my proposal
Vikram Atreya (Mar 24 2021 at 05:47):I got one more doubt as I was researching about how transactions are implemented in different systems, I found this in the journal/ledger based file system wikipedia page, The statement was made in context to deleting a file in linux and power failure during the process. "After a crash, recovery simply involves reading the journal from the file system and replaying changes from this journal until the file system is consistent again." , So i found it too simple that will just replaying instructions of the prev cmd stop the db from being corrupted.
Ref:https://en.wikipedia.org/wiki/Journaling_file_system#Rationale
Vikram Atreya (Mar 24 2021 at 19:50):For the UNDO command, I was thinking about an implementation where we store all commands applied on the file and when we undo, just rebuild the file from scratch applying all cmds given from the beginning excluding the number of undo specified, like if we implement the cmd as: undo <no of steps to go back> , we just pop the cmds from the applied_cmds stack and put them in the redo_cmd stack.
Vikram Atreya (Mar 24 2021 at 19:50):I wanted to check the viability of this method in terms of time complexity, so I first checked the primitives in tank_car.g which was around 500 primitives. I wrote a script to create 500 primitives in a file, the script took just 0.5 sec to run and all 500 primitives were created.
Vikram Atreya (Mar 24 2021 at 19:51):So this looked like a viable approach to implement the undo command (I could surely be wrong in saying its viable).
Vikram Atreya (Mar 24 2021 at 19:51):To further test viability could anyone suggest me commands that affect the db (unlike an rt, stat, summary ) and are very complex (in terms of time )to run.
Sean (Mar 25 2021 at 04:35):Vikram Atreya said:
I wanted to check the viability of this method in terms of time complexity, so I first checked the primitives in tank_car.g which was around 500 primitives. I wrote a script to create 500 primitives in a file, the script took just 0.5 sec to run and all 500 primitives were created.
@Vikram Atreya excellent question, but it's not really viable. Not that our simple example geometry files are just that ... simple examples. Real .g files in common use can be 1-10GB with 50,000 objects in them.
Sean (Mar 25 2021 at 04:38):So I think you're on the right track, but I think undo can actually be quite simple.
Vikram Atreya (Mar 25 2021 at 04:39):I feel it is quite simple for commands that don't have data being removed, like a rename or addition of a new object
Sean (Mar 25 2021 at 04:39):There's two possible approaches. One very simple method is to simply keep the last N undo states in the .g file as hidden objects.
Vikram Atreya (Mar 25 2021 at 04:39):But if a kill command is used or a dbconcat -r , then to recover data with an undo we will have to store data
Vikram Atreya (Mar 25 2021 at 04:40):Sean said:
There's two possible approaches. One very simple method is to simply keep the last N undo states in the .g file as hidden objects.
Nice idea
Sean (Mar 25 2021 at 04:48):That way, for example, if you deleted an object A, you might copy it to ._A;1 and then delete A. To undo, you'd rename it back, delete ._A;1 (and possibly create another hidden object like ._A;2 that would let you undo the undo)
Sean (Mar 25 2021 at 04:51):Ideally, it's some convention that lets you represent an undo chain (like what emacs and other programs do so you can undo and redo infinitely).
Sean (Mar 25 2021 at 04:53):here's an textual description of that process: https://stackoverflow.com/questions/3527142/how-do-you-redo-changes-after-undo-with-emacs
Vikram Atreya (Mar 25 2021 at 04:55):Sean said:
here's an textual description of that process: https://stackoverflow.com/questions/3527142/how-do-you-redo-changes-after-undo-with-emacs
Yeah I read this
Vikram Atreya (Mar 25 2021 at 04:58):But some actions aren't specific to an object like a dbconcat, there the code will have to be modified to give details to the undo stack
Vikram Atreya (Mar 25 2021 at 04:58):Im aware of only dbconcat for now, but there might be others
Vikram Atreya (Mar 25 2021 at 04:59):Which might delay the implementation of the undo, if we were to store previous versions of objects and not the entire db
Sean (Mar 25 2021 at 04:59):relevant background: https://www.gnu.org/software/emacs/manual/html_node/emacs/Undo.html
Sean (Mar 25 2021 at 05:00):Vikram Atreya said:
But some actions aren't specific to an object like a dbconcat, there the code will have to be modified to give details to the undo stack
There's a whole bunch of commands that act on multiple objects, so yes -- they'll have to be dealt with too, but they're essentially the same.
Sean (Mar 25 2021 at 05:01):maybe I run "kill *"
Sean (Mar 25 2021 at 05:02):a single undo should restore them all...
Vikram Atreya (Mar 25 2021 at 05:02):Yep
Vikram Atreya (Mar 25 2021 at 05:04):How about this idea where we take some N and store the entire db before N steps, and then the N commands that had followed. Thus we will need to apply only N steps before we access any of the previous states
Vikram Atreya (Mar 25 2021 at 05:04):N could be some 50-100
Vikram Atreya (Mar 25 2021 at 05:04):After every additional step, we update the copy of the db by applying the last cmd in the stack
Sean (Mar 25 2021 at 05:05):the way I've had in mind for libged is very much related to transactions but doesn't require transactions. that is, we could modify all commands to generate events instead of actually modifying the database, so "rm *" will generate a set of DELETE events, for example, or dbconcat will generate a set of WRITE or REPLACE events.
Vikram Atreya (Mar 25 2021 at 05:06):Okay, now I get what you meant by events, I hadnt understood fully what u meant the last time you had mentioned it
Sean (Mar 25 2021 at 05:08):inside the invoking parent wrapper (e.g., ged_exec()), it calls the command like ged_rm, gets back the set of DELETE events, then the wrapper does the work to "cp obj1 ._obj1;1" and "cp obj2 ._obj2;1" and record the undo entry for rm in the undo ledger/journal/transcript.
Sean (Mar 25 2021 at 05:08):and then actually deletes obj1 obj2 etc
Sean (Mar 25 2021 at 05:16):Notice that this is essentially the "partial checkpoint" method described using both command pattern and memento pattern.
Sean (Mar 25 2021 at 05:16):https://en.wikipedia.org/wiki/Undo
Vikram Atreya (Mar 25 2021 at 05:18):Sean said:
Ive done my basic research :grinning_face_with_smiling_eyes:
Sean (Mar 25 2021 at 05:18):That all said -- I started off saying there are two possible approaches. Two more easily implemented approaches, that is. The other method is a form of full checkpoint.
Vikram Atreya (Mar 25 2021 at 05:21):Yeah, but the partial checkpoint is even better right? and the implementations are not too far off either acc to me
Sean (Mar 25 2021 at 05:22):yes and no, there are complexity considerations
Sean (Mar 25 2021 at 05:22):Instead of modifying all commands to generate events and making a wrapper do all the work to act intelligently given a set of events, we could implement a completely agnostic system that is very simple.
Sean (Mar 25 2021 at 05:22):for example (and this is JUST an example of a possible implementation)...
Sean (Mar 25 2021 at 05:24):we could let a command (e.g., rm *) operate on the .g, and then check it into a repo. then another command (e.g., dbconcat file2.g) operates, check it into a repo, etc. Then an undo is requested, we restore a previous repo state, check it into the repo, and so on.
Vikram Atreya (Mar 25 2021 at 05:25):Like saving a version for every command right?
Sean (Mar 25 2021 at 05:25):right, you'd probably use something like https://libgit2.org
Vikram Atreya (Mar 25 2021 at 05:28):Wont this be very bad for space complexity to even consider, especially if files are 10gb or more
Sean (Mar 25 2021 at 05:30):this is very simple to implement. the downsides are it will unlikely be anywhere near as fast as the partial approach with events, but the simplicity is hard to argue against as it's command-agnostic and will not be susceptible to bugs, oversight, changes over time. For example, say someone makes the "rt" command start writing a counter into the .g file ... which would be very easy to overlook and not undo.
Sean (Mar 25 2021 at 05:32):Vikram Atreya said:
Wont this be very bad for space complexity to even consider, especially if files are 10gb or more
No, not necessarily. It actually could end up taking up less space under some conditions.
Vikram Atreya (Mar 25 2021 at 05:32):So you are suggesting like how patches are created in svn, where we just store changes in the file at a very low level?s
Sean (Mar 25 2021 at 05:37):the first method ("partial" linear undo with events) for example, we might do "cp A ._A;1" because the color changed. if A is a really big object, that'd be terribly wasteful to copy it like that. the second method ("full" checkpoint undo), it's up to the VCS to figure out what changed and only save those changes. In the case of something like libgit2, it's only going to save the bytes around what changed even for binary blobs. If A's a big object and we only change the 3 color bytes, it's only going to track that a small block of the file changed (maybe writing 256 bytes).
Sean (Mar 25 2021 at 05:37):Vikram Atreya said:
So you are suggesting like how patches are created in svn, where we just store changes in the file at a very low level?s
yes, in fact libsvn is another viable option
Vikram Atreya (Mar 25 2021 at 05:38):I also thought about this approach but haven't started reading about it. I'll read about this method as well
Sean (Mar 25 2021 at 05:47):There's two ways to use a version-control-system (VCS) like libgit2 or libsvn. One would be to simply use them as a temporary undo mechanism in a temporary scratch space whenever a .g is opened for write access, deleting the repo when the app quits.
Another would be to fundamentally change our database I/O layer (i.e., rel6 .g files) where we'd write out a .g as 1 file per object as an actual .git initialized repository, with every change checked in and preserved.
Sean (Mar 25 2021 at 05:49):The latter is probably ideal, but HIGH risk, so it's not really a good candidate for GSoC at this time.
Sean (Mar 25 2021 at 05:53):so I think it still boils down to either partial+events or full+libgit2 in a temp space.
Vikram Atreya (Mar 25 2021 at 05:54):Yeah
Vikram Atreya (Mar 25 2021 at 05:57):I'm a little pro-partial+events
Vikram Atreya (Mar 25 2021 at 05:58):Because I think there is a big learning curve for the second approach, and implementation might have to deal with mged backend rather than the way we write a file
starseeker (Mar 25 2021 at 11:05):A question... I'm not sure how well git or svn backends would do compacting (say) a large BoT with one vertex changed. It would be worth doing some testing creating minor variations on large meshes and checking them in to see what happens... based on the repo conversion I'd expect libgit2 to do a bit better than svn, but either way I think it's worth checking practically.
Erik (Mar 25 2021 at 14:22):"lfs" exists for dealing with large files, but it's a headache all it's own
starseeker (Mar 25 2021 at 16:18):https://github.com/mendsley/bsdiff I think is used for lots of binary diffing applications - might be interesting to see what could be done with it to adapt to .g object diffing, if libgit2 doesn't do well out of the box...
starseeker (Mar 25 2021 at 18:03):I should probably point out this work in case it's of use in this context: https://github.com/BRL-CAD/geomcore/tree/main/src/libgvm
Vikram Atreya (Mar 25 2021 at 18:06):Could you give me a brief on what this is? Is it a new geom editor being developed?
Vikram Atreya (Mar 25 2021 at 18:07):To be more specific, will I be working on this if I am to implement the UNDO command
starseeker (Mar 25 2021 at 18:08):geomcore is some old work primarily focused on a "geometry service" that would expose database contents over the network. As part of that work, some early work was done on an API for "versioning" geometry by breaking it up into individual objects and checking them into a VCS backend.
starseeker (Mar 25 2021 at 18:09):IIRC, I got it as far as being able to read in havoc.g from the examples, check all the objects into the SVN database, and then reconstitute the .g file from the SVN history as a ".g" checkout, but it's been a long time now...
starseeker (Mar 25 2021 at 18:09):Thanks to Sean there was some thought given to making the API modular to different VCS backends, but I only ever experimented with SVN. Today I'd prefer to try libgit2...
starseeker (Mar 25 2021 at 18:10):My vague memory was that the checking process, at least, was quite slow with SVN.
Vikram Atreya (Mar 25 2021 at 18:13):Ohkay
starseeker (Mar 25 2021 at 18:16):I don't know if it would be useful for what you've been discussing or not - Sean may have a better sense - but it's what I think of right off anytime discussion of version control backends for geometry comes up.
Vikram Atreya (Mar 25 2021 at 18:18):Yeah I think it will be useful if I go ahead on making a VCS for the undo mechanism
Vikram Atreya (Mar 25 2021 at 18:19):But I wont be implementing a VCS for files though, that is too complex for GSoC
starseeker (Mar 25 2021 at 18:27):OK, I got the gvmtest app running. Ha! One minor compile fix, but looks as if it still works.
starseeker (Mar 25 2021 at 18:31):@Vikram Atreya The libgvm work may well be relevant to both... If you're going to establish a temporary repository for UNDO operations, you'd in principle be doing a variation of what gvmtest is doing - takeing an existing .g file, writing out the individual objects into the VCS, and then at need pulling out old versions into the "working" .g file.
Vikram Atreya (Mar 25 2021 at 18:36):Yeah that could surely be done, I was expecting to work with it at the binary level though, since you have already made some progress on it, I will go through libgvm and then reconsider, because right now I "feel" implementing a VCS might be difficult for me :grimacing:
starseeker (Mar 25 2021 at 18:48):You mean keeping entire copies of the .g file?
Vikram Atreya (Mar 25 2021 at 18:49):Sean said:
the second method ("full" checkpoint undo), it's up to the VCS to figure out what changed and only save those changes. In the case of something like libgit2, it's only going to save the bytes around what changed even for binary blobs. If A's a big object and we only change the 3 color bytes, it's only going to track that a small block of the file changed (maybe writing 256 bytes).
Like this
starseeker (Mar 25 2021 at 18:50):Oh, OK, so feed the whole .g to libgit2 and let it figure out what the diff is?
Vikram Atreya (Mar 25 2021 at 18:50):Exactly!
starseeker (Mar 25 2021 at 18:51):/me winces a little - I'd for sure test that to verify Git does something smart. Grab a big mesh online, alter a couple vertices with Meshlab, then check both versions of the mesh into a Git repo and see how big it ends up being.
Vikram Atreya (Mar 25 2021 at 18:53):True that at times it might be large but wont it always be smaller than storing a copy of the whole .g file?
starseeker (Mar 25 2021 at 18:53):Yes, but for lots of edits on large meshes it may still quickly reach impractical sizes.
Vikram Atreya (Mar 25 2021 at 18:54):Got your point
starseeker (Mar 25 2021 at 18:54):It may still be worthwhile, but if that's the behavior we would need to guard against the VCS repo getting too big
starseeker (Mar 25 2021 at 18:54):I.e. throwing away the older history when we start approaching some size limit
Vikram Atreya (Mar 25 2021 at 18:55):We will be saving a the same no of diffs as we need to go back and that could add up to be impractcal
Vikram Atreya (Mar 25 2021 at 18:55):starseeker said:
I.e. throwing away the older history when we start approaching some size limit
Yeah
starseeker (Mar 25 2021 at 18:55):If the diffs are small we'd be in good shape, but that's only practical if Git can figure out the small delta between similar binary objects. That's what I'm not sure of.
Vikram Atreya (Mar 25 2021 at 18:56):We could let the user set this space as well, so that they can adjust themselves the no of undos they want vs space consumed
starseeker (Mar 25 2021 at 18:56):/me nods
Vikram Atreya (Mar 25 2021 at 18:58):starseeker said:
If the diffs are small we'd be in good shape, but that's only practical if Git can figure out the small delta between similar binary objects. That's what I'm not sure of.
Let me try this out
Vikram Atreya (Mar 25 2021 at 19:20):I uploaded a .g file to github 2.85 KiB,
First I added a rhc to it and then pushed again, this led to writing 104 bytes
Next I changed the colour of an object, this led to writing 160 bytes
Vikram Atreya (Mar 25 2021 at 19:23):@starseeker this looks good right?
starseeker (Mar 25 2021 at 19:24):Well, what is needed is to change the same object and see if the written delta is smaller than the object size
Vikram Atreya (Mar 25 2021 at 19:24):Yeah thats what i did in the second step
Vikram Atreya (Mar 25 2021 at 19:24):160 bytes < 2.85 KiB
starseeker (Mar 25 2021 at 19:25):Well, that's not a geometry change though - that's an attribute change.
starseeker (Mar 25 2021 at 19:25):That's why I suggested editing a mesh.
Vikram Atreya (Mar 25 2021 at 19:25):Like a change in length?
Vikram Atreya (Mar 25 2021 at 19:25):Let me do that
starseeker (Mar 25 2021 at 19:26):moving a couple vertices would be ideal
starseeker (Mar 25 2021 at 19:26):tiny change to a large mesh
starseeker (Mar 25 2021 at 19:26):Are you familiar with meshlab?
Vikram Atreya (Mar 25 2021 at 19:26):starseeker said:
Are you familiar with meshlab?
ummm no
starseeker (Mar 25 2021 at 19:32):OK, one sec.
starseeker (Mar 25 2021 at 19:38):https://brlcad.org/~starseeker/bunny_0.g and https://brlcad.org/~starseeker/bunny_1.g are two versions of the Stanford bunny. If you save those two, first _0 and then _1, what happens?
Vikram Atreya (Mar 25 2021 at 19:38):Yea let me check
Vikram Atreya (Mar 25 2021 at 19:43):So adding the bunny wrote 1.58 MB and after overwriting bunny0 with bunny1 and then pushing it wrote an additional 8.49KB
Vikram Atreya (Mar 25 2021 at 19:44):I think these values are not correct..... 1 sec
Vikram Atreya (Mar 25 2021 at 19:45):So when i did git push it said: writing 832.11 KiB
after overwriting it and then pushing, it said writing 385.18 KiB
Vikram Atreya (Mar 25 2021 at 19:46):Vikram Atreya said:
So when i did git push it said: writing 832.11 KiB
after overwriting it and then pushing, it said writing 385.18 KiB
This is in the terminal
Vikram Atreya (Mar 25 2021 at 19:46):Vikram Atreya said:
So adding the bunny wrote 1.58 MB and after overwriting bunny0 with bunny1 and then pushing it wrote an additional 8.49KB
This is actually increase in file size
Vikram Atreya (Mar 25 2021 at 19:47):Vikram Atreya said:
So when i did git push it said: writing 832.11 KiB
after overwriting it and then pushing, it said writing 385.18 KiB
@starseeker So what do you think?
starseeker (Mar 25 2021 at 19:48):OK, not too bad.
starseeker (Mar 25 2021 at 19:48):It didn't double in size, at least
Vikram Atreya (Mar 25 2021 at 19:48):starseeker said:
It didn't double in size, at least
Yeah
starseeker (Mar 25 2021 at 19:59):Yeah, the xdelta3 size is on that order - 349K. bsdiff gets it down to 219K.
starseeker (Mar 25 2021 at 20:04):Ideally what we'll be able to set up is something that uses the default VCS backend for most deltas, but will let us specialize as we start to refine some specific operations (rotate, translate and scale will have large data deltas when applied to meshes, but can in principle be versioned with reverse-edit operations applied to the "working" mesh, for example. That's a lot of complexity though, and not something to introduce at the beginning.)
Vikram Atreya (Mar 25 2021 at 20:12):Though Im clear on the concept, I still feel that I lack the knowledge of the codebase to implement a VCS into it :(
I think should take a deep dive into the code and think of ways to implement it
Sean (Mar 25 2021 at 22:32):starseeker said:
I.e. throwing away the older history when we start approaching some size limit
This. I think that'll be necessary no matter what approach is taken with undo. We're at a point where we need to know how full the disks are getting.
Sean (Mar 25 2021 at 22:35):starseeker said:
If the diffs are small we'd be in good shape, but that's only practical if Git can figure out the small delta between similar binary objects. That's what I'm not sure of.
Since we're talking about undo, I think we can proceed with an assumption that undo information disappears when the database is closed. Undo is the priority, not maintaining a full edit history, so we could simply plan to let it go or utilize an in-memory database and keep track of memory that way.
Sean (Mar 25 2021 at 22:37):The big decision that needs to happen first is partial+events or full/agnostic as they are dramatically different approaches and impact on existing commands.
Sean (Mar 25 2021 at 23:06):I'd suggest we (i.e., @Vikram Atreya ) either plan to just use libgvm/libgit2 completely as-is and let it be whatever it is for a preliminary cut at full/agnostic, so we can then focus on the undo/redo side of things in the GUI; or we (i.e., @Vikram Atreya ) plan to make commands report events, have a main processor that handles all events consistently, and focus on migrating commands. Both ways have merit and tradeoffs.
Partial+events will almost certainly be an undo/redo system that is more performant, efficient, reversible, and simple to implement well enough, but entails editing all commands (so only able to undo migrated commands) and requires designing+implementing the event handlers. It's not hard and is 100% in our control complexity-wise, but requires a lot of code change.
Full/agnostic will almost certainly result in an undo/redo system more quickly, robustly, reversibly, and works with any command, but it will be FAR less efficient and involves a new external dep that can derail focus and otherwise represents unknown risk. It's almost certainly less code, though, so it would be overall less risk if libgvm/libgit2 behave.
Sean (Mar 25 2021 at 23:09):I think they're both viable and good approaches for different reasons. If you were experienced in either, that would probably lean me towards one or the other but you're not so it could be a wash risk-wise. @Vikram Atreya do you understand the tradeoffs? What are your thoughts? Which do you feel you understand the most? What don't you feel you understand?
Sean (Mar 25 2021 at 23:09):Those answers would be good to put into your proposal ;)
Vikram Atreya (Mar 26 2021 at 06:07):Sean said:
Partial+events . . . . , but entails editing all commands (so only able to undo migrated commands) and requires designing+implementing the event handlers. It's not hard and is 100% in our control complexity-wise, but requires a lot of code change.
Cant we directly store the command instead of generating events and then storing them?
Vikram Atreya (Mar 26 2021 at 06:13):Sean said:
do you understand the tradeoffs? What are your thoughts?
Yeah I understand the tradeoffs between both the methods. In theory, I understand both methods well and also feel the VCS method is better in terms of features as it can handle all commands and works independently of commands, so will be easier in the future also if some newer/unique cmds come up.
Vikram Atreya (Mar 26 2021 at 06:19):Sean said:
Which do you feel you understand the most? What don't you feel you understand?
As I said, in theory both look fine to me. But implementation wise maybe I'll have to learn more if the VCS method is applied, and I'm fine actually with spending as much time as needed to learn (even pre-GSoC). The part that I don't understand (which I eventually should understand is) how these methods translate to code,
I'm more clear on the implementation of the partial+events, as I have some experience with that part of the code, So we could just kill * of the present db, copy the partial into the db and then apply commands, in the undo list, on it
Vikram Atreya (Mar 26 2021 at 06:23):Sean said:
If you were experienced in either, that would probably lean me towards one or the other but you're not so it could be a wash risk-wise.
The partial+events doesnt require any experience right? Its pretty simple imo
Whereas ill have to learn how to implement a VCS in the other method (which i 'm actually fine doing, just that it might have to more uncertainity)
Sean (Mar 26 2021 at 17:37):Vikram Atreya said:
Cant we directly store the command instead of generating events and then storing them?
I'm not sure what you mean. What I'm calling an "event" is sort of the lowest level database-changing commands possible that are invertible without any special knowledge. For example, a "DELETE" event that removes an object, or a "CREATE" event that adds an object to the database. You can undo a CREATE by deleting the object. You can undo a DELETE by restoring the object from a cached backup. They require no knowledge of our primitive types, no knowledge of object internals, and could even be used to handle a higher level events like a "RENAME" (CREATE+DELETE) or a "MODIFY" (DELETE+CREATE).
Sean (Mar 26 2021 at 17:45):Vikram Atreya said:
As I said, in theory both look fine to me. But implementation wise maybe I'll have to learn more if the VCS method is applied
Probably about the same. You'll learn more about libgvm or libgit2 and less about libged with one and a lot more about libged and commands in the other.
The part that I don't understand (which I eventually should understand is) how these methods translate to code,
That's a good observation. Maybe it would help to see if you can write a simple main.c that adds, deletes, and undeletes a sphere with both methods. With the partial/libged method, you'd probably make your program call ged_make(), then copy the object to another in-memory backup on delete, then copy it back to undo. With libgvm or libgit, you'll call ged_make() and check it into the vcs, delete it with ged_kill() and check that into the vcs, then restore it from the vcs to undo.
Both programs would probably be 20 lines or so and be a really good way to understand the lowest level implications.
Sean (Mar 26 2021 at 17:45):We could probably use that harness to see what some really big edit implications look like performance-wise.
Vikram Atreya (Mar 26 2021 at 17:50):Regarding the events, maybe I have misunderstood the partial + events method
So what I understand is, we store a copy of the db before N commands, and when we want to do an undo, we apply N-1 commands on the copy of the db we have stored and by overwriting that on the resent state we bring it to the undone state
Vikram Atreya (Mar 26 2021 at 17:50):So in my method we just store the commands
like make rhc rhc, kill example, etc etc and rerun hem when necessary
Vikram Atreya (Mar 26 2021 at 17:56):So what I understand by events is that we break down commands, as you have mentioned above, RENAME = CREATE + DELETE
But why r they necessary if we directly store the high level command and then execute it
eg : storing ( dbconcat ex1.g ) would be easier than storing ( create obj1 , create obj2 etc )
Vikram Atreya (Mar 26 2021 at 18:40):I think the counter argument is that, just taking the case of dbconcat only, the ex1.g might not exist by the time we are undoing
Vikram Atreya (Mar 26 2021 at 18:42):But generating events will be required for such few commands right, like a make, kill, rename, etc need not be split into events....... So just creating events for few cmds might suffice, right?
Sean (Mar 29 2021 at 03:31):@Vikram Atreya That's not the idea of partial undo. What you describe is a "rerun" approach, which I don't think will scale well for our use case.
Sean (Mar 29 2021 at 03:32):Rerun is briefly described with full and partial at https://en.wikipedia.org/wiki/Undo
Sean (Mar 29 2021 at 03:36):With partial, you're basically storing (partial) state as commands are run so we have whatever information is required to restore the previous state. Those most simple form of this is to keep a copy of any objects that are changed or deleted. Since we can undo objects added by simply deleting them, we don't need to do anything special other than keep track of them.
Sean (Mar 29 2021 at 03:45):Note that there's LOTS of ways to implement this sort of undo. For example, here's a really simple setup where (for example) we decide to store our backups in the same .g file using a simple numeric suffix that denotes what change. For example, db.g might have in it: objA objB objC objA;1 objA;2 objA;3 objB;2 objC;4
The current state is three objects: objA objB objC
Sean (Mar 29 2021 at 03:49):The most recent change was some edit to objC. When the change was made to objC, we save the previous state is in objC;4 .. so undoing the command is a simple to replace objC with objC;4
Sean (Mar 29 2021 at 03:51):The next undo restores objA to a previous state (via objA;3).
The one after that was a simultaneous change to objA and objB, so we restore both from objA;2 and objB;2
Sean (Mar 29 2021 at 03:54):Hopefully you get the gist. That example is intentionally simple, but hopefully you can see how that's easily extended to keep track of added/removed objects or could use a different naming convention, and/or could be stored in a different file (e.g., .db.g~) or no file at all (e.g., just keep them in memory).
Sean (Mar 29 2021 at 03:56):It'd be really simple to implement something preliminary like this in under a week, and it'd be easy to optimize (e.g., instead of storing an entire backup of an object, maybe we only really need to store a matrix or an attribute change or handle certain events special (e.g., object additions/deletions).
Vikram Atreya (Mar 29 2021 at 10:23):Yeah I understood, but wont this take up a lot of space, we might end up storing (no of UNDOs possible)*(size of db)
So we deal with this by putting some limit to storage for UNDO?
Vikram Atreya (Mar 29 2021 at 17:25):Sean said:
That's a good observation. Maybe it would help to see if you can write a simple main.c that adds, deletes, and undeletes a sphere with both methods. With the partial/libged method, you'd probably make your program call ged_make(), then copy the object to another in-memory backup on delete, then copy it back to undo. With libgvm or libgit, you'll call ged_make() and check it into the vcs, delete it with ged_kill() and check that into the vcs, then restore it from the vcs to undo.
Both programs would probably be 20 lines or so and be a really good way to understand the lowest level implications.
I'll try writing this program
Vikram Atreya (Mar 29 2021 at 19:07):Do I make this program inside libged or a separate file like I had done for the boolweave unit test?
Sean (Mar 29 2021 at 19:11):I think a separate standalone file is best. This isn't likely code that would be kept as-is very long, it's to understand the implications and make a decision.
Sean (Mar 29 2021 at 19:14):like my assumption that it's about 20 lines.. if it ends up being dramatically less or more work, that would be important to know. And if/once it's working, we can use it to do a couple tests. If one takes <1sec and the other 100sec to do the same work, that would be pretty guiding (I don't expect that to be the case, more like 100ms vs 200ms).
Vikram Atreya (Mar 29 2021 at 19:35):Okay, will make a standalone file
Vikram Atreya (Mar 29 2021 at 19:37):If i have to call ged_make(), I'll have to pass a struct ged *gedp, but how do I initialize it if its not coming via mged
Sean (Mar 30 2021 at 04:19):@Vikram Atreya see src/libged/tests/test_tops.c for an example. You either call ged_open() which returns a gedp, or you call GED_INIT() directly if you want to open it yourself.
Vikram Atreya (Mar 30 2021 at 04:19):Thanks, will look into it
Vikram Atreya (Mar 30 2021 at 10:46):@Sean My proposal is partially done, should I still go ahead and upload a draft in the GSoC website?
Vikram Atreya (Mar 30 2021 at 10:56):Vikram Atreya said:
I got one more doubt as I was researching about how transactions are implemented in different systems, I found this in the journal/ledger based file system wikipedia page, The statement was made in context to deleting a file in linux and power failure during the process. "After a crash, recovery simply involves reading the journal from the file system and replaying changes from this journal until the file system is consistent again." , So i found it too simple that will just replaying instructions of the prev cmd stop the db from being corrupted.
Ref:https://en.wikipedia.org/wiki/Journaling_file_system#Rationale
ping?
Sean (Mar 30 2021 at 16:54):pong?
Sean (Mar 30 2021 at 16:54):Vikram Atreya said:
Sean My proposal is partially done, should I still go ahead and upload a draft in the GSoC website?
Sure, upload it!
Sean (Mar 30 2021 at 16:58):Vikram Atreya said:
ping?
There wasn't a question in there, so I'm not sure what you're seeking input on. Did you have a question about journaling in general? There's some safety checks that are needed, but that is the gist of implementing safe undo. Assume a command is going to crash mid-work. The journal will say something like "modify A; create B; delete C;" and we can verify at any point whether A was modified, B was created, and C was deleted. If they weren't we can do them again. If they were, we just need to update the journal.
Vikram Atreya (Mar 30 2021 at 17:00):Question was like this, if we were doing "modify A" and system crashed while the process was running
Vikram Atreya (Mar 30 2021 at 17:01):which might corrupt the db
So will rerunning "modify A" rectify the db?
Vikram Atreya (Mar 30 2021 at 17:18):Vikram Atreya said:
Question was like this, if we were doing "modify A" and system crashed while the process was running
I forgot to ask it in the prev msg :grinning_face_with_smiling_eyes:
Vikram Atreya (Mar 30 2021 at 18:53):I have shared a draft of my proposal.
Its 70% done I would say
Sean (Mar 30 2021 at 19:25):Vikram Atreya said:
which might corrupt the db
So will rerunning "modify A" rectify the db?
In general, it depends on the lowest level write operations and how they're implemented. That said, librt does go to lengths to minimize potential situations. Should a write be interrupted, the effect should only affect the partially written object, not corrupt the entire file.
Sean (Mar 30 2021 at 19:26):so the fix would be to write over the partially written object again, picking up where it left off
Vikram Atreya (Mar 30 2021 at 19:27):But will that be possible to note down in the ledger, as that will be generating a very big file imo?
Vikram Atreya (Mar 30 2021 at 19:28):that = different checkpoints in an event, so that we can know where we left it
Sean (Mar 30 2021 at 19:28):This is not something exhaustively tested, but in practice has been very robust to production use and crashes. The database i/o layer first finds a section of the file that will fit the object being written/updated. Then it fully prepares the object being written in memory first, then it writes it out as 1 write operation, then is fflushes the file to make the operating system ensure everything is written to disk.
Sean (Mar 30 2021 at 19:28):the only thing in the ledger is what's currently changing
Sean (Mar 30 2021 at 19:30):it's not a full history, it's just what's happening right now that hasn't been applied yet. the actual undo data is in a separate space (e.g., another .g in memory)
Vikram Atreya (Mar 30 2021 at 19:31):Yeah, I wasnt talking abt that either
I was talking about if we could/would note down all minor steps being done under 1 event, but yeah now im clear that we will just store the thing thats being modified
Sean (Mar 30 2021 at 19:33):I think you'd note down all the events in a given transaction, then do them, then make sure they were all done.
Sean (Mar 30 2021 at 19:34):so not necessarily just what's being modified, but also what's being created or killed
Sean (Mar 30 2021 at 19:37):Sean said:
it's not a full history, it's just what's happening right now that hasn't been applied yet. the actual undo data is in a separate space (e.g., another .g in memory)
I guess I should clarify, because we're mixing journaling with undo... in that case the journal/ledger will either have a full history or you'll keep an integrity journal and separately keep an undo ledger.
Sean (Mar 30 2021 at 19:40):because there's two things we're talking about -- one is ensuring the .g is consistent -- that a command was either fully, partially, or not completed. for that, a journal of the current transaction is all that's required. You write what you're going to do, you do it to the .g, then you remove it from the journal. So if at any point the journal is non-empty, we know something is either in-progress or was interrupted/crashed/aborted/etc.
Sean (Mar 30 2021 at 19:44):in order to undo transactions and commands, we need a ledger of what changes were made, which is a little different. for that, a simple convention in a working space with copies of the data in its previous state should be enough.
Vikram Atreya (Mar 30 2021 at 20:00):Got it
Vikram Atreya (Mar 30 2021 at 20:16):Vikram Atreya said:
I have shared a draft of my proposal.
Its 70% done I would say
Do review if you find time
Sutirtha Bhattacharyya (Mar 31 2021 at 12:53):Hi, I have some experience with C/C++. Is it too late to work on the annotation support project?
Sean (Mar 31 2021 at 14:56):@Sutirtha Bhattacharyya it's only too late after the deadline passes. The challenge for you will be putting in adequate time and effort to demonstrate your coding ability. You need to submit a productively useful change to BRL-CAD's code. That typically can be done in less than a day, but it depends on your ability to read existing code and ask productive questions (i.e., don't ask questions that are easily discovered on your own, ask questions that help you understand and be productive).
Sutirtha Bhattacharyya (Mar 31 2021 at 15:06):Okay. I will try my best!
Vikram Atreya (Mar 31 2021 at 19:31):@Sean In the simple main.c that I'm writing should I write the backup objects in another file or putting it in the same db works?
Vikram Atreya (Mar 31 2021 at 19:35):Also shud I create a separate topic for my project idea (undo + transactions), so that this topic remains for people who want to introduce themselves
Sean (Mar 31 2021 at 20:15):When we do it for real, we'll probably want to put them in a separate space, a separate in-mem database. For this, though, I think the same db should give a close enough indication of the implications and performance.
Sean (Mar 31 2021 at 20:19):Vikram Atreya said:
Also shud I create a separate topic for my project idea (undo + transactions), so that this topic remains for people who want to introduce themselves
The title should be what you want it to be, what you think makes the most sense, and shoudl include what you want to do first and foremost -- it's your proposal. From there, we can discuss changes if needed. Our project ideas are just that -- ideas. They're not a menu of options, just starting points for others to create proposals.
Vikram Atreya (Mar 31 2021 at 20:23)::oh_no: Ummmm..... by topic in my previous msg I had meant topic (thread) in zulip
Vikram Atreya (Mar 31 2021 at 20:27):I felt that many people discussing things parallelly in the same thread/topic might get clumsy
So I was talking about creating a new thread/topic in zulip, so that people could introduce themselves in the "GSoC ideas" topic and discussion specific to my proposal could be in a separate newly created topic
Vikram Atreya (Apr 01 2021 at 05:27):I have written a main.c, similar to test_tops.c, but when I compile and run it, it doesnt work
I tried printing dbp->ged_result_str, It prints unknown command: make
Vikram Atreya (Apr 01 2021 at 05:28):I tried to run the exectuable of test_tops.c, even that prints unknown command: tops
Sean (Apr 01 2021 at 05:30):@Vikram Atreya how are you compiling and running? I suspect it has to do with the dynamic libged plugin loading
Vikram Atreya (Apr 01 2021 at 05:30):I wrote the file in libged/tests
Vikram Atreya (Apr 01 2021 at 05:30):added a line in the Cmakelists.txt
Sean (Apr 01 2021 at 05:31):k, that's good -- and running it?
Vikram Atreya (Apr 01 2021 at 05:31):Vikram Atreya said:
added a line in the Cmakelists.txt
BRLCAD_ADDEXEC(ged_test_undo test_undo.c libged TEST)
Vikram Atreya (Apr 01 2021 at 05:31):Sean said:
k, that's good -- and running it?
Ya, its running
Vikram Atreya (Apr 01 2021 at 05:31):It prints all the bu_log() statements
Vikram Atreya (Apr 01 2021 at 05:31):Doesnt throw any error either
Vikram Atreya (Apr 01 2021 at 05:32):Just that the ged_make or ged_kill isnt working and the file Im modifying remains the same
Sean (Apr 01 2021 at 05:32):No, I mean how are you running it.
Sean (Apr 01 2021 at 05:33):what did you type to run your program
Vikram Atreya (Apr 01 2021 at 05:34):./test_undo empty.g
Sean (Apr 01 2021 at 05:34):that's why
Sean (Apr 01 2021 at 05:34):check this:
morrison@agua .build % ./src/libged/tests/ged_test_tops share/db/moss.g
all.g
morrison@agua .build % cd src/libged
morrison@agua libged % ./tests/ged_test_tops ../../share/db/moss.g
unknown command: tops
it loads the plugins dynamically and since you are running from an uninstalled state, it doesn't know where to find them unless you're at the top of the build directory.
Sean (Apr 01 2021 at 05:35):something that can be fixed, but is a current limitation
Vikram Atreya (Apr 01 2021 at 05:37):Cool, Thank you
Vikram Atreya (Apr 01 2021 at 17:24):I have implemented the create and delete and then backup (by hardcoding it for now), got the basic libgit2 cmds running within the code, learning the libgit2 API to implement the undo using VCS
Vikram Atreya (Apr 03 2021 at 18:27):I'm a little busy with clg work and also I'm encountering a lot of errors in the libgit2 part
Will try to have a working version asap
Vikram Atreya (Apr 03 2021 at 18:28):If possible can any of the mentor(s) review the draft of my proposal and suggest changes or additions, so that I will have time to work on them before i make the final draft
Vikram Atreya (Apr 04 2021 at 07:16):I have successfully made 2 files: one which is a hardcoded implementation of the "partial + events " method and the other is an implementation which uses libgit2 to revert back to the undone state
Vikram Atreya (Apr 04 2021 at 07:21):I will create a patch on this in sometime
Vikram Atreya (Apr 04 2021 at 10:41):Created the patch, it has 2 files in libged/tests/ . They work fine except that ged_summary doesn't give the right output after reverting back using libgit2. The killed object does get recovered when I open with mged and check. but ged_summary doesn't reflect that there is an object while running the code
Vikram Atreya (Apr 04 2021 at 10:41):@Sean any ideas on how I can proceed further?
Sean (Apr 05 2021 at 19:34):@Vikram Atreya You've done great thus far. I'll update on the patches soon, but feel free to see if you can figure out what's going on with killed objects. Note that depending on how you add/delete objects, you will need to rebuild the geometry directory. That's especially true for altering the .g from outside librt, like you're doing with libgit2 (that's one of the downsides, have to fully rebuild because you don't know what changed).
Vikram Atreya (Apr 05 2021 at 19:40):Yeah, dint know there was a rebuild
I was thinking of closing and reopining the db and then doing ged_summary
But this week has been quite loaded in terms of academics so couldn't work much
Vikram Atreya (Apr 05 2021 at 19:41):@Sean I would really appreciate it if you just skimmed through my proposal and tell me what all I could/should add to my proposal
Vikram Atreya (Apr 05 2021 at 19:42):Rn I don't have any code snippets/pseudo code in the proposal, it is necessary to put them or only the theory is sufficient?
Sean (Apr 06 2021 at 05:33):@Vikram Atreya I'm done commenting. Your draft is very good. The biggest critique I have is that the timeline is very aggressive, maybe unrealistic. There's many things that could end up taking a week or two to figure out, and priority should be on completing undo/redo robustly and (adequately) performant first and foremost (my opinion) before, for example, tackling transactions or anything else really. That's really the essential feature from a user perspective that it may be best to just focus on it.
Sean (Apr 06 2021 at 05:35):I think your analysis of all the various options is good. Your patches are on-point as well. A couple decisions would need to be made before GSoC begins so there's less/no discovery in the shortened timeframe.
Sean (Apr 06 2021 at 05:37):Vikram Atreya said:
Rn I don't have any code snippets/pseudo code in the proposal, it is necessary to put them or only the theory is sufficient?
The theory and links are sufficient. It would be good to expand on the recommended plan and reasons for it. From my perspective, I think the main decision needed is whether to go with the libgit2 approach or not and having a good rationale for why (or why not), what the implication trade-off considerations were, etc.
Sean (Apr 06 2021 at 05:45):If you eliminate transactions from the plan and focus entirely on undo, it would be good to see an updated timeline. Keep in mind a potential new undo command (perhaps with a -r redo option), documentation (e.g., doc/docbook/system/mann/undo.xml and potentially other docs), behavioral/usability testing (important commands to consider include facetize, search -exec, dbconcat, dbupgrade, edit, and clone commands), and performance/robustness testing.
Vikram Atreya (Apr 06 2021 at 08:17):Sure, I will make an updated time line only for UNDO
Vikram Atreya (Apr 06 2021 at 21:12):Sean said:
From my perspective, I think the main decision needed is whether to go with the libgit2 approach or not and having a good rationale for why (or why not), what the implication trade-off considerations were, etc.
So I'll try to compare time and space complexity for both methods using some heavy database
Vikram Atreya (Apr 07 2021 at 18:36):Is the havoc.g (589 kB) big enough to use for comparing both methods for deletion and re-addition(undo)?
Or should I generate something bigger?
Amanjot Singh (Apr 08 2021 at 13:31):Greetings, I'm Amanjot Singh from India. I've been practicing front-end web development for 3 years. I'm a Btech student in 3rd-year Computer Science and also completed a 3-year diploma in Computer Science and Engineering. I also have 9 months of in-field job experience with the role of a frontend developer. I'm interested to contribute to open source during GSoC this year. I hope I'll get a cool and awesome experience here :)
Daniel Rossberg (Apr 08 2021 at 15:02):You could try this one: https://brlcad.org/~sean/tmp/hairball.g
https://brlcad.zulipchat.com/#narrow/stream/104062-general/topic/guidance.20for.20the.20first.20contribution/near/214412419
Daniel Rossberg (Apr 08 2021 at 15:04):Hi @Amanjot Singh, which project ideas sound most interesting for you?
Amanjot Singh (Apr 08 2021 at 16:07):Daniel Rossberg said:
Hi Amanjot Singh, which project ideas sound most interesting for you?
Hey, I've found the task Online Geometry Viewer most interesting and best suitable for me.
Vikram Atreya (Apr 08 2021 at 19:21):Vikram Atreya said:
So I'll try to compare time and space complexity for both methods using some heavy database
I used hairball.g (139MB), tried to delete the whole object and then recover it using both methods:
Vikram Atreya (Apr 08 2021 at 19:21):Using the partial+events method, where I stored a recovery copy of the object and used that to recover the deleted object, the file was 277MB when the recovery object was present
Vikram Atreya (Apr 08 2021 at 19:22):In the method using libgit2, the .git folder was 127.3MB and add that to the 139MB of the database, it becomes 266.3MB
Vikram Atreya (Apr 08 2021 at 19:29):Here the libgit2 approach saves 11MB which is about 8% of space compared to the initial database, which is pretty significant in my opinion
Vikram Atreya (Apr 08 2021 at 19:45):I tried to rename the hairball object using mv cmd and used libgit2 to bring it back to the original state, in this case the .git folder occupies 63.7 MB
starseeker (Apr 09 2021 at 00:31):/me idly wonders what git would do if we b64 encoded the .g files, the way we do for 3dm breps when we walk them through g2asc/asc2g - wonder if something that is nominally a text file might be easier for git to intelligently diff...
Amanjot Singh (Apr 09 2021 at 11:19):Hey Mentors and fellow participants. I want to clear a doubt. Is there any specific template for the proposal to apply in BRL-CAD or we should write it completely in our own way?
Daniel Rossberg (Apr 09 2021 at 14:09):There is no template, but guidelines. Have a look at our general GSoC page and the CHECKLIST too.
Amanjot Singh (Apr 09 2021 at 14:23):Thanks Daniel
Vikram Atreya (Apr 09 2021 at 17:50):Hey @starseeker or @Sean
Could you tell me what deliverables I could mention in my proposal?
I was considering the following as checkpoints:
Vikram Atreya (Apr 10 2021 at 08:52):As @Sean had mentioned that the timeline is too aggressive, should I still have the implementation of transactions as part of my GSoC proposal, as I might not have sufficient time to fully implement it
Amanjot Singh (Apr 11 2021 at 05:05):Hey @Shubham Rathore @Panda (Gauravjeet Singh) I was thinking of sending some patches before I draft my proposal. I looked at this issue https://github.com/BRL-CAD/OGV-meteor/issues/63 that says there is no message if upload fails for some reason. I tried uploading a png file so that I can see if it gives an error or not and it gave me an error. Can you please explain a little bit for what reason it may give an error so that I can reproduce the issue and try fixing that.
Amanjot Singh (Apr 11 2021 at 05:06):One more question, this is one of the checklist tasks provided for GSoC 2021. Am I I allowed sending fixes for the issues that are in the checklist?
Sean (Apr 12 2021 at 05:35):Vikram Atreya said:
Using the partial+events method, where I stored a recovery copy of the object and used that to recover the deleted object, the file was 277MB when the recovery object was present
Just a word of caution here -- .g files don't typically reclaim killed space until new objects are added. That means there is some dead some, i.e., phantom size, represented there that is an optimization but not actual utilization. It's a slow filesystem operation to shrink a file, so it doesn't do that unless explicitly requested.
Sean (Apr 12 2021 at 05:39):Vikram Atreya said:
Here the libgit2 approach saves 11MB which is about 8% of space compared to the initial database, which is pretty significant in my opinion
Even if the partial approach retained two copies (e.g., the edit case), that size difference is pretty negligible. Time is the more interesting factor -- how long did the two methods take? How does it scale with 10 hairballs being edited? that's likely a usability-impacting factor.
Sean (Apr 12 2021 at 05:48):starseeker said:
/me idly wonders what git would do if we b64 encoded the .g files, the way we do for 3dm breps when we walk them through g2asc/asc2g - wonder if something that is nominally a text file might be easier for git to intelligently diff...
The main/only benefit of using something like libgit2 would be agnostic simplicity and speed of implementation. If we have to layer in any complexity, that would pretty much eliminates the feature point in my view. Keeping in mind that this is for an undo/redo system too. It needs to be as simple as possible, reliable and performant first and foremost. Memory/space utilization is not a deciding feature as it's all temp state.
Sean (Apr 12 2021 at 05:54):Vikram Atreya said:
Hey starseeker or Sean
Could you tell me what deliverables I could mention in my proposal?
I was considering the following as checkpoints:
- A dummy UNDO command which just prints a line
- Storing a backup file/adding backup objects after every change
- Linking the backup to the UNDO command and basic testing
Do these look fine as deliverables, if not could you give me some direction
They look fine. It's more important that you just set checkpoints than what the checkpoints are. Personally, I would make them user-centric like a completely designed undo command (from a usage perspective) where all behavior is defined and documented, but nothing implemented. Then single undo/redo fully implemented and demonstrated via command. Then fully integrated into GUI (mged and/or archer) and display manager keybindings. Then multi-undo/redo fully implemented/demonstrated/integrated. Something like that.
Sean (Apr 12 2021 at 05:58):Amanjot Singh said:
One more question, this is one of the checklist tasks provided for GSoC 2021. Am I I allowed sending fixes for the issues that are in the checklist?
Of course. You can submit any fix/improvement for any issue (in any code/language/repo for BRL-CAD). It helps if it's related to your proposal, but not necessary. For OGV, it relies heavily on backend processing by other BRL-CAD tools, so they are heavily related too.
Vikram Atreya (Apr 12 2021 at 06:14):Sean said:
Just a word of caution here -- .g files don't typically reclaim killed space until new objects are added. That means there is some dead some, i.e., phantom size, represented there that is an optimization but not actual utilization. It's a slow filesystem operation to shrink a file, so it doesn't do that unless explicitly requested.
I deleted the objected and then recovered it back again, so will that still have phantom size
because I added back the same object (which will have the same size as the deleted object)
Vikram Atreya (Apr 12 2021 at 06:15):Sean said:
Even if the partial approach retained two copies (e.g., the edit case), that size difference is pretty negligible. Time is the more interesting factor -- how long did the two methods take? How does it scale with 10 hairballs being edited? that's likely a usability-impacting factor.
Haven't checked the time factor yet, let me check right now
Vikram Atreya (Apr 12 2021 at 06:17):Sean said:
They look fine. It's more important that you just set checkpoints than what the checkpoints are. Personally, I would make them user-centric like a completely designed undo command (from a usage perspective) where all behavior is defined and documented, but nothing implemented. Then single undo/redo fully implemented and demonstrated via command. Then fully integrated into GUI (mged and/or archer) and display manager keybindings. Then multi-undo/redo fully implemented/demonstrated/integrated. Something like that.
Sure, will update my proposal by tonight keeping these n mind
Vikram Atreya (Apr 12 2021 at 06:47):Vikram Atreya said:
As Sean had mentioned that the timeline is too aggressive, should I still have the implementation of transactions as part of my GSoC proposal, as I might not have sufficient time to fully implement it
So should I completely shift focus to UNDO and remove the transactions part from my proposal as I might not find time to implement it?
Vikram Atreya (Apr 12 2021 at 07:27):Time taken for following in libgit2 method: (haven't divided by CLOCK_CYCLES)
Intialize repo: 4,524,677
open db and make object: 193,198
commit: 4,446,718
delete: 81
Revert back to prev commit: 4,450,182
Vikram Atreya (Apr 12 2021 at 07:29):Time taken for following in partial :
Open and add: 193,797
Kill and add recovery ibject: 187,787
Recover: 195,973
Vikram Atreya (Apr 12 2021 at 07:29):These times are for a single hairball
Amanjot Singh (Apr 12 2021 at 11:06):Sean said:
Amanjot Singh said:
One more question, this is one of the checklist tasks provided for GSoC 2021. Am I I allowed sending fixes for the issues that are in the checklist?
Of course. You can submit any fix/improvement for any issue (in any code/language/repo for BRL-CAD). It helps if it's related to your proposal, but not necessary. For OGV, it relies heavily on backend processing by other BRL-CAD tools, so they are heavily related too.
Thanks for clearing out :raised_hands:
Sean (Apr 12 2021 at 13:52):Vikram Atreya said:
Sean said:
I deleted the objected and then recovered it back again, so will that still have phantom size
because I added back the same object (which will have the same size as the deleted object)
What I would expect to happen is 100 MB object in a file.g gets modified/deleted (i.e., moved to a undo_cache.g), so file.g is at 0MB+100MB (data+padding) for a total of 100MB file size. The undo_cache.g is a 100MB+0MB=100MB file. Then you undo.
In that case, file.g either changes to 100MB+0MB (data+padding) or doubles to 100MB+100MB (data+padding)=200MB file size depending on the nature of the edit, and the undo_cache.g is 0MB+100MB (data+padding) in size.
Sean (Apr 12 2021 at 13:56):Vikram Atreya said:
So should I completely shift focus to UNDO and remove the transactions part from my proposal as I might not find time to implement it?
I think so. I mean it's still part of undo in a way, like if someone runs the "clone" command, I expect undo to completely revert the clone, not just the last object copied, but transactions aren't strictly necessary -- it's just one way that can be achieved.
Sean (Apr 12 2021 at 13:58):Vikram Atreya said:
These times are for a single hairball
Now that's very informative. What are those rates in MB/sec ?
Vikram Atreya (Apr 12 2021 at 15:55):Sean said:
I think so. I mean it's still part of undo in a way, like if someone runs the "clone" command, I expect undo to completely revert the clone, not just the last object copied, but transactions aren't strictly necessary -- it's just one way that can be achieved.
Regarding this, if I go with the libgit2 approach that will be completely different from transactions, right?
Vikram Atreya (Apr 12 2021 at 16:10):Sean said:
Now that's very informative. What are those rates in MB/sec ?
These are in clock ticks ( values returned by clock() in C)
Amanjot Singh (Apr 12 2021 at 16:23):Hey @Shubham Rathore @Panda (Gauravjeet Singh) @Harmanpreet Singh I've drafted the proposal to the GSoC website. Can you please check? I'll send the final proposal tomorrow 2PM IST.
Vikram Atreya (Apr 12 2021 at 18:49):Sean said:
I think so. I mean it's still part of undo in a way, like if someone runs the "clone" command, I expect undo to completely revert the clone, not just the last object copied, but transactions aren't strictly necessary -- it's just one way that can be achieved.
I'll put the theory that I have for the transactions at the end as part of an appendix
Vikram Atreya (Apr 12 2021 at 18:57):Vikram Atreya said:
Time taken for following in partial :
Open and add: 193,797
Kill and add recovery ibject: 187,787
Recover: 195,973
Right now the recovery object is made in the same file, even if it is made in a separate file and then accessed I don't think there will be a huge change in time
Vikram Atreya (Apr 12 2021 at 19:47):@Sean any changes I might need to do to the proposal based on the above results or any more tests to do before proposal submission?
Vikram Atreya (Apr 12 2021 at 20:58):I have submitted a final draft as of now
I have changed the timeline, made changes based on your previous comments, added a comparison between both methods in terms of space (as of now, will add the time comparisons ), and added basic implementation details in the appendix
Vikram Atreya (Apr 12 2021 at 20:58):Please review if you find time
Sean (Apr 13 2021 at 03:14):Vikram Atreya said:
Regarding this, if I go with the libgit2 approach that will be completely different from transactions, right?
Depends what you mean. It's SUPER SIMPLE to implement transactions using the libgit2 approach because you can simply note the start and check-in at the end of each transaction. So undoing a transaction is simply rolling back to a previous commit. Even if you have recursive transactions within transactions, it just works because it keeps track the full state change automatically.
Sean (Apr 13 2021 at 03:14):Amanjot Singh said:
Hey Shubham Rathore Panda (Gauravjeet Singh) Harmanpreet Singh I've drafted the proposal to the GSoC website. Can you please check? I'll send the final proposal tomorrow 2PM IST.
Excellent @Amanjot Singh. Make sure you don't wait until the last couple hours. Submit early if you can.
Sean (Apr 13 2021 at 03:15):Vikram Atreya said:
Sean said:
Now that's very informative. What are those rates in MB/sec ?
These are in clock ticks ( values returned by
clock()in C)
I realized that -- was asking you to do the math ;) also, to look at rate, not just time since the dominant factor right now is almost certainly I/O and the quantity of data being read/written to disk.
Sean (Apr 13 2021 at 03:17):Vikram Atreya said:
Right now the recovery object is made in the same file, even if it is made in a separate file and then accessed I don't think there will be a huge change in time
I agree -- even more important, though, is that we have an in-memory-only database concept that is relevant here as it would avoid the disk entirely. There I suspect would be the fastest option by quite a huge amount.
Sean (Apr 13 2021 at 03:18):Vikram Atreya said:
Please review if you find time
At this point, we're in details that are subject to change. For application purposes, you've done well.
Vikram Atreya (Apr 13 2021 at 03:28):Sean said:
I realized that -- was asking you to do the math ;) also, to look at rate, not just time since the dominant factor right now is almost certainly I/O and the quantity of data being read/written to disk.
Is there some function to get the rate, or should i do it manually?
Vikram Atreya (Apr 13 2021 at 03:29):Because you had mentioned about padding size, I might not be able to calculate exactly how much memory is being written after each change
Sean (Apr 13 2021 at 03:34):No you just calculate it.
Sean (Apr 13 2021 at 03:41):I mean just looking quickly, if we disregard open and add actvity, it took approx 188+196=384k clock cycles to do+undo which is conveniently 384ms or 0.384 sec. I think you said hairball is nominally 139MB of data, so the do+undo rate is 362MB/sec.
Sean (Apr 13 2021 at 03:43):The libgit2 method took 8.896 sec, so it's effective rate was 15.6MB/sec
Vikram Atreya (Apr 13 2021 at 03:45):Right.
Sean (Apr 13 2021 at 03:45):which at a glance, doesn't seem right .. that's a LOT of time for a file-based activity. you sure there's not a mistake in there?
Sean (Apr 13 2021 at 03:45):basically 4 sec to check-in
Vikram Atreya (Apr 13 2021 at 03:45):Yeah does seem like a lot of time
Sean (Apr 13 2021 at 03:47):I would expect it to be more, but that's 23x more. I would have thought 2-10x more likely.
Vikram Atreya (Apr 13 2021 at 03:48):Yes, I also realized that the git method was taking too much time
Sean (Apr 13 2021 at 03:49):Just as I would expect in-mem to take 0.15-0.04 sec or 2x-10x faster than that 368MB/sec rate.
Vikram Atreya (Apr 13 2021 at 03:49):Maybe theres some redundant code in the git method, like I'm creating a tree for every commit (which I wasn't very sure on if it was required or not)
Sean (Apr 13 2021 at 03:50):Well that can be something to figure out later/next. Will have to look more closely at your harness code that's pending review.
Sean (Apr 13 2021 at 03:50):Since that's really the deciding factor of whehter libgit2's overhead slowness is worth it to get transactions for free.
Sean (Apr 13 2021 at 03:51):8 sec would be a non-starter if that's real, but I suspect it's not.
Sean (Apr 13 2021 at 03:53):if it went from .3s to .5s or similar, then libgit2 might be compelling, though it's more likely going to be competing with 0.1s vs 0.5s
Vikram Atreya (Apr 13 2021 at 04:22):So I have submitted my final draft again with a few minor changes
And I don't plan on making any further changes
Amanjot Singh (Apr 13 2021 at 04:55):Sean said:
Amanjot Singh said:
Hey Shubham Rathore Panda (Gauravjeet Singh) Harmanpreet Singh I've drafted the proposal to the GSoC website. Can you please check? I'll send the final proposal tomorrow 2PM IST.
Excellent Amanjot Singh. Make sure you don't wait until the last couple hours. Submit early if you can.
Okay @Sean I don't have a plan to make any change I can submit this right now as well. Do you think I should make any changes?
shubham shah (Apr 13 2021 at 06:18):hey @Sean . i went through project ideas and last year projects and i wanted to contribute to appleseed project. i cloned it from github and getting to read the sourcecode. but i guess i am a bit late and should submit my draft first. how do you propose i make my timeline?
Amanjot Singh (Apr 13 2021 at 07:54):Hey @Sean I've submitted my final proposal. I'm also going to send some patches to the application today.
Sean (Apr 13 2021 at 14:02):shubham shah said:
hey Sean . i went through project ideas and last year projects and i wanted to contribute to appleseed project. i cloned it from github and getting to read the sourcecode. but i guess i am a bit late and should submit my draft first. how do you propose i make my timeline?
That's great, look forward to reading your proposal as that's a very important project for us. The deadline is four hours away, though, so best of luck to you. You'll not only need to write and submit your proposal by the deadline, but you will need to submit some code improvement (a patch or pull request) before the end of the week as well.
shubham shah (Apr 13 2021 at 14:02):@sean i went through tasks of appleseed project, i dont understand much about multithreading support and not able to figure out my timeline around that
Sean (Apr 13 2021 at 14:05):If you don't understand multithreading, then you probably shouldn't be discussing it in your proposal. There's not much time during this year's GSoC to learn and explore. That needed to happen the past 2-3 months.
Sean (Apr 13 2021 at 14:05):It'll be enough work to propose improvements and projects using techniques and technology you do understand.
Vikram Atreya (Apr 14 2021 at 04:37):I have my end-semester exams starting soon, So won't be able to contribute much till they end on May 2. I will regularly check zulip though and if there is anything to be done before the community bonding period I will work on it from May 2 -17.
Vikram Atreya (Apr 14 2021 at 04:38):Also, It has been a wonderful journey till now in BRL-CAD and I would like to thank the whole community for guiding me from the beginning and being so welcoming. Hope to get selected for GSoC '21 and contributing a lot more in the future.
Amanjot Singh (Apr 16 2021 at 14:22):Hey there @Sean, I was preparing for my exams. Would that be okay if I send the fix by the end of Sunday?
Sean (Apr 16 2021 at 15:24):Amanjot Singh said:
Hey there Sean, I was preparing for my exams. Would that be okay if I send the fix by the end of Sunday?
That's certainly your choice and leaves no time for interaction, no time to help you navigate the code, but yes -- that is technically okay.
Sean (Apr 16 2021 at 15:28):Vikram Atreya said:
I have my end-semester exams starting soon, So won't be able to contribute much till they end on May 2. I will regularly check zulip though and if there is anything to be done before the community bonding period I will work on it from May 2 -17.
Thanks for the update @Vikram Atreya not a problem.
Amanjot Singh (Apr 17 2021 at 06:17):Sean said:
Amanjot Singh said:
Hey there Sean, I was preparing for my exams. Would that be okay if I send the fix by the end of Sunday?
That's certainly your choice and leaves no time for interaction, no time to help you navigate the code, but yes -- that is technically okay.
Thanks for the reply @Sean . I've sent a code improvement and surely will send more this weekend. I hope I'm not bothering you with the updates. Have a great day :blush:
Amanjot Singh (Apr 17 2021 at 06:21):https://github.com/BRL-CAD/OGV-meteor/pull/101 here is the link of PR in case required
scorp08 (Apr 17 2021 at 10:26):Sean said:
That's a good thing to check - Vikram Atreya see what performance looks like with rt -P1 for ocl vs non. That requests a single CPU core. Also notice in that non-opencl profile rt_boolweave() and rt_boolfinal() ... that's the two functions mentioned earlier.
I was wondering if I build librt and other dlls with openCL enabled, after call rt_shootray with ocl supported primitives does it make difference vs non ocl dlls ?
Sean (Apr 19 2021 at 16:12):@scorp08 Yes, it can make a huge difference, but that's definitely a work in progress. Some primitives are 10-20x faster, some are about the same, some aren't implemented. There's a report on the wiki that talks about the performance implications.
One test case was havoc, if I recall correctly, that went from taking about 1-10 seconds to shoot 1M rays to taking less than a second. It didn't have boolean weaving yet, though, when that study was conducted, so that's easily 25% of the time.
scorp08 (Apr 21 2021 at 19:55):Sean said:
scorp08 Yes, it can make a huge difference, but that's definitely a work in progress. Some primitives are 10-20x faster, some are about the same, some aren't implemented. There's a report on the wiki that talks about the performance implications.
One test case was havoc, if I recall correctly, that went from taking about 1-10 seconds to shoot 1M rays to taking less than a second. It didn't have boolean weaving yet, though, when that study was conducted, so that's easily 25% of the time.
@Sean oh great, the working primitives are the ones in common.cl?
Sean (Apr 23 2021 at 03:55):scorp08 said:
Sean oh great, the working primitives are the ones in common.cl?
Yes, I believe so. There may be patches for a couple more pending review too.
scorp08 (Apr 23 2021 at 05:20):Sean said:
scorp08 said:
Sean oh great, the working primitives are the ones in common.cl?
Yes, I believe so. There may be patches for a couple more pending review too.
@Sean I installed intel sdk , env variables are set but cmake-gui did not show any flag about openCL?
scorp08 (Apr 23 2021 at 05:55):(deleted)
scorp08 (Apr 23 2021 at 05:56):disabled mark_as_advanced and done , cmake seems sometimes very complicated to me :joy:
scorp08 (Apr 23 2021 at 09:10):@Sean I failed to run rt with opencl. It throws nothing(works without opencl). Which projects do I need to compile for enable opencl ?
Vikram Atreya (Apr 23 2021 at 09:12):I guess you are on windows, in linux you are supposed to add the -DBRLCAD_ENABLE_OPENCL=ON if that helps
Vikram Atreya (Apr 23 2021 at 09:12):When you run the Cmake command it shows whether OpenCL is found or not
Vikram Atreya (Apr 23 2021 at 09:13):If it doesnt find OpenCL and you try running rt -z, it just works like the normal rt (without any involvement of OpenCL )
scorp08 (Apr 23 2021 at 10:17):Vikram Atreya said:
I guess you are on windows, in linux you are supposed to add the
-DBRLCAD_ENABLE_OPENCL=ONif that helps
I already generated projects with opencl enabled but with rt -z 1 it print out "compiling opencl programs" and no nothing. I think I need to compile some projects in brlcad.sln but I do not know which (compiled all shared libs)
Sean (Apr 23 2021 at 17:18):Compile the ALL_BUILD target... that should be everything.
scorp08 (Apr 24 2021 at 04:48):Sean said:
Compile the ALL_BUILD target... that should be everything.
@Sean no luck, still "compiling opencl programs" and doing nothing and finish.
scorp08 (Apr 24 2021 at 05:19):scorp08 said:
Sean said:
Compile the ALL_BUILD target... that should be everything.
Sean no luck, still "compiling opencl programs" and doing nothing and finish.
It is "__stdio_common_vsprintf " from stdio.h - access violation. I had compiled some libraries with different SDK version because of Tcl/tk latest supported version.
Sean (Apr 27 2021 at 04:12):You're definitely in advanced territory there @scorp08 .. I think you need to first confirm that opencl has been enabled. What's the entire output from running cmake? You'll have to copy-paste the log from the cmake transcript / log window into a text file.
scorp08 (Apr 27 2021 at 14:30):Sean said:
You're definitely in advanced territory there scorp08 .. I think you need to first confirm that opencl has been enabled. What's the entire output from running cmake? You'll have to copy-paste the log from the cmake transcript / log window into a text file.
@Sean yeah, USE_OPENCL seems enabled after cmake builds. also selected BRLCAD_ENABLE_OPENCL before generate. I 'll try with different SDK version but could not do with cmake, altough set (CMAKE_SYSTEM_VERSION 10.0 "10.0.15063.0 " CACHE STRING INTERNAL FORCE) and #set(CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION "10.0.15063.0 "). I do not know which libraries are need to be changed from VS. Maybe librt dependencies?
scorp08 (Apr 27 2021 at 19:11):@Sean debugged. It seems that bu_brlcad_root returning null from clt_init() . I think need to define a BRLCAD_ROOT env. variable. After all the messy search of SDK version change within cmake here the BRLCAD_ROOT :joy: . I thought that It already defined after cmake generations but seems did not why?
scorp08 (May 01 2021 at 15:28):scorp08 said:
Sean debugged. It seems that bu_brlcad_root returning null from clt_init() . I think need to define a BRLCAD_ROOT env. variable. After all the messy search of SDK version change within cmake here the BRLCAD_ROOT :joy: . I thought that It already defined after cmake generations but seems did not why?
I solved include app.h to get correct bu_brlcad_root. But clt_get_program returning error(igc64.dll ). I have laptop with 2 graphic cards . What am I missing ?
Vikram Atreya (May 02 2021 at 10:47):Hey @Sean
My exams are done, I have a few project submissions and presentations to give till May 10 but nothing too hectic. I can start working on comparing both methods and ideas on how to implement them
Sean (May 05 2021 at 04:12):@scorp08 what all you're saying isn't making a whole lot of sense to me, lots of missing information. There's a couple things you can try.
Sean (May 05 2021 at 04:20):First thing to check/try if you're seeing odd bu_brlcad_root behavior is to make sure that you compile and install so that resources are where they're expected, i.e., don't try running from the build directory.
Sean (May 05 2021 at 04:21):Assuming that doesn't fix everything, next I'd suggest you delete your cmake cache, and run cmake again -- and then post a log of the entire output. There is a lot of relevant info on whether opencl was detected properly or not in the output. Would also help to post the CMakeOutput.txt and CMakeError.txt log files.
Sean (May 05 2021 at 04:21):Vikram Atreya said:
Hey Sean
My exams are done, I have a few project submissions and presentations to give till May 10 but nothing too hectic. I can start working on comparing both methods and ideas on how to implement them
That sounds excellent. Congratulations on finishing your exams!
Sean (May 05 2021 at 04:22):@scorp08 you should also start a new thread to discuss opencl instead of posting to the gsoc ideas thread. ;)
Vikram Atreya (May 05 2021 at 16:37):Sean said:
Well that can be something to figure out later/next. Will have to look more closely at your harness code that's pending review.
Could you take a look when you are free, so that I can get some leads on how to proceed next
Sean (May 05 2021 at 16:58):@Vikram Atreya absolutely can and will.. the trouble is getting to that free time. I get little pockets, but they're more breaks from handling other urgent deadlines! I've had too much going on for a few months now, need to make some commitment adjustments.
Ajay Pal (Mar 28 2023 at 05:43):I am excited to express my interest in participating in the Google Summer of Code program as a mentee with BRL-CAD. After reviewing your project ideas, I am particularly interested in proposing a project that would greatly benefit many developers.
As we all know, reviewing code is one of the most important parts of the development cycle, and saving time in this process can greatly help developers. With this in mind, I propose to develop a GitHub pull request code reviewer that provides helpful code reviews on the changes made in a pull request. This tool will streamline the code review process, making it faster and more reliable for developers.
I am confident that this project aligns with the goals and objectives of BRL-CAD, and I have relevant experience in [mention your relevant experience] that makes me well-suited for the project. I am excited to apply my skills to this project and contribute to BRL-CAD's mission.
Thank you for your consideration, and I look forward to the opportunity to work with BRL-CAD as a mentee this summer.
Sincerely,
Ajay Pal
Sean (Mar 28 2023 at 06:10):Hello @Ajay Pal and welcome!
You're definitely seem to be considering a different type of project. Why is that project interesting to you? What's the motivation for wanting to develop something new over tools like ReviewBoard, Gerritt, Phabricator, etc.?
Also, you claim this project would align with goals and objectives of BRL-CAD -- how so?
Sean (Mar 28 2023 at 06:10):Also, you didn't "[mention your relevant experience]".
Ajay Pal (Mar 28 2023 at 06:21):Sean said:
Hello Ajay Pal and welcome!
You're definitely seem to be considering a different type of project. Why is that project interesting to you? What's the motivation for wanting to develop something new over tools like ReviewBoard, Gerritt, Phabricator, etc.?
Also, you claim this project would align with goals and objectives of BRL-CAD -- how so?
Hello!
I am Ajay Pal, 4th year (2023 batch) 𝐂𝐨𝐦𝐩𝐮𝐭𝐞𝐫 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 student from Delhi Technological University(𝐃𝐓𝐔) formerly 𝐃𝐂𝐄 (CGPA:9.24).
I have 𝐢𝐧𝐭𝐞𝐫𝐧𝐞𝐝 at 𝐓𝐰𝐢𝐭𝐭𝐞𝐫 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐢𝐨𝐧 𝐈𝐧𝐝𝐢𝐚 𝐏𝐯𝐭. 𝐋𝐭𝐝. in ETL and Workflow Tools (𝐄𝐖𝐓) team and delivered the Airflow Business Metric dashboard to various internal customers, which is needed for maximal adoption of Airflow at Twitter. I have also interned at 91Squarefeet (YC W22) where I have set up services that are needed for development and contributed to the portfolio project i.e rbdash. I have led many teams to 𝐇𝐚𝐜𝐤𝐚𝐭𝐡𝐨𝐧 and have also won 𝐈𝐍𝐍𝐎𝐕𝐀𝐓𝐄𝐍𝐒𝐔𝐓` 21. I have Solved 850+ 𝐃𝐒𝐀 𝐩𝐫𝐨𝐛𝐥𝐞𝐦𝐬( Leetcode + HackerRank + Code Studio).
I am proficient in 𝐩𝐫𝐨𝐛𝐥𝐞𝐦 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 and 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭.
This project is interesting to me because while doing internships and participating in open source community I have found out that a lot of org lack this kind of integration. I can see that it will help a lot of developers in their review cycle as most of the review would be provided by bots according to some preset rules which can be easily modified according to user needs.
The example you have given of review platforms is to make the review process easy in some ways but they don't give a review by themselves.
Hence I believe this can increase the efficiency of any org.
Sean (Mar 28 2023 at 06:34):Honestly you’re being way too generic and vague. You’re making a claim that a tool would be helpful and is so widespread needed, yet don’t really explain what or how it will help (other than saying that it will). I also don’t see anything specific about BRL-CAD in what you are wanting to propose. How exactly will this help our code and community? Not general notions. Please elaborate if you can, otherwise my inclination is this would be better fit in a tooling community (eg gitlab).
Zhejia Hu (Mar 15 2024 at 19:48):Hello. I am computer science student from Delft University of Technology and I am quite interested in the neural tracing project affiliated of GSOC 2024? I have project experiences with both neural network, which is a competition, and projects with top design companies in China. I have got full marks in my Computer Graphics course
Zhejia Hu (Mar 15 2024 at 22:45):If you need more information on me, feel free to private message me and I will send you the CV and I am also happy to discuss with you the contents of the paper.
Sean (Mar 16 2024 at 14:01):Hello @Zhejia Hu and welcome. Sounds like your experience with NNets and CG could be a benefit to this project. You'll need to research it enough to make a compelling proposal as that's obviously not an easy topic. We will definitely need to learn more about you, and see how you work and interact. The CV can go with your proposal submission, but it's only a piece. What's more important is demonstrating your coding skills on our codebase in some meaningful manner. Towards that, we recommend compiling, installing, and running BRL-CAD from a github cloned checkout, and checking over prior work (AMD's paper and our previous work).
Zhejia Hu (Mar 17 2024 at 18:27):Sean said:
Hello Zhejia Hu and welcome. Sounds like your experience with NNets and CG could be a benefit to this project. You'll need to research it enough to make a compelling proposal as that's obviously not an easy topic. We will definitely need to learn more about you, and see how you work and interact. The CV can go with your proposal submission, but it's only a piece. What's more important is demonstrating your coding skills on our codebase in some meaningful manner. Towards that, we recommend compiling, installing, and running BRL-CAD from a github cloned checkout, and checking over prior work (AMD's paper and our previous work).
Thank you very much for the reply. May I reconfirm the deadline of application, submitting the proposal, and especially interacting in some manner with the codebase is the same as the deadline provided on the GSOC website: Tuesday, April 2, 2024 18:00 UTC? If so, then I would begin to prepare and look into the code. Thank you very much for the instruction.
Danni W (Mar 19 2024 at 15:28):Hi There! I'm a computer science student interested in graphics. I have experience in web application development, OpenGL, and GLSL. Also, I used to be a heavy user of CAD, 3D software, etc. I'm really excited about this year's GSoC topics but a bit overwhelmed by their broadness. Would you give me some suggestions on how to start? Thanks!
Daniel Rossberg (Mar 19 2024 at 16:49):Hi @Danni W, welcome!
You should start with installing BRL-CAD and going through the tutorial. The following depends on your interests and where you see your strengths (e.g., which programming languages you like).
Sean (Mar 23 2024 at 21:00):@Zhejia Hu that submission deadline is a FIRM deadline meaning there are absolutely no exceptions no matter the reason. I suggest submitting several days before the deadline if you can, and then make updates. Any pull requests or other code submissions should be made by that time as well, but there is a little flexibility for that if something is prepared and/or submitted afterwards. Just be aware that we will be reviewing applications after the deadline and may rank proposals at any point (and if there's not a code submission at that time, the application will be disqualified).
Sean (Apr 03 2024 at 12:54):For anyone still working on a coding change, there's a bunch of new GCC14 static analyzer checks that you could try going through on the codebase. Could see if you can find/fix any issues by enabling the -fanalyzer check (using latest gcc14):
https://developers.redhat.com/articles/2024/04/03/improvements-static-analysis-gcc-14-compiler#try_it_out_
Sean (Apr 03 2024 at 15:36):Other really easy tasks including adding support for arbitrary units to the g-obj or gcv converters, fixing "killtree /" bug in mged, add openmesh as a gcv plugin, make src/util/terrain.c output a binunif in the .g instead of an external file, convert teapot to brep/opennurbs instead of bspline, function that converts a non-planar arb8 into brep/opennurbs, implement a function that returns top-level objects, make g-obj export colors, add support for importing ply-g point clouds, import xyz files (gcv), etc.
Basically anything conversion-related is always a good starting point since it avoids having to navigate GUI, build, and dependency issues. Any BUGS or TODO or Issue is fair game, though.
Sean (Apr 04 2024 at 23:29):Daniel makes a good point -- it's good if you can align a coding change with your proposal. Barring that, any BRL-CAD coding change is better than no coding change. Sharing code you implemented on your own is not terribly helpful as it's more about demonstrating your ability to communicate, interact, and navigate existing code.
Last updated: Oct 09 2024 at 00:44 UTC