| 00:46.43 | *** join/#brlcad kintel (~kintel@unaffiliated/kintel) | |
| 01:46.34 | *** join/#brlcad scscvntnrezesqze (~armin@dslc-082-083-185-231.pools.arcor-ip.net) | |
| 02:02.07 | starseeker | brlcad: was lmdb (https://github.com/LMDB/lmdb/tree/mdb.master/libraries/liblmdb) one of the key/value storage solutions on your "to-investigate" list? |
| 02:07.28 | *** join/#brlcad gcibot (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 02:15.36 | *** join/#brlcad gcibot (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 02:17.03 | *** join/#brlcad gcibot (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 02:23.17 | *** join/#brlcad gcibot (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 02:24.14 | *** join/#brlcad gcibot (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 02:24.55 | *** join/#brlcad kintel (~kintel@unaffiliated/kintel) | |
| 03:04.52 | *** join/#brlcad gcibot (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 03:56.52 | *** join/#brlcad merzo (~merzo@93-195-113-92.pool.ukrtel.net) | |
| 04:55.51 | *** join/#brlcad kintel (~kintel@unaffiliated/kintel) | |
| 05:37.29 | *** join/#brlcad ickby (~stefan@x5d846fd2.dyn.telefonica.de) | |
| 06:04.07 | *** join/#brlcad ickby_ (~stefan@x5d846fd2.dyn.telefonica.de) | |
| 06:19.54 | *** join/#brlcad ickby_ (~stefan@x5d846fd2.dyn.telefonica.de) | |
| 06:26.38 | *** join/#brlcad amarjeet (~Amarjeet@169.149.141.232) | |
| 06:28.23 | *** join/#brlcad ickby (~stefan@x5d846fd2.dyn.telefonica.de) | |
| 06:33.49 | *** join/#brlcad ickby_ (~stefan@x5d846fd2.dyn.telefonica.de) | |
| 06:37.28 | *** join/#brlcad Aprtm (67cbe8b1@gateway/web/freenode/ip.103.203.232.177) | |
| 07:06.50 | *** join/#brlcad teepee_ (~teepee@unaffiliated/teepee) | |
| 07:08.22 | *** join/#brlcad Caterpillar2 (~caterpill@unaffiliated/caterpillar) | |
| 07:36.40 | *** join/#brlcad Aprtm (67cbe8b1@gateway/web/freenode/ip.103.203.232.177) | |
| 07:37.20 | *** join/#brlcad Caterpillar (~caterpill@unaffiliated/caterpillar) | |
| 07:38.43 | *** join/#brlcad merzo (~merzo@91.217.179.122) | |
| 07:44.57 | *** join/#brlcad gcibot_ (~gcibot@r167-59-14-192.dialup.adsl.anteldata.net.uy) | |
| 08:24.21 | *** join/#brlcad gcibot (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 08:32.55 | *** join/#brlcad gcibot-afk (~gcibot@r167-59-14-192.dialup.adsl.anteldata.net.uy) | |
| 08:32.55 | *** join/#brlcad gcibot-afk (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 08:33.45 | *** join/#brlcad gcibot-afk (~gcibot@r167-59-14-192.dialup.adsl.anteldata.net.uy) | |
| 08:33.45 | *** join/#brlcad gcibot (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 08:36.20 | *** join/#brlcad ignacio (bip@unaffiliated/ignacio) | |
| 08:38.18 | *** join/#brlcad Apra (67cbe8b1@gateway/web/freenode/ip.103.203.232.177) | |
| 08:38.45 | Apra | is this blog ready to be submitted http://openscadtutorial.blogspot.in/2016/12/openscad-program-used-to-make-3-d-models.html |
| 08:39.11 | *** join/#brlcad KimK (~Kim__@2600:8803:7a85:6d00:d893:6bde:784b:7b8f) | |
| 09:44.48 | *** join/#brlcad teepee] (bc5c2133@gateway/web/freenode/ip.188.92.33.51) | |
| 09:56.34 | *** join/#brlcad amarjeet (~Amarjeet@2405:205:4087:aedf:d472:d1a3:9fb6:b30f) | |
| 10:01.34 | *** join/#brlcad Lord_of_Codes (~Lord_of_C@122.163.195.246) | |
| 10:11.46 | *** join/#brlcad caen23 (~caen23@79.112.95.83) | |
| 11:07.08 | *** join/#brlcad Arc__ (67cbe8b1@gateway/web/freenode/ip.103.203.232.177) | |
| 11:45.02 | *** join/#brlcad yorik (~yorik@2804:431:f720:25fd:290:f5ff:fedc:3bb2) | |
| 13:03.48 | *** join/#brlcad amarjeet (~Amarjeet@2405:205:4087:aedf:d472:d1a3:9fb6:b30f) | |
| 14:06.48 | brlcad | caen23: sorry I missed your question from a couple days ago -- yes there is an API or they can scrape, though I doubt they'll be able to scrape without being a mentor |
| 14:08.21 | brlcad | caen23: the CI tasks are all completely open-ended at this point. we have an older jenkins and buildbot both installed, but how to manage them properly is not defined. what might make the most sense is to create an empty repo on github, and have them set everything up there |
| 14:10.34 | brlcad | Stragus: those opencl gci tasks aren't for performance (they don't have the time or experience), it's just to transcode |
| 14:11.23 | brlcad | Stragus: basically they have a time limit of about 3 hours effort, half of which is time spent getting set up, compiled, and submtting their work |
| 14:18.18 | brlcad | starseeker: lmdb looks worth investigating, but I see now my database notes were kept uncommitted: on my short list after a bit of research was unqlite, leveldb, and tarantool |
| 14:20.38 | brlcad | I would add lmdb to that short list |
| 14:25.26 | brlcad | would make a good little github repo project, to set those four up and compare them |
| 14:37.15 | brlcad | oh, rocksdb was also on the list |
| 14:37.38 | brlcad | ahh, and this was why I'd discounted lmdb, but presumably something is wrong in the setup: https://www.influxdata.com/benchmarking-leveldb-vs-rocksdb-vs-hyperleveldb-vs-lmdb-performance-for-influxdb/ |
| 14:45.23 | caen23 | brlcad: so about the script that downloads gci data, what can a student do? is the api public? |
| 14:46.09 | caen23 | since scraping doesn't really look like a good option |
| 14:48.21 | caen23 | and for CI, if buildbot is up, i suppose we should set up some clients -- maybe look into some free service that provides those? since i don't think it's a good idea to use personal computers as buildbot clients (like we did last time) |
| 14:49.41 | caen23 | oh, and another thing :D is there any way i could unsubscribe myself from design tasks? i don't think there's much i can do there (wouldn't know what to look for to evaluate/help properly), so it's mostly noise for me |
| 14:50.06 | *** join/#brlcad kintel (~kintel@unaffiliated/kintel) | |
| 15:00.24 | *** join/#brlcad amarjeet (~Amarjeet@2405:205:4087:aedf:d472:d1a3:9fb6:b30f) | |
| 15:25.19 | *** join/#brlcad kintel (~kintel@unaffiliated/kintel) | |
| 15:58.11 | *** join/#brlcad dracary1083 (dracarys98@nat/iiit/x-dmuatyfpdvzahfvj) | |
| 16:03.45 | Notify | 03BRL-CAD:starseeker * 69261 brlcad/trunk/doc/docbook/system/mann/search.xml: Clarify options vs search plans, add an example showing verbose output. |
| 16:04.03 | *** join/#brlcad Caterpillar2 (~caterpill@unaffiliated/caterpillar) | |
| 16:31.09 | *** join/#brlcad amarjeet (~Amarjeet@2405:205:4180:5ef1:1a2:9bd2:34ba:de6d) | |
| 16:45.44 | *** join/#brlcad kintel (~kintel@unaffiliated/kintel) | |
| 16:59.42 | brlcad | caen23: the api is public: https://developers.google.com/open-source/gci/resources/downloads/TaskAPISpec.pdf |
| 17:02.49 | brlcad | caen23: yes, you can either go to each task and click the "x" on your name ... or use that API to auto-remove yourself from all of them :) |
| 17:07.57 | caen23 | thanks :D |
| 17:08.30 | brlcad | as admin, I can bulk add you to tasks easily, but there is no bulk remove |
| 17:37.56 | starseeker | brlcad: the page you pointed to indicated lmdb came out ahead for 30M values or less... maybe that's what it's optimized for? |
| 17:39.21 | starseeker | also: "You could also potentially create a configuration with smaller shards and use LMDB for screaming fast performance." |
| 17:44.17 | starseeker | also ran across and flagged rocksdb, unqlite and leveldb... |
| 17:49.42 | brlcad | lmdb is heavily optimized for fast reads (which is a good thing, probably our case too) |
| 17:50.10 | brlcad | whereas leveldb is supposedly optimized for a balancing of reads and writes |
| 17:50.29 | starseeker | brlcad: what's your preferred starting test? unpack a .g into them in some way and do some random reads? |
| 17:52.56 | brlcad | basically, but I'd structure it in context of our API |
| 17:53.23 | starseeker | um. how so? |
| 17:54.14 | brlcad | thinking about something like http://brlcad.org/wiki/Example_Application minus the rt_i |
| 17:56.23 | starseeker | erm. so, db_open, db_dirbuild, db_update_nref? |
| 17:57.32 | brlcad | update_nref shouldn't exist |
| 17:57.35 | brlcad | but yeah, something like that |
| 17:57.37 | Stragus | brlcad: But wouldn't it be a problem to naively rewrite these functions to OpenCL, if later all APIs and memory layouts have to be changed for parallel hardware? |
| 17:57.55 | Stragus | What happens inside the function is no big deal, but the interface and storage... |
| 17:58.03 | brlcad | Stragus: it's a problem if the code were used and performance gain were expected |
| 17:58.42 | Stragus | OpenCL with no expected performance gain... I see, eh |
| 17:58.43 | brlcad | it's all about breaking the work up into 2-4 hour chunks |
| 17:58.49 | Stragus | All right then |
| 17:59.03 | brlcad | first step, convert and compile ... that's probably already more than 2-4 hours for most of those kids |
| 17:59.16 | Stragus | :) Okay then |
| 17:59.43 | brlcad | next step, maybe try to optimize one of them, or maybe just profile and report, or we come up with a pattern and they apply the pattern |
| 18:00.23 | brlcad | either way, definitely not an end-state by any means, just one (tiny) incremental step forward that they can (maybe) manage |
| 18:00.32 | brlcad | so far not even one of those tasks has been claimed and completed |
| 18:01.10 | Stragus | I see. Thanks, I'll keep all that in mind when answering questions on the topic |
| 18:01.41 | brlcad | starseeker: so at heart, I think there just needs to be three basic timing tests we'll care about |
| 18:02.02 | brlcad | Stragus: feel free to push them further if you get a kid that understands, any help is appreciated |
| 18:02.22 | brlcad | but honestly, this is pushing the limits for most of them in many regards |
| 18:02.28 | brlcad | GCI kids can be as young as 13 |
| 18:02.54 | Stragus | Sure, it was fun with that guy that did some CSG raytracing in OpenCL |
| 18:03.01 | Stragus | Interesting |
| 18:03.02 | brlcad | some *are* outstanding coders, but I've not seen any so far (we don't have most of our coding tasks uploaded) |
| 18:03.26 | brlcad | that guy is still here, he's helping mentor them too (vasc) |
| 18:03.38 | Stragus | Cool |
| 18:03.53 | brlcad | he actually has a phd candidate that might be picking up where he left off as a thesis topic |
| 18:04.03 | starseeker | favors lmdb and unqlite from a "dirt simple to integrate" standpoint at a quick glance, but acknowledges that performance and robustness have to be king at this level of db IO |
| 18:04.26 | brlcad | starseeker: yep, I was leaning towards that as well -- unqlite is particularly basic |
| 18:05.13 | starseeker | lmdb is (i think) 4 files that matter - lmdb.h, mdb.c midl.c and midl.h |
| 18:05.14 | brlcad | starseeker: the three basic timing tests are the ones you mentioned day before yesterday: reads, writes, updates |
| 18:05.32 | brlcad | and even on those, I think really it's the read test that matters most |
| 18:05.55 | Stragus | I once took libdbh (disk based hashtables, GPL) and did a complete rewrite for higher performance. It might not be what you have in mind though |
| 18:06.16 | starseeker | unfortunately GPL is a non-starter |
| 18:06.30 | brlcad | how long does it take to db_open, dirbuild and/or lookup now .. then stub a new version of those three using unqlite and lmdb |
| 18:07.07 | starseeker | am I correct that the type of primitive doesn't matter at this point? |
| 18:07.19 | brlcad | yeah, I don't think that matters |
| 18:07.25 | starseeker | i.e. we're at struct directory, not concerned with any of the rt_db_internal info... |
| 18:07.55 | brlcad | db_lookup is internals |
| 18:08.11 | starseeker | so maybe we should generate a mean large .g with sphflake to push things... |
| 18:08.40 | starseeker | ?? - I thought db_lookup got the directory pointer given a string key |
| 18:08.53 | starseeker | isn't that the "read" part of the test? |
| 18:08.59 | Stragus | I see, right. I assume it's still GPL even if one rewrites every single line of a GPL library, only keeping most of the "internal logic" |
| 18:09.29 | brlcad | starseeker: sorry, you're right -- lookup is a directory |
| 18:11.04 | starseeker | figured initializing the data from the .g into the container was the "write" (db_open/db_dirbuild), a randomized set of db_lookups was "read", and replacing an existing entry with a new one would be update? |
| 18:11.27 | brlcad | Stragus: yep, still a derivative |
| 18:11.53 | brlcad | some argue it's potentially a derivative even if you don't start with their code, but you looked at it |
| 18:13.00 | brlcad | same as with proprietary code, the only safe route was to put two guys in two different rooms, one looking at the code and talking about it to the other guy, the other guy never looking at the code but just using spoken descriptions from his buddy |
| 18:13.08 | brlcad | at least, that's the old school way |
| 18:13.59 | Stragus | In this case, my code is definitely a direct derivative, no doubt about it :) |
| 18:19.26 | starseeker | in fairness, I've never heard of an open source dev suing another open source dev on the basis of simply reading code without copying it... doesn't mean it couldn't happen I suppose |
| 18:21.12 | starseeker | doesn't actually know if a geometry edit invalidates a struct directory pointer as we're currently set up... |
| 18:21.27 | brlcad | starseeker: the goal of this would be to make something like Example_Application's prep run faster, so what gets caller there is what should be timed |
| 18:21.50 | brlcad | which I think is what we're saying, but needs checking -- maybe that's a high-level test |
| 18:21.55 | starseeker | what threw me though is not using rt_i - pretty much all the work in the example assumes the rt_i struct |
| 18:22.23 | starseeker | and rt preping does need the db internal structs, IIRC |
| 18:25.47 | brlcad | so two different issues |
| 18:26.58 | brlcad | all I meant is that it's the db layer that is affected by reading into an lmdb store vs into our dbi_Head[] array |
| 18:27.14 | starseeker | right |
| 18:27.27 | brlcad | if that's sped up (or slowed down), that will propagate through to rt_dirbuild() |
| 18:27.40 | starseeker | oh, gotcha |
| 18:29.13 | brlcad | but you do have a point .. there are some things that rt_dirbuild() does that db_dirbuild() does not and probably should |
| 18:30.11 | brlcad | so maybe a better setup is to define a specific high-level metric, like the time it takes to call ged_tops() |
| 18:30.12 | starseeker | was afraid it might be hard to separate out dbio times from other prep times if the prep dominated the test time, but if the dbio time is long enough in the first place it shouldn't matter |
| 18:31.13 | brlcad | those times are already separated out in rt, check the usual timing output |
| 18:31.33 | starseeker | ah, OK |
| 18:31.45 | starseeker | isn't as familiar with what we report there as he should be |
| 18:32.22 | brlcad | the only funky one is calls to prep moved from rt_prep_parallel() to rt_dirbuild() way back when |
| 18:32.41 | brlcad | i.e., the ft_prep calls are aggregated into the rt_dirbuild timings() |
| 18:32.54 | brlcad | but the read-from-disk times are a separate line iirc |
| 18:33.46 | brlcad | want me to whip up a tops timer? |
| 18:34.09 | brlcad | then you can try hooking in different db's underneath |
| 18:34.18 | starseeker | sure - sounds like a plan |
| 18:34.55 | starseeker | just wants to be sure that whatever numbers come out of this are the numbers brlcad wants |
| 18:37.24 | brlcad | if we had our act together, we'd already have some libged or regression tests that exercise a database |
| 18:37.40 | starseeker | nods |
| 18:38.16 | starseeker | would be instructive to push .g to its limits - how many objects can it hold practically, what are the failure modes as it reaches its limits, etc. |
| 18:38.35 | brlcad | there is also still value in timing the low-level operations if you want to work on that as well, a simple timer around db_dirbuild() would do that |
| 18:39.05 | brlcad | absolutely! ... I asked nick to work on that for the caching stuff but he didn't get there |
| 18:39.39 | starseeker | nods - I'm concerned with what our decision criteria will be for going from the existing hash to "something else" - for this level of IO we need to know things are rock solid, and I'm not confident right now that I at least can tell that for certain |
| 18:39.45 | brlcad | how many objects, how fast can it read/write/update them now, where does it break if two procs access simultaneously |
| 18:48.39 | *** join/#brlcad bug (783bad84@gateway/web/freenode/ip.120.59.173.132) | |
| 18:48.54 | brlcad | hi bug |
| 18:49.12 | bug | hi |
| 18:50.06 | bug | i dont know where is the blog post to submit the tutorial task. can i have some help? |
| 18:52.12 | brlcad | bug: which task? |
| 18:52.48 | brlcad | the task description probably tells you to post your tutorial to a blog -- this would be some blog that you set up |
| 18:53.59 | bug | brlcad: thank you |
| 18:58.29 | caen23 | brlcad: what kind of data should they download? i see google also has some scripts here https://code.googlesource.com/codein/api |
| 18:59.31 | brlcad | caen23: something that can be massaged into http://brlcad.org/gci/data/ |
| 19:00.28 | brlcad | which presently is the task instance int a dir with a scrape of the url and all url linked files downloaded |
| 19:00.44 | brlcad | with a timestamp so we know which came before/after what |
| 19:01.10 | brlcad | e.g., http://brlcad.org/gci/data/uncategorized/2014-6453861430591488-Design_a_new_website_favicon_5_-_BRL-CAD/ |
| 19:01.22 | caen23 | yup, seen that |
| 19:02.06 | caen23 | looking at the api spec however, i can't see anything regarding the files |
| 19:02.18 | brlcad | comments? |
| 19:03.43 | brlcad | I'll ask robert to see if he has a hook |
| 19:03.56 | caen23 | nope :-? |
| 19:07.35 | brlcad | sent |
| 19:09.47 | caen23 | could also be that the api docs are incomplete :-?? |
| 19:10.37 | caen23 | as for scraping, i think they'd have to use a headless browser, since the bulk of the page is generated dynamically from js |
| 19:31.58 | brlcad | caen23: doable, I wonder if the old script that scraped the old site might work |
| 19:32.04 | brlcad | (with some edits) |
| 19:47.51 | *** join/#brlcad gcibot-afk (~gcibot@r167-59-47-242.dialup.adsl.anteldata.net.uy) | |
| 19:47.52 | *** join/#brlcad gcibot-afk (~gcibot@unaffiliated/ignaciouy/bot/gcibot) | |
| 19:57.02 | caen23 | brlcad: is the old script somewhere i can see? |
| 21:25.38 | brlcad | caen23: yes ... somewhere .... |
| 21:49.48 | brlcad | maths22: do you recall where the script is that generated brlcad.org/gci/data? |
| 21:50.17 | brlcad | I know peter wrote a script ... and it's probably in that uncategorized GCI task list |
| 22:23.02 | caen23 | brlcad: found it http://brlcad.org/gci/data/uncategorized/2013-5807450632486912-Write_a_script_to_download_GCI_files_-_BRL-CAD/ |
| 22:29.08 | caen23 | i see he parsed some json, but i can't seem to find any for the current app |