starseeker (Sep 01 2020 at 18:13): starseeker (Sep 01 2020 at 18:13):One of our eventual website tasks is to set up an automatically updated "generate the doxygen docs" process that runs the "make dox" target whenever the tree is updated.
Right now the doxygen output only get sporadic attention, but there's actually been a fair bit of work put into it over the years.
Sumagna Das (Sep 01 2020 at 18:14):i could have tried to help with website stuff but i can only do website stuff in python
Sean (Sep 01 2020 at 18:14):it's long since been on our to-do for someone to get doxygen output set up on the website. getting it set up is rarely ever a problem. keeping it updated has been a challenge. would be nice if jenkins did it for us.
Sumagna Das (Sep 01 2020 at 18:18):should i try?
Sumagna Das (Sep 01 2020 at 18:23):making a doxygen generated online documentation?
Sean (Sep 01 2020 at 18:23):you're certainly welcome to, it would be quite helpful.
Sean (Sep 01 2020 at 18:23):the way I would think best to set it up is as a Jenkins job at ci.brlcad.org
Sean (Sep 01 2020 at 18:23):I believe you already have an account
Sumagna Das (Sep 01 2020 at 18:24):i would want to keep the generated output somewhere. so is there a way in jenkins to do that?
Sean (Sep 01 2020 at 18:24):yeah, I think it's all in how the job is set up - there's always an active workspace
Sean (Sep 01 2020 at 18:24):I think the job can have file copy or random shell script phases too
Sumagna Das (Sep 01 2020 at 18:25):i will try to do that. if i get any problems i will tell you.
Sean (Sep 01 2020 at 18:25):looks like this is a tutorial: http://dnaeon.github.io/building-projects-documentation-with-jenkins-and-doxygen/
Sean (Sep 01 2020 at 18:27):that uses a doxygen plugin and a web publshing plugin
Sumagna Das (Sep 01 2020 at 18:28):yea
Sean (Sep 01 2020 at 18:28):looks like this plugin would be an alternative possibility, or perhaps useful for the other doc/ products: https://plugins.jenkins.io/doclinks/
Sean (Sep 01 2020 at 18:29):we can also set up a job on the server that copies files into the web root if needed, but looks like jenkins might be able to handle it
Sumagna Das (Sep 01 2020 at 18:31):i should "install without a restart" i think
Sumagna Das (Sep 01 2020 at 18:33):as i can see a job running
Sumagna Das (Sep 01 2020 at 18:36):should i?
Sean (Sep 01 2020 at 18:37):install what? plugins?
Sean (Sep 01 2020 at 18:37):you can interrupt it all you need to
Sumagna Das (Sep 01 2020 at 18:37):yea
Sean (Sep 01 2020 at 18:37):kill jobs, whatever
Sean (Sep 01 2020 at 18:37):it's fine
Sumagna Das (Sep 01 2020 at 18:37):ok
Sean (Sep 01 2020 at 18:38):the main build job is presently failing about 50% of the time right now anyways when it gets to our unit tests because of some obscure semaphore/threading issue on freebsd that we haven't solved yet
Sean (Sep 01 2020 at 18:38):so don't be discouraged if you see that the build failed.
Sean (Sep 01 2020 at 18:40):if you check the log, you'll see it is usually during testing and it'll say Aborted, Failed, or something similar in the middle of "make regress"
Sumagna Das (Sep 01 2020 at 18:41):(deleted)
Sumagna Das (Sep 01 2020 at 18:46):how are you triggering the builds?
starseeker (Sep 01 2020 at 18:46):"make dox"
Sean (Sep 01 2020 at 18:46):The builds are triggered automatically on commit. Jenkins watches the repository.
Sumagna Das (Sep 01 2020 at 18:46):which option to use for that?
starseeker (Sep 01 2020 at 18:47):Oh, you mean Jenkins - sorry. Yeah, it monitors the repo automatically if you provide it with the URL
Sean (Sep 01 2020 at 18:47):you can trigger a build manually in jenkins too
Sean (Sep 01 2020 at 18:48):ah, I see -- you created a new project
Sumagna Das (Sep 01 2020 at 18:48):starseeker said:
Oh, you mean Jenkins - sorry. Yeah, it monitors the repo automatically if you provide it with the URL
i dont have to choose anything from this area? image.png
Sean (Sep 01 2020 at 18:49):yeah, select "Build after other projects are built"
Sean (Sep 01 2020 at 18:49):and select the BRL-CAD project
Sumagna Das (Sep 01 2020 at 18:50):trigger even if the build fails?
Sean (Sep 01 2020 at 18:50):we should probably break up that main BRL-CAD project into a source checkout project, a compile-it project, and a test it project
Sean (Sep 01 2020 at 18:51):doxygen can probably be set to run with/without the build actually succeeding or tests passing
Sumagna Das (Sep 01 2020 at 18:51):ok then
Sumagna Das (Sep 01 2020 at 18:51):i am choosing that option
Sean (Sep 01 2020 at 18:51):only thing it'll be missing is libbu's mime types... which we can think about later
Sumagna Das (Sep 01 2020 at 18:52):hmm
Sumagna Das (Sep 01 2020 at 18:53):does this look ok? image.png
Sean (Sep 01 2020 at 18:54):don't know.
Sean (Sep 01 2020 at 18:54):depends what your current working directory is (could add a pwd or ls -la in there and then check the log)
Sean (Sep 01 2020 at 18:55):learn something new every day. actually didn't know about -B ! that can be used to reduce our onboarding docs by one step...
Sumagna Das (Sep 01 2020 at 18:55):i learnt that option while trying to configure cmake within atom
Sean (Sep 01 2020 at 18:56):I don't think it existed when we got started...
Sumagna Das (Sep 01 2020 at 18:56):might be new then
Sumagna Das (Sep 01 2020 at 18:59):should have added ls -la :face_palm:
Sumagna Das (Sep 01 2020 at 18:59):it is probably cloning directly into the workspace
Sumagna Das (Sep 01 2020 at 19:02):should i make a directory to checkout the repo?
Sumagna Das (Sep 01 2020 at 19:09):should i use -j4 while using cmake --build?
Sumagna Das (Sep 01 2020 at 19:10):i mean is it safe or will it cause problem for other builds?
Sean (Sep 01 2020 at 19:11):I don't think we want multiple checkouts just because that's a lot of filesystem strain and delay, and it adds no value
Sumagna Das (Sep 01 2020 at 19:11):ok then
Sumagna Das (Sep 01 2020 at 19:11):so i just need to make the target dox for this job?
Sean (Sep 01 2020 at 19:12):that's why that " run this when the other project completes" option is good - when it completes, the sources should be in that project's workspace.
Sumagna Das (Sep 01 2020 at 19:12):i can use that build's resources if i want
Sean (Sep 01 2020 at 19:12):i would run cmake again in your own workspace just for good measure, but yeah -- then run "make dox"
Sumagna Das (Sep 01 2020 at 19:15):it seems like it is going to checkout the svn repo everytime it is built
Sumagna Das (Sep 01 2020 at 19:15):is there an option to not checkout the repo if it is already present
Sean (Sep 01 2020 at 19:15):well yeah, you turned on Subversion
Sean (Sep 01 2020 at 19:16):set that to "None"
Sean (Sep 01 2020 at 19:16):it's getting it's sources from the BRL-CAD project
Sumagna Das (Sep 01 2020 at 19:16):will configure after this build
Sumagna Das (Sep 01 2020 at 19:16):Sean said:
it's getting it's sources from the BRL-CAD project
what?
Sean (Sep 01 2020 at 19:17):there are two projects: "online-docgen" and another named "BRL-CAD"
Sumagna Das (Sep 01 2020 at 19:17):you are saying i can use the sources from the "BRL-CAD" project
Sean (Sep 01 2020 at 19:18):the "BRL-CAD" one is already checking out from Subversion, so you can configure the "online-docgen" one to use "BRL-CAD" project's files instead of also checking it out
Sean (Sep 01 2020 at 19:18):that's what that "Build after other projects are built" implies
Sean (Sep 01 2020 at 19:19):it's just probably in a workspace path next to where "online-docgen" has its workspace
Sumagna Das (Sep 01 2020 at 19:19):so should i copy the sources from the "BRL-CAD" workspace to the "online-docgen"
Sean (Sep 01 2020 at 19:19):(so it'd be something like mkdir build && cmake -B build -S ../BRL-CAD/workspace/... or something
Sumagna Das (Sep 01 2020 at 19:20):Sean said:
(so it'd be something like mkdir build && cmake -B build -S ../BRL-CAD/workspace/... or something
let me configure it then
Sean (Sep 01 2020 at 19:20):I wouldn't bother copying them
Sean (Sep 01 2020 at 19:22):take a look at the "BRL-CAD" project, look at the "Build" section
Sean (Sep 01 2020 at 19:22):might help you figure out what you need in yours
Sean (Sep 01 2020 at 19:22):and a bit more readable ;)
Sumagna Das (Sep 01 2020 at 19:23):i am making a conditional step to see if there is a build directory. if there is a build directory, then it will not trigger mkdir build
Sumagna Das (Sep 01 2020 at 19:23):you have any other ideas?
Sean (Sep 01 2020 at 19:24):that sounds great
Sumagna Das (Sep 01 2020 at 19:24):it made the previous build fail
Sean (Sep 01 2020 at 19:25):that's akin to what I was suggesting that we should break up the "BRL-CAD" project into one that doesn't do testing, just checks out sources. another one can run testing. I think there's a way to do this all within one project too using pipelines, but I'm not that familiar with Jenkins yet.
starseeker (Sep 01 2020 at 19:29):At least for doxygen 1.8.17, r77031 seems to clear the warnings.
Sumagna Das (Sep 01 2020 at 19:37):waiting for the brlcad project build to end
Sumagna Das (Sep 01 2020 at 20:00):@Sean i built the dox target but it seems a bit off
Sumagna Das (Sep 01 2020 at 20:00):the index.html seems a bit off
starseeker (Sep 01 2020 at 20:21):How so?
Sumagna Das (Sep 01 2020 at 20:21):Screenshot-from-2020-09-02-01-51-33.png
Sumagna Das (Sep 01 2020 at 20:21):this is from jenkins build
Sumagna Das (Sep 01 2020 at 20:22):Screenshot-from-2020-09-02-01-52-29.png
Sumagna Das (Sep 01 2020 at 20:22):this is from my build
Sumagna Das (Sep 01 2020 at 20:26):check for yourself
Sumagna Das (Sep 01 2020 at 20:26):https://ci.brlcad.org/status/job/online-docgen/ws/build/doc/doxygen_output/html/index.html
Sean (Sep 01 2020 at 20:26):yeah, clearly some feature of doxygen isn't getting enabled, perhaps due to relative paths or current working directory. can you check the log?
Sean (Sep 01 2020 at 20:27):it clearly did work, as can be seen in other paths: https://ci.brlcad.org/status/job/online-docgen/ws/build/doc/doxygen_output/html/annotated.html
Sumagna Das (Sep 01 2020 at 20:27):yea
Sumagna Das (Sep 01 2020 at 20:27):Sean said:
yeah, clearly some feature of doxygen isn't getting enabled, perhaps due to relative paths or current working directory. can you check the log?
which one?
starseeker (Sep 01 2020 at 20:27):I saw that happen once with my local build too, actually - chromium displayed it, but firefox didn't want to. Not sure why.
Sean (Sep 01 2020 at 20:29):looking at the page source gives clues
Sean (Sep 01 2020 at 20:30):Screen-Shot-2020-09-01-at-4.29.36-PM.png
Sean (Sep 01 2020 at 20:30):looks like the menu is dynamically generated
Sean (Sep 01 2020 at 20:30):check stackoverflow for that error. has to be common.
Sean (Sep 01 2020 at 20:33):yep, looks like it's this: https://stackoverflow.com/questions/34315723/jenkins-error-blocked-script-execution-in-url-because-the-documents-frame
Sumagna Das (Sep 01 2020 at 20:35):see if you can do anything about that
Sumagna Das (Sep 01 2020 at 20:35):i cant understand the solution
Sumagna Das (Sep 01 2020 at 20:35):what it tells to follow
Sumagna Das (Sep 01 2020 at 20:35):but i understand what has happened
Sean (Sep 01 2020 at 20:36):jenkins apparently has a security setting that prevents javascript and css from loading
Sumagna Das (Sep 01 2020 at 20:36):yea
Sean (Sep 01 2020 at 20:36):so either we disable it, or (better) ignore it and deploy it outside of jenkins
Sean (Sep 01 2020 at 20:36):the intent isn't to view from within jenkins
Sumagna Das (Sep 01 2020 at 20:36):where can we deploy it?
Sean (Sep 01 2020 at 20:36):anywhere on brlcad.org
Sumagna Das (Sep 01 2020 at 20:37): Sean (Sep 01 2020 at 20:37):I think best might be something like brlcad.org/api or brlcad.org/docs/api
Sumagna Das (Sep 01 2020 at 20:38):which one sounds better?
Sean (Sep 01 2020 at 20:39):heh, looks like we were previously using brlcad.org/docs/doxygen-r######
Sumagna Das (Sep 01 2020 at 20:39):yea
Sumagna Das (Sep 01 2020 at 20:39):i used it once
Sean (Sep 01 2020 at 20:39):there's a couple doxygens there now, a bit out of date
Sumagna Das (Sep 01 2020 at 20:39):but the revision number was out of date
Sean (Sep 01 2020 at 20:39):yeah, it's about 5000 commits out of date
Sumagna Das (Sep 01 2020 at 20:40):its best for brlcad.org/api or brlcad.org/docs/api
Sean (Sep 01 2020 at 20:40):lets go with the latter to keep docs grouped
Sumagna Das (Sep 01 2020 at 20:40):yea
Sumagna Das (Sep 01 2020 at 20:41):so how to deploy it there?
Sean (Sep 01 2020 at 20:41):I created the dir, so just have to figure out how to safely deploy
Sumagna Das (Sep 01 2020 at 20:42):what are the possible ways to deploy?
Sean (Sep 01 2020 at 20:43):option 1) almost certainly can just have a build rule that copies the files from the workspace to the web directory;
Sumagna Das (Sep 01 2020 at 20:43):can be a post-build action
Sean (Sep 01 2020 at 20:43):option 2) use a plugin that has a deploy/copy rule (and presumably with smarts on how to copy more safely/smartly)
Sean (Sep 01 2020 at 20:43):right, post-build action would be option 1
Sean (Sep 01 2020 at 20:43):custom script or the same rule that runs make dox even
Sean (Sep 01 2020 at 20:44):those are all opt1
Sean (Sep 01 2020 at 20:44):option 3) use a job on the server that pulls from jenkins periodically
Sean (Sep 01 2020 at 20:44):3 is the old way and isn't obvious so lets try to avoid that one
Sumagna Das (Sep 01 2020 at 20:45):for option2 will this work? -> https://plugins.jenkins.io/publish-over-ssh/
Sean (Sep 01 2020 at 20:45):overkill -- jenkins is on the same host
Sean (Sep 01 2020 at 20:45):doesn't need to ssh
Sumagna Das (Sep 01 2020 at 20:46):i didnt know about that so i suggested it
Sean (Sep 01 2020 at 20:46):looks like there's a copy artifacts step
Sumagna Das (Sep 01 2020 at 20:47):in post build action?
Sean (Sep 01 2020 at 20:47):https://plugins.jenkins.io/copyartifact/
Sumagna Das (Sep 01 2020 at 20:48):it seems like it can be used to copy artifacts from one project to another
Sumagna Das (Sep 01 2020 at 20:48):will it work for the deployment?
Sean (Sep 01 2020 at 20:49):I don't see why not
Sean (Sep 01 2020 at 20:49):it can either be added to this project to deploy or we can have another "deploy api" project that does the copy from this one
Sean (Sep 01 2020 at 20:50):I don't see why it can't be added to an existing though
Sean (Sep 01 2020 at 20:50):that said, might be nice to separate deployment into its own project
Sumagna Das (Sep 01 2020 at 20:50):lets keep this project only for generating docs
Sean (Sep 01 2020 at 20:50):agreed
Sean (Sep 01 2020 at 20:51):that way a deployment project can focus on just deployment like deploying the docs that got built during "make docs" earlier
Sumagna Das (Sep 01 2020 at 20:51):yea
Sumagna Das (Sep 01 2020 at 20:52):you set up the deployment project
Sean (Sep 01 2020 at 20:53):I'm currently working on the github permissions
Sumagna Das (Sep 01 2020 at 20:53):then what to do for the deployment
Sumagna Das (Sep 01 2020 at 20:53):like what are the steps?
Sumagna Das (Sep 01 2020 at 20:54):the docgen job is set up
Sean (Sep 01 2020 at 20:55):enable the plugin, create a deployment project just like you did for the online-docgen project, then tell it to copy to [path_i'll_send_in_pm]
Sumagna Das (Sep 01 2020 at 20:56):the copy artifact plugin you mean?
Sean (Sep 01 2020 at 20:57):yeah
Sumagna Das (Sep 01 2020 at 21:00):should i keep it to "trigger if build is stable"?
Sean (Sep 01 2020 at 21:03):I guess, I don't know what stable means
Sumagna Das (Sep 01 2020 at 21:03):can you check up on the error?
Sean (Sep 01 2020 at 21:03):what error?
Sumagna Das (Sep 01 2020 at 21:04):ERROR: Failed to copy artifacts from online-docgen with filter: build/doc/doxygen_output/html/*
Sean (Sep 01 2020 at 21:05): Sumagna Das (Sep 01 2020 at 21:07):i might have to go to sleep after this last attempt
Sumagna Das (Sep 01 2020 at 21:07):so if you can please check up on the error if you can
Sean (Sep 01 2020 at 21:07):I did, that link looks like the fix
Sean (Sep 01 2020 at 21:08):either didn't specify workspace or need to archive in the online-docgen project
Sean (Sep 01 2020 at 21:08):what time is it for you?
Sumagna Das (Sep 01 2020 at 21:08):2 38 am
Sean (Sep 01 2020 at 21:08):nice!
Sumagna Das (Sep 01 2020 at 21:08):please see up on the error
Sumagna Das (Sep 01 2020 at 21:09):i have to go to sleep now
Sumagna Das (Sep 01 2020 at 21:09):have a last exam tomorrow
Sean (Sep 01 2020 at 21:09):okay, I'll see if I can once I get done with github here
Sean (Sep 01 2020 at 21:09):this is excellent work
Sean (Sep 01 2020 at 21:09):actual doxygen progress!
Sean (Sep 01 2020 at 21:10):and thank you for your work on this, it's great
Sumagna Das (Sep 01 2020 at 21:10):no problem
Sean (Sep 01 2020 at 21:10):I think I'll break out the testing from the compilation afterwards too
Sumagna Das (Sep 01 2020 at 21:10):just doing this so that future contributors can find at least some api reference online and not build it locally
Sean (Sep 01 2020 at 21:10):and checkout from compile
Sumagna Das (Sep 01 2020 at 21:23):Its giving errors from the copy artifact plugin
Sumagna Das (Sep 01 2020 at 21:23):Something about accesdenied
Sean (Sep 01 2020 at 21:23):ah, right
Sean (Sep 01 2020 at 21:24):the web root is owned by the webserver and a different user
Sean (Sep 01 2020 at 21:24):hmmmmmmm
Sean (Sep 01 2020 at 21:24):let me try this
Sean (Sep 01 2020 at 21:24):try again
Sean (Sep 01 2020 at 21:25):made jenkins the group
Sean (Sep 01 2020 at 21:37):upgrading all plugins
Sumagna Das (Sep 02 2020 at 03:02):So what is left to do?
Sumagna Das (Sep 02 2020 at 03:15):(deleted)
Sumagna Das (Sep 02 2020 at 03:15):@Sean The API documentation is live on brlcad.org/docs/api
Sumagna Das (Sep 02 2020 at 03:19):It will be automatically deployed after every revision from now
starseeker (Sep 02 2020 at 03:42):@Sumagna Das It seems like some of the links aren't working?
starseeker (Sep 02 2020 at 03:42):https://brlcad.org/docs/api/dd/d89/group__vmath.html
Sumagna Das (Sep 02 2020 at 03:43):I think I messed up the directory structure
Sumagna Das (Sep 02 2020 at 03:44):Its all in one directory
Sumagna Das (Sep 02 2020 at 03:44):Don't know how to fix it
Sumagna Das (Sep 02 2020 at 03:44):brlcad.org/docs/api/group__vmath.html
Sumagna Das (Sep 02 2020 at 03:45):See
Sumagna Das (Sep 02 2020 at 03:46):Any ideas how to fix this?
Sean (Sep 02 2020 at 07:40):working on it @Sumagna Das ... you got it super close
Sean (Sep 02 2020 at 07:42):the problem is the "Files to archive" option that saves the doxygen output in online-docgen is stashing archive/build/doc/doxygen_output/html ... i.e., the whole path specified, not the matching pattern
Sean (Sep 02 2020 at 07:42):that gets copied in the deployment phase, so it' installing into the web root as doc/api/build/doc/doxygen_output/html ...
Sean (Sep 02 2020 at 08:41):what a pain in the culo...
Sean (Sep 02 2020 at 08:41):the artifact archiving system sucks. it appears to only copy the whole tree, no matter how I specify it. and if I flatten, it will only flatten the whole hierarchy.
Sean (Sep 02 2020 at 08:44):there doesn't seem to be a general copy mechanism -- it assumes/wants single files or to copy trees exactly as they are in a workspace.
Sean (Sep 02 2020 at 08:45):so I modified the test to result in exactly the dir it needs to install into brlcad.org/docs (i.e., api) and now it's working
Sumagna Das (Sep 02 2020 at 09:39):Its working now
Sumagna Das (Sep 02 2020 at 09:47):Its not in brlcad.org/docs but in brlcad.org/docs/api
Sean (Sep 02 2020 at 15:17):It installs "api" into brlcad.org/docs, thus brlcad.org/docs/api :)
Sean (Sep 02 2020 at 15:17):Good luck on your exam @Sumagna Das !
Sean (Sep 02 2020 at 15:18):and thanks again. awesome to see doxygen up and running again in what will hopefully be a more sustainable way now
Sumagna Das (Sep 02 2020 at 17:32):Sean said:
It installs "api" into brlcad.org/docs, thus brlcad.org/docs/api :smile:
ooh
Last updated: Oct 09 2024 at 00:44 UTC