Sean (Mar 26 2021 at 17:30): Sean (Mar 26 2021 at 17:30):Hi all, just FYI, I did submit an application for BRL-CAD to participate in the 2021 Season of Docs. There were a number of significant changes this year including the proposal and stipend format. As such, if we're approved, there is the option for up to three mentors to be reimbursed $500 for their efforts, and $7560 to a technical writer.
If you are interested in mentoring, please send me a private message. If you're interested in participating as a technical writer, please open a topic thread here and introduce yourself. As always, priority will go towards technical writers interested in getting involved in open source more, interested in learning technical infrastructure. Students that apply to GSoC are generally not ideally suited for the Season of Docs.
Seun (Apr 18 2021 at 13:49):Hello everyone,i am seun by name and i am interested in migrating BRL-CAD's Documentation Infrastructure,i am excited and energised to see Docbook files are fully converted to AsciiDoc.
Thank you
Indermohan Singh (Apr 24 2021 at 11:30):Hi everyone. My name is Indermohan Singh. I'm a technical writer, author, and software developer interested in working with BRL-CAD in this GSoD. I'd like to work on the "Organize and publish developer docs" project. I have extensive experience in writing developer-centric docs and tutorials. Looking forward to it.
Kritika Srivastava (Apr 24 2021 at 17:37):Hi @Sean ,
I'm Kritika Srivastava and interested in working as a Technical Writer for BRL-CAD under GSoD '21.
I have gone through the details mentioned here.
You can find my previous work on my GitHub and personal blog.
I had to ask about the prerequisites, getting started, and the application process for this project.
I also wish to draft a detailed proposal for this project with detailed information on the points mentioned in the application along with the set timelines once approved by project mentors to start working on my proposal.
Sean (Apr 27 2021 at 04:08):Welcome @Indermohan Singh and @Kritika Srivastava ! Thank you for your introductions. What can you tell me about the project and how you'd intend to approach it?
Indermohan Singh (Apr 30 2021 at 15:43):Hi, @Sean thanks for the welcome message. For the project "Organize and publish developer docs". Following are the things I would like to do:
This is an informal and brief proposal. I'll be soon finishing my formal proposal and submit that. Meantime here is some of my work on writing documentation and tutorials: Auth0 Blog, Logrocket Blog, RxJS-Audio Docs
Looking forward to your comment.
Inder
Sean (May 17 2021 at 17:29):Hello all! I'm happy to announce that we have invited @Dashamir Hoxha to work for us this season on our documentation infrastructure.
Thank you all for your interest, proposals, discussion, and consideration getting selected for the 2021 Google Season of Docs. It was a very complicated decision this year with several very strong candidates under final consideration. Among them, Dashamir was selected for his early, frequent, and strong communication skills; extensive Docbook experience (author of docbookwiki); technical writing experience; and solid alignment with the needs of BRL-CAD's documentation project this year.
If any that were not accepted are interested in providing documentation infrastructure testing and review support for the Season of Docs project, we do have a mentoring position open to anyone. It entails approximately 10 hours of effort over 4 weeks, compensated. Please send me a private message if you're interested.
Dashamir Hoxha (May 18 2021 at 03:14):@Sean thank you for trusting me this year. I will try to do my best.
I am starting by reading HACKING_BRL-CAD.pdf, while at the same time getting familiar with Antora and AsciiDoc. I estimate that this should take me about 2-3 weeks.
If you don't have any other preferences, I am going to report on this channel at the end of each week about the progress and what I plan to do next.
Sahibpreet Kaur (May 18 2021 at 06:03):Awesome! Welcome @Dashamir Hoxha We're glad to have you here.
Dashamir Hoxha (May 21 2021 at 20:07):I had a look at HACKING_BRL-CAD.pdf and at the directory doc/docbook/. I also started to read the docs of Antora. It seems that this project is going to be more exciting than I thought. I am trying to figure out how to categorize the docs of BRL-CAD and how to display them on the Antora structure.
Usually the docs of a project are divided into user docs and developer docs (which includes also how to contribute). The user docs are usually categorized further into tutorials (including how to get started), guides, howto-s, and manuals. I should find out (maybe with some help) how to fit the existing docs into these categories, and how to represent them in the new docs site.
Another thing is versioning of the docs, which is supported by Antora, and translations to other languages, which I am not sure yet whether it can be supported, but I have to see (Antora is extensible and maybe can be customized to support this as well).
Although HACKING_BRL-CAD.pdf recommends using the SVN repo on SourceForge, I would prefer to start a new repo for the new docs on https://github.com/BRL-CAD, separate from the main code repo. Maybe later it can also be merged with the wiki repo and the homepage repo.
For the next week I will continue to learn more about the details of Antora and AsciiDoc, from https://docs.antora.org/
starseeker (May 21 2021 at 20:33):Yes, the SVN advice is obsolete - we have transitioned to Github now.
Sean (May 25 2021 at 02:02):@Dashamir Hoxha that's great and all your observations look spot-on. Historically, I would have agreed about user docs vs. dev docs too, but there has been a trend the last few years to adjust that categorization/characterization.
Sean (May 25 2021 at 02:04):In a nutshell, the trend has been towards recognizing all the potential roles of participants whether users or not, as anyone can be a contributor, but to also value other non-dev forms of contribution including graphic designers, community management, public relations, and developers.
Sean (May 25 2021 at 02:06):We emphasize contributors in our HACKING BRL-CAD book, and I do think that is a central theme to our project in general. The project is also lacking several important governance docs that I started working on a couple months back but will be a process.
Sean (May 25 2021 at 02:09):A good article on the subject: https://opensource.com/article/19/12/open-source-contributors
Sean (May 25 2021 at 02:13):As for docs, definitely good idea to plan for a new github repo, so you have a clean slate to work with at least initially. I suggest just starting without any categorization or organization, and focus on setting up the About BRL-CAD article which presently resides in the repo in a couple languages. From there, maybe do the MGED tutorial series next (which is essentially 16 tutorials).
That will give some time and familiar context/content to consider next steps and organizational needs next. I don't think we'll have time to solve all our doc organizing needs, but that way will give more time for thinking about it and not having you spend too much time on something that might change (a lot).
Dashamir Hoxha (May 25 2021 at 02:28):I agree. Let me first finish the conversion/migration of the docs without worrying much about the "perfect" structure, and later we can think about better organization, de-duplication, etc.
Dashamir Hoxha (May 28 2021 at 21:49):I finished reading the Antora docs (or at least skimming through them).
I also read some other blogs and tutorials about Antora and AsciiDoc, like:
I also tested a few tools for converting DocBook to AsciiDoc, like:
The best one seems to be docbookrx, although it is not perfect. For example it does not handle <part> elements (I submitted a pull request for fixing it: https://github.com/asciidoctor/docbookrx/pull/69).
Then I started to convert the HACKING book into the Antora format:
It seems like splitting each chapter into a separate page gives the best results.
For the time being I am using a single component for all the docs, where each book is a module (and also a group of related articles is going to be a module).
To handle translations of docs to other languages, I am using a separate component for each language. Another solution could have been to use a different playbook for each language. This approach could also allow the UI template to be different for each language (and potentially translated and customized). However the first approach (using separate components) allows different language docs to (potentially) use the same figures, the same examples, etc.
It seems that the issue of multiple-language support is still being discussed and developed in the Antora community:
For the time being I am not planning any versioning for the docs, but this can be handled easily in Antora with a (git) branch of the docs matching each version of the software.
Next week I am going to continue with converting some more docs and adding them to the test site.
Sean (Jun 03 2021 at 04:01):@Dashamir Hoxha this is excellent progress. Love that you submitted a fix to the docbookrx folks.
Sean (Jun 03 2021 at 04:02):A couple issues I noticed, https://brlcad.fs.al/docs/hacking-brlcad/part2/chapter3.html has stray footnote tags that don't appear to have converted cleanly.
Sean (Jun 03 2021 at 04:02):Also, the spanish about page appears to be in english -- what happened there?
Dashamir Hoxha (Jun 04 2021 at 03:33):Sean said:
https://brlcad.fs.al/docs/hacking-brlcad/part2/chapter3.html has stray footnote tags that don't appear to have converted cleanly.
The footnotes of that page seem to be correct. I don't see where is the problem.
Dashamir Hoxha (Jun 04 2021 at 03:39):Sean said:
Also, the spanish about page appears to be in english -- what happened there?
Actually the about page is not translated in Spanish (at least I cannot find it).
Maybe I used the English page just just for mocking, and later forgot to delete it.
I am going to remove it.
Dashamir Hoxha (Jun 04 2021 at 03:45):By the way, I have started using https://github.com/dashohoxha/brlcad-docs/issues and https://github.com/dashohoxha/brlcad-docs/projects/1 to manage the tasks, bugs, etc. and track/record the progress of the project.
Dashamir Hoxha (Jun 04 2021 at 03:48):I created an issue there about the Spanish about page: https://github.com/dashohoxha/brlcad-docs/issues/11
But you (the mentors etc.) can create bug reports yourself, if you wish.
Dashamir Hoxha (Jun 04 2021 at 04:03):In general, the automatic conversion has lots of problems (as you probably know), and there are many things that need to be fixed manually. For example image paths need to be fixed, <informalfigure> is not converted, sometimes titles are broken, paragraphs have a broken formatting, sometimes tables are not converted properly, etc.
I am trying to fix as many of these problems as I can, but sometimes it is a bit difficult for me to spot a problem because I am not quite familiar with the BRL-CAD itself and its docs. For example I suspect that some of the tables in the MGED tutorials have not been converted properly, however I cannot tell quickly which ones are broken.
If you (the mentors and the community) could make a quick review of the converted docs and report any problems (for example by opening bug issues at https://github.com/dashohoxha/brlcad-docs/issues), it would be great.
But let me make the first pass of the review myself. For the time being I am still reviewing the MGED tutorials myself, and then I am going to review the About pages and the HACKING book. I will let you know when I am done, so that you can start reviewing them.
Dashamir Hoxha (Jun 04 2021 at 04:05):For the next week, besides the review of the existing docs, I will also try to convert and add the articles and the rest of the books.
Dashamir Hoxha (Jun 08 2021 at 14:23):Sean said:
https://brlcad.fs.al/docs/hacking-brlcad/part2/chapter3.html has stray footnote tags that don't appear to have converted cleanly.
Now I see the problem.
https://github.com/dashohoxha/brlcad-docs/issues/13
I am fixing it.
Dashamir Hoxha (Jun 12 2021 at 09:38):But let me make the first pass of the review myself.
I am done with the About pages and the HACKING book. I also fixed the MGED lessons (English): https://brlcad.fs.al/docs/lessons/mged.html
However I am still working to fix the tables on the Spanish lessons (I should be able to do it quickly since I already know how to fix them). Besides the tables, there were many other things to be fixed manually in the lessons, so it took me longer than expected.
I am planning to make an update of the English lessons, once the conversion of all the docs is finished:
https://github.com/dashohoxha/brlcad-docs/issues/17
Any suggestions about what to fix or improve are welcome.
Please add them as comments to the issue above, or create new issues at:
https://github.com/dashohoxha/brlcad-docs/issues/
Dashamir Hoxha (Jun 12 2021 at 17:51):I am still working to fix the tables on the Spanish lessons
Done: https://brlcad.fs.al/docs/es/lecciones/mged.html
Sean (Jun 14 2021 at 04:18):@Dashamir Hoxha awesome progress! This is looking really great.
Sean (Jun 14 2021 at 04:23):So right now, it looks like translations are basically being handled as separate documents, manually inserted into the hierarchy?
Sean (Jun 14 2021 at 04:24):Is there a way to register them automatically so that while looking at an article or lesson, for example, it's possible to see a list of translations available?
Sean (Jun 14 2021 at 04:43):I'm seeing in https://gitlab.com/antora/antora/-/issues/208 that multilingual support appears to be lacking for anything beyond having separate doc sets. Maybe adopting a convention internally, but not something Antora itself can help manage.
Sean (Jun 14 2021 at 04:47):looks like what the fedora folks are doing is running their english docs through something that extracts pot/po files during a compilation step, which are then translated and re-constituted into new .adoc files. so instead of having es/ru/whatever copies of docs, they're just extracting the phrases and maintaining the translations through toolings.
Sean (Jun 14 2021 at 04:51):we're definitely not set up to go in that direction right now, so I'm thinking your work is strong evidence that we should re-focus our efforts on just english for now. the rest is
Sean (Jun 14 2021 at 04:52):the rest can be revisited, and by then maybe antora will have something more to offer. looks like they're somewhat working on it.
Dashamir Hoxha (Jun 15 2021 at 17:42):Sean said:
So right now, it looks like translations are basically being handled as separate documents, manually inserted into the hierarchy?
Currently I am generating a separate documentation site for the Spanish docs (with antora translations/build-es.yml). This HTML files for this site are saved at /var/www/html/docs/es/.
For the rest of the languages (including English) I am currently using a separate antora component for each language. All of them are generated with antora build.yml (or antora build-dev.yml for a local site). The HTML output is saved at /var/www/html/docs/, /var/www/html/docs-it/, etc.
So, the Spanish docs are inserted inside the English docs directory, and English docs directory is inserted inside the BRL-CAD website directory. Hence the would be accessed like https://brlcad.org/docs/, https://brlcad.org/docs/es/, https://brlcad.org/docs-it/, etc.
It is also possible to use subdomains, if wished, like: https://docs.brlcad.org/, https://es.docs.brlcad.org (or https://docs.brlcad.org/es/), https://docs-it.brlcad.org/, etc. But it seems like the first approach is simpler.
Dashamir Hoxha (Jun 15 2021 at 17:50):Sean said:
Is there a way to register them automatically so that while looking at an article or lesson, for example, it's possible to see a list of translations available?
For the time being I am using a list like this, which is included on the docs site and looks like this. This approach can be made more granular (for each article or lesson) and maybe embedded nicely on a doc page (for example like a drop-down list on a corner). These translation lists can be maintained manually, or maybe can be generated automatically with some scripts. But I think it's not worth (to make them more granular and to make efforts to generate them automatically), especially since BRL-CAD does not have lots or doc pages translated.
Dashamir Hoxha (Jun 15 2021 at 17:53):Sean said:
looks like what the fedora folks are doing is running their english docs through something that extracts pot/po files during a compilation step, which are then translated and re-constituted into new .adoc files. so instead of having es/ru/whatever copies of docs, they're just extracting the phrases and maintaining the translations through toolings.
Some people on that discussion issue think that this approach is not a good one (because translation gets out of context, it is easy to break the structure of asciidoc and it is difficult to debug and fix, etc.) I agree with them.
Dashamir Hoxha (Jun 15 2021 at 18:01):Sean said:
the rest can be revisited, and by then maybe antora will have something more to offer. looks like they're somewhat working on it.
In my opinion translation is a difficult topic, and maybe there is no correct approach that works for all the cases.
I even think that translation is not necessary, unless the software is a very popular one (like GNOME, KDE, LibreOffice, etc.) BRL-CAD does not seem to me like a software that needs to be translated, but maybe I am wrong.
For example I tried to translate the About page into Albanian, and I gave up, because I was not able to translate properly the very first paragraph :smiley:
Sean (Jun 19 2021 at 01:47):I agree that the dir-approach is simpler (and with the point that editing po/pot statements out of context is suboptimal). We could always introduce rewrite rules on the server too. But again, I think it's fine to just focus on getting the english site working as best possible, keep what you've done with the translations, but put them on a back shelf for now.
Sean (Jun 19 2021 at 01:49):Dashamir Hoxha said:
For example I tried to translate the About page into Albanian, and I gave up, because I was not able to translate properly the very first paragraph :smiley:
LOL!
Sean (Jun 19 2021 at 01:51):The about page is overdue a good re-writing. It was originally written for a very different audience.
Dashamir Hoxha (Jun 19 2021 at 20:25):I have already converted the articles: https://brlcad.fs.al/docs/articles/index.html
I had to split the MGED article into smaller parts, because it was too big.
However the user commands and developer commands of the MGED article still need to be reformatted and fixed (manually):
I have decide to postpone this for later (after the conversion of all the other docs is done), because there is too much manual work to be done and may take a long time: https://github.com/dashohoxha/brlcad-docs/issues/19
Or maybe this could be a good task for a student.
Next week I am going to continue with converting the tutorials: https://github.com/dashohoxha/brlcad-docs/issues/10
Sean (Jun 21 2021 at 14:37):That's looking fantastic @Dashamir Hoxha !
Sean (Jun 21 2021 at 14:38):I like the MGED breakup. Begs for re-organization as it obviously has a Tutorial section and then there's the top-level Tutorials section.. and they overlap heavily in what they teach.
Sean (Jun 21 2021 at 14:42):Dashamir Hoxha said:
I had to split the MGED article into smaller parts, because it was too big.
Too big of a page or too big for Antora to handle?
Sean (Jun 21 2021 at 14:51):I did notice the command links on https://brlcad.fs.al/docs/articles/mged/developer-commands.html aren't working -- is that part of the reformatting and fixing you mention?
Dashamir Hoxha (Jun 21 2021 at 16:11):Sean said:
Too big of a page or too big for Antora to handle?
Too big of an article, Antora does not mind. Also, as you noticed, it mixes tutorials, command references, etc.
Dashamir Hoxha (Jun 21 2021 at 16:16):Sean said:
I did notice the command links on https://brlcad.fs.al/docs/articles/mged/developer-commands.html aren't working -- is that part of the reformatting and fixing you mention?
Yes. The automatic conversion has failed to convert the id-s to anchors for each command, so the links are broken.
Dashamir Hoxha (Jun 21 2021 at 16:19):Actually, to be honest, the XML file itself does not seem to be very well formatted, so maybe the converter is not to be blamed for all the mistakes and problems.
Sean (Jun 21 2021 at 20:16):If memory serves, it was originally a word doc, converted to an html doc, converted to docbook xml ... I think.
Sean (Jun 21 2021 at 20:17):So not terribly surprising that the format could be less than optimal. That said, it "should" be valid xml at least. I'm pretty sure it goes through our conversion process which includes a validation pass and produces html/pdf output during doc compilation.
Dashamir Hoxha (Jun 21 2021 at 21:07):It is valid XML, and maybe valid DocBook too, but still the semantic structure is broken.
For example look at the details for %. There are some example paragraphs outside the definition of the term.
Even if we try to fix the XML file first, or fix the converter, it would still need to be done manually.
Dashamir Hoxha (Jun 25 2021 at 21:22):Finished the tutorials: https://brlcad.fs.al/docs/tutorials/index.html
Custom CSS styles: https://github.com/dashohoxha/brlcad-docs/blob/main/ui/src/css/custom.css
I have also added some AsciiDoc examples on the TEMPLATE: https://brlcad.fs.al/docs/articles/TEMPLATE.html#_asciidoc_examples
However I am not sure whether this is the right place for them. Maybe they should be added as a section to the HACKING book. What do you think?
Dashamir Hoxha (Jun 25 2021 at 21:24):Next week I am going to look at the rest of the docs: devguides, presentations and specifications (https://github.com/dashohoxha/brlcad-docs/issues/22)
Sean (Jun 26 2021 at 04:04):I don't remember my docbook xml well enough to know if a para is valid at the same level as a variablelist, but I see what you mean.. it's not grouped into a section or other logical construct. It looks like several if not most of the examples are like that.
Sean (Jun 26 2021 at 04:05):Not to say it's okay, but I imagine it's technically valid structure and a side-effect of the html-to-xml conversion. It's like the whole thing is just one big stream of para's and lists.
Sean (Jun 26 2021 at 04:11):Dashamir Hoxha said:
Finished the tutorials: https://brlcad.fs.al/docs/tutorials/index.html
That's looking really good. Where are the navigation controls through? Is that a setting or something that has to be manaully set up somewhere?
Sean (Jun 26 2021 at 04:12):I'm referering to seeing Prev and Next controls at the bottom of each page so one can continue reading from section to section, doc to doc
Sean (Jun 26 2021 at 04:40):Custom CSS styles: https://github.com/dashohoxha/brlcad-docs/blob/main/ui/src/css/custom.css
I have also added some AsciiDoc examples on the TEMPLATE: https://brlcad.fs.al/docs/articles/TEMPLATE.html#_asciidoc_examples
However I am not sure whether this is the right place for them. Maybe they should be added as a section to the HACKING book. What do you think?
Yeah, I think it belongs elsewhere. The template is intended to be just that -- a simple template for starting a new document, a new article. An authorship guide like that is really necessary too, but probably belongs with either the HACKING BRL-CAD document or a stand-alone article of some sort incorporated into it.
Sean (Jun 26 2021 at 04:41):So this is all looking really good enough that we'll soon want to look at what's needed to transition this into the production setup on brlcad.org and into our repo. Do you have thoughts on that -- what do you need?
Dashamir Hoxha (Jun 26 2021 at 05:57):Sean said:
Not to say it's okay, but I imagine it's technically valid structure and a side-effect of the html-to-xml conversion. It's like the whole thing is just one big stream of para's and lists.
The AsciiDoc output generated by the converter looks like a mess. But maybe I can fix some of the problems automatically using Emacs macros. I haven't tried it yet. Otherwise it would be too much manual work, because it is a huge section.
Dashamir Hoxha (Jun 26 2021 at 06:01):Sean said:
Where are the navigation controls through? Is that a setting or something that has to be manaully set up somewhere?
I am not sure. So far I have been using the default settings. Maybe it is a setting or maybe I need to customize the UI template. I am going to look into it: https://github.com/dashohoxha/brlcad-docs/issues/26
Sean (Jun 26 2021 at 06:06):Example is right in antora's own docs, e.g., https://docs.antora.org/antora/2.3/run-antora/
Sean (Jun 26 2021 at 06:07):It's interesting that it even jumps from doc to doc, like this one: https://docs.antora.org/antora/2.3/antora-container/
Dashamir Hoxha (Jun 26 2021 at 06:16):Sean said:
So this is all looking really good enough that we'll soon want to look at what's needed to transition this into the production setup on brlcad.org and into our repo. Do you have thoughts on that -- what do you need?
We can fork or migrate the repo to https://github.com/BRL-CAD/ at anytime. I am not sure whether the issues and the project (https://github.com/dashohoxha/brlcad-docs/projects/1) can be migrated as well.
I would need some info about how you host and manage your website (https://brlcad.org/). Maybe we can schedule an online meeting with the webmaster(s) and discuss about how to publish the antora docs on the website.
Basically, antora needs to be installed, repository should be cloned, and then these commands can generate the docs: antora build.yml and antora translations/build-es.yml.
Dashamir Hoxha (Jun 26 2021 at 19:39):Sean said:
Where are the navigation controls through? Is that a setting or something that has to be manaully set up somewhere?
It was fixed by setting the attribute page-pagination: https://github.com/dashohoxha/brlcad-docs/issues/26
Dashamir Hoxha (Jun 27 2021 at 20:26):Sean said:
So this is all looking really good enough that we'll soon want to look at what's needed to transition this into the production setup on brlcad.org and into our repo. Do you have thoughts on that -- what do you need?
It seems that if you give me access to create new repos on https://github.com/BRL-CAD/, I should be able to transfer my repo there (including issues and everything).
Dashamir Hoxha (Jun 27 2021 at 20:30):The other way may also be possible:
if I give you full access to my repo, you should be able to transfer it to https://github.com/BRL-CAD/
Dashamir Hoxha (Jun 29 2021 at 20:37):BRL_CAD_g_format_V5 and BRL_CAD_g_format_V6 seem to be almost identical, so I published only the latest. Please let me know if I am missing something.
Sean (Jun 29 2021 at 21:33):Dashamir Hoxha said:
Sean said:
It seems that if you give me access to create new repos on https://github.com/BRL-CAD/, I should be able to transfer my repo there (including issues and everything).
Awesome, let's do that then. I sent you an invitation request.
Sean (Jun 29 2021 at 21:37):Dashamir Hoxha said:
BRL_CAD_g_format_V5andBRL_CAD_g_format_V6seem to be almost identical, so I published only the latest. Please let me know if I am missing something.
We're currently on v5. Our next major release that breaks file compatibility is going to be v6, so that document was a placeholder for all the hard-breaking changes. There's not so much on 'main' but there may be more changes on the v6 development branch but I don't know that for sure. There might not be much to say about v6. If there's not, I'd suggest we eliminate the document until there's something important to say. What were the differences you noticed?
Dashamir Hoxha (Jun 29 2021 at 21:47):Sean said:
Awesome, let's do that then. I sent you an invitation request.
I did it and it worked: https://github.com/BRL-CAD/brlcad-docs
I also wanted to rename the repo from brlcad-docs to docs, but it seems that now I don't have access to the settings of the repo.
Dashamir Hoxha (Jun 29 2021 at 21:53):Sean said:
What were the differences you noticed?
Almost no differences at all (I used diff on the asciidoc files).
So, I should keep v5 instead of v6, at least for the time being.
When the development branch is released, we can update the docs accordingly.
Sean (Jun 29 2021 at 22:00):Yeah, keep v5
Dashamir Hoxha (Jul 01 2021 at 02:35):I have tried to convert the man pages to AsciiDoc.
A problem is that while DocBook uses lots of details and information for a man page, AsciiDoc uses a very constraint formatting, so the conversion will result in loss of details and information. For example an AsciiDoc man page looks like this: https://raw.githubusercontent.com/asciidoctor/asciidoctor/master/man/asciidoctor.adoc
It basically uses only bold and underline inline formatting (which actually is the only formatting used on a man page displayed on a terminal). The information about <command>, <prompt>, <option>, <replaceable> etc. is not present. This makes it impossible to generate a nice HTML page from an AsciiDoc manpage document. For example in Antora this man page looks like this: https://brlcad.fs.al/docs/man/1/asciidoctor.html
An HTML man page generated from a DocBook document usually has much more details that can be styled.
In the man pages that I have converted I have tried to preserve some formatting, so that the HTML page could be a bit nicer. For example the man page of mged looks like this:
But this is a bit tricky because some formatting may cause failure to convert to a man page (that can be viewed with the man command). For example the command in SYNOPSIS has to be enclosed in stars ( * ) otherwise it will fail. This is what the converter assumes and expects.
This extra formatting also makes the document cluttered and more difficult to edit. So I am not sure it is worth keeping it, just to make the HTML page a little bit nicer.
What do you think?
Dashamir Hoxha (Jul 02 2021 at 17:20):I have already finished the rest of the docs:
I have also customized a bit the UI templates, but still need some help (for example to find a better logo, etc.):
I have played as well with converting man pages from DocBook to AsciiDoc (see also the previous message). I tried different ways and the best one seemed to be converting DocBook to man, and then using pandoc to convert man to asciidoc. However this would completely loose any DocBook formatting. So, I tried to customize the docbookrx to convert man pages as well (it currently does not support this):
It does preserve most of the formatting, but I am still not happy with the result. I am going to look further at it. But in any case, the resulting asciidoc pages would need a lot of manual review and corrections.
Sean (Jul 06 2021 at 07:35):This is outstanding progress @Dashamir Hoxha. I'm going to review this in more detail and should have feedback for you tomorrow. I'd also like to get you connected with one of our other documenters to test/review and iterate on some feedback as well.
Dashamir Hoxha (Jul 06 2021 at 09:04):I have tried to migrate the wiki pages to GitHub (using https://github.com/outofcontrol/mediawiki-to-gfm).
The result does not look very impressive: https://github.com/BRL-CAD/brlcad-docs/wiki
GitHub wiki pages can be cloned with git clone https://github.com/BRL-CAD/brlcad-docs.wiki, and then processed locally with advanced tools or editors, so maybe some problems can be fixed automatically.
This attempt was the first step towards merging wiki pages into the Antora documentation. It seems like not all the wiki pages are about documentation, so not all of them should be merged to Antora. Deciding which ones should to be moved to Antora most probably cannot be done automatically.
Sean (Jul 06 2021 at 17:54):So the wiki is tricky. Half the docs really belong in the repository. The other "working documents" arguably belong in the repo or belong on the wiki because they're works in progress. We don't have pedigree/criteria spelled out for when a document is "good enough" but there are definitely a slew on the wiki that would be best in the repo, if not most of the wiki pages.
Sean (Jul 06 2021 at 17:55):So yeah, your analysis is spot on. They're not all docs ;)
Dashamir Hoxha (Jul 06 2021 at 18:30):I am not quite familiar with mediawiki, so I didn't notice it at first, but actually the categories provide a good guidance about what wiki pages belong to the docs, and even how to organize (categorize) them on the Antora site.
Sean (Jul 06 2021 at 19:02):Except we've not been terribly consistent with using the category labels except for the developer working docs label, and even that is spotty
Sean (Jul 06 2021 at 19:02):It's all ad hoc.
Dashamir Hoxha (Jul 14 2021 at 14:22):I finished converting and fixing the man pages: https://brlcad.fs.al/docs/man/1.html
The automatic conversion was far from perfect and I had to do a lot of manual correction. There are too many man pages, so it took me a lot of time. Since I am not familiar with the details of BRL-CAD, some mistakes in the content may have slipped me, so another review (by someone who is familiar) might be useful.
I also added a search box for searching the docs.
Added also a README with installation instructions etc.
Sean (Jul 14 2021 at 18:13):That is awesome @Dashamir Hoxha !! You are rocking this project. I met with a web dev last week to see if they can help you get the docs instantiated on brlcad.org, but they weren't sure they'd have time - so looks like it'll fall on me (which is great, but we'll have to schedule some time to sort through things). In the meantime, I'll take a look at the README and see how it goes!
Dashamir Hoxha (Jul 15 2021 at 02:37):I am happy that you like it, and I am enjoying it too :smile:
But I am not done yet.
Please let me know when is a suitable meeting time for you.
I don't think that it is going to take more than 1 hour.
Dashamir Hoxha (Jul 15 2021 at 03:06):What do you think about replacing the image gallery with a static one? This would get rid of the need for MySQL+PHP on the host.
I have listed some possible solutions on this issue, but I think that Antora can handle an image gallery quite well too.
Dashamir Hoxha (Jul 17 2021 at 07:27):I have been playing with GitHub pages: https://github.com/dashohoxha/brlcad-docs/tree/gh-pages
One of the advantages of hosting the docs (and the website) on GitHub pages is that it is possible to automate rebuilding and publishing them whenever there are some changes. The playbook looks like this: https://github.com/dashohoxha/brlcad-docs/blob/main/.github/workflows/build-and-deploy-gh-pages.yml
For the time being it seems to work, but maybe it can be optimized further, I am not sure.
However GitHub pages require a website that does not depend on MySQL and dynamic PHP.
Dashamir Hoxha (Jul 19 2021 at 16:31):So the wiki is tricky.
I am giving it a try anyway.
There are lots of pages about MGED commands, which start with "MGED CMD" (265 of them). These seem like generated automatically, maybe from man pages (having the footer "Page Generated by David Loman on: 10/11/2007").
Is it worth including them in the Antora site, given that we already have the man pages included?
Sean (Jul 20 2021 at 06:07):Hm, probably not, but we'll need to reconcile the list to make sure. I think the were gereated from the manpages, but then dave augmented them too if I recall. Still, I think you can safely ignore them.
Dashamir Hoxha (Jul 23 2021 at 19:29):Just for the record, I have started already with the wiki pages:
https://brl-cad.fs.al/docs/wiki/Documentation.html
But there is lots of manual work to be done and it is going slowly.
My aim is to move to Antora the wiki pages that are related to documentation,
and the rest of them (about GSoC etc.) will be moved to GitHub wiki:
https://github.com/BRL-CAD/brlcad-docs/wiki
The final goal is to get rid of MediaWiki, since it depends on MySQL and PHP.
Dashamir Hoxha (Aug 18 2021 at 17:06):The new image gallery, which is hosted on GitHub pages:
Sean (Aug 20 2021 at 15:16):I saw the github pages site, that's awesome. Stashing images in a repo is far better than the manual maintenance hell that is the current piwigo site. Will just need to eventually pull all the newer/better images from the facebook gallery... let me know if you know a way to automate/facilitate that as it's a lot of images.
starseeker (Aug 20 2021 at 16:50):Is there some way to capture the captions as well? (see, for example, https://brlcad.org/gallery/picture.php?/32/category/3)
Even imgname.txt files in the repo next to the images would be better than losing the info, if we can't display it...
Dashamir Hoxha (Aug 20 2021 at 17:31):Sean said:
Will just need to eventually pull all the newer/better images from the facebook gallery... let me know if you know a way to automate/facilitate that as it's a lot of images.
I know that it should be possible with the Facebook API, but I have never used it before.
But it seems that it is also possible to download all the images in bulk:
I am not sure if this download includes the status/description of the photo as well. Also I am afraid that the photos may need to be renamed and categorized manually.
If nothing else works, downloading them manually should still be possible (for example as a Code-In task or something like this), because there are only about 50 photos published:
Dashamir Hoxha (Aug 20 2021 at 17:34):starseeker said:
Is there some way to capture the captions as well? (see, for example, https://brlcad.org/gallery/picture.php?/32/category/3)
Even imgname.txt files in the repo next to the images would be better than losing the info, if we can't display it...
There are only 105 photos in all the albums, so doing it manually is also an option (as a Code-In task or something).
I used some bash kung-fu to download the images. Maybe it is possible to scrape the description as well, but I haven't tried it.
Sean (Aug 20 2021 at 18:28):It may be better to just upload originals again anyways as facebook downsamples them pretty heavily, now that I think about it.
Sean (Aug 20 2021 at 18:28):I'm not sure they keep the original uploaded resolution
Sean (Aug 20 2021 at 18:29):but if you already got them, that's great too!
Dashamir Hoxha (Aug 20 2021 at 20:27):Sean said:
but if you already got them, that's great too!
To clarify, I downloaded only the images from the gallery (https://brlcad.org/gallery/), not the photos from FB.
I downloaded the original versions, not the resized ones.
Sean (Aug 21 2021 at 19:49):Ah, okay, I did misunderstand you there. I did notice the /gallery images were nice full res, and the static gallery generator it's using
Dashamir Hoxha (Aug 26 2021 at 18:43):Fareha Nousheen said:
Is there any scope I can contribute for the selected GSoD project? Kindly guide me
You can start by installing Antora and building the docs locally:
https://github.com/BRL-CAD/brlcad-docs
Once you do that, have a look at AsciiDoc (if you are not already familiar with it):
https://docs.antora.org/antora/2.3/asciidoc/section-headings/
Finally have a look at the currently open issues and see if there is any that you can do:
https://github.com/BRL-CAD/brlcad-docs/issues
Actually it is my duty to complete them, but I am also happy to collaborate with the community, if possible.
Dashamir Hoxha (Aug 26 2021 at 18:58):I am done with migrating doc pages from wiki to Antora: https://brl-cad.fs.al/docs/wiki/all-pages.html
Maybe some of the migrated pages do not belong to the docs (and should be removed), maybe I have missed some pages (they are doc pages but have not been migrated). If you spot such cases please let me know.
Dashamir Hoxha (Aug 27 2021 at 03:57):I am moving the rest of the wiki pages to a separate GitHub repo, generating the HTML with mkdocs and hosting them on GitHub:
I tried initially to host them on the Wiki of a github repo: https://github.com/BRL-CAD/brlcad-docs/wiki
However the GitHub wiki (which is based on Gollum) does not have a good support for organizing pages into directories and subdirectories, so I tried other solutions. I considered MDWiki and Docusaurus, but finally decided to use MkDocs.
Currently the wiki pages still need some reorganization, fixing some broken internal links (which are a bit tricky), etc.
Also a few pages are missing because somehow they failed to be converted from Mediawiki.
Dashamir Hoxha (Aug 27 2021 at 17:11):starseeker said:
Is there some way to capture the captions as well?
It is done.
I also use this script to add the description to each image.
Dashamir Hoxha (Aug 31 2021 at 19:55):Dashamir Hoxha said:
I am moving the rest of the wiki pages to a separate GitHub repo, generating the HTML with
mkdocsand hosting them on GitHub:
I am done with the wiki. Unfortunately the automatic conversion was not perfect and most of the pages need some manual corrections, but at least they serve as an archive of the past activity, in case someone is interested.
@Sean what do you think about it?
Sean (Sep 01 2021 at 15:52):@Dashamir Hoxha I've been looking through it. I think it looks really great overall. I'm following your installation steps to get antora instanced on brlcad.org and will let you know if I run into any roadblocks, but it looks like you've made it as simple as possible. Hopefully the FreeBSD environment doesn't pose any challenges! (usually doesn't, but ... sometimes codes assume Linux)
Dashamir Hoxha (Sep 01 2021 at 17:09):I hope that it should not be too difficult to install Antora on FreeBSD.
However I was planning to move the whole website and documentation infrastructure to GitHub pages.
This is usually done by adding a CNAME file on the gh-pages branch of the repo, and adding some records on the DNS configuration of the site. More details here:
The documentation (Antora) site can be hosted on the GitHub pages:
The gallery is already hosted on the GitHub pages (it only needs the configuration of a custom domain):
The wiki is already hosted on the GitHub pages (needs only the configuration of a custom domain):
The main website (brlcad.org) has not been transferred yet to GitHub pages, but it should not bee too difficult.
Is it just a plain HTML website, a static generated site (with Jekyll, Hugo, etc.) or something else?
If you give me more details about it I can try to migrate it to GitHub pages.
In case it is a plain HTML website, maybe a migration to a statically generated site (Jekyll or Hugo) should be done too.
Dashamir Hoxha (Sep 01 2021 at 20:13):If you give me more details about it I can try to migrate it to GitHub pages.
I realize that the code of the website is here: https://github.com/BRL-CAD/web
It seems that it is a plain HTML website. The wiki/ subdir must be about the mediawiki pages. I am not sure what the WordPress (wp/ subdir) is used for.
I have created an issue for updating it: https://github.com/BRL-CAD/brlcad-docs/issues/43
Sean (Sep 02 2021 at 02:39):Hm, I hadn't really considered that. What are the implications / benefits? I mean, beside the obvious that it's hosted and maintained by the GH folks, probably nice replication and such going on -- but are there benefits from a deployment/maintenance/integration/editing or some other perspective?
Sean (Sep 02 2021 at 02:44):@Dashamir Hoxha I noticed with the wiki that you used mkdocs instead of antora. Was there a particular reason they're different? Just easier to export+import? Just wanting to try it out? Easier than antora?
What are the implciations for merging them all to one or the other? My original thinking at least was that we would merge most of the wiki docs (at least the non-working docs) with the new site/repo in asciidoc alongside our other documentation, and either retire/migrate/relegate the wiki to temporal content (like gsoc project logs, working docs, etc).
Dashamir Hoxha (Sep 02 2021 at 07:37):Sean said:
Dashamir Hoxha I noticed with the wiki that you used mkdocs instead of antora. Was there a particular reason they're different? Just easier to export+import? Just wanting to try it out? Easier than antora?
What are the implciations for merging them all to one or the other? My original thinking at least was that we would merge most of the wiki docs (at least the non-working docs) with the new site/repo in asciidoc alongside our other documentation, and either retire/migrate/relegate the wiki to temporal content (like gsoc project logs, working docs, etc).
Actually I did merge the wiki docs to Antora: https://brl-cad.fs.al/docs/wiki/all-pages.html
At least the doc pages that I thought were important (please let me know if I have missed some important doc pages).
Then I converted the rest of the wiki to a MkDocs site, which is quite similar to wiki, but better (in my opinion, since all the docs are saved as plain files and versioned by GitHub).
Actually I included in the MkDocs site the doc pages as well, for convenience, since there are some references to them from the other pages.
The migration of wiki pages to MkDocs was done in order to retire the MediaWiki site, since it depends on PHP, MySQL, etc. Because MkDocs does not use dynamically generated pages (PHP+MySQL) it is much easier to deploy, and safer. It can also be deployed on GitHub pages.
Dashamir Hoxha (Sep 02 2021 at 07:51):Sean said:
Hm, I hadn't really considered that. What are the implications / benefits? I mean, beside the obvious that it's hosted and maintained by the GH folks, probably nice replication and such going on -- but are there benefits from a deployment/maintenance/integration/editing or some other perspective?
If your website is hosted on GitHub pages, you don't need to maintain your own web server (if this counts as a benefit). And it is for free.
Combined with GitHub Actions, your website can be updated automatically whenever a pull request is approved (or there is a commit on the main branch). This is CI/CD (Continuous Integration and Deployment). But maybe there are ways to do this with your own web server too.
Using a suitable Jekyll or Hugo template, you have to care only about the content of the site; the styles, layout and other formatting is taken care by the template. This makes the maintenance and update of the web site easier.
I am not sure if these reason are convincing, but it is up to you to decide. For me it is fine if you continue to host your website on your own webserver.
Sean (Sep 02 2021 at 13:06):Dashamir Hoxha said:
Actually I did merge the wiki docs to Antora: https://brl-cad.fs.al/docs/wiki/all-pages.html
At least the doc pages that I thought were important (please let me know if I have missed some important doc pages).
Aha! I didn't realize some were migrated and some not, but that totally makes sense. Is there a list of what wasn't converted? It looks like the mkdocs site has just about everything that was on the wiki. I was actually just really enjoying the categorized views you set up in mkdocs as it makes it really easy to sort through which docs are important that weren't on the main doc listing.
The migration of wiki pages to MkDocs was done in order to retire the MediaWiki site, since it depends on PHP, MySQL, etc. Because MkDocs does not use dynamically generated pages (PHP+MySQL) it is much easier to deploy, and safer. It can also be deployed on GitHub pages.
I guess I'm wondering what, if any benefits it has over Antora, not MediaWiki. Def want to retire MW if we can. The only question we'll need to decide is whether/how we want to support anonymous edits and how new pages from new contributors are supported (the one feature MW made super easy).
Sean (Sep 02 2021 at 13:12):As for the GH hosting, I was originally thinking we'd still do all of that just on our server. GH action that pushes updates to the server whenever there's a commit/merge and CI passes (if we have any validation/review criteria, which we don't currently).
The reason being that we've been bitten hard in the past by notable service providers having outages and the idea of their outage taking our website down is an unpleasant risk and would affect our industry partners. Of course, there's always the risk our server has an issue too, but then that's on us and we have a pretty good track record in that department.
Sean (Sep 02 2021 at 13:13):Will have to mull and think that one over a bit. Not opposed, just wasn't in the original thinking/plan for the doc restructuring.
Sean (Sep 02 2021 at 13:14):Thanks for your perspective on this. Definitely helps to hear things from others'. We get tied up in our bubble sometimes, focused on our industry users more than the bigger picture.
Dashamir Hoxha (Sep 02 2021 at 14:42):Is there a list of what wasn't converted? It looks like the mkdocs site has just about everything that was on the wiki.
The wiki pages that start with MGED_CMD_ were not converted, for example:
This information is already present on the Antora docs, for example:
Actually I am working right now to fix their formatting (which is a huge work, because there are too many commands and everything has to be fixed manually):
Dashamir Hoxha (Sep 02 2021 at 14:59):I guess I'm wondering what, if any benefits it has over Antora, not MediaWiki.
MkDocs uses GFM (GitHub Flavored Markdown), which is closer to MediaWiki markdown compared to AsciiDoc. This means that the conversion is a bit easier.
GFM is also much easier than AsciiDoc for users to get familiar to, which makes it more suitable for a Wiki site.
Dashamir Hoxha (Sep 02 2021 at 15:05):The only question we'll need to decide is whether/how we want to support anonymous edits and how new pages from new contributors are supported (the one feature MW made super easy).
With GitHub based MkDocs, anybody who has a GitHub account can contribute by submitting a pull request to https://github.com/BRL-CAD/wiki. Of course it has to be approved by someone before the contributions get published, but I think this is better than anonymous editing. I have noticed spam pages on the current MW site.
Dashamir Hoxha (Sep 09 2021 at 16:18):Dashamir Hoxha said:
The wiki pages that start with
MGED_CMD_were not converted, for example:This information is already present on the Antora docs, for example:
Actually I am working right now to fix their formatting (which is a huge work, because there are too many commands and everything has to be fixed manually):
I just finished fixing mged/user-commands.adoc:
I still have to fix mged/developer-commands.adoc:
I notice that the information on these pages is almost identical to these man pages:
I am not sure which one of them is more complete, up to date, etc. But I think that maintaining both of them is a pain. It seems to me that the doc pages (that I just fixed) are more practical for users and easier to maintain. So, I would suggest to drop the corresponding man pages. @Sean what do you think?
Dashamir Hoxha (Sep 10 2021 at 13:26):I finished fixing mged/developer-commands.adoc:
I realized that the user-commands and developer-commands references in these articles are somehow related to the appendixes of Introduction to MGED (pdf). I don't know which one is the primary source of information, but I noticed that some user-commands on the PDF file are not listed on article (for example: make_bb, memprint, polybinout, pov, rmmats, etc.) I'd like to believe that the article is more up to date than the PDF book and these are deprecated commands, but I am not sure. Otherwise, if they are missing by mistake, we should create a task/issue for adding them to the antora docs.
Dashamir Hoxha (Sep 10 2021 at 13:30):Next, I am going to work on this task: Update the HACKING book
@Sean do you think there is something else I should do, before we call this GSoD project finished?
Dashamir Hoxha (Sep 13 2021 at 10:37):I have pushed some changes on this Pull Request: https://github.com/BRL-CAD/brlcad-docs/pull/46
You can view most of them here:
Anyone willing to review this pull request?
Sean (Sep 17 2021 at 05:39):Dashamir Hoxha said:
Sean do you think there is something else I should do, before we call this GSoD project finished?
I'd love to have a non-dev go over the docs and your work to review and give feedback from their fresh perspective. We have a mentor that's agreed to do that, so I'll reach out to them to have them get started on that now.
Dashamir Hoxha (Sep 17 2021 at 07:22):Sean said:
I'd love to have a non-dev go over the docs and your work to review and give feedback from their fresh perspective. We have a mentor that's agreed to do that, so I'll reach out to them to have them get started on that now.
This would be really useful. I guess the reviewer can submit any noticed problems to the issues, or (why not) submit any pull requests for fixing them, if it is something that can be fixed easily.
Dashamir Hoxha (Sep 17 2021 at 07:24):For the time being I would like to shift my focus to other projects, but I am still available to reply to any questions or to help with integrating the docs to the website.
Sahibpreet Kaur (Oct 17 2021 at 12:59):@Sean Can I do that? If you are still looking?
Last updated: Jan 10 2025 at 00:48 UTC