Docs Publisher issueshttps://lab.civicrm.org/documentation/docs-publisher/-/issues2017-04-30T22:16:43Zhttps://lab.civicrm.org/documentation/docs-publisher/-/issues/63Move this repo to GitLab2017-04-30T22:16:43ZSean ColsenMove this repo to GitLab*Created by: seanmadsen*
We are starting to test the waters with GitLab at https://lab.civicrm.org. I'd like to migrate this repository (and our workflow) there. Is that okay with others? I'd especially like to get the go-ahead from @mi...*Created by: seanmadsen*
We are starting to test the waters with GitLab at https://lab.civicrm.org. I'd like to migrate this repository (and our workflow) there. Is that okay with others? I'd especially like to get the go-ahead from @michaelmcandrew here. What do you think?https://lab.civicrm.org/documentation/docs-publisher/-/issues/62Re-title wiki.civicrm.org so Docs site is preferred2017-04-30T22:16:43ZxurizaemonRe-title wiki.civicrm.org so Docs site is preferredCurrently searches for "civicrm docs" have the wiki at the top. By now I think we should be preferring to land people on docs.civicrm.org and linking to "legacy" docs in the wiki when necessary.
Not sure what is required to re-title w...Currently searches for "civicrm docs" have the wiki at the top. By now I think we should be preferring to land people on docs.civicrm.org and linking to "legacy" docs in the wiki when necessary.
Not sure what is required to re-title wiki.civicrm.org "CiviCRM Wiki" or "Legacy Documentation" to denote this, so opening an issue to propose it.
![screen shot 2017-04-22 at 11 51 08 pm](https://cloud.githubusercontent.com/assets/105608/25304220/f191eb42-27b6-11e7-8990-86498772896b.png)https://lab.civicrm.org/documentation/docs-publisher/-/issues/61Open external links in a new tab/window.2019-05-21T10:20:19ZSean ColsenOpen external links in a new tab/window.*Created by: RandyTobias*
Personally speaking, I really hate to be directed away from a site or guide when I am following a link. I'd like to have external links open in a new tab (normally with target=_blank"). I know markdown does not...*Created by: RandyTobias*
Personally speaking, I really hate to be directed away from a site or guide when I am following a link. I'd like to have external links open in a new tab (normally with target=_blank"). I know markdown does not specifically support this behavior, so it was suggested that if people find this to be a worthwhile change, we could use JS to accomplish this task on a global scale.
If desired, this could be accomplished with:
```JS
$( document ).ready(function() {
Array.from( document.querySelectorAll( 'a' ) ).forEach( a => {
a.classList.add( location.hostname === a.hostname || !a.hostname.length ? 'local' : 'external' );
});
$('a.external').each(function(){
$(this).attr('target','_blank');
});
})
```Docs Publisher 2019homotechsualhomotechsualhttps://lab.civicrm.org/documentation/docs-publisher/-/issues/60Automatic publishing for books hosted in GitLab2019-05-21T10:18:44ZSean ColsenAutomatic publishing for books hosted in GitLab*Created by: seanmadsen*
We can automatically publish books when changes are made, but only if the book is hosted on GitHub (and if the owner configures the webhooks correctly). It'd be great to get something like this working for GitLa...*Created by: seanmadsen*
We can automatically publish books when changes are made, but only if the book is hosted on GitHub (and if the owner configures the webhooks correctly). It'd be great to get something like this working for GitLab too, so that we can automatically publish books which are hosted in GitLab.michaelmichaelhttps://lab.civicrm.org/documentation/docs-publisher/-/issues/59When acting on GitHub webhooks, only publish if docs content was modified2019-05-21T10:19:35ZSean ColsenWhen acting on GitHub webhooks, only publish if docs content was modified*Created by: seanmadsen*
Now that we have books being published from repos with actual code (in addition to just docs) I think it'd be nice to avoid the server load (and unnecessary notification emails) of re-publishing books when non-d...*Created by: seanmadsen*
Now that we have books being published from repos with actual code (in addition to just docs) I think it'd be nice to avoid the server load (and unnecessary notification emails) of re-publishing books when non-docs changes are made to the repo. For example, if an extension's maintainer makes a code change to their extension, civicrm-docs will currently re-publish the book and send a notification email every time. We could check the paths modified and publish only if we see `/docs` or `/mkdocs.yml`Docs Publisher 2019homotechsualhomotechsualhttps://lab.civicrm.org/documentation/docs-publisher/-/issues/58Allow aliases for book slugs2019-05-21T10:17:41ZSean ColsenAllow aliases for book slugs*Created by: seanmadsen*
It'd be nice if, in a book's yaml config, we could specify aliases for the book slug which would redirect to the book's correct URL if requested. For example, we have a book with the slug `volunteer`. I'd be nic...*Created by: seanmadsen*
It'd be nice if, in a book's yaml config, we could specify aliases for the book slug which would redirect to the book's correct URL if requested. For example, we have a book with the slug `volunteer`. I'd be nice if we could also reach the docs (through a redirect) by requesting URLs with `civivolunter`. This would make it safer to re-name booksDocs Publisher 2019homotechsualhomotechsualhttps://lab.civicrm.org/documentation/docs-publisher/-/issues/56Say "guides" instead of "books"2017-04-30T22:16:43ZSean ColsenSay "guides" instead of "books"*Created by: seanmadsen*
Remove language about "books", and change it to "guides" instead.
I think the home page is the only place where this change is needed within this repository.
Convo originally started by @michaelmcandrew ...*Created by: seanmadsen*
Remove language about "books", and change it to "guides" instead.
I think the home page is the only place where this change is needed within this repository.
Convo originally started by @michaelmcandrew [here](https://chat.civicrm.org/civicrm/pl/ksz16ognw3dxim6s8mmixh69pe)https://lab.civicrm.org/documentation/docs-publisher/-/issues/55don't use "admin" as a path component for app functionality2017-10-09T22:42:55ZSean Colsendon't use "admin" as a path component for app functionality*Created by: seanmadsen*
Currently we have these two routes which Symfony handles: `/admin/publish` and `/admin/listen`. I think it's likely that we'll eventually want to have a book with the slug `admin` which complicates this routing....*Created by: seanmadsen*
Currently we have these two routes which Symfony handles: `/admin/publish` and `/admin/listen`. I think it's likely that we'll eventually want to have a book with the slug `admin` which complicates this routing.
I'd like to change those routes to `/system/publish` and `/system/listen` respectively. I think it's unlikely enough that we'll ever want a book with the slug `system`. https://lab.civicrm.org/documentation/docs-publisher/-/issues/54Implement "watchers" email addresses in book config files2017-04-30T22:16:43ZSean ColsenImplement "watchers" email addresses in book config files*Created by: seanmadsen*
This feature was there before my refactor, but I didn't carry it through since none of the books were actually using it.
The idea here is that in a book's yaml config file, one could specify a list of email ...*Created by: seanmadsen*
This feature was there before my refactor, but I didn't carry it through since none of the books were actually using it.
The idea here is that in a book's yaml config file, one could specify a list of email addresses which should always be notified during the publishing process. This list should be specified at the "language" level for a book.https://lab.civicrm.org/documentation/docs-publisher/-/issues/53Get details for email notifications from git repo instead of POST request2019-05-21T10:18:15ZSean ColsenGet details for email notifications from git repo instead of POST request*Created by: seanmadsen*
When a book is updated, GitHub sends a webhook POST request to the app. The app then uses data from that request to (1) publish a specific book if necessary, and (2) send notification emails. We should consider ...*Created by: seanmadsen*
When a book is updated, GitHub sends a webhook POST request to the app. The app then uses data from that request to (1) publish a specific book if necessary, and (2) send notification emails. We should consider the request as untrusted data. Therefor for step 2, we should not use data from the request. Instead, we should validate the `before` and `after` refs from the request and use them to inspect the git repo locally and glean whatever info (e.g. email addresses, commit messages, etc.) is necessary for sending this notification email.Docs Publisher 2019homotechsualhomotechsualhttps://lab.civicrm.org/documentation/docs-publisher/-/issues/52Better handle concurrent requests to publish the same edition2019-05-21T10:00:19ZSean ColsenBetter handle concurrent requests to publish the same edition*Created by: seanmadsen*
Deal more gracefully with the situation where a request to publish an edition is submitted while the system is currently in the process of publishing that edition.
Probably we want to create a lock file when ...*Created by: seanmadsen*
Deal more gracefully with the situation where a request to publish an edition is submitted while the system is currently in the process of publishing that edition.
Probably we want to create a lock file when beginning the publishing process and remove this file upon completion.
The second request could be queued or discarded. I think it's best to discard it.https://lab.civicrm.org/documentation/docs-publisher/-/issues/51User Guide - label 4.7 as "Latest" and 4.6 as "LTS"2017-10-19T15:00:17ZSean ColsenUser Guide - label 4.7 as "Latest" and 4.6 as "LTS"*Created by: seanmadsen*
This is in response to a request from @nganivet [here](https://github.com/civicrm/civicrm-docs/pull/49#issuecomment-286637921). Specific changes to implement:
* [x] Update [civicrm.org/documentation](https://ci...*Created by: seanmadsen*
This is in response to a request from @nganivet [here](https://github.com/civicrm/civicrm-docs/pull/49#issuecomment-286637921). Specific changes to implement:
* [x] Update [civicrm.org/documentation](https://civicrm.org/documentation) to change the URL for User Guide 4.7 from `/user/en/stable` to `/user/en/latest` **(Done by Sean on 2017-03-15)**
* [x] Redirect `user/en/stable` to `/user/en/latest` **(Blocked by #50)** *(Note: this change is just to avoid giving 404s to any "stable" URLs which still exist in the wild, since people have been using this URL for the 4.7 version of the User Guide for quite some time now. The term "stable" would not be displayed to any readers anywhere, but would just function as a redirect)*
* [x] Redirect `/user/en/lts` to `/user/en/4.6` **(Blocked by #50)**
* [x] When describing the available editions of the User Guide to readers, do it as follows: **(Blocked by #50 and #48 and #46 )** *(Note: these descriptors will eventually be used in the auto-generated book home pages, as well as at the top of every page within the book when the book is built with the Material theme customizations)*
* "English – 4.7 (Latest)"
* "English – 4.6 (LTS)"
* "françias (French) – 4.7 (Latest)"https://lab.civicrm.org/documentation/docs-publisher/-/issues/50Allow book versions to be defined in a more flexible way2017-10-19T14:59:04ZSean ColsenAllow book versions to be defined in a more flexible way*Created by: seanmadsen*
## Our current config schema
The way this app currently works, you define a book like this:
```yaml
name: User Guide
langs:
en:
repo: 'https://github.com/civicrm/civicrm-user-guide'
latest:...*Created by: seanmadsen*
## Our current config schema
The way this app currently works, you define a book like this:
```yaml
name: User Guide
langs:
en:
repo: 'https://github.com/civicrm/civicrm-user-guide'
latest: master
stable: 4.7
history:
- 4.6
```
## End-user example from Symfony
There are a number of limitations with our current approach. For the sake of time, I won't summarize them all. But I will offer an example of the end-user experience of choosing a version within [Symfony's documentation](http://symfony.com/doc/current/setup.html):
![screenshot13](https://cloud.githubusercontent.com/assets/42411/23837987/42fc6204-0756-11e7-8ddd-cbf115ca701b.png)
I like the way this works. When you click on "3.2 / Current" you're taken to `/doc/current/`. If you manually enter the URL `/doc/3.2/` you get redirected to `/doc/current`. Very slick. Very usable.
## Proposed new config schema
I want to change it to this:
```yaml
name: User Guide
default_language: en
languages:
en:
repo: 'https://github.com/civicrm/civicrm-user-guide'
default_version: 4.7
watchers:
- sean@seanmadsen.com
versions:
4.7: # This is stored as $slug
name: 4.7 / Current # If omitted, use $slug
path: current # If omitted, use a URL-safe version of $slug
branch: master # If omitted, use $slug
visible: true # If omitted, use "true"
maintained: true # If omitted, use "true"
redirects: # These are not displayed anywhere
- latest
4.6:
branch: 4.6
name: 4.6 / LTS
redirects:
- lts
4.5:
branch: 4.5
visible: false
maintained: false
```
Here's what each of those settings would actually *do*:
- `slug` - basically just used to organize the settings within the yaml file
- `name` - this is what the user sees when choosing between different versions, and at the top of every page when viewing the book
- `path` - this is used for the version component of the URL
- `branch` - tells us what branch to use in the git repo
- `visible` - when true, display this version in lists of different versions presented to the user. When false, don't display this version in navigation, but still allow its content to be published and viewed
- `maintained` - when this is false, display something like "(unmaintained)" to the user in various contexts such as choosing between different versions and (possible) when viewing the book
- `redirects` - when a user requests a URL with one of the listed redirects in place of `path`, the app will redirect the user to the correct URL that uses `path`. Both `branch` and `slug` will automatically function as redirects when their values differ from `path`.
I'm curious to hear feedback from others about this proposal. If I don't hear any concerns within 5 days, I'll begin the work to implement this proposal. https://lab.civicrm.org/documentation/docs-publisher/-/issues/44remove docs:serve command2017-04-30T22:16:43ZSean Colsenremove docs:serve command*Created by: seanmadsen*
Hey @totten I [see](https://github.com/civicrm/civicrm-docs/commit/5ea1844066be02463bc71042dd8521dcb2a8936c) that last year you added the `docs:serve` command. Do you use this command? I'm wondering if we can re...*Created by: seanmadsen*
Hey @totten I [see](https://github.com/civicrm/civicrm-docs/commit/5ea1844066be02463bc71042dd8521dcb2a8936c) that last year you added the `docs:serve` command. Do you use this command? I'm wondering if we can remove it. I've been doing a [whole bunch of refactoring](https://github.com/seanmadsen/civicrm-docs/commits/refactor-model) (so far just in my own fork) and when I came across `docs:serve` I said "hmm, this is a cool idea, but I'm not sure it's worth maintaining, given the ease of running `mkdocs serve`", so I [removed it](https://github.com/seanmadsen/civicrm-docs/commit/1e9da6a0e40e38ef3da4277ca85e183bec46eb31). It wouldn't be too hard for me to put it back, and update the code to conform to some of the changes in my refactor, but I wonder how you'd feel about dropping this functionality from civicrm-docs to keep the app's purpose more focused on publishing. https://lab.civicrm.org/documentation/docs-publisher/-/issues/42Set up theme customizations2017-04-30T22:16:43ZSean ColsenSet up theme customizations*Created by: seanmadsen*
*Created by: seanmadsen*
https://lab.civicrm.org/documentation/docs-publisher/-/issues/40better CSS for definition lists2017-04-30T22:16:43ZSean Colsenbetter CSS for definition lists*Created by: seanmadsen*
Some definition lists on the [extension life cycle](https://docs.civicrm.org/dev/en/master/extend-stages/#definitions) page. the `dt` items looked a little nicer with the readthedocs theme. Would be quick and us...*Created by: seanmadsen*
Some definition lists on the [extension life cycle](https://docs.civicrm.org/dev/en/master/extend-stages/#definitions) page. the `dt` items looked a little nicer with the readthedocs theme. Would be quick and useful to add a little style that makes the terms pop out a bit morehttps://lab.civicrm.org/documentation/docs-publisher/-/issues/39book publishing should fail more gracefully when page-building errors are enc...2018-11-08T04:23:30ZSean Colsenbook publishing should fail more gracefully when page-building errors are encountered*Created by: seanmadsen*
On 2017-02-04 we had a little snafu with the Dev Guide. Here's what happened, chronologically:
1. On 2017-02-03 Michael (at my request) took [these steps](https://github.com/squidfunk/mkdocs-material/issues/...*Created by: seanmadsen*
On 2017-02-04 we had a little snafu with the Dev Guide. Here's what happened, chronologically:
1. On 2017-02-03 Michael (at my request) took [these steps](https://github.com/squidfunk/mkdocs-material/issues/138#issuecomment-276567696) to install a more bleeding edge version of `pymdown-extensions` in attempt at addressing [civicrm-docs-material#7](https://github.com/civicrm/civicrm-docs-material/issues/7)
1. Shortly thereafter, I made [3 commits](https://github.com/civicrm/civicrm-dev-docs/commits/hooks-op-4), tested them locally with success (with the same version of `pymdown-extensions`), and then pushed them to the `hooks-op-4` branch of `civicrm-dev-docs`.
1. After this push, the docs system attempted to publish the `hooks-op-4` branch but encountered an error that I am fairly certain stemmed from a bug within `pymdown-extensions` and caused by the more bleeding edge version. Unfortunately I have not been able to reproduce this bug locally :slightly_frowning_face:
* The full error output is below
* Most notably: `ERROR - Error building page markdownrules.md`
1. Unfortunately MkDocs didn't handle this error gracefully because it built all the pages up to `markdownrules.md` (as listed in `mkdocs.yml`) and then stopped there without attempting to build any pages listed *after* `markdownrules.md`. (That's too bad, but seems like an upstream issue with MkDocs that we probably can't control so easily.)
* **(Issue A)**: What *would* be good, is if civicrm-docs could *attempt* to build a book, and then only make it live if the build process completes succesfully.
1. The especially bad (and strange) thing that happened here is that *somehow* the `latest` and `master` versions of the Dev Guide also ended up pointing to this botched branch of `hooks-op-4`.
* **(Issue B)**: The errors in publishing one branch ended up messing up another branch. I imagine we should be able to prevent this somehow.
1. Then on 2017-02-04 @ErichBSchulz [pointed out](https://chat.civicrm.org/civicrm/pl/rk56wfjx9bfqbj69hfw6fyfj3c) the problem with the master branch, and I tried to re-publish that branch (without publishing `hooks-op-4`) by loading `https://docs.civicrm.org/admin/publish/dev/en/master`
* **(Issue C)**: publishing the master branch with the URL above didn't work during this time when the Dev Guide was "messed up". I forget the exact error I got when I tried. Either 404 or 500 or 503.
1. Then I asked @bgm to do the following on the server, and these steps got the `master` branch up and running again:
```bash
cd /path/to/civicrm-docs/site
./bin/console docs:publish dev/en/master
```
Again a can't reproduce any of this locally, so I'm not sure how best to proceed here. Hoping that @michaelmcandrew will think of something.
## Full error given by the publishing system
(This was emailed to me)
```text
Running 'mkdocs build -c -f /var/www/civicrm-docs/var/repos/buildfiles/dev.en.master.yml -d /var/www/civicrm-docs/app/../web/static/dev/en/master'
INFO: mkdocs output: '
INFO - Cleaning site directory
INFO - Building documentation to directory: /var/www/civicrm-docs/web/static/dev/en/master
ERROR - Error building page markdownrules.md Traceback (most recent call last):
File "/usr/local/bin/mkdocs", line 11, in sys.exit(cli())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/mkdocs/__main__.py", line 156, in build_command ), dirty=not clean)
File "/usr/local/lib/python2.7/dist-packages/mkdocs/commands/build.py", line 379, in build build_pages(config, dirty=dirty)
File "/usr/local/lib/python2.7/dist-packages/mkdocs/commands/build.py", line 332, in build_pages dump_json)
File "/usr/local/lib/python2.7/dist-packages/mkdocs/commands/build.py", line 188, in _build_page site_navigation=site_navigation
File "/usr/local/lib/python2.7/dist-packages/mkdocs/commands/build.py", line 59, in convert_markdown extension_configs=config['mdx_configs']
File "/usr/local/lib/python2.7/dist-packages/mkdocs/utils/__init__.py", line 366, in convert_markdown html_content = md.convert(markdown_source)
File "/usr/local/lib/python2.7/dist-packages/markdown/__init__.py", line 368, in convert self.lines = prep.run(self.lines)
File "/usr/local/lib/python2.7/dist-packages/pymdown_extensions-1.8-py2.7.egg/pymdownx/superfences.py", line 484, in run lines = self.search_nested(lines)
File "/usr/local/lib/python2.7/dist-packages/pymdown_extensions-1.8-py2.7.egg/pymdownx/superfences.py", line 385, in search_nested self.eval(m, start, end)
File "/usr/local/lib/python2.7/dist-packages/pymdown_extensions-1.8-py2.7.egg/pymdownx/superfences.py", line 281, in eval self.process_nested_block(m, start, end)
File "/usr/local/lib/python2.7/dist-packages/pymdown_extensions-1.8-py2.7.egg/pymdownx/superfences.py", line 320, in process_nested_block code = entry["formatter"](self.rebuild_block(self.code), self.lang)
File "/usr/local/lib/python2.7/dist-packages/pymdown_extensions-1.8-py2.7.egg/pymdownx/superfences.py", line 423, in highlight lexer = guess_lexer(self.src) AttributeError: 'SuperFencesBlockPreprocessor' object has no attribute 'src'
```https://lab.civicrm.org/documentation/docs-publisher/-/issues/38"watch page" feature2017-04-30T22:16:43ZSean Colsen"watch page" feature*Created by: seanmadsen*
It'd be great if a person choose to get email notifications any time someone else changed a page, kind of like in a wiki. *Created by: seanmadsen*
It'd be great if a person choose to get email notifications any time someone else changed a page, kind of like in a wiki. https://lab.civicrm.org/documentation/docs-publisher/-/issues/36How do we include theme customizations in this repo?2017-04-30T22:16:43ZSean ColsenHow do we include theme customizations in this repo?*Created by: seanmadsen*
So, we have a separate repository, [civicrm-docs-material](https://github.com/civicrm/civicrm-docs-material) for the theme customizations which was part of my [plan](https://github.com/civicrm/civicrm-docs/issue...*Created by: seanmadsen*
So, we have a separate repository, [civicrm-docs-material](https://github.com/civicrm/civicrm-docs-material) for the theme customizations which was part of my [plan](https://github.com/civicrm/civicrm-docs/issues/15#issuecomment-276156953), but now I'm doubting this choice a bit, after talking with @michaelmcandrew today. I'd like some advice on the best way to proceed here, especially from @totten
The theme customizations need to be accessible (and current)
* (A): on www-prod when publishing books
* (B): locally for theme developers to use with `mkdocs serve`
* (C): locally for civicrm-docs developers after doing `civibuild docs`
So the question is, how do we store the theme customizations to best meet all these needs. Here are ideas that emerged from my convo with Michael
1. `civicrm-docs-material` as a git submodule within `civicrm-docs`
1. theme customizations stored within `civicrm-docs` directly (and we get rid of the separate repo)
1. something else?
I don't have an opinion. Do others? (Sorry if it was a bit premature to create that new repo and add 6 issues within it! :grimacing: )https://lab.civicrm.org/documentation/docs-publisher/-/issues/35Restrict web crawler bots to crawl only one branch per book?2019-05-21T10:12:45ZSean ColsenRestrict web crawler bots to crawl only one branch per book?*Created by: seanmadsen*
Sometimes I am searching for docs in other projects and the google results take me to a version-specific docs page, which sometimes is outdated. Other times google will display separate results for separate vers...*Created by: seanmadsen*
Sometimes I am searching for docs in other projects and the google results take me to a version-specific docs page, which sometimes is outdated. Other times google will display separate results for separate versions, both with the same content, which is somewhat confusing. Also my limited understanding of SEO tells me that this is bad SEO, too.
I wonder if it would be easy/possible/desirable for us to make civicrm-docs generate a `robots.txt` file or something else that will tell search engines to only index *one* version of each book. Maybe "latest", maybe "stable".
This issue will only become relevant once the auto-generated docs home page is in place.
Low priority. Just a thought. Docs Publisher 2019homotechsualhomotechsual