How to generate docs from multiple repositories

Some static site generators sell multi-repository functionality as a feature. This feature is mainly achieved through a built-in git client.

Since we almost always work on systems which already have git installed, docToolchain does not come with its own git client.

The solution we propose instead is just a simple bash script which does the magic for you. Here is how.

Imagine you have a documentation repository set up with the docToolchain wrapper dtcw and some code documents. Now you would like to include the documents from another repository.

git clone solution

To get the contents of the other repository, one way is to do a git clone of it right to the build folder of your own repository. Let’s call the script in which we store this command clonerefs.sh, because we clone a reference to the remote documentation.

clonerefs.sh
#!/usr/bin/env bash
git clone git@github.com:docToolchain/docToolchain.git build/refs/docToolchain

this works fine if you execute it once, but the second time it will complain that the folder build/refs/docToolchain is not empty. So let’s create a cloneOrPull function which tries first to clone the repo and pulls an update if the clone fails.

clonerefs.sh
#!/usr/bin/env bash
function cloneOrPull {
  echo ""
  echo $1
  (git clone "$1" "$2" 2> /dev/null && echo "cloned repo" )|| git -C "$2" pull
}

cloneOrPull git@github.com:docToolchain/docToolchain.git build/refs/docToolchain

That’s better. We can now include the docs from our main docs.

To do so, we still need a little Trick. We need to reference the docs in the build folder.

We could do something like include::../../../build/refs/docToolchain/somefile.adoc[] but this will break. For this, docToolchain sets the attribute projectRootDir. With this, we can write the include as include::/opt/build/repo/build/refs/docToolchain/somefile.adoc[].

In order to get projectRootDir also in your local preview, create a file called .asciidoctorconfig in the root folder of your project and add

:projectRootDir: {asciidoctorconfigdir}

as content.

But what if we don’t want to include them but just let them render? We could clone the repository directly to our src/docs folder. But that would require that it only contains docs and no src/docs folder itself. So we have to copy it over to our src/docs folder.

clonerefs.sh
#!/usr/bin/env bash
function cloneOrPull {
  echo ""
  echo $1
  (git clone "$1" "$2" 2> /dev/null && echo "cloned repo" )|| git -C "$2" pull
}

cloneOrPull git@github.com:docToolchain/docToolchain.git build/refs/docToolchain
cp build/refs/docToolchain/src/docs/manual src/docs/.

But we don’t want to add these folders to our main repository, so let’s add it to the .gitignore file.

.gitignore
[...]
src/docs/manual
[...]

There is one more thing we can optimize. Currently, the script clones the repository with the full history. This is far too much traffic if you only want to use the latest version of your docs. Let’s add --depth 1 to the `git clone`command to only fetch the latest version.

clonerefs.sh
#!/usr/bin/env bash
function cloneOrPull {
  echo ""
  echo $1
  (git clone --depth 1 "$1" "$2" 2> /dev/null && echo "cloned repo" )|| git -C "$2" pull
}

cloneOrPull git@github.com:docToolchain/docToolchain.git build/refs/docToolchain
cp build/refs/docToolchain/src/docs/manual src/docs/.

This script will now let you clone remote repositories and merge them with your main documentation before building.

git submodule solution

Another solution is to reference your sub-repositories as git submodules. Git submodules are pointers in your main repository to a certain version of another repository. Most CI/CD systems clone your repository together with all the configured submodules. This makes this approach quite convenient.

A drawback is that submodules are not often used and thus developers are not used to them.

artifact solution

A third solution could be that sub-repositories publish their docs as zip file somewhere. Maybe to an artifactory instance. Your main repository could fetch the published artifacts and use them in the build process. That would be exactly what you do with code.

Use Antora integration

The Antora integration is currently in beta and we are happy to get your feedback.

If you use Antora to build your docs, docToolchain provides a special Antora integration. Essentially docToolchains task downloadTemplate is now capable to set up your project as Antora module and still leverage the various docToolchain features. This enables you to register your docToolchain project to an existing Antora playbook. The Antora integration currently supports arc42 and req42 templates out of the box. If you want to use your own template, you need to ensure the directory follows the default Antora structure. Docs are currently expected to be in ${inputPath}/modules/ROOT/.

Include content

Antora comes along with some custom behaviour and non-standard asciidoc syntax. Read the following sections to learn how to include content.

Images

Images are expected to be within assets/images.

Other files

In order to include other files, eg. generated output from docToolchain tasks like collectIncludes, you must ensure there is a symlink from the docToolchain output directory to ${inputPath}/modules/ROOT/examples/<outputFile/folder>.