= Asciidoctor::DocTest
Jakub Jirutka <https://github.com/jirutka[@jirutka]>
:page-layout: base
:idprefix:
ifdef::env-github[:idprefix: user-content-]
:idseparator: -
:source-language: ruby
:language: {source-language}
// custom
:gem-name: asciidoctor-doctest
:gh-name: asciidoctor/{gem-name}
:gh-branch: master
:badge-style: flat
:doctor-man-uri: http://asciidoctor.org/docs/user-manual
:rawgit-base: https://cdn.rawgit.com/{gh-name}/master
:src-base: lib/asciidoctor/doctest

image:https://img.shields.io/travis/{gh-name}/{gh-branch}.svg?style={badge-style}[Build Status, link="https://travis-ci.org/{gh-name}"]
image:https://img.shields.io/codeclimate/coverage/github/{gh-name}.svg?style={badge-style}[Test Coverage, link="https://codeclimate.com/github/{gh-name}"]
image:https://img.shields.io/codeclimate/github/{gh-name}.svg?style={badge-style}[Code Climate, link="https://codeclimate.com/github/{gh-name}"]
image:https://inch-ci.org/github/{gh-name}.svg?branch={gh-branch}&style={badge-style}[Inline docs, link="http://inch-ci.org/github/{gh-name}"]
image:https://img.shields.io/gem/v/{gem-name}.svg?style={badge-style}[Gem Version, link="https://rubygems.org/gems/{gem-name}"]
image:https://img.shields.io/badge/yard-docs-blue.svg?style={badge-style}[Yard Docs, link="http://www.rubydoc.info/github/{gh-name}/frames"]

DocTest is a tool for end-to-end testing of Asciidoctor _backends_ based on comparing of textual output.

It provides a collection of categorized <<input-examples, input _examples_>> (documents in AsciiDoc syntax) to simplify and systematize writing tests for new backends.
You just write or <<generate-examples, generate>> the expected output, i.e. what the backend should produce for the given input.

image::{rawgit-base}/doc/img/doctest-diag.svg[diagram, align="center"]

Each example should be focused on one use case, so when writing a new backend, you can incrementally implement new features following the reference input examples.
However, they are not strictly isolated like unit tests.
For example, if you change a format of a paragraph, it may affect a variety of other examples.

When test fails, DocTest prints a nicely formatted diff of the expected and actual output (see <<run-tests>>), so you see exactly what went wrong.
Insignificant differences, such as attributes order in HTML, are ignored.

DocTest supports HTML-based backends and can be easily <<how-to-extend-it, extended>> to support any other backend with textual output.


== Setup DocTest

Let’s say that you’re developing a new shiny HTML template-based backend named “shiny” and assume that you have templates in the directory `data/templates`.

. Create a directory for your output examples:
+
[source, sh]
mkdir -p test/examples/shiny
+
and optionally a directory for your extra input examples:
+
[source, sh]
mkdir -p test/examples/asciidoc

. Add development dependency on `asciidoctor-doctest` to your gemspec:
+
[source]
s.add_development_dependency 'asciidoctor-doctest', '~> 1.5.2'
+
or Gemfile if you’re not distributing the backend as a gem:
+
[source]
gem 'asciidoctor-doctest', '~> 1.5.2'
+
and run `bundle install`.

. Create or edit `test/test_helper.rb`; require test dependencies and setup `examples_path`:
+
[source]
----
require 'asciidoctor/doctest'
require 'minitest/autorun'

# used to colorize output
require 'minitest/rg'

# needed if you're testing templates-based backend
require 'tilt'

# extra input examples (optional)
DocTest.examples_path.unshift 'test/examples/asciidoc'

# output examples
DocTest.examples_path.unshift 'test/examples/shiny'
----

. Create test file `test/templates_test.rb` and extend class link:{src-base}/test.rb[DocTest::Test]. Specify `converter_opts` and then call `generate_tests!` macro with a specific subclass of `BaseExamplesSuite`:
+
[source]
----
require 'test_helper'

class TestTemplates < DocTest::Test
  converter_opts template_dirs: 'data/templates'
  generate_tests! DocTest::HTML::ExamplesSuite
end
----

. Create or edit `Rakefile`; add tasks to run tests and optionally a generator of output examples:
+
[source]
----
require 'asciidoctor/doctest'
require 'rake/testtask'
require 'thread_safe'
require 'tilt'

Rake::TestTask.new(:test) do |task|
  task.description = 'Run tests for templates'
  task.pattern = 'test/templates_test.rb'
  task.libs << 'test'
end

DocTest::GeneratorTask.new(:generate) do |task|
  task.output_suite = DocTest::HTML::ExamplesSuite.new(examples_path: 'test/examples/shiny')
  task.converter_opts[:template_dirs] = 'data/templates'
  #
  # add extra input examples (optional)
  task.examples_path.unshift 'test/examples/asciidoc'
end

# When no task specified, run test.
task :default => :test
----


== Run tests

Assume that you have defined the test Rake task named `:test` (see above).
Then you can simply run:

[source, sh]
bundle exec rake test

image::doc/img/failing-test-term.gif[Failing test in term, align="center"]


== Examples

Test _example_ is just a document fragment in AsciiDoc syntax (a reference input), or the backend’s target syntax (an expected output).
Example should consider one case of the generated output, i.e. it should reflect one code branch in a converter or template.
Examples are grouped in _example groups_.
Each group focuses on one block or inline element — more precisely Asciidoctor’s AST node (paragraph, table, anchor, footnote…).

Examples group is a text file named similar to Asciidoctor templates, i.e. the AST node name with an extension according to syntax, for example `block_table.adoc`, `block_table.html`. See below for a list of the AST nodes.
Individual examples in the group file are separated by a special header with the name of the example, an optional description and options.

Each example is identified by its name and the group name like this: `{group_name}:{example_name}` (e.g. `block_table:with_title`).
Input and output examples are paired — for every input example there should be an output example with the same identifier.
When you <<run-tests, run tests>>, the input example is converted using the tested backend (or templates) and its actual output is compared with the expected output example.

[horizontal]
.List of Asciidoctor’s AST nodes
document:: TODO
embedded:: TODO
section:: {doctor-man-uri}/#sections[document sections], i.e. headings
block_admonition:: {doctor-man-uri}/#admonition[an admonition block]
block_audio:: {doctor-man-uri}/#audio[an audio block]
block_colist:: {doctor-man-uri}/#callouts[a code callouts] list
block_dlist:: {doctor-man-uri}/#labeled-list[a labeled list] (aka definition list) and {doctor-man-uri}/#question-and-answer-style-list[a Q&A style list]
block_example:: {doctor-man-uri}/#example[an example block]
block_floating_title:: {doctor-man-uri}/#discrete-or-floating-section-titles[a discrete or floating section title]
block_image:: {doctor-man-uri}/#images[an image block]
block_listing:: {doctor-man-uri}/#listing-blocks[a listing and source code block]
block_literal:: {doctor-man-uri}/#literal-text-and-blocks[a literal block]
block_olist:: {doctor-man-uri}/#ordered-lists[an ordered list] (i.e. numbered list)
block_open:: {doctor-man-uri}/#open-blocks[open blocks], {doctor-man-uri}/#user-abstractabstract[abstract], …
block_outline:: an actual {doctor-man-uri}/#user-toc[TOC] content (i.e. list of links), usually recursively called
block_page_break:: {doctor-man-uri}/#page-break[page break]
block_paragraph:: {doctor-man-uri}/#paragraph[a paragraph]
block_pass:: {doctor-man-uri}/#pass-bl[a passthrough block]
block_preamble:: {doctor-man-uri}/#doc-preamble[a preamble], optionally with a TOC
block_quote:: {doctor-man-uri}/#quote[a quote block]
block_sidebar:: {doctor-man-uri}/#sidebar[a sidebar]
block_stem:: {doctor-man-uri}/#stem[a STEM block] (Science, Technology, Engineering and Math)
block_table:: {doctor-man-uri}/#tables[a table]
block_thematic_break:: {doctor-man-uri}/#horizontal-rules[a thematic break] (i.e. horizontal rule)
block_toc:: {doctor-man-uri}/#manual-placement[a TOC macro] (i.e. manually placed TOC); This block is used for `toc::[]` macro only and it’s responsible just for rendering of a the TOC “envelope,” not an actual TOC content.
block_ulist:: {doctor-man-uri}/#unordered-lists[an unordered list] (aka bullet list) and a {doctor-man-uri}/#checklist[checklist] (e.g. TODO list)
block_verse:: {doctor-man-uri}/#verse[a verse block]
block_video:: {doctor-man-uri}/#video[a video block]
inline_anchor:: {doctor-man-uri}/#url[anchors] (links, cross references and bibliography references)
inline_break:: {doctor-man-uri}/#line-breaks[line break]
inline_button:: {doctor-man-uri}/#ui-buttons[UI button]
inline_callout:: {doctor-man-uri}/#callouts[code callout] icon/mark inside a code block
inline_footnote:: {doctor-man-uri}/#user-footnotes[footnote]
inline_image:: {doctor-man-uri}/#images[inline image] and {doctor-man-uri}/#inline-icons[inline icon]
inline_kbd:: {doctor-man-uri}/#keyboard-shortcuts[keyboard shortcut]
inline_menu:: {doctor-man-uri}/#menu-selections[menu section]
inline_quoted:: {doctor-man-uri}/#quotes[text formatting]; emphasis, strong, monospaced, superscript, subscript, curved quotes and inline STEM


=== Input examples

DocTest provides a collection of the reference input examples that should be suitable for most backends.
You can find them in link:data/examples/asciidoc[].footnote:[Since GitHub implicitly renders them as a plain AsciiDoc, you must view a Raw source if you want to see what’s going on there.]
There are a lot of test examples and some of them may not be relevant to your backend — that’s okay, when some output example is not found, it’s marked as skipped in test.

You can also write your own input examples and use them together with those provided (or replace them).
Just add another directory to your examples_path (e.g. `test/examples/asciidoc`) and create example group files with `.adoc` suffix here (e.g. `block_video.adoc`).
When DocTest is looking for examples to test, it indexes all examples found in files with `.adoc` suffix on the examples_path.
If there are two files with the same name, it simply merges their content, and if they contain two examples with the same name, then the first wins (i.e. that from the file that is ahead on the examples_path).

==== Format

[source, asciidoc]
----
// .first_example
// Each block must be preceded by a header (comment); the first line must
// contain the example’s name prefixed with a dot. This text is interpreted
// as a description.
The example’s content in *Asciidoc*.

NOTE: The trailing new line (below this) will be removed.

// .second_example
* List item level 1
** List item level 2

----

=== HTML-based examples

HtmlTest assumes that paragraphs are enclosed in `<p></p>` tags and implicitly sets the _include_ option to `./p/node()` for `inline_*:*` examples (if _include_ is not already set).
If it’s not your case, then you must overwrite it:

[source]
----
class TestShiny < DocTest::Test
  converter_opts template_dirs: 'data/templates'
  generate_tests! DocTest::HTML::ExamplesSuite.new(paragraph_xpath: './div/p/node()')
end
----

==== Options

List of options that can be set in the header of HTML example.

include::
  XPath expression that specifies a subsection of the document that should be
  compared (asserted). Default is `./p/node()` for `inline_*:*` groups and
  empty (i.e. `.`) for others.
exclude::
  XPath expression that specifies parts of the document that should _not_ be
  compared (asserted). Always start the expression with a dot (e.g. `.//h1`).
  This option may be used multiple times per example.
header_footer::
  Option for Asciidoctor to render a full document (instead of embedded).
  This is default for `document:*` group.

==== Format

[source, html]
----
<!-- .first_example
  Each example must be preceded by a header (comment); the first line must
  contain the example’s name prefixed with a dot. This text is interpreted
  as a description.
-->
<p>The example’s content in <strong>HTML</strong>.</p>

<div class="note">The trailing new line (below this) will be removed.</div>

<!-- .second_example
  You may also specify options for comparing or Asciidoctor renderer. Option
  line starts with a semicolon, then comes the option name ended by a
  semicolon and after that the option’s value (may be omitted for boolean
  options).
  :option_1: value 1
  :option_2: value 1
  :option_2: value 2
  :boolean_option:
-->
<div class="colist">
  <ol>
    <li>Method signature</li>
    <li>Some stuff inside</li>
    <li>Return statement</li>
  </ol>
</div>

----


=== Generate examples

Writing examples of an expected output for all the input examples from scratch is quite a chore.
Therefore DocTest provides a generator.
When you have at least partially working Asciidoctor _backend_ (converter or a set of templates), you can pass the input examples through it and generate your output examples.
Then you should verify them and modify if needed.

Assume that you have defined the generator Rake task named `:generator` (see <<setup-doctest>>).

Now you can generate output examples from all the input examples (those with `.adoc` extension) found on the examples_path that doesn’t already exist (i.e. it doesn’t rewrite existing):

[source, sh]
bundle exec rake generate

Same as previous, but rewrite existing tested examples:

[source, sh]
bundle exec rake generate FORCE=yes

Generate just examples for `block_ulist` node (i.e. all examples in `block_ulist.adoc` file(s) found on the examples_path) that doesn’t exist yet:

[source, sh]
bundle exec rake generate PATTERN='block_ulist:*'

(Re)generate examples which name starts with `basic` for all _block_ nodes (i.e. files that starts with `block_`):

[source, sh]
bundle exec rake generate PATTERN='block_*:basic*' FORCE=yes


== How to extend it

You can extend DocTest to support any textual format you want.
All what you need is to subclass link:{src-base}/base_examples_suite.rb[BaseExamplesSuite] and usually also link:{src-base}/base_example.rb[BaseExample].


== Contributing

. Fork it
. Create your feature branch (`git checkout -b my-new-feature`)
. Commit your changes (`git commit -am 'Add some feature'`)
. Push to the branch (`git push origin my-new-feature`)
. Create new Pull Request

== License

This project is licensed under http://opensource.org/licenses/MIT/[MIT License]. For the full text of the license, see the link:LICENSE[LICENSE] file.