Usage¶
Sybil
works by discovering a series of
documents
as part of the
test runner integration. These documents are then
parsed into a set of non-overlapping
regions
. When the tests are run, each Region
is turned into an Example
that is evaluated in the document’s
namespace
. The examples are evaluated
in the order in which they appear in the document.
If an example does not evaluate as expected, a test failure is reported and Sybil
continues on to evaluate the remaining
examples
in the
Document
.
Test runner integration¶
Sybil aims to integrate with all major Python test runners. Those currently catered for explicitly are listed below, but you may find that one of these integration methods will work as required with other test runners. If not, please file an issue on GitHub.
To show how the integration options work, the following documentation examples
will be tested. They use doctests,
code blocks
and require a temporary directory:
A fairly pointless function:
.. code-block:: python
import sys
def write_message(filename, message):
with open(filename, 'w') as target:
target.write(message)
Now we can use a doctest REPL to show it in action:
>>> write_message('test.txt', 'is this thing on?')
>>> with open('test.txt') as source:
... print(source.read())
is this thing on?
pytest¶
To have pytest check the examples, Sybil makes use of the
pytest_collect_file
hook. To use this, configuration is placed in
a confest.py
in your documentation source directory, as shown below.
pytest
should be invoked from a location that has the opportunity to
recurse into that directory:
from os import chdir, getcwd
from shutil import rmtree
from tempfile import mkdtemp
import pytest
from sybil import Sybil
from sybil.parsers.codeblock import PythonCodeBlockParser
from sybil.parsers.doctest import DocTestParser
@pytest.fixture(scope="module")
def tempdir():
# there are better ways to do temp directories, but it's a simple example:
path = mkdtemp()
cwd = getcwd()
try:
chdir(path)
yield path
finally:
chdir(cwd)
rmtree(path)
pytest_collect_file = Sybil(
parsers=[
DocTestParser(),
PythonCodeBlockParser(future_imports=['print_function']),
],
pattern='*.rst',
fixtures=['tempdir']
).pytest()
The file glob passed as pattern
should match any documentation source
files that contain examples which you would like to be checked.
As you can see, if your examples require any fixtures, these can be requested
by passing their names to the fixtures
argument of the
Sybil
class.
These will be available in the Document
namespace
in a way that should feel natural
to pytest
users.
The setup
and teardown
parameters can still be used to pass
Document
setup and teardown callables.
The path
parameter, however, is ignored.
Note
pytest provides its own doctest plugin, which can cause problems. It should be disabled by including the following in your pytest configuration file:
[pytest]
addopts = -p no:doctest
unittest¶
To have Test Discovery check the example, Sybil makes use of
the load_tests protocol. As such, the following should be placed in a test
module, say test_docs.py
, where the unit test discovery process can find it:
from os import chdir, getcwd
from shutil import rmtree
from tempfile import mkdtemp
from sybil import Sybil
from sybil.parsers.codeblock import PythonCodeBlockParser
from sybil.parsers.doctest import DocTestParser
def sybil_setup(namespace):
# there are better ways to do temp directories, but it's a simple example:
namespace['path'] = path = mkdtemp()
namespace['cwd'] = getcwd()
chdir(path)
def sybil_teardown(namespace):
chdir(namespace['cwd'])
rmtree(namespace['path'])
load_tests = Sybil(
parsers=[
DocTestParser(),
PythonCodeBlockParser(future_imports=['print_function']),
],
path='../docs', pattern='*.rst',
setup=sybil_setup, teardown=sybil_teardown
).unittest()
The path
parameter gives the path, relative to the file containing this
code, that contains the documentation source files.
The file glob passed as pattern
should match any documentation source
files that contain examples which you would like to be checked.
Any setup or teardown necessary for your tests can be carried out in
callables passed to the setup
and teardown
parameters,
which are both called with the Document
namespace
.
The fixtures
parameter is ignored.
Parsers¶
Sybil parsers are what extract examples from source files and turns them into parsed examples with evaluators that can check if they are correct. The parsers available depend on the source language of the files containing the examples you wish to check:
For ReStructured Text, typically
.rst
or.txt
files, see ReST Parsers.For Markdown, typically
.md
files, CommonMark, GitHub Flavored Markdown and MyST, along with other flavours, are supported.For Python source code, typically
.py
files, it depends on the markup used in the docstrings; both the ReST parsers and MyST parsers will work. The source files are presented asPythonDocument
instances that import the document’s source file as a Python module, making names within it available in the document’snamespace
.
It’s also relatively easy to develop your own parsers as shown in the section below.
Developing your own parsers¶
Sybil parsers
are callables that take a
Document
and yield a sequence of regions. A Region
contains
the character position of the start
and end
of the example in the document’s
text
, along with a parsed
version of the
example and a callable evaluator
.
Parsers are free to access any documented attribute of the Document
although
will most likely only need to work with text
.
The namespace
attribute should not be modified.
The parsed
version can take any form and only needs to be understood by the
evaluator
.
That evaluator will be called with an
Example
constructed from the
Document
and the Region
and should return a false value if the example is as expected. Otherwise, it should
either raise an exception or return a textual description in the
event of the example not being as expected. Evaluators may also
modify the document’s namespace
or push
and
pop
evaluators.
Example
instances are used to wrap up
all the attributes you’re likely to need when writing an evaluator and all
documented attributes are fine to use. In particular,
parsed
is the parsed value provided by the parser
when instantiating the Region
and
namespace
is a reference to the document’s
namespace. Evaluators are free to modify the
namespace
if they need to.
If you need to write your own parser, you should consult the API Reference so see if suitable
lexers
already exist for the source language containing your examples.
Failing that, parsers quite often use regular expressions to extract the text for examples
from the document. There’s no hard requirement for this, but if you find you need to, then
find_region_sources()
may be of help.
Worked example¶
As an example, let’s look at a parser suitable for evaluating bash commands in a subprocess and checking the output is as expected:
.. code-block:: bash
$ echo hi there
hi there
Since this is a ReStructured Text code block, the simplest thing we could do would be to use the existing support for other languages:
from subprocess import check_output
from sybil import Sybil
from sybil.parsers.rest import CodeBlockParser
def evaluate_bash_block(example):
command, expected = example.parsed.strip().split('\n')
assert command.startswith('$ ')
command = command[2:].split()
actual = check_output(command).strip().decode('ascii')
assert actual == expected, repr(actual) + ' != ' + repr(expected)
parser = CodeBlockParser(language='bash', evaluator=evaluate_bash_block)
sybil = Sybil(parsers=[parser], pattern='*.rst')
Another alternative would be to start with the
lexer for ReST directives
.
Here, the parsed version consists of a tuple of the command to run and the expected output:
from subprocess import check_output
from typing import Iterable
from sybil import Sybil, Document, Region, Example
from sybil.parsers.rest.lexers import DirectiveLexer
from subprocess import check_output
def evaluate_bash_block(example: Example):
command, expected = example.parsed
actual = check_output(command).strip().decode('ascii')
assert actual == expected, repr(actual) + ' != ' + repr(expected)
def parse_bash_blocks(document: Document) -> Iterable[Region]:
lexer = DirectiveLexer(directive='code-block', arguments='bash')
for lexed in lexer(document):
command, output = lexed.lexemes['source'].strip().split('\n')
assert command.startswith('$ ')
parsed = command[2:].split(), output
yield Region(lexed.start, lexed.end, parsed, evaluate_bash_block)
sybil = Sybil(parsers=[parse_bash_blocks], pattern='*.rst')
Finally, the parser could be implemented from scratch, with the parsed version again consisting of a tuple of the command to run and the expected output:
from subprocess import check_output
import re, textwrap
from sybil import Sybil, Region
BASHBLOCK_START = re.compile(r'^\.\.\s*code-block::\s*bash')
BASHBLOCK_END = re.compile(r'(\n\Z|\n(?=\S))')
def evaluate_bash_block(example):
command, expected = example.parsed
actual = check_output(command).strip().decode('ascii')
assert actual == expected, repr(actual) + ' != ' + repr(expected)
def parse_bash_blocks(document):
for start_match, end_match, source in document.find_region_sources(
BASHBLOCK_START, BASHBLOCK_END
):
command, output = textwrap.dedent(source).strip().split('\n')
assert command.startswith('$ ')
parsed = command[2:].split(), output
yield Region(start_match.start(), end_match.end(),
parsed, evaluate_bash_block)
sybil = Sybil(parsers=[parse_bash_blocks], pattern='*.rst')