Python doctest Module | Document and Test Code

In this tutorial, we will learn about the doctest module. It is a testing framework that helps us to document and test code simultaneously. This module allows us to document and test our code, which is essential to coding. By default, we can use the docstring to write the class description or function to provide a better understanding of the code.

We won't need to install any third party to use the doctest module. However, one should have a basic understanding of Python.

Documentation Example

Documenting the code is the best practice, and the senior developer suggests to follow this practice often. It would be right to say coding and its documentation are equally important. Python offers various methods to document a project, application, module, or script. External documentation is typically needed for larger projects, while smaller projects can use descriptive names, comments, and docstrings for documentation purposes.

It's crucial to keep to a standard structure and write docstrings. Principles for writing docstrings are provided in the Python documentation, and several third-party tools and packages can assist in enforcing these principles. The numpydoc style, used to document scientific Python projects, is one well-liked docstring format. It is based on reStructuredText and offers a consistent technique to clearly and concisely document functions, classes, and methods. It might be simpler for other developers to understand how to use and interact with your code if you utilize a standard docstring format.

Lets See an Example which shows how to effectively use Comments in a programme:

In this illustration, comments are used to describe each code component. A docstring that describes the function's purpose, the parameters it accepts, and the expected output come before the function itself. After initializing the first two values of the sequence, the function's code uses a while loop to generate the remainder of the series up to the maximum number of n. The function then returns the sequence as a list.

We use the input value of 100 to execute the Fibonacci algorithm outside and report the outcome. It is simpler for other developers to comprehend what the code does and how it operates when comments are used to clarify it.

Drawbacks of Comments

• Comments can become outdated: If the code is changed, but the comments need to be updated, the comments can no longer correctly represent what the code is doing. This might be deceptive and confusing for other developers who rely on the comments to understand the code.
• Comments can be redundant: Sometimes, comments merely repeat information that is clear from the code itself. This might make the code cluttered and more difficult to read.
• Comments can be unclear or ambiguous: Comments can be imprecise or poorly worded, which might lead to confusion and misunderstandings. Code flaws and mistakes may result from this.
• Comments can be ignored: Other developers may dismiss them if there are too many or too many lengthy comments. This may cause crucial information to be missed.
• Comments may be a crutch; if they are used too frequently, other developers might not make an effort to comprehend the code thoroughly. This could result in a lack of understanding and bad coding techniques.

Although comments may help clarify code and make it simpler to comprehend, it's crucial to use them sparingly and ensure that they are correct, clear, and required.

Documentation strings are a very useful feature of Python that can help us to document our code as we code. And it has an advantage over the comments because the interpreter doesn't ignore them.

The docstring is the active part of the code; we can access it at runtime. Python provides __doc__ special attributes on our packages, modules, classes, methods, and functions.

Python allows for the inclusion of docstrings in packages, modules, classes, methods, and functions. To write effective docstrings, it is recommended to follow the conventions and guidelines outlined in PEP 257.

Introduction to Doctest Module

Developers may create tests and check code examples embedded into docstrings using the built-in testing structures provided by Python's doctest module. It offers a practical method to concurrently document and test code, guaranteeing that the code examples contained in docstrings deliver the desired results.

The doctest module's primary goal is to encourage developers to provide executable code samples in their docstrings to advance best practices for documentation. These program examples show how to utilize certain functions, classes, or modules while automatically confirming their accuracy. They act as both documentation and test cases.

The doctest package finds interactive Python sessions within docstrings by looking for the >>> prompt. It runs the code in these sessions and contrasts the results with what was anticipated and stated in the docstring. The test is deemed successful if the results line up; otherwise, a failure is noted.

One benefit of utilizing doctest is that the tests are integrated into the documentation, making it simpler to maintain correct and current content. To check that the code examples in the docstrings still generate the correct results after code modifications, the doctest may be executed. Doing so keeps the code reliable and possible declines are caught.

Since the doctest module is a built-in component of Python, it has no external dependencies. It is appropriate for testing small to medium-sized codebases and is lightweight and simple. Other testing frameworks, such as unit test or pytest, could be better appropriate for larger, more complicated programs.

Writing doctest Tests in Python

We have got the understanding of the doctest so far. Now, we will learn to check how to return value of the functions, methods, and other callable. We will also learn how to create test cases for code. The most common use case of the testing is checking the return value of functions, methods and other callables. In the following example, we have a function multiply(a, b) that accepts two arguments and return the product of two values.

You may take the following actions to test this method using doctest:

• Import the doctest module.
• To run the tests, call the testmod() function:
• The testmod() function automatically locates and runs the tests included in the method's docstrings. It reports faults or mistakes after comparing the output with the anticipated outcomes.
• If you run the code in this example, the doctest module will run the code examples included in the multiply function's docstring. No output will be shown if all tests pass. However, doctest will raise an exception and offer information about the failed test if any test fails.
• The Python interactive shell may be used to run the code and view the test results, or you can run the complete script.
• This method of doctest guarantees that the code examples in the docstring continue to be valid and generate the desired results. It assists in maintaining proper documentation and works as a testing tool for the code.

Output:

TestResults(failed=0, attempted=3)

This result shows that each of the three tests was successful. It indicates that there were no failures or problems and that the code examples in the docstring delivered the desired results.

The output would have included information on the failed test, the predicted output, and the actual result if any of the tests had failed.

Some More Examples to Understand it Better

Example 1: Testing a Function with Multiple Test Cases

Output:

Trying:
    is_even(4)
Expecting:
    True
ok
Trying:
    is_even(7)
Expecting:
    False
ok
Trying:
    is_even(0)
Expecting:
    True
ok
1 items had no tests:
    __main__
1 items passed all tests:
   3 tests in __main__.is_even
3 tests in 2 items.
3 passed and 0 failed.
Test passed.

Explanation:

• The is_even() function takes a number as input and checks if it is even.
• The if __name__ == "__main__": block ensures that the doctests are run only when the script is executed directly.
• The doctest verifies three test cases:
• is_even(4) should return True, as 4 is an even number.
• is_even(7) should return False, as 7 is not an even number.
• is_even(0) should return True, as 0 is considered an even number.
• The output shows that all three tests failed. The is_even() function returned the expected results, but the test cases were marked as failures. This could be due to a mismatch in the expected and actual outputs or incorrect implementation of the function.

Example 2: Testing a Function with Complex Outputs

Output:

Trying:
    reverse_list([1, 2, 3, 4])
Expecting:
    [4, 3, 2, 1]
ok
Trying:
    reverse_list(['a', 'b', 'c'])
Expecting:
    ['c', 'b', 'a']
ok
1 items had no tests:
    __main__
1 items passed all tests:
   2 tests in __main__.reverse_list
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

Explanation

• The program defines a function reverse_list() that reverses a given list.
• The docstring of the function contains two test cases with expected results using the >>> notation.
  • The reverse_list() function takes a list as input and returns the reversed version of the list.
  • The doctest checks two test cases:
  • reverse_list([1, 2, 3, 4]) should return [4, 3, 2, 1], as the list is reversed.
  • reverse_list(['a', 'b', 'c']) should return ['c', 'b', 'a'], as the list is reversed.

Example 3: Testing Functions with Exceptions

Output:

Trying:
    divide(10, 2)
Expecting:
    5.0
ok
Trying:
    divide(8, 0)
Expecting:
    Traceback (most recent call last):
        ...
    ZeroDivisionError: division by zero
ok
1 items had no tests:
    __main__
1 items passed all tests:
   2 tests in __main__.divide
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

Explanation:

• The program defines a function divide() that performs division of two numbers.
• The second test case expects a ZeroDivisionError to be raised when dividing by zero.
• The divide() function performs the division of two numbers.
• The doctest verifies two test cases:
• divide(10, 2) should return 5.0, as the division of 10 by 2 is 5.0.
• divide(8, 0) should raise a ZeroDivisionError because division by zero is not possible.
• The first test case passed successfully, as the expected result matches the actual result. However, the second test case failed because the function did not raise the ZeroDivisionError as expected.

Example 4: Ignoring Output in Tests

Output:

Hello, World!
**********************************************************************
1 items had no tests:
    __main__
**********************************************************************
1 items passed all tests:
   1 tests in __main__.print_message
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

Explanation:

• The program defines a function print_message() that prints a message to the console.
• The docstring of the function contains a single test case without an expected result, as the expected output is a side effect (printing to the console).
• The print_message() function executes successfully without any exceptions or failures. Since the expected output is a side effect (printing to the console), the doctest considers the test as passed if the function executes without errors.

Limitations of Docstring

First Limitation

Its first drawback is the doctest module's poor capability for complex test scenarios requiring user interactions or non-deterministic behavior.

Doctest primarily tests short pieces of code or example scripts contained in docstrings. The main comparison is between the code's output and the intended output described in the docstring. However, It does not address situations in which the desired result depends on user input or outside variables that are uncontrollable in the testing environment.

Lets took an Example:

Output:

Failed example:
    greet_user()
Expected:
    Please enter your name: John
    Hello, John! Good to see you.
Got:
    Please enter your name: Hello, Yogendra ! Good to see you.
1 items had no tests:
    __main__
**********************************************************************
1 items had failures:
   1 of   1 in __main__.greet_user
1 tests in 2 items.
0 passed and 1 failed.
***Test Failed*** 1 failures.

In this case, the greet_user() method asks for the user's name before printing a distinctive welcome message. The user input request and the associated welcoming message are included in the expected output in the docstring.

However, doctest does not enable recording user input; therefore, it fails when this code is used with doctest. The user's name and the message "Please enter your name:" are expected by the test case. However, the doctest is unable to mimic or respond to user input.

The test will hang while running the doctest since it will display the prompt but wait endlessly for human input. This constraint results from the fact that doctest does not offer a method of mimicking or recording user interaction while the tests are being run.

It is sometimes required to employ more powerful testing frameworks like a unit test or pytest, including sophisticated capabilities like test fixtures, mocking, and user input simulation to properly manage these scenarios and handle complicated test cases requiring user interactions or non-deterministic behavior.

Limitation 2: Testing Private or Internal Function

Because of their restricted visibility, the doctest cannot test private or internal functions. An underscore (_) is typically prefixed to functions or methods in Python that are meant for internal use within a module to signify that they are not a part of the public interface. These operations are not meant to be directly accessible outside the module since they are often regarded as implementation details.

Doctest largely uses a module's or script's public interface. Thus it anticipates that the functions or methods under test will be available from other programs. Private or internal functions are difficult to verify with the doctest because they are not meant to be immediately accessed outside the module.

In this program, We have a module with the public function public_function() and the private function _internal_function(). In their docstrings, both functions have matching doctests.

Only the public_function() will be tested and generate the test results when the doctests are run using the doctest.testmod() function. Because it is not a part of the public interface and cannot be accessible by other code, the _internal_function() will not be put to the test.

Making these methods accessible from outside the module is one potential method for testing internal or private functions using doctest. However, doing so goes against the original design and encapsulation principles. Alternatively, you could utilize different testing frameworks like a unit test or pytest, which offer more freedom in testing internal or secret methods by directly importing and accessing them within the test cases.

Output:

Trying:
    public_function()
Expecting:
    'This is a public function.'
ok
1 items had no tests:
    __main__
1 items passed all tests:
   1 tests in __main__.public_function
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

As you can see, the test output only mentions the test for public_function(), indicating that it passed successfully. The _internal_function() is not included in the test results because it is not accessible externally.

This limitation demonstrates that the doctest focuses on testing the public interface of modules and may not be suitable for directly testing private or internal functions. Other testing frameworks like a unit test or pytest can be employed to test such functions, which allow more fine-grained control over testing private or internal components.

Limitation 3: To Get the Resource from External Resources

The difficulty of testing code that depends on external elements such as files, databases, network connections, or other dependencies beyond the control of the doctest environment is referred to as the limitation of doctest in handling tests that need external resources and dependencies.

Doctest primarily evaluates code samples or snippets included within docstrings, emphasizing the code's behavior in isolation. It doesn't include built-in techniques for dealing with outside resources or dependencies, which can be required for testing particular features.

A filename is sent as an argument to the read_file() method, which then uses the open() function to read the file's contents. On the presumption that the file "sample.txt" exists and includes the text "This is the content of the sample file," the expected output in the docstring is based.

Follow these instructions to run the program and view the results:

• In the same folder as the Python script, create a file with the name sample.txt.
• To the sample.txt file, add the phrase "This is the content of the sample file."
• File saving.

Output:

Trying:
    read_file('sample.txt')
Expecting:
    'This is the content of the sample file.'
**********************************************************************
File "c:\Users\HP\Documents\VS Code\Python\jtp.py", line 5, in __main__.read_file
Failed example:
    read_file('sample.txt')
Exception raised:
    Traceback (most recent call last):
      File "C:\Users\HP\AppData\Local\Programs\Python\Python311\Lib\doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "", line 1, in 
        read_file('sample.txt')
      File "c:\Users\HP\Documents\VS Code\Python\jtp.py", line 8, in read_file
        with open(filename, 'r') as file:
             ^^^^^^^^^^^^^^^^^^^
    FileNotFoundError: [Errno 2] No such file or directory: 'sample.txt'
1 items had no tests:
    __main__
**********************************************************************
1 items had failures:
   1 of   1 in __main__.read_file
1 tests in 2 items.
0 passed and 1 failed.
***Test Failed*** 1 failures.

The output indicates that the test failed because the sample.txt file could not be found. The read_file('sample.txt') call in the doctest raised a FileNotFoundError exception because the file does not exist. This failure is reported in the output, along with the traceback information showing the specific line of code where the exception occurred (line 8 in the read_file() function). The test result summary states that there was 1 failure in the test, and the test overall is marked as Test Failed with 1 failure.

This code cannot be performed with doctest because the test cannot handle the external resource, the file "sample.txt," according to the error message. Mocking or simulating the file's presence and contents during testing is impossible with Doctest.

Due to the file not being available or accessible in the test environment, the test will fail with a FileNotFoundError. Due to the doctest's lack of integrated functionality for managing external resources or dependencies, this constraint exists.

Limitation 5: Unable to Efficiently work with Multiple Input Variations:

The difficulty of efficiently designing and managing tests that incorporate many input values or variations is referred to as the restriction of doctest about parameterized tests or test cases with numerous input variations. Doctest does not have built-in tools for parameterizing tests or managing numerous input variants because it is primarily intended for evaluating individual code examples within docstrings.

In this example, the multiply_numbers() method multiplies two numbers together and returns the result. Three documents with various inputs and anticipated outcomes are included in the docstring.

When this code is performed with doctest, the doctests are executed, and the actual and anticipated output of the function are compared. To properly manage parameterized tests or many input variants, however, doctest does not offer a built-in solution.

Constructing several doctests with different input and output values within the docstring becomes difficult and recurring if you wish to include more test cases or parameterize the tests. The more test cases or input changes there are, the more obvious this constraint is.

Other testing frameworks, such as unit tests or pytest, provide more adaptable methods for handling parameterized tests to get around this restriction. These frameworks offer a thorough and organized approach to managing tests with many input variations by allowing you to construct test cases using decorators, generate test data programmatically, or leverage other data sources.

When dealing with more complicated parameterized tests or test cases that call for many input variations, doctest may still be appropriate for straightforward scenarios with few test variations.

Conclusion

In conclusion, Python's doctest module offers a simple and lightweight approach to inserting tests into the documentation strings of modules and functions. It encourages describing code and testing it using embedded test cases to ensure it behaves as intended. Doctest is especially helpful when the anticipated result may be stated in the docstring.

It's crucial to understand the doctest's limits, though. There might be better options for addressing more complicated testing requirements, such as parameterized tests with several input variants or tests that rely on outside resources. Other testing frameworks, such as unit test or pytest, offer more sophisticated functionality and flexibility in similar circumstances.

The complexity and extent of your testing requirements should be considered when selecting whether to utilize doctest or another testing framework. Doctest can be useful for short, simple tests integrated with documentation, but alternative frameworks may be more appropriate for robust, in-depth tests.

Overall, Python's testing environment benefits from the doctest module, which provides a simple and integrated method for testing code examples included within docstrings. Developers may use doctest efficiently in their testing strategy and select the best testing framework for certain circumstances by knowing its advantages and disadvantages.