Python TOML

In this tutorial, we will learn about the TOML which is a Tom's Obvious Minimal Language. It is a reasonably new configuration file format that widely used by the Python community. We will discuss syntax of TOML, use tomli and tomllib to parse TOML document and use tomli_w to write data structure as TOML.

Use TOML as a Configuration Format

TOML, which stands for Tom's Obvious Minimal Language, is a configuration file format named after its creator, Tom Preston-Werner. Its design objective was to create a format that is effortlessly parseable into data structures across a broad range of programming languages.

Configurations and Configuration Files

Configuration plays a crucial role in virtually every application or system, enabling the flexibility to modify settings and behavior without altering the source code. Configurations are employed in various scenarios, such as specifying essential details for connecting to external services like databases or cloud storage. Additionally, configuration settings facilitate the customization of user experiences within a project, empowering users to tailor the application according to their preferences.

Utilizing a configuration file in your project offers several benefits, including the separation of code from its settings. It prompts you to carefully identify which aspects of your system should be genuinely configurable, providing a means to assign meaningful names to values that would otherwise be considered "magic" within your source code. Let's consider the usage of a configuration file in a hypothetical tic-tac-toe game for now.

Config File

While the standard tic-tac-toe game is traditionally played on a three-by-three grid, the configurability of the board size is uncertain. The existing logic may not be compatible with different board sizes. However, it could still be beneficial to include the board size value in the configuration file. This approach serves two purposes: providing a meaningful name to the value and making it visible, even if it remains unchanged for the standard game.

The project URL holds significant importance when deploying an application. While it may not be subject to change for the average user, power users might require the ability to redeploy the game on a different server.

To provide clarity and accommodate diverse use cases, it can be beneficial to organize the configuration file accordingly. One popular approach is to separate the configuration into multiple files, each addressing a distinct concern. Alternatively, you can group related configuration values together for better organization. Here is a possible reorganization of your hypothetical configuration file:

There are various methods exist for specifying configurations, depending on the operating system and specific requirements. For instance, Windows has traditionally employed INI files, which bear resemblance to the configuration file example provided earlier. On Unix systems, plain-text configuration files are commonly used, although the specific format may vary across different services. Regardless of the format, the emphasis is typically placed on maintaining human-readability and simplicity for ease of understanding and modification.

As time has progressed, an increasing number of applications have adopted well-defined formats such as XML, JSON, or YAML for their configuration requirements. These formats were originally developed as data interchange or serialization formats, primarily intended for computer communication purposes. However, due to their structured nature and widespread support across programming languages, they have found utility in representing configuration settings as well. Their adoption simplifies parsing, manipulation, and interoperability between different systems and programming languages, making them popular choices for configuration file formats.

TOML - Tom's Obvious Minimal Language

TOML is a relatively recent format, with its initial specification (version 0.1.0) being released in 2013. From its inception, TOML has prioritized being a minimal and human-readable configuration file format. The goals outlined on the TOML website are as follows:

"TOML aims to be a minimal configuration file format that's easy to read due to obvious semantics. TOML is designed to map unambiguously to a hash table. TOML should be easy to parse into data structures in a wide variety of languages. (Source, emphasis added)"

Let's define the above file in the TOML.

TOML draws inspiration from traditional configuration files in its syntax. However, its significant advantage over Windows INI files and UNIX configuration files lies in its well-defined specification. Unlike its counterparts, TOML has a comprehensive specification that precisely outlines the allowed elements in a TOML document and provides clear guidelines on how different values should be interpreted. This specification has evolved and reached stability and maturity, culminating in version 1.0.0 in early 2021.

In contrast to TOML, the INI format lacks a formal specification. Instead, it comprises numerous variants and dialects, typically defined by specific implementations. Python, for instance, includes support for reading INI files in its standard library through ConfigParser. Although ConfigParser is fairly flexible, it may not support all variations of INI files.

Another notable distinction between TOML and many traditional formats is that TOML assigns types to its values. In the provided example, "blue" is interpreted as a string, while 3 is considered a number. One potential criticism of TOML is that human authors need to be conscious of these types when writing TOML documents. In simpler formats, such responsibility is typically handled by the programmer who parses the configuration file

TOML Schema Validation

At present, TOML does not incorporate a schema language that allows for specifying required and optional fields within a TOML document. Various proposals for adding this capability have been put forward, but it remains uncertain if any of them will be accepted and implemented in the near future.

When dealing with more complex TOML documents, the approach of relying solely on the TOML format may not scale effectively. Additionally, if the goal is to provide informative error messages, it requires additional effort. A superior alternative is to leverage a library like pydantic, which utilizes type annotations to perform data validation during runtime. By using pydantic, one can benefit from its precise and helpful built-in error messages. This approach not only simplifies data validation but also enhances the scalability and usability of handling TOML configurations.

More about TOML: Key-Value Pair

TOML store the value in the key-value format. TOML values have different types. A value must be a one of following type.

• String
• Integer
• Float
• Boolean
• Offset date-time
• Local date
• Local time
• Array
• Inline table

Note - TOML embraces comments using the same syntax as Python. By utilizing the hash symbol (#), you can designate the remainder of a line as a comment. Introducing comments in your configuration files serves the purpose of enhancing clarity and facilitating comprehension for both yourself and your users.

The key-value pairs are basic building blocks in a TOML document. We mention them with key=value syntax where key is separated from the value with an equal sign. Following is an example of TOML document with one key-value pair.

In TOML, keys are consistently interpreted as strings, irrespective of whether quotation marks surround them or not. Let's consider the following example to illustrate this point.

In TOML, a key like "12" is considered valid, but it will be interpreted as a string rather than a number. Typically, it is recommended to use bare keys whenever possible. Bare keys consist of ASCII letters, numbers, underscores, and dashes. These keys can be written without quotation marks, as demonstrated in the examples provided earlier. By using bare keys, you can enhance the readability and simplicity of your TOML configuration files.

TOML documents must be encoded in UTF-8 Unicode, offering considerable flexibility in representing values. While there are restrictions on bare keys, you can still utilize Unicode characters when defining your keys. However, using Unicode keys necessitates enclosing them in quotation marks. It's important to note that employing Unicode keys comes with an added requirement of using quotation marks to delimit them.

In TOML, dots (.) have a specific significance when used in keys. While you can include dots in unquoted keys, doing so will result in the grouping of values. The dotted key will be split at each dot, creating nested groupings within the data structure. Let's consider the following example to illustrate this behavior.

In the given example, there are two keys that include dots. As both keys start with "match_x", they will be grouped together under a section named "match_x". The keys "symbol" and "color" will be nested within this section, allowing for a structured organization of related data.

String, Numbers, and Boolean

TOML uses similar data type as Python. The only difference is Boolean values are lowercase: true and false. In TOML, it is common to use double quotation marks (") for defining strings. Within strings, you can utilize backslashes to escape special characters. For instance, "\u03c0 is less than four" includes the escape sequence \u03c0, which represents the Unicode character with codepoint U+03c0, corresponding to the Greek letter π. When interpreted, this string will be rendered as "π is less than four".

In TOML, an alternative way to specify strings is by using single quotation marks ('). These single-quoted strings are known as literal strings and exhibit behavior similar to raw strings in Python. In a literal string, no characters are escaped or interpreted, meaning that '\u03c0 is the Unicode codepoint of π' will treat the initial characters '\u03c0' as literals, rather than interpreting them as a Unicode escape sequence.