A frequent requirement for productive Python application is that they are configurable via configuration files and/or the command-line-interface (CLI). This allows you to change the behavior of your application without touching the source code, e.g. configuring another database URL or the logging verbosity. For the CLI-part, argparse or click is often used and with PyYAML configuration files can be easily read, so where is the problem?
Configuration of a Python application by CLI or a YAML file have many things in common, i.e., both
- configure the runtime behaviour of your application,
- need to implement validations, e.g. is the port an integer above 1024,
- need to be consistent and mergeable, i.e. a CLI flag should be named like the YAML key and if both are passed the CLI overwrites the YAML configuration.
Thus implementing configuration by a CLI and a YAML file separately, leads often to code duplication and inconsistent behavior, not to mention the enormous amount of work that must be done to get this right.
With this in mind, Facebook implemented the Hydra library, which allows you to do hierarchical configuration by composition and override it through config files and the command-line. In this blog post, we demonstrate in an example project the most important features of Hydra and how it can be used in conjunction with pydantic, which extends its validation capabilities. To follow along, check out this repository that serves as a demonstration, but also as a playground for you.
Ok, so give me the gist of how Hydra works
Sure, just take a look into cli.py and config.py first as these are the only files we added, roughly 70 lines of code. The hierarchical configuration can be found in the configs folder.
├── configs
│ ├── main.yaml <- entry point for configuration
│ ├── db <- database configuration group
│ │ ├── mysql.yaml <- configuration of MySQL
│ │ └── postgresql.yaml <- configuration for PostgreSQL
│ └── experiment <- experiment configuration group
│ ├── exp1.yaml <- configuration for experiment 1
│ ├── exp2.yaml <- configuration for experiment 2
│ ├── missing_key.yaml <- wrong configuration with missing key
│ └── wrong_type.yaml <- wrong configuration with wrong type
Basically, this structure allows you to mix and match your configuration by choosing for instance the MySQL database with the setup of experiment 2. Each group then corresponds to an attribute in the configuration object, which includes other attributes, just think of a dictionary where some keys are again dictionaries.
In the repository of our example project, we defined the CLI command hydra-test by changing in setup.cfg the following lines:
# Add here console scripts like:
console_scripts =
hydra-test = my_pkg.cli:main
We can thus invoke our application with the console command hydra-test and this will execute the main function in cli.py:
@hydra.main(config_path=None, config_name="main")
def main(cfg: Config) -> None:
# this line actually runs the checks of pydantic
OmegaConf.to_object(cfg)
# log to console and into the `outputs` folder per default
log.info(f"\n{OmegaConf.to_yaml(cfg)}")
# note that IDEs allow auto-complete for accessing the attributes!
time.sleep(cfg.main.sleep)
Looking at the actual code, we see that we only trigger some pydantic checks to see if the configuration and CLI parameters are alright, then we log the current configuration and sleep for some time as defined in the configuration.
So executing just hydra-test results in:
Cannot find primary config 'main'. Check that it's in your config search path.
Config search path:
provider=hydra, path=pkg://hydra.conf
provider=main, path=pkg://my_pkg
provider=schema, path=structured://
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
This is due to the fact that we set config_path=None, which is desirable for productive application. The application
itself doesn’t know where it is gonna be installed and thus defining a path to the configuration files doesn’t make sense.
For this reason we pass the configuration at execution with -cd, short for --config-dir:
hydra-test -cd configs
This results in the error:
Error executing job with overrides: []
Traceback (most recent call last):
File ".../hydra-example-project/src/my_pkg/cli.py", line 11, in main
OmegaConf.to_object(cfg)
omegaconf.errors.MissingMandatoryValue: Structured config of type `Config` has missing mandatory value: experiment
full_key: experiment
object_type=Config
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
This is exactly as we want it, since by taking a look into config.py, we see that the schema of the main configuration is:
@dataclass
class Config:
main: Main
db: DataBase
neptune: Neptune
experiment: Experiment = MISSING
Therefore, experiment is a mandatory parameter that the CLI user needs to provide. Consequently, we add +experiment=exp1 to select the
configuration from exp1.yaml and finally get what we would expect:
❯ hydra-test -cd configs +experiment=exp1
[2022-01-27 08:14:34,257][my_pkg.cli][INFO] -
main:
sleep: 3
neptune:
project: florian.wilhelm/my_expriments
api_token: ~/.neptune_api_token
tags:
- run-1
description: Experiment run on GCP
mode: async
db:
driver: mysql
host: server_string
port: ${oc.env:MYSQL_PORT,1028}
username: myself
password: secret
experiment:
model: XGBoost
l2: 0.01
n_steps: 1000
Note that we had to use a plus sign in the flag +experiment as we added the mandatory experiment parameter. Conveniently,
Hydra has also set up the logging for us and besides logging to the terminal, all output will also be collected in the ./outputs
folder.
So the section main and neptune are directly defined in main.yaml, but why did Hydra now choose the MySQL database?
This is due to fact that in main.yaml, we defined some defaults:
# hydra section to build up the config hierarchy with defaults
defaults:
- _self_
- base_config
- db: mysql.yaml
# experiment: is not mentioned here but in config.py to have a mandatory setting
Taking a look into mysql.yaml, we see that Hydra also allows accessing environment variables easily to help with
configuration. As an example we defined the database port to be whatever the environment variable MYSQL_PORT is set
to or 1028 if undefined. So Hydra does not only unify the configuration by YAML and CLI but also by environment variables.
driver: mysql
host: server_string
port: ${oc.env:MYSQL_PORT,1028}
username: myself
password: secret
We can also override the default database by adding the flag db=postgresql and this time without + as we override a default:
❯ hydra-test -cd configs +experiment=exp1 db=postgresql
Error executing job with overrides: ['+experiment=exp1', 'db=postgresql']
Traceback (most recent call last):
File ".../hydra-example-project/src/my_pkg/cli.py", line 11, in main
OmegaConf.to_object(cfg)
pydantic.error_wrappers.ValidationError: 1 validation error for DataBase
port
Choose a non-privileged port! (type=value_error)
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Nice! This worked as expected by telling us that our port configuration is actually wrong as we chose a privileged port!
This is the magic of pydantic doing its validation work. Taking a look into config.py, we see the check that assures
a port greater than 1023.
@dataclass
class DataBase:
driver: str
host: str
@validator("port")
def check_non_privileged_port(cls, port: int) -> int:
if port < 1024:
raise ValueError("Choose a non-privileged port!")
return port
port: int
username: str
password: str
Good, so we could fix our configuration file or pass an extra parameter if we are in a hurry, i.e.:
❯ hydra-test -cd configs +experiment=exp1 db=postgresql db.port=1832
[2022-01-27 08:13:52,148][my_pkg.cli][INFO] -
main:
sleep: 3
neptune:
project: florian.wilhelm/my_expriments
api_token: ~/.neptune_api_token
tags:
- run-1
description: Experiment run on GCP
mode: async
db:
driver: postgreqsql
host: server_string
port: 1832
username: me
password: birthday
experiment:
model: XGBoost
l2: 0.01
n_steps: 1000
And this works! So much flexibility and robustness in just 70 lines of code, awesome! While you are at it, you can also
run hydra-test -cd configs +experiment=missing_key and hydra-test -cd configs +experiment=wrong_type to see some
nice errors from pydantic telling you about a missing key and wrong type of the configuration value, respectively.
By the way, also passing the port parameter wrong, e.g. with db.port=72, would have triggered the same exception, so
CLI and the YAML configurations share the same checks and validations. Hydra and pydantic work nicely together
to make this possible and pydantic greatly extends the validation capabilities of OmegaConf, which Hydra uses
as default. Just remember to use the dataclass from pydantic, not the standard library
and call OmegaConf.to_object(cfg) at the start of your application to fail as early as possible.
Hydra has many more, really nice features. Imagine you want to run now the experiments exp1 and exp2 consecutively,
you can just use the --multirun feature, or -m for short:
hydra-test -m -cd configs +experiment=exp1,exp2
Or in case you have hundreds of experiments, you can also use globbing like:
hydra-test -m -cd configs "+experiment=glob(exp*)"
There’s so much more to Hydra and several plugins even for hyperparameter optimization exist. Also note that with the
flag --hydra-help, you can see the hydra-specific parameters of your application. Using just --help returns some
automatic generated help according to your application’s configuration. This can of course be customized easily with
the help of a powerful templating system as described in Customizing Application’s help docs.
Hydra makes configuration by CLI, YAML and environment variables a bliss and the time of learning Hydra is well invested as your application’s codebase will be more flexible configurable, less complex and thus more robust.
Comments
comments powered by Disqus