How to use a YAML file or environment variables to populate credentials¶
This guide will explain how to use a YAML file and/or environment variables to populate credentials (or any value) in your great_expectations.yml
project config.
Prerequisites: This how-to guide assumes you have already:
Steps¶
Decide where you would like to save the desired credentials or config values - in a YAML file, environment variables, or a combination - then save the values. In most cases, we suggest using a config variables YAML file. YAML files make variables more visible, easily editable, and allow for modularization (e.g. one file for dev, another for prod).
Note:
In the
great_expectations.yml
config file, environment variables take precedence over variables defined in a config variables YAMLEnvironment variable substitution is supported in both the
great_expectations.yml
and config variablesconfig_variables.yml
config file.If using a YAML file, save desired credentials or config values to
great_expectations/uncommitted/config_variables.yml
or another YAML file of your choosing:# great_expectations/uncommitted/config_variables.yml my_postgres_db_yaml_creds: drivername: postgres host: 127.0.0.778 port: '7987' username: administrator password: ${MY_DB_PW} database: postgresNote:
If you wish to store values that include the dollar sign character
$
, please escape them using a backslash\
so substitution is not attempted. For example in the above example for postgres credentials you could setpassword: pa\$sword
if your password ispa$sword
. Say that 5 times fast, and also please choose a more secure password!When you save values via the CLI, they are automatically escaped if they contain the
$
character.You can also have multiple substitutions for the same item, e.g.
database_string: ${USER}:${PASSWORD}@${HOST}:${PORT}/${DATABASE}
If using environment variables, set values by entering
export ENV_VAR_NAME=env_var_value
in the terminal or adding the commands to your~/.bashrc
file:export POSTGRES_DRIVERNAME=postgres export POSTGRES_HOST=localhost export POSTGRES_PORT='5432' export POSTGRES_USERNAME=postgres export POSTGRES_PW='' export POSTGRES_DB=postgres export MY_DB_PW=password
If using a YAML file, set the
config_variables_file_path
key in yourgreat_expectations.yml
or leave the default.
# great_expectations/great_expectations.yml config_variables_file_path: uncommitted/config_variables.yml
Replace credentials or other values in your
great_expectations.yml
with${}
-wrapped variable names (i.e.${ENVIRONMENT_VARIABLE}
or${YAML_KEY}
).
# great_expectations/great_expectations.yml datasources: my_postgres_db: class_name: SqlAlchemyDatasource data_asset_type: class_name: SqlAlchemyDataset module_name: great_expectations.dataset module_name: great_expectations.datasource credentials: ${my_postgres_db_yaml_creds} my_other_postgres_db: class_name: SqlAlchemyDatasource data_asset_type: class_name: SqlAlchemyDataset module_name: great_expectations.dataset module_name: great_expectations.datasource credentials: drivername: ${POSTGRES_DRIVERNAME} host: ${POSTGRES_HOST} port: ${POSTGRES_PORT} username: ${POSTGRES_USERNAME} password: ${POSTGRES_PW} database: ${POSTGRES_DB}
Additional Notes¶
The default
config_variables.yml
file located atgreat_expectations/uncommitted/config_variables.yml
applies to deployments created usinggreat_expectations init
.