great_expectations.data_context.util¶
Module Contents¶
Classes¶
Used to mask passwords in Datasources. Does not mask sqlite urls. |
Functions¶
|
Build a GE class from configuration dictionaries. |
|
|
|
This method takes a string, and if it contains a pattern ${SOME_VARIABLE} or $SOME_VARIABLE, |
This method takes a value, tries to parse the value to fetch a secret from a secret manager |
|
This methods uses a boto3 client and the secretsmanager service to try to retrieve the secret value |
|
This methods uses a google.cloud.secretmanager.SecretManagerServiceClient to try to retrieve the secret value |
|
This methods uses a azure.identity.DefaultAzureCredential to authenticate to the Azure SDK for Python |
|
|
Substitute all config variables of the form ${SOME_VARIABLE} in a dictionary-like |
|
This function is useful when one needs to load a file that is |
|
Parse and check whether the string contains a substitution variable of the case insensitive form ${SOME_VAR} or $SOME_VAR |
-
great_expectations.data_context.util.SecretClient¶
-
great_expectations.data_context.util.boto3¶
-
great_expectations.data_context.util.secretmanager¶
-
great_expectations.data_context.util.sa¶
-
great_expectations.data_context.util.logger¶
-
great_expectations.data_context.util.instantiate_class_from_config(config, runtime_environment, config_defaults=None)¶ Build a GE class from configuration dictionaries.
-
great_expectations.data_context.util.build_store_from_config(store_name: str = None, store_config: dict = None, module_name: str = 'great_expectations.data_context.store', runtime_environment: dict = None)¶
-
great_expectations.data_context.util.format_dict_for_error_message(dict_)¶
-
great_expectations.data_context.util.substitute_config_variable(template_str, config_variables_dict, dollar_sign_escape_string: str = '\$')¶ This method takes a string, and if it contains a pattern ${SOME_VARIABLE} or $SOME_VARIABLE, returns a string where the pattern is replaced with the value of SOME_VARIABLE, otherwise returns the string unchanged. These patterns are case sensitive. There can be multiple patterns in a string, e.g. all 3 will be substituted in the following: $SOME_VARIABLE${some_OTHER_variable}$another_variable
If the environment variable SOME_VARIABLE is set, the method uses its value for substitution. If it is not set, the value of SOME_VARIABLE is looked up in the config variables store (file). If it is not found there, the input string is returned as is.
If the value to substitute is not a string, it is returned as-is.
If the value to substitute begins with dollar_sign_escape_string it is not substituted.
If the value starts with the keyword secret|, it tries to apply secret store substitution.
- Parameters
template_str – a string that might or might not be of the form ${SOME_VARIABLE} or $SOME_VARIABLE
config_variables_dict – a dictionary of config variables. It is loaded from the config variables store (by default, “uncommitted/config_variables.yml file)
dollar_sign_escape_string – a string that will be used in place of a $ when substitution is not desired.
- Returns
a string with values substituted, or the same object if template_str is not a string.
-
great_expectations.data_context.util.substitute_value_from_secret_store(value)¶ This method takes a value, tries to parse the value to fetch a secret from a secret manager and returns the secret’s value only if the input value is a string and contains one of the following patterns:
AWS Secrets Manager: the input value starts with
secret|arn:aws:secretsmanagerGCP Secret Manager: the input value matches the following regex
^secret\|projects\/[a-z0-9\_\-]{6,30}\/secretsAzure Key Vault: the input value matches the following regex
^secret\|https:\/\/[a-zA-Z0-9\-]{3,24}\.vault\.azure\.net
Input value examples:
AWS Secrets Manager:
secret|arn:aws:secretsmanager:eu-west-3:123456789012:secret:my_secretGCP Secret Manager:
secret|projects/gcp_project_id/secrets/my_secretAzure Key Vault:
secret|https://vault-name.vault.azure.net/secrets/my-secret
- Parameters
value – a string that might or might not start with secret|
- Returns
a string with the value substituted by the secret from the secret store, or the same object if value is not a string.
-
great_expectations.data_context.util.substitute_value_from_aws_secrets_manager(value)¶ This methods uses a boto3 client and the secretsmanager service to try to retrieve the secret value from the elements it is able to parse from the input value.
value: string with pattern
secret|arn:aws:secretsmanager:${region_name}:${account_id}:secret:${secret_name}optional : after the value above, a secret version can be added
:${secret_version}optional : after the value above, a secret key can be added
|${secret_key}region_name: AWS region used by the secrets manager
account_id: Account ID for the AWS account used by the secrets manager
This value is currently not used.
secret_name: Name of the secret
secret_version: UUID of the version of the secret
secret_key: Only if the secret’s data is a JSON string, which key of the dict should be retrieve
- Parameters
value – a string that starts with
secret|arn:aws:secretsmanager- Returns
a string with the value substituted by the secret from the AWS Secrets Manager store
- Raises
ImportError, ValueError
-
great_expectations.data_context.util.substitute_value_from_gcp_secret_manager(value)¶ This methods uses a google.cloud.secretmanager.SecretManagerServiceClient to try to retrieve the secret value from the elements it is able to parse from the input value.
value: string with pattern
secret|projects/${project_id}/secrets/${secret_name}optional : after the value above, a secret version can be added
/versions/${secret_version}optional : after the value above, a secret key can be added
|${secret_key}project_id: Project ID of the GCP project on which the secret manager is implemented
secret_name: Name of the secret
secret_version: ID of the version of the secret
secret_key: Only if the secret’s data is a JSON string, which key of the dict should be retrieve
- Parameters
value – a string that matches the following regex
^secret|projects/[a-z0-9_-]{6,30}/secrets- Returns
a string with the value substituted by the secret from the GCP Secret Manager store
- Raises
ImportError, ValueError
-
great_expectations.data_context.util.substitute_value_from_azure_keyvault(value)¶ This methods uses a azure.identity.DefaultAzureCredential to authenticate to the Azure SDK for Python and a azure.keyvault.secrets.SecretClient to try to retrieve the secret value from the elements it is able to parse from the input value.
value: string with pattern
secret|https://${vault_name}.vault.azure.net/secrets/${secret_name}optional : after the value above, a secret version can be added
/${secret_version}optional : after the value above, a secret key can be added
|${secret_key}vault_name: Vault name of the secret manager
secret_name: Name of the secret
secret_version: ID of the version of the secret
secret_key: Only if the secret’s data is a JSON string, which key of the dict should be retrieve
- Parameters
value – a string that matches the following regex
^secret|https://[a-zA-Z0-9-]{3,24}.vault.azure.net- Returns
a string with the value substituted by the secret from the Azure Key Vault store
- Raises
ImportError, ValueError
-
great_expectations.data_context.util.substitute_all_config_variables(data, replace_variables_dict, dollar_sign_escape_string: str = '\$')¶ Substitute all config variables of the form ${SOME_VARIABLE} in a dictionary-like config object for their values.
The method traverses the dictionary recursively.
- Parameters
data –
replace_variables_dict –
dollar_sign_escape_string – a reserved character for specifying parameters
- Returns
a dictionary with all the variables replaced with their values
-
great_expectations.data_context.util.file_relative_path(dunderfile, relative_path)¶ This function is useful when one needs to load a file that is relative to the position of the current file. (Such as when you encode a configuration file path in source file and want in runnable in any current working directory)
It is meant to be used like the following: file_relative_path(__file__, ‘path/relative/to/file’)
-
great_expectations.data_context.util.parse_substitution_variable(substitution_variable: str) → Optional[str]¶ Parse and check whether the string contains a substitution variable of the case insensitive form ${SOME_VAR} or $SOME_VAR :param substitution_variable: string to be parsed
- Returns
string of variable name e.g. SOME_VAR or None if not parsable. If there are multiple substitution variables this currently returns the first e.g. $SOME_$TRING -> $SOME_
-
class
great_expectations.data_context.util.PasswordMasker¶ Used to mask passwords in Datasources. Does not mask sqlite urls.
Example usage masked_db_url = PasswordMasker.mask_db_url(url) where url = “postgresql+psycopg2://username:password@host:65432/database” and masked_url = “postgresql+psycopg2://username:***@host:65432/database”
-
MASKED_PASSWORD_STRING= ***¶
-
static
mask_db_url(url: str, use_urlparse: bool = False, **kwargs)¶ Mask password in database url. Uses sqlalchemy engine parsing if sqlalchemy is installed, otherwise defaults to using urlparse from the stdlib which does not handle kwargs. :param url: Database url e.g. “postgresql+psycopg2://username:password@host:65432/database” :param use_urlparse: Skip trying to parse url with sqlalchemy and use urlparse :param **kwargs: passed to create_engine()
-