Access secrets managers
GX Core supports the AWS Secrets Manager, Google Cloud Secret Manager, and Azure Key Vault secrets managers.
Use of a secrets manager is optional. Credentials can be securely stored as environment variables or entries in a yaml file without referencing content stored in a secrets manager.
- AWS Secrets Manager
- GCP Secret Manager
- Azure Key Vault
Prerequisites
- An AWS Secrets Manager instance. See AWS Secrets Manager.
- The ability to install Python packages with
pip
. - A preconfigured File Data Context.
Procedure
-
Set up AWS Secrets Manager support.
To use the AWS Secrets Manager with GX Core you will first need to install the
great_expectations
Python package with theaws_secrets
requirement. To do this, run the following command:Terminalpip install 'great_expectations[aws_secrets]'
-
Reference AWS Secrets Manager variables in
config_variables.yml
.By default,
config_variables.yml
is located at: 'gx/uncomitted/config_variables.yml' in your File Data Context.Values in
config_variables.yml
that start withsecret|arn:aws:secretsmanager
will be substituted with corresponding values from the AWS Secrets Manager. However, if the keywords followingsecret|arn:aws:secretsmanager
do not correspond to keywords in AWS Secrets Manager no substitution will occur.You can reference other stored credentials within the keywords by wrapping their corresponding variable in
${
and}
. When multiple references are present in a value, the secrets manager substitution takes place after all other substitutions have occurred.An entire connection string can be referenced from the secrets manager. In this example,
dev_db_credentials
is the Secret Name in AWS Secrets Manager, andconnection_string
is the Secret Key that corresponds to the value to be retrieved:config_variables.ymlmy_aws_creds: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|connection_string
Or each component of the connection string can be referenced separately. In these examples,
dev_db_credentials
remains the Secret Name in AWS Secrets Manager. However, rather than retrieving the value of the Secret Keyconnection_string
, Secret Keys for individual parts of the connection string are provided for retrieval:config_variables.ymldrivername: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|drivername
host: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|host
port: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|port
username: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|username
password: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|password
database: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|databaseNote that the last seven characters of an AWS Secrets Manager arn are automatically generated by AWS and are not mandatory to retrieve the secret. For example, the following two values retrieve the same secret:
config_variables.ymlsecret1: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:my_secret-1zAyu6
secret2: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:my_secret -
Optional. Reference versioned secrets.
Unless otherwise specified, the latest version of the secret is returned by default. To get a specific version of the secret you want to retrieve, specify its version UUID. For example:
config_variables.ymlversioned_secret: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:my_secret:00000000-0000-0000-0000-000000000000
-
Optional. Retrieve specific secrets from a JSON string.
To retrieve a specific secret from a JSON string, include the JSON key after a pipe character
|
at the end of the secrets keywords. For example:config_variables.ymljson_secret: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:my_secret|<KEY>
versioned_json_secret: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:my_secret:00000000-0000-0000-0000-000000000000|<KEY>
Prerequisites
- A GCP Secret Manager instance with configured secrets.
- The ability to install Python packages with
pip
. - A preconfigured File Data Context.
Procedure
-
Set up GCP Secret Manager support.
To use GCP Secret Manager with GX Core you will first need to install the
great_expectations
Python package with thegcp
requirement. To do this, run the following command:Terminalpip install 'great_expectations[gcp]'
-
Reference GCP Secret Manager variables in
config_variables.yml
.By default,
config_variables.yml
is located at: 'gx/uncomitted/config_variables.yml' in your File Data Context.Values in
config_variables.yml
that match the regex^secret\|projects\/[a-z0-9\_\-]{6,30}\/secrets
will be substituted with corresponding values from GCP Secret Manager. However, if the keywords in the matching regex do not correspond to keywords in GCP Secret Manager no substitution will occur.You can reference other stored credentials within the regex by wrapping their corresponding variable in
${
and}
. When multiple references are present in a value, the secrets manager substitution takes place after all other substitutions have occurred.An entire connection string can be referenced from the secrets manager:
config_variables.ymlmy_gcp_creds: secret|projects/${PROJECT_ID}/secrets/dev_db_credentials|connection_string
Or each component of the connection string can be referenced separately:
config_variables.ymldrivername: secret|projects/${PROJECT_ID}/secrets/PROD_DB_CREDENTIALS_DRIVERNAME
host: secret|projects/${PROJECT_ID}/secrets/PROD_DB_CREDENTIALS_HOST
port: secret|projects/${PROJECT_ID}/secrets/PROD_DB_CREDENTIALS_PORT
username: secret|projects/${PROJECT_ID}/secrets/PROD_DB_CREDENTIALS_USERNAME
password: secret|projects/${PROJECT_ID}/secrets/PROD_DB_CREDENTIALS_PASSWORD
database: secret|projects/${PROJECT_ID}/secrets/PROD_DB_CREDENTIALS_DATABASE -
Optional. Reference versioned secrets.
Unless otherwise specified, the latest version of the secret is returned by default. To get a specific version of the secret you want to retrieve, specify its version id. For example:
config_variables.ymlversioned_secret: secret|projects/${PROJECT_ID}/secrets/my_secret/versions/1
-
Optional. Retrieve specific secrets for a JSON string.
To retrieve a specific secret for a JSON string, include the JSON key after a pipe character
|
at the end of the secrets regex. For example:config_variables.ymljson_secret: secret|projects/${PROJECT_ID}/secrets/my_secret|<KEY>
versioned_json_secret: secret|projects/${PROJECT_ID}/secrets/my_secret/versions/1|<KEY> -
Run the following code to use the
connection_string
parameter values when you add adatasource
to a Data Context:# We can use a single connection string
pg_datasource = context.data_sources.add_or_update_sql(
name="my_postgres_db", connection_string="${my_gcp_creds}"
)
# Or each component of the connection string separately
pg_datasource = context.data_sources.add_or_update_sql(
name="my_postgres_db", connection_string="${drivername}://${username}:${password}@${host}:${port}/${database}"
)
Prerequisites
- An Azure Key Vault instance with configured secrets.
- The ability to install Python packages with
pip
. - A preconfigured File Data Context.
Procedure
-
Set up Azure Key Vault support.
To use Azure Key Vault with GX Core you will first need to install the
great_expectations
Python package with theazure_secrets
requirement. To do this, run the following command:Terminalpip install 'great_expectations[azure_secrets]'
-
Reference Azure Key Vault variables in
config_variables.yml
.By default,
config_variables.yml
is located at: 'gx/uncomitted/config_variables.yml' in your File Data Context.Values in
config_variables.yml
that match the regex^secret\|https:\/\/[a-zA-Z0-9\-]{3,24}\.vault\.azure\.net
will be substituted with corresponding values from Azure Key Vault. However, if the keywords in the matching regex do not correspond to keywords in Azure Key Vault no substitution will occur.You can reference other stored credentials within the regex by wrapping their corresponding variable in
${
and}
. When multiple references are present in a value, the secrets manager substitution takes place after all other substitutions have occurred.An entire connection string can be referenced from the secrets manager:
config_variables.ymlmy_abs_creds: secret|https://${VAULT_NAME}.vault.azure.net/secrets/dev_db_credentials|connection_string
Or each component of the connection string can be referenced separately:
config_variables.ymldrivername: secret|https://${VAULT_NAME}.vault.azure.net/secrets/dev_db_credentials|host
host: secret|https://${VAULT_NAME}.vault.azure.net/secrets/dev_db_credentials|host
port: secret|https://${VAULT_NAME}.vault.azure.net/secrets/dev_db_credentials|port
username: secret|https://${VAULT_NAME}.vault.azure.net/secrets/dev_db_credentials|username
password: secret|https://${VAULT_NAME}.vault.azure.net/secrets/dev_db_credentials|password
database: secret|https://${VAULT_NAME}.vault.azure.net/secrets/dev_db_credentials|database -
Optional. Reference versioned secrets.
Unless otherwise specified, the latest version of the secret is returned by default. To get a specific version of the secret you want to retrieve, specify its version id (32 alphanumeric characters). For example:
config_variables.ymlversioned_secret: secret|https://${VAULT_NAME}.vault.azure.net/secrets/my-secret/a0b00aba001aaab10b111001100a11ab
-
Optional. Retrieve specific secrets for a JSON string.
To retrieve a specific secret for a JSON string, include the JSON key after a pipe character
|
at the end of the secrets regex. For example:config_variables.ymljson_secret: secret|https://${VAULT_NAME}.vault.azure.net/secrets/my-secret|<KEY>
versioned_json_secret: secret|https://${VAULT_NAME}.vault.azure.net/secrets/my-secret/a0b00aba001aaab10b111001100a11ab|<KEY>