great_expectations.data_context.store

Package Contents

Classes

Store(store_backend=None, runtime_environment=None, store_name=’no_store_name’)

A store is responsible for reading and writing Great Expectations objects

StoreBackend(fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = ‘’, store_name=’no_store_name’)

A store backend acts as a key-value store that can accept tuples as keys, to abstract away

InMemoryStoreBackend(runtime_environment=None, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = ‘’, store_name=None)

Uses an in-memory dictionary as a store backend.

TupleFilesystemStoreBackend(base_directory, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, root_directory=None, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = ‘’, base_public_path=None, store_name=None)

Uses a local filepath as a store.

TupleGCSStoreBackend(bucket, project, prefix=’’, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = ‘’, public_urls=True, base_public_path=None, store_name=None)

Uses a GCS bucket as a store.

TupleS3StoreBackend(bucket, prefix=’’, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = ‘’, base_public_path=None, endpoint_url=None, store_name=None)

Uses an S3 bucket as a store.

TupleStoreBackend(filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = ‘’, base_public_path=None, store_name=None)

If filepath_template is provided, the key to this StoreBackend abstract class must be a tuple with

DatabaseStoreBackend(table_name, key_columns, fixed_length_key=True, credentials=None, url=None, connection_string=None, engine=None, store_name=None, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = ‘’, **kwargs)

A store backend acts as a key-value store that can accept tuples as keys, to abstract away

ConfigurationStore(store_name: str, store_backend: Optional[dict] = None, overwrite_existing: bool = False, runtime_environment: Optional[dict] = None)

Configuration Store provides a way to store any Marshmallow Schema compatible Configuration (using the YAML format).

CheckpointStore(store_name: str, store_backend: Optional[dict] = None, overwrite_existing: bool = False, runtime_environment: Optional[dict] = None)

A CheckpointStore manages Checkpoints for the DataContext.

EvaluationParameterStore(store_backend=None, store_name=None)

A MetricStore stores ValidationMetric information to be used between runs.

MetricStore(store_backend=None, store_name=None)

A MetricStore stores ValidationMetric information to be used between runs.

ExpectationsStore(store_backend=None, runtime_environment=None, store_name=None)

An Expectations Store provides a way to store Expectation Suites accessible to a Data Context.

ValidationsStore(store_backend=None, runtime_environment=None, store_name=None)

A ValidationsStore manages Validation Results to ensure they are accessible via a Data Context for review and rendering into Data Docs.

SqlAlchemyQueryStore(credentials, queries=None, store_backend=None, runtime_environment=None, store_name=None)

SqlAlchemyQueryStore stores queries by name, and makes it possible to retrieve the resulting value by query

HtmlSiteStore(store_backend=None, runtime_environment=None)

A HtmlSiteStore facilitates publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results.

class great_expectations.data_context.store.Store(store_backend=None, runtime_environment=None, store_name='no_store_name')

A store is responsible for reading and writing Great Expectations objects to appropriate backends. It provides a generic API that the DataContext can use independently of any particular ORM and backend.

An implementation of a store will generally need to define the following:
  • serialize

  • deserialize

  • _key_class (class of expected key type)

All keys must have a to_tuple() method.

_key_class
_validate_key(self, key)
property store_backend(self)
property store_name(self)
property store_backend_id(self)

Report the store_backend_id of the currently-configured StoreBackend :returns: store_backend_id which is a UUID(version=4)

property store_backend_id_warnings_suppressed(self)

Report the store_backend_id of the currently-configured StoreBackend, suppressing warnings for invalid configurations. :returns: store_backend_id which is a UUID(version=4)

serialize(self, key, value)
key_to_tuple(self, key)
tuple_to_key(self, tuple_)
deserialize(self, key, value)
get(self, key)
set(self, key, value)
list_keys(self)
has_key(self, key)
self_check(self, pretty_print)
property config(self)
class great_expectations.data_context.store.StoreBackend(fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = '', store_name='no_store_name')

A store backend acts as a key-value store that can accept tuples as keys, to abstract away reading and writing to a persistence layer.

In general a StoreBackend implementation must provide implementations of:
  • _get

  • _set

  • list_keys

  • _has_key

IGNORED_FILES = ['.ipynb_checkpoints']
STORE_BACKEND_ID_KEY = ['.ge_store_backend_id']
STORE_BACKEND_ID_PREFIX = store_backend_id =
STORE_BACKEND_INVALID_CONFIGURATION_ID = 00000000-0000-0000-0000-00000000e003
property fixed_length_key(self)
property store_name(self)
_construct_store_backend_id(self, suppress_warning: bool = False)

Create a store_backend_id if one does not exist, and return it if it exists If a valid UUID store_backend_id is passed in param manually_initialize_store_backend_id and there is not already an existing store_backend_id then the store_backend_id from param manually_initialize_store_backend_id is used to create it. :param suppress_warning: boolean flag for whether warnings are logged

Returns

store_backend_id which is a UUID(version=4)

property store_backend_id(self)
property store_backend_id_warnings_suppressed(self)
get(self, key, **kwargs)
set(self, key, value, **kwargs)
move(self, source_key, dest_key, **kwargs)
has_key(self, key)
get_url_for_key(self, key, protocol=None)
_validate_key(self, key)
_validate_value(self, value)
abstract _get(self, key)
abstract _set(self, key, value, **kwargs)
abstract _move(self, source_key, dest_key, **kwargs)
abstract list_keys(self, prefix=())
abstract remove_key(self, key)
abstract _has_key(self, key)
is_ignored_key(self, key)
property config(self)
class great_expectations.data_context.store.InMemoryStoreBackend(runtime_environment=None, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = '', store_name=None)

Bases: great_expectations.data_context.store.store_backend.StoreBackend

Uses an in-memory dictionary as a store backend.

_get(self, key)
_set(self, key, value, **kwargs)
_move(self, source_key, dest_key, **kwargs)
list_keys(self, prefix=())
_has_key(self, key)
remove_key(self, key)
property config(self)
class great_expectations.data_context.store.TupleFilesystemStoreBackend(base_directory, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, root_directory=None, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = '', base_public_path=None, store_name=None)

Bases: great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend

Uses a local filepath as a store.

The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added. The filepath_template is a string template used to convert the key to a filepath.

_get(self, key)
_set(self, key, value, **kwargs)
_move(self, source_key, dest_key, **kwargs)
list_keys(self, prefix=())
rrmdir(self, mroot, curpath)

recursively removes empty dirs between curpath and mroot inclusive

remove_key(self, key)
get_url_for_key(self, key, protocol=None)
get_public_url_for_key(self, key, protocol=None)
_has_key(self, key)
property config(self)
class great_expectations.data_context.store.TupleGCSStoreBackend(bucket, project, prefix='', filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = '', public_urls=True, base_public_path=None, store_name=None)

Bases: great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend

Uses a GCS bucket as a store.

The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added.

The filepath_template is a string template used to convert the key to a filepath.

_build_gcs_object_key(self, key)
_move(self, source_key, dest_key, **kwargs)
_get(self, key)
_set(self, key, value, content_encoding='utf-8', content_type='application/json')
list_keys(self)
get_url_for_key(self, key, protocol=None)
get_public_url_for_key(self, key, protocol=None)
_get_path_url(self, path)
remove_key(self, key)
_has_key(self, key)
property config(self)
class great_expectations.data_context.store.TupleS3StoreBackend(bucket, prefix='', filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = '', base_public_path=None, endpoint_url=None, store_name=None)

Bases: great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend

Uses an S3 bucket as a store.

The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added. The filepath_template is a string template used to convert the key to a filepath.

_build_s3_object_key(self, key)
_get(self, key)
_set(self, key, value, content_encoding='utf-8', content_type='application/json')
_move(self, source_key, dest_key, **kwargs)
list_keys(self)
get_url_for_key(self, key, protocol=None)
get_public_url_for_key(self, key, protocol=None)
remove_key(self, key)
_has_key(self, key)
property config(self)
class great_expectations.data_context.store.TupleStoreBackend(filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, fixed_length_key=False, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = '', base_public_path=None, store_name=None)

Bases: great_expectations.data_context.store.store_backend.StoreBackend

If filepath_template is provided, the key to this StoreBackend abstract class must be a tuple with fixed length equal to the number of unique components matching the regex r”{d+}”

For example, in the following template path: expectations/{0}/{1}/{2}/prefix-{2}.json, keys must have three components.

_validate_key(self, key)
_validate_value(self, value)
_convert_key_to_filepath(self, key)
_convert_filepath_to_key(self, filepath)
verify_that_key_to_filepath_operation_is_reversible(self)
property config(self)
class great_expectations.data_context.store.DatabaseStoreBackend(table_name, key_columns, fixed_length_key=True, credentials=None, url=None, connection_string=None, engine=None, store_name=None, suppress_store_backend_id=False, manually_initialize_store_backend_id: str = '', **kwargs)

Bases: great_expectations.data_context.store.store_backend.StoreBackend

A store backend acts as a key-value store that can accept tuples as keys, to abstract away reading and writing to a persistence layer.

In general a StoreBackend implementation must provide implementations of:
  • _get

  • _set

  • list_keys

  • _has_key

property store_backend_id(self)

Create a store_backend_id if one does not exist, and return it if it exists Ephemeral store_backend_id for database_store_backend until there is a place to store metadata :returns: store_backend_id which is a UUID(version=4)

_build_engine(self, credentials, **kwargs)

Using a set of given credentials, constructs an Execution Engine , connecting to a database using a URL or a private key path.

_get_sqlalchemy_key_pair_auth_url(self, drivername: str, credentials: dict)

Utilizing a private key path and a passphrase in a given credentials dictionary, attempts to encode the provided values into a private key. If passphrase is incorrect, this will fail and an exception is raised.

Parameters
  • drivername (str) –

  • credentials (dict) –

Returns

a tuple consisting of a url with the serialized key-pair authentication, and a dictionary of engine kwargs.

_get(self, key)
_set(self, key, value, allow_update=True)
abstract _move(self)
get_url_for_key(self, key)
_convert_engine_and_key_to_url(self, key)
_has_key(self, key)
list_keys(self, prefix=())
remove_key(self, key)
property config(self)
class great_expectations.data_context.store.ConfigurationStore(store_name: str, store_backend: Optional[dict] = None, overwrite_existing: bool = False, runtime_environment: Optional[dict] = None)

Bases: great_expectations.data_context.store.store.Store

Configuration Store provides a way to store any Marshmallow Schema compatible Configuration (using the YAML format).

_key_class
_configuration_class
remove_key(self, key)
serialize(self, key, value)
deserialize(self, key, value)
property overwrite_existing(self)
property config(self)
self_check(self, pretty_print: bool = True)
abstract serialization_self_check(self, pretty_print: bool)
class great_expectations.data_context.store.CheckpointStore(store_name: str, store_backend: Optional[dict] = None, overwrite_existing: bool = False, runtime_environment: Optional[dict] = None)

Bases: great_expectations.data_context.store.ConfigurationStore

A CheckpointStore manages Checkpoints for the DataContext.

_configuration_class
serialization_self_check(self, pretty_print: bool)
class great_expectations.data_context.store.EvaluationParameterStore(store_backend=None, store_name=None)

Bases: great_expectations.data_context.store.metric_store.MetricStore

A MetricStore stores ValidationMetric information to be used between runs.

get_bind_params(self, run_id)
property config(self)
class great_expectations.data_context.store.MetricStore(store_backend=None, store_name=None)

Bases: great_expectations.data_context.store.store.Store

A MetricStore stores ValidationMetric information to be used between runs.

_key_class
_validate_value(self, value)
serialize(self, key, value)
deserialize(self, key, value)
class great_expectations.data_context.store.ExpectationsStore(store_backend=None, runtime_environment=None, store_name=None)

Bases: great_expectations.data_context.store.store.Store

An Expectations Store provides a way to store Expectation Suites accessible to a Data Context.

Feature Maturity

icon-f4457072612611ebae8a0242ac110002 Expectation Store - Git - How-to Guide
Use a git repository to store expectation suites.
Maturity: Production
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: N/A
Documentation Completeness: Complete
Bug Risk: Low
icon-f4457234612611ebae8a0242ac110002 Expectation Store - Filesystem - How-to Guide
Filesystem-based Expectations Store
Maturity: Production
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: N/A
Documentation Completeness: Complete
Bug Risk: Low
icon-f4457310612611ebae8a0242ac110002 Expectation Store - S3 - How-to Guide
Use an Amazon Web Services S3 bucket to store expectations.
Maturity: Beta
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Complete
Bug Risk: Low
icon-f44573ce612611ebae8a0242ac110002 Expectation Store - GCS - How-to Guide
Use a Google Cloud Platform Cloud Storage bucket to store expectations.
Maturity: Beta
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Partial
Bug Risk: Low
icon-f4457482612611ebae8a0242ac110002 Expectation Store - Azure - How-to Guide
Use Microsoft Azure Blob Storage to store expectations.
Maturity: N/A
Details:
API Stability: Stable
Implementation Completeness: Minimal
Unit Test Coverage: Minimal
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Minimal
Bug Risk: Moderate
_key_class
remove_key(self, key)
serialize(self, key, value)
deserialize(self, key, value)
self_check(self, pretty_print)
class great_expectations.data_context.store.ValidationsStore(store_backend=None, runtime_environment=None, store_name=None)

Bases: great_expectations.data_context.store.store.Store

A ValidationsStore manages Validation Results to ensure they are accessible via a Data Context for review and rendering into Data Docs.

Feature Maturity

icon-f445c68a612611ebae8a0242ac110002 Validations Store - Filesystem - How-to Guide
Use a locally-mounted filesystem to store validation results.
Maturity: Production
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: N/A
Documentation Completeness: Complete
Bug Risk: Low
icon-f445c824612611ebae8a0242ac110002 Validations Store - S3 - How-to Guide
Use an Amazon Web Services S3 bucket to store validation results.
Maturity: Beta
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Complete
Bug Risk: Low
icon-f445c900612611ebae8a0242ac110002 Validations Store - GCS - How-to Guide
# What it does <br /> Store validation results in a Google Cloud Storage bucket. You may optionally specify a key to use. <br /> <br /> See the GCS Store backend [module docs](https://docs.greatexpectations.io/en/latest/autoapi/great_expectations/data_context/store/tuple_store_backend/index.html#great_expectations.data_context.store.tuple_store_backend.TupleGCSStoreBackend) for more information.”
Maturity: Beta
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Partial
Bug Risk: Low
icon-f445c9c8612611ebae8a0242ac110002 Validations Store - Azure - How-to Guide
Use Microsoft Azure Blob Storage to store validation results.
Maturity: N/A
Details:
API Stability: Stable
Implementation Completeness: Minimal
Unit Test Coverage: Minimal
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Minimal
Bug Risk: Moderate
_key_class
serialize(self, key, value)
deserialize(self, key, value)
self_check(self, pretty_print)
property config(self)
class great_expectations.data_context.store.SqlAlchemyQueryStore(credentials, queries=None, store_backend=None, runtime_environment=None, store_name=None)

Bases: great_expectations.data_context.store.store.Store

SqlAlchemyQueryStore stores queries by name, and makes it possible to retrieve the resulting value by query name.

_key_class
_convert_key(self, key)
get(self, key)
set(self, key, value)
get_query_result(self, key, query_parameters=None)
property config(self)
class great_expectations.data_context.store.HtmlSiteStore(store_backend=None, runtime_environment=None)

A HtmlSiteStore facilitates publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results.

Feature Maturity

icon-f4464b0a612611ebae8a0242ac110002 HTML Site Store - Filesystem - How-to Guide
For publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on the Filesystem
Maturity: Production
Details:
API Stability: Mostly Stable (profiling)
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: N/A
Documentation Completeness: Partial
Bug Risk: Low
icon-f4464c90612611ebae8a0242ac110002 HTML Site Store - S3 - How-to Guide
For publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on S3
Maturity: Beta
Details:
API Stability: Mostly Stable (profiling)
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Complete
Bug Risk: Moderate
icon-f4464d6c612611ebae8a0242ac110002 HTMLSiteStore - GCS - How-to Guide
For publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on GCS
Maturity: Beta
Details:
API Stability: Mostly Stable (profiling)
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Partial (needs auth)
Bug Risk: Moderate (resource URL may have bugs)
icon-f4464e2a612611ebae8a0242ac110002 HTMLSiteStore - Azure - How-to Guide
For publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on Azure Blob Storage
Maturity: N/A
Details:
API Stability: Mostly Stable (profiling)
Implementation Completeness: Minimal
Unit Test Coverage: Minimal
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Minimal
Bug Risk: Moderate
_key_class
get(self, key)
set(self, key, serialized_value)
get_url_for_resource(self, resource_identifier=None, only_if_exists=True)

Return the URL of the HTML document that renders a resource (e.g., an expectation suite or a validation result).

Parameters

resource_identifier – ExpectationSuiteIdentifier, ValidationResultIdentifier or any other type’s identifier. The argument is optional - when not supplied, the method returns the URL of the index page.

Returns

URL (string)

_validate_key(self, key)
list_keys(self)
write_index_page(self, page)

This third param_store has a special method, which uses a zero-length tuple as a key.

clean_site(self)
copy_static_assets(self, static_assets_source_dir=None)

Copies static assets, using a special “static_assets” backend store that accepts variable-length tuples as keys, with no filepath_template.

property config(self)