great_expectations.data_context.store
¶
Submodules¶
great_expectations.data_context.store.database_store_backend
great_expectations.data_context.store.expectations_store
great_expectations.data_context.store.html_site_store
great_expectations.data_context.store.metric_store
great_expectations.data_context.store.query_store
great_expectations.data_context.store.store
great_expectations.data_context.store.store_backend
great_expectations.data_context.store.tuple_store_backend
great_expectations.data_context.store.validations_store
Package Contents¶
Classes¶
|
A store backend acts as a key-value store that can accept tuples as keys, to abstract away |
|
An Expectations Store provides a way to store Expectation Suites accessible to a Data Context. |
|
A HtmlSiteStore facilitates publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results. |
|
A MetricStore stores ValidationMetric information to be used between runs. |
|
A MetricStore stores ValidationMetric information to be used between runs. |
|
SqlAlchemyQueryStore stores queries by name, and makes it possible to retrieve the resulting value by query |
|
A store is responsible for reading and writing Great Expectations objects |
|
Uses an in-memory dictionary as a store backend. |
|
A store backend acts as a key-value store that can accept tuples as keys, to abstract away |
|
Uses a local filepath as a store. |
|
Uses a GCS bucket as a store. |
|
Uses an S3 bucket as a store. |
|
If filepath_template is provided, the key to this StoreBackend abstract class must be a tuple with |
|
A ValidationsStore manages Validation Results to ensure they are accessible via a Data Context for review and rendering into Data Docs. |
-
class
great_expectations.data_context.store.
DatabaseStoreBackend
(credentials, table_name, key_columns, fixed_length_key=True)¶ Bases:
great_expectations.data_context.store.store_backend.StoreBackend
A store backend acts as a key-value store that can accept tuples as keys, to abstract away reading and writing to a persistence layer.
- In general a StoreBackend implementation must provide implementations of:
_get
_set
list_keys
_has_key
-
_get
(self, key)¶
-
_set
(self, key, value, allow_update=True)¶
-
abstract
_move
(self)¶
-
get_url_for_key
(self, key)¶
-
_convert_engine_and_key_to_url
(self, key)¶
-
_has_key
(self, key)¶
-
list_keys
(self, prefix=())¶
-
remove_key
(self, key)¶
-
class
great_expectations.data_context.store.
ExpectationsStore
(store_backend=None, runtime_environment=None)¶ Bases:
great_expectations.data_context.store.store.Store
An Expectations Store provides a way to store Expectation Suites accessible to a Data Context.
Expectation Store - Git - How-to GuideUse a git repository to store expectation suites.Maturity: ProductionDetails:API Stability: StableImplementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: N/ADocumentation Completeness: CompleteBug Risk: LowExpectation Store - Filesystem - How-to GuideFilesystem-based Expectations StoreMaturity: ProductionDetails:API Stability: StableImplementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: N/ADocumentation Completeness: CompleteBug Risk: LowExpectation Store - S3 - How-to GuideUse an Amazon Web Services S3 bucket to store expectations.Maturity: BetaDetails:API Stability: StableImplementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: CompleteBug Risk: LowExpectation Store - GCS - How-to GuideUse a Google Cloud Platform Cloud Storage bucket to store expectations.Maturity: BetaDetails:API Stability: StableImplementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: PartialBug Risk: LowExpectation Store - Azure - How-to GuideUse Microsoft Azure Blob Storage to store expectations.Maturity: N/ADetails:API Stability: StableImplementation Completeness: MinimalUnit Test Coverage: MinimalIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: MinimalBug Risk: Moderate-
_key_class
¶
-
remove_key
(self, key)¶
-
serialize
(self, key, value)¶
-
deserialize
(self, key, value)¶
-
-
class
great_expectations.data_context.store.
HtmlSiteStore
(store_backend=None, runtime_environment=None)¶ Bases:
object
A HtmlSiteStore facilitates publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results.
HTML Site Store - Filesystem - How-to GuideFor publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on the FilesystemMaturity: ProductionDetails:API Stability: Mostly Stable (profiling)Implementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: N/ADocumentation Completeness: PartialBug Risk: LowHTML Site Store - S3 - How-to GuideFor publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on S3Maturity: BetaDetails:API Stability: Mostly Stable (profiling)Implementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: CompleteBug Risk: ModerateHTMLSiteStore - GCS - How-to GuideFor publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on GCSMaturity: BetaDetails:API Stability: Mostly Stable (profiling)Implementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: Partial (needs auth)Bug Risk: Moderate (resource URL may have bugs)HTMLSiteStore - Azure - How-to GuideFor publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on Azure Blob StorageMaturity: N/ADetails:API Stability: Mostly Stable (profiling)Implementation Completeness: MinimalUnit Test Coverage: MinimalIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: MinimalBug Risk: Moderate-
_key_class
¶
-
get
(self, key)¶
-
set
(self, key, serialized_value)¶
-
get_url_for_resource
(self, resource_identifier=None, only_if_exists=True)¶ Return the URL of the HTML document that renders a resource (e.g., an expectation suite or a validation result).
- Parameters
resource_identifier – ExpectationSuiteIdentifier, ValidationResultIdentifier or any other type’s identifier. The argument is optional - when not supplied, the method returns the URL of the index page.
- Returns
URL (string)
-
_validate_key
(self, key)¶
-
list_keys
(self)¶
-
write_index_page
(self, page)¶ This third param_store has a special method, which uses a zero-length tuple as a key.
-
clean_site
(self)¶
-
copy_static_assets
(self, static_assets_source_dir=None)¶ Copies static assets, using a special “static_assets” backend store that accepts variable-length tuples as keys, with no filepath_template.
-
-
class
great_expectations.data_context.store.
EvaluationParameterStore
(store_backend=None)¶ Bases:
great_expectations.data_context.store.metric_store.MetricStore
A MetricStore stores ValidationMetric information to be used between runs.
-
get_bind_params
(self, run_id)¶
-
-
class
great_expectations.data_context.store.
MetricStore
(store_backend=None)¶ Bases:
great_expectations.data_context.store.store.Store
A MetricStore stores ValidationMetric information to be used between runs.
-
_key_class
¶
-
_validate_value
(self, value)¶
-
serialize
(self, key, value)¶
-
deserialize
(self, key, value)¶
-
-
class
great_expectations.data_context.store.
SqlAlchemyQueryStore
(credentials, queries=None, store_backend=None, runtime_environment=None)¶ Bases:
great_expectations.data_context.store.store.Store
SqlAlchemyQueryStore stores queries by name, and makes it possible to retrieve the resulting value by query name.
-
_key_class
¶
-
_convert_key
(self, key)¶
-
get
(self, key)¶
-
set
(self, key, value)¶
-
get_query_result
(self, key, query_parameters=None)¶
-
-
class
great_expectations.data_context.store.
Store
(store_backend=None, runtime_environment=None)¶ Bases:
object
A store is responsible for reading and writing Great Expectations objects to appropriate backends. It provides a generic API that the DataContext can use independently of any particular ORM and backend.
- An implementation of a store will generally need to define the following:
serialize
deserialize
_key_class (class of expected key type)
All keys must have a to_tuple() method.
-
_key_class
¶
-
_validate_key
(self, key)¶
-
property
store_backend
(self)¶
-
serialize
(self, key, value)¶
-
key_to_tuple
(self, key)¶
-
tuple_to_key
(self, tuple_)¶
-
deserialize
(self, key, value)¶
-
get
(self, key)¶
-
set
(self, key, value)¶
-
list_keys
(self)¶
-
has_key
(self, key)¶
-
class
great_expectations.data_context.store.
InMemoryStoreBackend
(runtime_environment=None, fixed_length_key=False)¶ Bases:
great_expectations.data_context.store.store_backend.StoreBackend
Uses an in-memory dictionary as a store backend.
-
_get
(self, key)¶
-
_set
(self, key, value, **kwargs)¶
-
_move
(self, source_key, dest_key, **kwargs)¶
-
list_keys
(self, prefix=())¶
-
_has_key
(self, key)¶
-
remove_key
(self, key)¶
-
-
class
great_expectations.data_context.store.
StoreBackend
(fixed_length_key=False)¶ Bases:
object
A store backend acts as a key-value store that can accept tuples as keys, to abstract away reading and writing to a persistence layer.
- In general a StoreBackend implementation must provide implementations of:
_get
_set
list_keys
_has_key
-
IGNORED_FILES
= ['.ipynb_checkpoints']¶
-
property
fixed_length_key
(self)¶
-
get
(self, key, **kwargs)¶
-
set
(self, key, value, **kwargs)¶
-
move
(self, source_key, dest_key, **kwargs)¶
-
has_key
(self, key)¶
-
get_url_for_key
(self, key, protocol=None)¶
-
_validate_key
(self, key)¶
-
_validate_value
(self, value)¶
-
abstract
_get
(self, key)¶
-
abstract
_set
(self, key, value, **kwargs)¶
-
abstract
_move
(self, source_key, dest_key, **kwargs)¶
-
abstract
list_keys
(self, prefix=())¶
-
abstract
remove_key
(self, key)¶
-
abstract
_has_key
(self, key)¶
-
is_ignored_key
(self, key)¶
-
class
great_expectations.data_context.store.
TupleFilesystemStoreBackend
(base_directory, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, root_directory=None, fixed_length_key=False)¶ Bases:
great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend
Uses a local filepath as a store.
The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added. The filepath_template is a string template used to convert the key to a filepath.
-
_get
(self, key)¶
-
_set
(self, key, value, **kwargs)¶
-
_move
(self, source_key, dest_key, **kwargs)¶
-
list_keys
(self, prefix=())¶
-
rrmdir
(self, mroot, curpath)¶ recursively removes empty dirs between curpath and mroot inclusive
-
remove_key
(self, key)¶
-
get_url_for_key
(self, key, protocol=None)¶
-
_has_key
(self, key)¶
-
-
class
great_expectations.data_context.store.
TupleGCSStoreBackend
(bucket, project, prefix='', filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, public_urls=True)¶ Bases:
great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend
Uses a GCS bucket as a store.
The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added.
The filepath_template is a string template used to convert the key to a filepath.
-
_move
(self, source_key, dest_key, **kwargs)¶
-
_get
(self, key)¶
-
_set
(self, key, value, content_encoding='utf-8', content_type='application/json')¶
-
list_keys
(self)¶
-
get_url_for_key
(self, key, protocol=None)¶
-
remove_key
(self, key)¶
-
_has_key
(self, key)¶
-
-
class
great_expectations.data_context.store.
TupleS3StoreBackend
(bucket, prefix=None, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False)¶ Bases:
great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend
Uses an S3 bucket as a store.
The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added. The filepath_template is a string template used to convert the key to a filepath.
-
_build_s3_object_key
(self, key)¶
-
_get
(self, key)¶
-
_set
(self, key, value, content_encoding='utf-8', content_type='application/json')¶
-
_move
(self, source_key, dest_key, **kwargs)¶
-
list_keys
(self)¶
-
get_url_for_key
(self, key, protocol=None)¶
-
remove_key
(self, key)¶
-
_has_key
(self, key)¶
-
-
class
great_expectations.data_context.store.
TupleStoreBackend
(filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, fixed_length_key=False)¶ Bases:
great_expectations.data_context.store.store_backend.StoreBackend
If filepath_template is provided, the key to this StoreBackend abstract class must be a tuple with fixed length equal to the number of unique components matching the regex r”{d+}”
For example, in the following template path: expectations/{0}/{1}/{2}/prefix-{2}.json, keys must have three components.
-
_validate_key
(self, key)¶
-
_validate_value
(self, value)¶
-
_convert_key_to_filepath
(self, key)¶
-
_convert_filepath_to_key
(self, filepath)¶
-
verify_that_key_to_filepath_operation_is_reversible
(self)¶
-
-
class
great_expectations.data_context.store.
ValidationsStore
(store_backend=None, runtime_environment=None)¶ Bases:
great_expectations.data_context.store.store.Store
A ValidationsStore manages Validation Results to ensure they are accessible via a Data Context for review and rendering into Data Docs.
Validations Store - Filesystem - How-to GuideUse a locally-mounted filesystem to store validation results.Maturity: ProductionDetails:API Stability: StableImplementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: N/ADocumentation Completeness: CompleteBug Risk: LowValidations Store - S3 - How-to GuideUse an Amazon Web Services S3 bucket to store validation results.Maturity: BetaDetails:API Stability: StableImplementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: CompleteBug Risk: LowValidations Store - GCS - How-to Guide# What it does <br /> Store validation results in a Google Cloud Storage bucket. You may optionally specify a key to use. <br /> <br /> See the GCS Store backend [module docs](https://docs.greatexpectations.io/en/latest/autoapi/great_expectations/data_context/store/tuple_store_backend/index.html#great_expectations.data_context.store.tuple_store_backend.TupleGCSStoreBackend) for more information.”Maturity: BetaDetails:API Stability: StableImplementation Completeness: CompleteUnit Test Coverage: CompleteIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: PartialBug Risk: LowValidations Store - Azure - How-to GuideUse Microsoft Azure Blob Storage to store validation results.Maturity: N/ADetails:API Stability: StableImplementation Completeness: MinimalUnit Test Coverage: MinimalIntegration Infrastructure/Test Coverage: MinimalDocumentation Completeness: MinimalBug Risk: Moderate-
_key_class
¶
-
serialize
(self, key, value)¶
-
deserialize
(self, key, value)¶
-