Store
A Store is a connector to store and retrieve information about metadata in Great Expectations. Stores are available from your Data Context.
Great Expectations supports a variety of Stores for different purposes, but the most common Stores are Expectation Stores, Validations Stores, Checkpoint Stores, Metric Stores, and Evaluation Parameter Stores. Data Docs Stores can also be configured for Data Doc Sites. Each of these Stores is tailored to a specific type of information.
- Expectation Store: a connector to store and retrieve information about collections of verifiable assertions about data. These are Stores for Expectation Suites.
- Validation Result Store: a connector to store and retrieve information about objects generated when data is Validated against an Expectation Suite.
- Checkpoint Store: a connector to store and retrieve information about means for validating data in a production deployment of Great Expectations.
- Evaluation Parameter Store: a connector to store and retrieve information about parameters used during Validation of an Expectation which reference simple expressions or previously generated Metrics.
- Data Docs Store: a connector to store and retrieve information pertaining to Human readable documentation generated from Great Expectations metadata detailing Expectations, Validation Results, etc.
- Metric Stores: a connector to store and retrieve information about computed attributes of data, such as the mean of a column. These differ from the Evaluation Parameter Store in how they are formatted. The data in a Metrics Store is intended to be used for generating reports and analyzing trends, rather than as Evaluation Parameter values.
Relationship to other objects
Each type of Store is designed to interact with a specific subset of information, and thus interacts with a specific subset of objects in Great Expectations. However, all Stores can be listed and modified through your Data Context. For further information on how a given type of Store relates to other objects, please see the corresponding Store type's technical term page:
- Expectation Store technical term page: Relationship to other objects
- Checkpoint Store technical term page: Relationship to other objects
- Validation Result Store technical term page: Relationship to other objects
- Evaluation Parameter Store (or Metric Store) technical term page: Relationship to other objects
- Data Docs Store technical term page: Relationship to other objects
- Metric Store technical term page: Relationship to other objects
Use cases
All Stores are configured during the Setup step of working with Great Expectations. For the most part, the default configurations will permit you to work with Great Expectations in a local environment. You can, however, configure your Stores to be hosted elsewhere.
When creating Expectations, you will use Expectation Stores to store your Expectation SuitesA collection of verifiable assertions about data.. You may also use Validation Result Stores or Evaluation Parameter Stores to configure some ExpectationsA verifiable assertion about data. that require Evaluation ParametersA dynamic value used during Validation of an Expectation which is populated by evaluating simple expressions or by referencing previously generated metrics. as part of their definition, provided you have previously stored relevant MetricsA computed attribute of data such as the mean of a column. in those Stores and then come back to create a new Expectation Suite that references them.
When Validating data, you may store new CheckpointsThe primary means for validating data in a production deployment of Great Expectations. (or retrieve existing ones) from your Checkpoint Store. Checkpoints may also use Expectation Suites retrieved by an Expectation Store, and Expectations that are run by a Checkpoint may retrieve Metrics as input (Evaluation Parameters) from Validation ResultsGenerated when data is Validated against an Expectation or Expectation Suite. in a Validation Results Store, or from values stored in an Evaluation Parameter Store. Checkpoints may also write information about Validation Results into a Validation Results Store, or information about Metrics into an Evaluation Parameter Store (which is why they are also known as Metric Stores).
Access
The stores
key in your great_expectations.yml
file lists your available Stores.
The data_docs_sites
key in your great_expectations.yml
file defines your Data Docs Store configurations. store_backend
keys below data_docs_sites
identify the Data Doc Store used by the Data Docs site.
Run the following Python command to view your Stores:
import great_expectations as gx
context = gx.get_context()
context.stores
Data Docs Stores are the one exception to the above. They are instead configured under the data_docs_sites
key in the great_expectations.yml
file. Each entry under data_docs_sites
will have a store_backend
key. The information under store_backend
will correspond to the Data Doc Store used by that site. Data Doc Stores are not listed by the great_expectations store list
command.
Configure
Your Stores configurations are located in your great_expectations.yml
file below the stores
key. Three stores are required: an Expectation Store, a Validation Result Store, and an Evaluation Parameter Store must always have a valid entry in the great_expectations.yml
file. Additional Stores can be configured for uses such as Data Docs and Checkpoints (and a Checkpoint Store configuration entry will automatically be added if one does not exist when you attempt to save a Checkpoint for the first time.)
Note that unlike other Stores, your Data Docs Stores are configured under each individual site under the data_docs_sites
key in the great_expectations.yml
file.
Related documentation
For more information about a specific type of Store, see the Store type's technical term definition page: