How to configure an Expectation store on a filesystem¶
By default, newly profiled Expectations are stored in JSON format in the expectations/
subdirectory of your great_expectations
folder. This guide will help you configure a new storage location for Expectations on your filesystem.
Prerequisites: This how-to guide assumes that you have already:
Configured a Data Context.
Configured an Expectation Suite.
Determined a new storage location where you would like to store Expectations. This can either be a local path, or a path to a network filesystem.
Steps¶
Show Docs for V2 (Batch Kwargs) API
Configure a new folder on your filesystem where Expectations will be stored.
Create a new folder where you would like to store your Expectations, and move your existing Expectation files over to the new location. In our case, the name of the Expectations file is
npi_expectations
and the path to our new storage location is/shared_expectations
.# in the great_expectations/ folder mkdir shared_expectations mv expectations/npi_expectations.json shared_expectations/
Identify your Data Context Expectations Store
In your
great_expectations.yml
, look for the following lines. The configuration tells Great Expectations to look for Expectations in a store calledexpectations_store
. Thebase_directory
forexpectations_store
is set toexpectations/
by default.expectations_store_name: expectations_store stores: expectations_store: class_name: ExpectationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: expectations/
Update your configuration file to include a new store for Expectations results on your filesystem
In the example below, Expectations Store is being set to
shared_expectations_filesystem_store
with thebase_directory
set toshared_expectations/
.expectations_store_name: shared_expectations_filesystem_store stores: shared_expectations_filesystem_store: class_name: ExpectationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: shared_expectations/
Confirm that the location has been updated by running
great_expectations store list
.Notice the output contains two Expectation stores: the original
expectations_store
on the local filesystem and theshared_expectations_filesystem_store
we just configured. This is ok, since Great Expectations will look for Expectations in theshared_expectations/
folder as long as we set theexpectations_store_name
variable toshared_expectations_filesystem_store
. The config forexpectations_store
can be removed if you would like.great_expectations store list 2 Stores found: - name: expectations_store class_name: ExpectationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: expectations/ - name: shared_expectations_filesystem_store class_name: ExpectationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: shared_expectations/
Confirm that Expectations can be read from the new storage location by running
great_expectations suite list
.great_expectations suite list 1 Expectation Suite found: - npi_expectations
Show Docs for V3 (Batch Request) API
Configure a new folder on your filesystem where Expectations will be stored.
Create a new folder where you would like to store your Expectations, and move your existing Expectation files over to the new location. In our case, the name of the Expectations file is
npi_expectations
and the path to our new storage location is/shared_expectations
.# in the great_expectations/ folder mkdir shared_expectations mv expectations/npi_expectations.json shared_expectations/
Identify your Data Context Expectations Store
In your
great_expectations.yml
, look for the following lines. The configuration tells Great Expectations to look for Expectations in a store calledexpectations_store
. Thebase_directory
forexpectations_store
is set toexpectations/
by default.expectations_store_name: expectations_store stores: expectations_store: class_name: ExpectationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: expectations/
Update your configuration file to include a new store for Expectations results on your filesystem
In the example below, Expectations Store is being set to
shared_expectations_filesystem_store
with thebase_directory
set toshared_expectations/
.expectations_store_name: shared_expectations_filesystem_store stores: shared_expectations_filesystem_store: class_name: ExpectationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: shared_expectations/
Confirm that the location has been updated by running
great_expectations --v3-api store list
.Notice the output contains two Expectation stores: the original
expectations_store
on the local filesystem and theshared_expectations_filesystem_store
we just configured. This is ok, since Great Expectations will look for Expectations in theshared_expectations/
folder as long as we set theexpectations_store_name
variable toshared_expectations_filesystem_store
. The config forexpectations_store
can be removed if you would like.great_expectations --v3-api store list 2 Stores found: - name: expectations_store class_name: ExpectationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: expectations/ - name: shared_expectations_filesystem_store class_name: ExpectationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: shared_expectations/
Confirm that Expectations can be read from the new storage location by running
great_expectations --v3-api suite list
.great_expectations --v3-api suite list 1 Expectation Suite found: - npi_expectations
Additional Notes¶
For best practices, we highly recommend that you store Expectations in a version-control system like Git. The JSON format of Expectations will allow for informative diff-statements and effective tracking of modifications. In the example below, 2 changes have been made to
npi_expectations
. The Expectation`expect_table_column_count_to_equal
was changed from330
to333
to331
.git log -p npi_expectations.json commit cbc127fb27095364c3c1fcbf6e7f078369b07455 changed expect_table_column_count_to_equal to 331 diff --git a/great_expectations/expectations/npi_expectations.json b/great_expectations/expectations/npi_expectations.json --- a/great_expectations/expectations/npi_expectations.json +++ b/great_expectations/expectations/npi_expectations.json @@ -17,7 +17,7 @@ { "expectation_type": "expect_table_column_count_to_equal", "kwargs": { - "value": 333 + "value": 331 } commit 05b3c8c1ed35d183bac1717d4877fe13bc574963 changed expect_table_column_count_to_equal to 333 diff --git a/great_expectations/expectations/npi_expectations.json b/great_expectations/expectations/npi_expectations.json --- a/great_expectations/expectations/npi_expectations.json +++ b/great_expectations/expectations/npi_expectations.json { "expectation_type": "expect_table_column_count_to_equal", "kwargs": { - "value": 330 + "value": 333 }
If it would be useful to you, please comment with a +1 and feel free to add any suggestions or questions below. Also, please reach out to us on Slack if you would like to learn more, or have any questions.