great_expectations.core
¶
Subpackages¶
great_expectations.core.expectation_diagnostics
great_expectations.core.usage_statistics
great_expectations.core.usage_statistics.anonymizers
great_expectations.core.usage_statistics.anonymizers.types
great_expectations.core.usage_statistics.anonymizers.action_anonymizer
great_expectations.core.usage_statistics.anonymizers.anonymizer
great_expectations.core.usage_statistics.anonymizers.base
great_expectations.core.usage_statistics.anonymizers.batch_anonymizer
great_expectations.core.usage_statistics.anonymizers.batch_request_anonymizer
great_expectations.core.usage_statistics.anonymizers.checkpoint_anonymizer
great_expectations.core.usage_statistics.anonymizers.data_connector_anonymizer
great_expectations.core.usage_statistics.anonymizers.data_docs_anonymizer
great_expectations.core.usage_statistics.anonymizers.datasource_anonymizer
great_expectations.core.usage_statistics.anonymizers.expectation_anonymizer
great_expectations.core.usage_statistics.anonymizers.profiler_anonymizer
great_expectations.core.usage_statistics.anonymizers.store_anonymizer
great_expectations.core.usage_statistics.anonymizers.store_backend_anonymizer
great_expectations.core.usage_statistics.anonymizers.validation_operator_anonymizer
great_expectations.core.usage_statistics.events
great_expectations.core.usage_statistics.execution_environment
great_expectations.core.usage_statistics.package_dependencies
great_expectations.core.usage_statistics.schemas
great_expectations.core.usage_statistics.usage_statistics
great_expectations.core.usage_statistics.util
Submodules¶
great_expectations.core._docs_decorators
great_expectations.core.async_executor
great_expectations.core.batch
great_expectations.core.batch_manager
great_expectations.core.batch_spec
great_expectations.core.config_peer
great_expectations.core.config_provider
great_expectations.core.config_substitutor
great_expectations.core.configuration
great_expectations.core.data_context_key
great_expectations.core.domain
great_expectations.core.evaluation_parameters
great_expectations.core.expectation_configuration
great_expectations.core.expectation_suite
great_expectations.core.expectation_validation_result
great_expectations.core.http
great_expectations.core.id_dict
great_expectations.core.metric
great_expectations.core.metric_domain_types
great_expectations.core.profiler_types_mapping
great_expectations.core.run_identifier
great_expectations.core.serializer
great_expectations.core.urn
great_expectations.core.util
great_expectations.core.yaml_handler
Package Contents¶
Classes¶
|
ExpectationConfiguration defines the parameters and name of a specific expectation. |
|
This ExpectationSuite object has create, read, update, and delete functionality for its expectations: |
|
A convenience class for migrating away from untyped dictionaries to stronger typed objects. |
|
A convenience class for migrating away from untyped dictionaries to stronger typed objects. |
|
dict() -> new empty dictionary |
|
A RunIdentifier identifies a run (collection of validations) by run_name and run_time. |
Functions¶
|
-
class
great_expectations.core.
ExpectationConfiguration
(expectation_type: str, kwargs: dict, meta: Optional[dict] = None, success_on_last_run: Optional[bool] = None, ge_cloud_id: Optional[str] = None, expectation_context: Optional[ExpectationContext] = None, rendered_content: Optional[List[RenderedAtomicContent]] = None)¶ Bases:
great_expectations.types.SerializableDictDot
ExpectationConfiguration defines the parameters and name of a specific expectation.
-
kwarg_lookup_dict
¶
-
runtime_kwargs
= ['result_format', 'include_config', 'catch_exceptions']¶
-
process_evaluation_parameters
(self, evaluation_parameters, interactive_evaluation: bool = True, data_context: Optional[DataContext] = None)¶
-
get_raw_configuration
(self)¶
-
patch
(self, op: str, path: str, value: Any)¶ - Parameters
op – A jsonpatch operation. One of ‘add’, ‘replace’, or ‘remove’
path – A jsonpatch path for the patch operation
value – The value to patch
- Returns
The patched ExpectationConfiguration object
-
property
ge_cloud_id
(self)¶
-
property
expectation_context
(self)¶
-
property
expectation_type
(self)¶
-
property
kwargs
(self)¶
-
property
rendered_content
(self)¶
-
_get_default_custom_kwargs
(self)¶
-
get_domain_kwargs
(self)¶
-
get_success_kwargs
(self)¶
-
get_runtime_kwargs
(self, runtime_configuration: Optional[dict] = None)¶
-
applies_to_same_domain
(self, other_expectation_configuration: ExpectationConfiguration)¶
-
isEquivalentTo
(self, other: Union[dict, ExpectationConfiguration], match_type: str = 'success')¶ ExpectationConfiguration equivalence does not include meta, and relies on equivalence of kwargs.
-
__eq__
(self, other)¶ ExpectationConfiguration equality does include meta, but ignores instance identity.
-
__ne__
(self, other)¶ Return self!=value.
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
to_json_dict
(self)¶ # TODO: <Alex>2/4/2022</Alex> A reference implementation can be provided, once circular import dependencies, caused by relative locations of the “great_expectations/types/__init__.py” and “great_expectations/core/util.py” modules are resolved.
-
get_evaluation_parameter_dependencies
(self)¶
-
static
_update_dependencies_with_expectation_suite_urn
(dependencies: dict, urn: ParseResults)¶
-
_get_expectation_impl
(self)¶
-
validate
(self, validator: Validator, runtime_configuration=None)¶
-
metrics_validate
(self, metrics: Dict, runtime_configuration: Optional[dict] = None, execution_engine: Optional[ExecutionEngine] = None, **kwargs: dict)¶
-
get_domain_type
(self)¶ Return “domain_type” of this expectation.
-
render
(self)¶ Renders content using the atomic prescriptive renderer for this Expectation Configuration to ExpectationConfiguration.rendered_content.
-
-
class
great_expectations.core.
ExpectationConfigurationSchema
¶ Bases:
marshmallow.Schema
-
expectation_type
¶
-
kwargs
¶
-
meta
¶
-
ge_cloud_id
¶
-
expectation_context
¶
-
rendered_content
¶
-
REMOVE_KEYS_IF_NONE
= ['ge_cloud_id', 'expectation_context', 'rendered_content']¶
-
clean_null_attrs
(self, data: dict, **kwargs: dict)¶ Removes the attributes in ExpectationConfigurationSchema.REMOVE_KEYS_IF_NONE during serialization if their values are None.
-
make_expectation_configuration
(self, data: dict, **kwargs)¶
-
-
class
great_expectations.core.
ExpectationSuite
(expectation_suite_name: str, data_context: Optional[AbstractDataContext] = None, expectations: Optional[List[Union[dict, ExpectationConfiguration]]] = None, evaluation_parameters: Optional[dict] = None, data_asset_type: Optional[str] = None, execution_engine_type: Optional[Type[ExecutionEngine]] = None, meta: Optional[dict] = None, ge_cloud_id: Optional[str] = None)¶ Bases:
great_expectations.types.SerializableDictDot
This ExpectationSuite object has create, read, update, and delete functionality for its expectations: -create: self.add_expectation() -read: self.find_expectation_indexes() -update: self.add_expectation() or self.patch_expectation() -delete: self.remove_expectation()
-
property
name
(self)¶
-
add_citation
(self, comment: str, batch_request: Optional[Union[str, Dict[str, Union[str, Dict[str, Any]]]]] = None, batch_definition: Optional[dict] = None, batch_spec: Optional[dict] = None, batch_kwargs: Optional[dict] = None, batch_markers: Optional[dict] = None, batch_parameters: Optional[dict] = None, profiler_config: Optional[dict] = None, citation_date: Optional[Union[str, datetime.datetime]] = None)¶
-
isEquivalentTo
(self, other)¶ ExpectationSuite equivalence relies only on expectations and evaluation parameters. It does not include: - data_asset_name - expectation_suite_name - meta - data_asset_type
-
__eq__
(self, other)¶ ExpectationSuite equality ignores instance identity, relying only on properties.
-
__ne__
(self, other)¶ Return self!=value.
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
__deepcopy__
(self, memo: dict)¶
-
to_json_dict
(self)¶ # TODO: <Alex>2/4/2022</Alex> A reference implementation can be provided, once circular import dependencies, caused by relative locations of the “great_expectations/types/__init__.py” and “great_expectations/core/util.py” modules are resolved.
-
get_evaluation_parameter_dependencies
(self)¶
-
get_citations
(self, sort: bool = True, require_batch_kwargs: bool = False, require_batch_request: bool = False, require_profiler_config: bool = False)¶
-
static
_filter_citations
(citations: List[Dict[str, Any]], filter_key)¶
-
static
_sort_citations
(citations: List[Dict[str, Any]])¶
-
append_expectation
(self, expectation_config)¶ Appends an expectation.
- Parameters
expectation_config (ExpectationConfiguration) – The expectation to be added to the list.
Notes
May want to add type-checking in the future.
-
remove_expectation
(self, expectation_configuration: Optional[ExpectationConfiguration] = None, match_type: str = 'domain', remove_multiple_matches: bool = False, ge_cloud_id: Optional[Union[str, uuid.UUID]] = None)¶ - Parameters
expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against for the removal of expectations.
match_type –
This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters
that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters
remove_multiple_matches – If True, will remove multiple matching expectations. If False, will raise a ValueError.
Returns: The list of deleted ExpectationConfigurations
- Raises
No match –
More than 1 match, if remove_multiple_matches = False –
-
remove_all_expectations_of_type
(self, expectation_types: Union[List[str], str])¶
-
find_expectation_indexes
(self, expectation_configuration: Optional[ExpectationConfiguration] = None, match_type: str = 'domain', ge_cloud_id: Optional[str] = None)¶ Find indexes of Expectations matching the given ExpectationConfiguration on the given match_type. If a ge_cloud_id is provided, match_type is ignored and only indexes of Expectations with matching ge_cloud_id are returned.
- Parameters
expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the index of any matching Expectation Configurations on the suite.
match_type –
This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters
that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters
ge_cloud_id – Great Expectations Cloud id
Returns: A list of indexes of matching ExpectationConfiguration
- Raises
-
find_expectations
(self, expectation_configuration: Optional[ExpectationConfiguration] = None, match_type: str = 'domain', ge_cloud_id: Optional[str] = None)¶ Find Expectations matching the given ExpectationConfiguration on the given match_type. If a ge_cloud_id is provided, match_type is ignored and only Expectations with matching ge_cloud_id are returned.
- Parameters
expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the index of any matching Expectation Configurations on the suite.
match_type –
This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters
that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters
ge_cloud_id – Great Expectations Cloud id
Returns: A list of matching ExpectationConfigurations
-
replace_expectation
(self, new_expectation_configuration: Union[ExpectationConfiguration, dict], existing_expectation_configuration: Optional[ExpectationConfiguration] = None, match_type: str = 'domain', ge_cloud_id: Optional[str] = None)¶ Find Expectations matching the given ExpectationConfiguration on the given match_type. If a ge_cloud_id is provided, match_type is ignored and only Expectations with matching ge_cloud_id are returned.
- Parameters
expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the index of any matching Expectation Configurations on the suite.
match_type –
This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters
that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters
ge_cloud_id – Great Expectations Cloud id
Returns: A list of matching ExpectationConfigurations
-
patch_expectation
(self, expectation_configuration: ExpectationConfiguration, op: str, path: str, value: Any, match_type: str)¶ - Parameters
expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the expectation to patch.
op – A jsonpatch operation (one of ‘add’,’update’, or ‘remove’) (see http://jsonpatch.com/)
path – A jsonpatch path for the patch operation (see http://jsonpatch.com/)
value – The value to patch (see http://jsonpatch.com/)
match_type – The match type to use for find_expectation_index()
Returns: The patched ExpectationConfiguration
- Raises
No match –
More than 1 match –
-
_add_expectation
(self, expectation_configuration: ExpectationConfiguration, send_usage_event: bool = True, match_type: str = 'domain', overwrite_existing: bool = True)¶ This is a private method for adding expectations that allows for usage_events to be suppressed when Expectations are added through internal processing (ie. while building profilers, rendering or validation). It takes in send_usage_event boolean.
- Parameters
expectation_configuration – The ExpectationConfiguration to add or update
send_usage_event – Whether to send a usage_statistics event. When called through ExpectationSuite class’ public add_expectation() method, this is set to True.
match_type – The criteria used to determine whether the Suite already has an ExpectationConfiguration and so whether we should add or replace.
overwrite_existing – If the expectation already exists, this will overwrite if True and raise an error if False.
- Returns
The ExpectationConfiguration to add or replace.
- Raises
More than one match –
One match if overwrite_existing = False –
-
send_usage_event
(self, success: bool)¶
-
add_expectation_configurations
(self, expectation_configurations: List[ExpectationConfiguration], send_usage_event: bool = True, match_type: str = 'domain', overwrite_existing: bool = True)¶ - Parameters
expectation_configurations – The List of candidate new/modifed “ExpectationConfiguration” objects for Suite.
send_usage_event – Whether to send a usage_statistics event. When called through ExpectationSuite class’ public add_expectation() method, this is set to True.
match_type – The criteria used to determine whether the Suite already has an “ExpectationConfiguration” object, matching the specified criteria, and thus whether we should add or replace (i.e., “upsert”).
overwrite_existing – If “ExpectationConfiguration” already exists, this will cause it to be overwritten if True and raise an error if False.
- Returns
The List of “ExpectationConfiguration” objects attempted to be added or replaced (can differ from the list of “ExpectationConfiguration” objects in “self.expectations” at the completion of this method’s execution).
- Raises
More than one match –
One match if overwrite_existing = False –
-
add_expectation
(self, expectation_configuration: ExpectationConfiguration, send_usage_event: bool = True, match_type: str = 'domain', overwrite_existing: bool = True)¶ - Parameters
expectation_configuration – The ExpectationConfiguration to add or update
send_usage_event – Whether to send a usage_statistics event. When called through ExpectationSuite class’ public add_expectation() method, this is set to True.
match_type – The criteria used to determine whether the Suite already has an ExpectationConfiguration and so whether we should add or replace.
overwrite_existing – If the expectation already exists, this will overwrite if True and raise an error if False.
- Returns
The ExpectationConfiguration to add or replace.
- Raises
More than one match –
One match if overwrite_existing = False –
-
show_expectations_by_domain_type
(self)¶ Displays “ExpectationConfiguration” list, grouped by “domain_type”, in predetermined designated order.
-
show_expectations_by_expectation_type
(self, expectation_configurations: Optional[List[ExpectationConfiguration]] = None)¶ Displays “ExpectationConfiguration” list, grouped by “expectation_type”, in predetermined designated order.
-
get_grouped_and_ordered_expectations_by_domain_type
(self)¶ Returns “ExpectationConfiguration” list in predetermined order by passing appropriate methods for retrieving “ExpectationConfiguration” lists by corresponding “domain_type” (with “table” first; then “column”, and so on).
-
get_grouped_and_ordered_expectations_by_expectation_type
(self)¶ Returns “ExpectationConfiguration” list, grouped by “expectation_type”, in predetermined designated order.
-
get_table_expectations
(self)¶ Return a list of table expectations.
-
get_column_expectations
(self)¶ Return a list of column map expectations.
-
get_column_pair_expectations
(self)¶ Return a list of column_pair map expectations.
-
get_multicolumn_expectations
(self)¶ Return a list of multicolumn map expectations.
-
get_grouped_and_ordered_expectations_by_column
(self, expectation_type_filter: Optional[str] = None)¶
-
static
_get_expectations_by_domain_using_accessor_method
(domain_type: str, accessor_method: Callable)¶
-
render
(self)¶ Renders content using the atomic prescriptive renderer for each expectation configuration associated with this ExpectationSuite to ExpectationConfiguration.rendered_content.
-
property
-
class
great_expectations.core.
ExpectationSuiteSchema
¶ Bases:
marshmallow.Schema
-
expectation_suite_name
¶
-
ge_cloud_id
¶
-
expectations
¶
-
evaluation_parameters
¶
-
data_asset_type
¶
-
meta
¶
-
clean_empty
(self, data)¶
-
prepare_dump
(self, data, **kwargs)¶
-
-
great_expectations.core.
expectationSuiteSchema
¶
-
class
great_expectations.core.
ExpectationSuiteValidationResult
(success: Optional[bool] = None, results: Optional[list] = None, evaluation_parameters: Optional[dict] = None, statistics: Optional[dict] = None, meta: Optional[ExpectationSuiteValidationResultMeta] = None, ge_cloud_id: Optional[UUID] = None)¶ Bases:
great_expectations.types.SerializableDictDot
A convenience class for migrating away from untyped dictionaries to stronger typed objects.
Can be instantiated with arguments:
- my_A = MyClassA(
foo=”a string”, bar=1,
)
Can be instantiated from a dictionary:
- my_A = MyClassA(
- **{
“foo”: “a string”, “bar”: 1,
}
)
Can be accessed using both dictionary and dot notation
my_A.foo == “a string” my_A.bar == 1
my_A[“foo”] == “a string” my_A[“bar”] == 1
Pairs nicely with @dataclass:
@dataclass() class MyClassA(DictDot):
foo: str bar: int
Can be made immutable:
@dataclass(frozen=True) class MyClassA(DictDot):
foo: str bar: int
For more examples of usage, please see test_dataclass_serializable_dot_dict_pattern.py in the tests folder.
-
__eq__
(self, other)¶ ExpectationSuiteValidationResult equality ignores instance identity, relying only on properties.
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
to_json_dict
(self)¶ # TODO: <Alex>2/4/2022</Alex> A reference implementation can be provided, once circular import dependencies, caused by relative locations of the “great_expectations/types/__init__.py” and “great_expectations/core/util.py” modules are resolved.
-
get_metric
(self, metric_name, **kwargs)¶
-
get_failed_validation_results
(self)¶
-
class
great_expectations.core.
ExpectationSuiteValidationResultSchema
¶ Bases:
marshmallow.Schema
-
success
¶
-
results
¶
-
evaluation_parameters
¶
-
statistics
¶
-
meta
¶
-
ge_cloud_id
¶
-
checkpoint_name
¶
-
prepare_dump
(self, data, **kwargs)¶
-
make_expectation_suite_validation_result
(self, data, **kwargs)¶
-
-
class
great_expectations.core.
ExpectationValidationResult
(success: Optional[bool] = None, expectation_config: Optional['ExpectationConfiguration'] = None, result: Optional[dict] = None, meta: Optional[dict] = None, exception_info: Optional[dict] = None, rendered_content: Optional[RenderedAtomicContent] = None, **kwargs: dict)¶ Bases:
great_expectations.types.SerializableDictDot
A convenience class for migrating away from untyped dictionaries to stronger typed objects.
Can be instantiated with arguments:
- my_A = MyClassA(
foo=”a string”, bar=1,
)
Can be instantiated from a dictionary:
- my_A = MyClassA(
- **{
“foo”: “a string”, “bar”: 1,
}
)
Can be accessed using both dictionary and dot notation
my_A.foo == “a string” my_A.bar == 1
my_A[“foo”] == “a string” my_A[“bar”] == 1
Pairs nicely with @dataclass:
@dataclass() class MyClassA(DictDot):
foo: str bar: int
Can be made immutable:
@dataclass(frozen=True) class MyClassA(DictDot):
foo: str bar: int
For more examples of usage, please see test_dataclass_serializable_dot_dict_pattern.py in the tests folder.
-
__eq__
(self, other)¶ ExpectationValidationResult equality ignores instance identity, relying only on properties.
-
__ne__
(self, other)¶ Return self!=value.
-
__repr__
(self)¶ # TODO: <Alex>5/9/2022</Alex> This implementation is non-ideal (it was agreed to employ it for development expediency). A better approach would consist of “__str__()” calling “__repr__()”, while all output options are handled through state variables.
-
__str__
(self)¶ # TODO: <Alex>5/9/2022</Alex> This implementation is non-ideal (it was agreed to employ it for development expediency). A better approach would consist of “__str__()” calling “__repr__()”, while all output options are handled through state variables.
-
render
(self)¶ Renders content using the: - atomic prescriptive renderer for the expectation configuration associated with this
ExpectationValidationResult to self.expectation_config.rendered_content
atomic diagnostic renderer for the expectation configuration associated with this ExpectationValidationResult to self.rendered_content.
-
static
validate_result_dict
(result)¶
-
to_json_dict
(self)¶ # TODO: <Alex>2/4/2022</Alex> A reference implementation can be provided, once circular import dependencies, caused by relative locations of the “great_expectations/types/__init__.py” and “great_expectations/core/util.py” modules are resolved.
-
get_metric
(self, metric_name, **kwargs)¶
-
class
great_expectations.core.
ExpectationValidationResultSchema
¶ Bases:
marshmallow.Schema
-
success
¶
-
expectation_config
¶
-
result
¶
-
meta
¶
-
exception_info
¶
-
rendered_content
¶
-
REMOVE_KEYS_IF_NONE
= ['rendered_content']¶
-
convert_result_to_serializable
(self, data, **kwargs)¶
-
clean_null_attrs
(self, data: dict, **kwargs: dict)¶ Removes the attributes in ExpectationValidationResultSchema.REMOVE_KEYS_IF_NONE during serialization if their values are None.
-
make_expectation_validation_result
(self, data, **kwargs)¶
-
-
great_expectations.core.
expectationSuiteValidationResultSchema
¶
-
great_expectations.core.
expectationValidationResultSchema
¶
-
great_expectations.core.
get_metric_kwargs_id
(metric_name, metric_kwargs)¶
-
class
great_expectations.core.
IDDict
¶ Bases:
dict
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s
(key, value) pairs
- dict(iterable) -> new dictionary initialized as if via:
d = {} for k, v in iterable:
d[k] = v
- dict(**kwargs) -> new dictionary initialized with the name=value pairs
in the keyword argument list. For example: dict(one=1, two=2)
-
_id_ignore_keys
:Set[str]¶
-
to_id
(self, id_keys=None, id_ignore_keys=None)¶
-
__hash__
(self)¶ Overrides the default implementation
-
class
great_expectations.core.
RunIdentifier
(run_name: Optional[str] = None, run_time: Optional[Union[datetime.datetime, str]] = None)¶ Bases:
great_expectations.core.data_context_key.DataContextKey
A RunIdentifier identifies a run (collection of validations) by run_name and run_time.
-
property
run_name
(self)¶
-
property
run_time
(self)¶
-
to_tuple
(self)¶
-
to_fixed_length_tuple
(self)¶
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
to_json_dict
(self)¶
-
classmethod
from_tuple
(cls, tuple_)¶
-
classmethod
from_fixed_length_tuple
(cls, tuple_)¶
-
property
-
class
great_expectations.core.
RunIdentifierSchema
¶ Bases:
marshmallow.Schema
-
run_name
¶
-
run_time
¶
-
make_run_identifier
(self, data, **kwargs)¶
-
-
great_expectations.core.
ge_urn
¶