great_expectations.core
¶
Subpackages¶
great_expectations.core.usage_statistics
great_expectations.core.usage_statistics.anonymizers
great_expectations.core.usage_statistics.anonymizers.action_anonymizer
great_expectations.core.usage_statistics.anonymizers.anonymizer
great_expectations.core.usage_statistics.anonymizers.batch_anonymizer
great_expectations.core.usage_statistics.anonymizers.batch_kwargs_anonymizer
great_expectations.core.usage_statistics.anonymizers.data_docs_site_anonymizer
great_expectations.core.usage_statistics.anonymizers.datasource_anonymizer
great_expectations.core.usage_statistics.anonymizers.expectation_suite_anonymizer
great_expectations.core.usage_statistics.anonymizers.site_builder_anonymizer
great_expectations.core.usage_statistics.anonymizers.store_anonymizer
great_expectations.core.usage_statistics.anonymizers.store_backend_anonymizer
great_expectations.core.usage_statistics.anonymizers.validation_operator_anonymizer
great_expectations.core.usage_statistics.schemas
great_expectations.core.usage_statistics.usage_statistics
Submodules¶
Package Contents¶
Classes¶
DataContextKey objects are used to uniquely identify resources used by the DataContext. |
|
|
Base schema class with which to define custom schemas. |
|
|
|
A RunIdentifier identifies a run (collection of validations) by run_name and run_time. |
|
Base schema class with which to define custom schemas. |
|
ExpectationConfiguration defines the parameters and name of a specific expectation. |
|
Base schema class with which to define custom schemas. |
|
This ExpectationSuite object has create, read, update, and delete functionality for its expectations: |
|
Base schema class with which to define custom schemas. |
|
|
|
Base schema class with which to define custom schemas. |
|
|
|
Base schema class with which to define custom schemas. |
Functions¶
|
Parse a parameter expression to identify dependencies including GE URNs. |
|
|
|
Register a method to invoke after deserializing an object. The method |
|
Register a method to invoke before serializing an object. The method |
|
|
Helper function to convert an object to one that is json serializable |
|
|
Helper function to convert an object to one that is json serializable |
|
-
great_expectations.core.
ge_version
¶
-
class
great_expectations.core.
DataContextKey
¶ Bases:
object
DataContextKey objects are used to uniquely identify resources used by the DataContext.
A DataContextKey is designed to support clear naming with multiple representations including a hashable version making it suitable for use as the key in a dictionary.
-
abstract
to_tuple
(self)¶
-
classmethod
from_tuple
(cls, tuple_)¶
-
abstract
to_fixed_length_tuple
(self)¶
-
abstract classmethod
from_fixed_length_tuple
(cls, tuple_)¶
-
__eq__
(self, other)¶ Return self==value.
-
__ne__
(self, other)¶ Return self!=value.
-
__hash__
(self)¶ Return hash(self).
-
__repr__
(self)¶ Return repr(self).
-
abstract
-
great_expectations.core.
find_evaluation_parameter_dependencies
(parameter_expression)¶ Parse a parameter expression to identify dependencies including GE URNs.
- Parameters
parameter_expression – the parameter to parse
- Returns
“urns”: set of strings that are valid GE URN objects
”other”: set of non-GE URN strings that are required to evaluate the parameter expression
- Return type
a dictionary including
-
great_expectations.core.
ge_urn
¶
-
great_expectations.core.
nested_update
(d, u)¶
-
exception
great_expectations.core.
DataContextError
(message)¶ Bases:
great_expectations.exceptions.GreatExpectationsError
Common base class for all non-exit exceptions.
-
exception
great_expectations.core.
InvalidCacheValueError
(result_dict)¶ Bases:
great_expectations.exceptions.GreatExpectationsError
Common base class for all non-exit exceptions.
-
exception
great_expectations.core.
InvalidExpectationConfigurationError
(message)¶ Bases:
great_expectations.exceptions.GreatExpectationsError
Common base class for all non-exit exceptions.
-
exception
great_expectations.core.
InvalidExpectationKwargsError
(message)¶ Bases:
great_expectations.exceptions.GreatExpectationsError
Common base class for all non-exit exceptions.
-
exception
great_expectations.core.
ParserError
(message)¶ Bases:
great_expectations.exceptions.GreatExpectationsError
Common base class for all non-exit exceptions.
Bases:
great_expectations.exceptions.GreatExpectationsError
Common base class for all non-exit exceptions.
-
class
great_expectations.core.
Schema
(*, only: types.StrSequenceOrSet = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: typing.Dict = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: typing.Union[bool, types.StrSequenceOrSet] = False, unknown: str = None)¶ Bases:
great_expectations.marshmallow__shade.base.SchemaABC
Base schema class with which to define custom schemas.
Example usage:
import datetime as dt from dataclasses import dataclass from great_expectations.marshmallow__shade import Schema, fields @dataclass class Album: title: str release_date: dt.date class AlbumSchema(Schema): title = fields.Str() release_date = fields.Date() album = Album("Beggars Banquet", dt.date(1968, 12, 6)) schema = AlbumSchema() data = schema.dump(album) data # {'release_date': '1968-12-06', 'title': 'Beggars Banquet'}
- Parameters
only – Whitelist of the declared fields to select when instantiating the Schema. If None, all fields are used. Nested fields can be represented with dot delimiters.
exclude – Blacklist of the declared fields to exclude when instantiating the Schema. If a field appears in both only and exclude, it is not used. Nested fields can be represented with dot delimiters.
many – Should be set to True if
obj
is a collection so that the object will be serialized to a list.context – Optional context passed to
fields.Method
andfields.Function
fields.load_only – Fields to skip during serialization (write-only fields)
dump_only – Fields to skip during deserialization (read-only fields)
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE.
Changed in version 3.0.0: prefix parameter removed.
Changed in version 2.0.0: __validators__, __preprocessors__, and __data_handlers__ are removed in favor of marshmallow.decorators.validates_schema, marshmallow.decorators.pre_load and marshmallow.decorators.post_dump. __accessor__ and __error_handler__ are deprecated. Implement the handle_error and get_attribute methods instead.
-
class
Meta
¶ Options object for a Schema.
Example usage:
class Meta: fields = ("id", "email", "date_created") exclude = ("password", "secret_attribute")
Available options:
fields
: Tuple or list of fields to include in the serialized result.additional
: Tuple or list of fields to include in addition to theexplicitly declared fields.
additional
andfields
are mutually-exclusive options.
include
: Dictionary of additional fields to include in the schema. It isusually better to define fields as class variables, but you may need to use this option, e.g., if your fields are Python keywords. May be an OrderedDict.
exclude
: Tuple or list of fields to exclude in the serialized result.Nested fields can be represented with dot delimiters.
dateformat
: Default format for Date <fields.Date> fields.datetimeformat
: Default format for DateTime <fields.DateTime> fields.render_module
: Module to use for loads <Schema.loads> and dumps <Schema.dumps>.Defaults to json from the standard library.
ordered
: If True, order serialization output according to theorder in which fields were declared. Output of Schema.dump will be a collections.OrderedDict.
index_errors
: If True, errors dictionaries will include the indexof invalid items in a collection.
load_only
: Tuple or list of fields to exclude from serialized results.dump_only
: Tuple or list of fields to exclude from deserializationunknown
: Whether to exclude, include, or raise an error for unknownfields in the data. Use EXCLUDE, INCLUDE or RAISE.
register
: Whether to register the Schema with marshmallow’s internalclass registry. Must be True if you intend to refer to this Schema by class name in Nested fields. Only set this to False when memory usage is critical. Defaults to True.
-
TYPE_MAPPING
:typing.Dict[type, typing.Type[ma_fields.Field]]¶
-
error_messages
:typing.Dict[str, str]¶
-
_default_error_messages
:typing.Dict[str, str]¶
-
OPTIONS_CLASS
:type¶
-
opts
:SchemaOpts¶
-
_declared_fields
:typing.Dict[str, ma_fields.Field]¶
-
_hooks
:typing.Dict[types.Tag, typing.List[str]]¶
-
__repr__
(self)¶ Return repr(self).
-
property
dict_class
(self)¶
-
property
set_class
(self)¶
-
classmethod
from_dict
(cls, fields: typing.Dict[str, typing.Union[ma_fields.Field, type]], *, name: str = 'GeneratedSchema')¶ Generate a Schema class given a dictionary of fields.
from great_expectations.marshmallow__shade import Schema, fields PersonSchema = Schema.from_dict({"name": fields.Str()}) print(PersonSchema().load({"name": "David"})) # => {'name': 'David'}
Generated schemas are not added to the class registry and therefore cannot be referred to by name in Nested fields.
- Parameters
fields (dict) – Dictionary mapping field names to field instances.
name (str) – Optional name for the class, which will appear in the
repr
for the class.
New in version 3.0.0.
-
handle_error
(self, error: ValidationError, data: typing.Any, *, many: bool, **kwargs)¶ Custom error handler function for the schema.
- Parameters
error – The ValidationError raised during (de)serialization.
data – The original input data.
many – Value of
many
on dump or load.partial – Value of
partial
on load.
New in version 2.0.0.
Changed in version 3.0.0rc9: Receives many and partial (on deserialization) as keyword arguments.
-
get_attribute
(self, obj: typing.Any, attr: str, default: typing.Any)¶ Defines how to pull values from an object to serialize.
New in version 2.0.0.
Changed in version 3.0.0a1: Changed position of
obj
andattr
.
-
static
_call_and_store
(getter_func, data, *, field_name, error_store, index=None)¶ Call
getter_func
withdata
as its argument, and store any ValidationErrors.- Parameters
getter_func (callable) – Function for getting the serialized/deserialized value from
data
.data – The data passed to
getter_func
.field_name (str) – Field name.
index (int) – Index of the item being validated, if validating a collection, otherwise None.
-
_serialize
(self, obj: typing.Union[_T, typing.Iterable[_T]], *, many: bool = False)¶ Serialize
obj
.- Parameters
obj – The object(s) to serialize.
many (bool) – True if
data
should be serialized as a collection.
- Returns
A dictionary of the serialized data
Changed in version 1.0.0: Renamed from
marshal
.
-
dump
(self, obj: typing.Any, *, many: bool = None)¶ Serialize an object to native Python data types according to this Schema’s fields.
- Parameters
obj – The object to serialize.
many – Whether to serialize obj as a collection. If None, the value for self.many is used.
- Returns
A dict of serialized data
- Return type
dict
New in version 1.0.0.
Changed in version 3.0.0b7: This method returns the serialized data rather than a
(data, errors)
duple. AValidationError
is raised ifobj
is invalid.Changed in version 3.0.0rc9: Validation no longer occurs upon serialization.
-
dumps
(self, obj: typing.Any, *args, many: bool = None, **kwargs)¶ Same as
dump()
, except return a JSON-encoded string.- Parameters
obj – The object to serialize.
many – Whether to serialize obj as a collection. If None, the value for self.many is used.
- Returns
A
json
string- Return type
str
New in version 1.0.0.
Changed in version 3.0.0b7: This method returns the serialized data rather than a
(data, errors)
duple. AValidationError
is raised ifobj
is invalid.
-
_deserialize
(self, data: typing.Union[typing.Mapping[str, typing.Any], typing.Iterable[typing.Mapping[str, typing.Any]]], *, error_store: ErrorStore, many: bool = False, partial=False, unknown=RAISE, index=None)¶ Deserialize
data
.- Parameters
data (dict) – The data to deserialize.
error_store (ErrorStore) – Structure to store errors.
many (bool) – True if
data
should be deserialized as a collection.partial (bool|tuple) – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE.
index (int) – Index of the item being serialized (for storing errors) if serializing a collection, otherwise None.
- Returns
A dictionary of the deserialized data.
-
load
(self, data: typing.Union[typing.Mapping[str, typing.Any], typing.Iterable[typing.Mapping[str, typing.Any]]], *, many: bool = None, partial: typing.Union[bool, types.StrSequenceOrSet] = None, unknown: str = None)¶ Deserialize a data structure to an object defined by this Schema’s fields.
- Parameters
data – The data to deserialize.
many – Whether to deserialize data as a collection. If None, the value for self.many is used.
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE. If None, the value for self.unknown is used.
- Returns
Deserialized data
New in version 1.0.0.
Changed in version 3.0.0b7: This method returns the deserialized data rather than a
(data, errors)
duple. AValidationError
is raised if invalid data are passed.
-
loads
(self, json_data: str, *, many: bool = None, partial: typing.Union[bool, types.StrSequenceOrSet] = None, unknown: str = None, **kwargs)¶ Same as
load()
, except it takes a JSON string as input.- Parameters
json_data – A JSON string of the data to deserialize.
many – Whether to deserialize obj as a collection. If None, the value for self.many is used.
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE. If None, the value for self.unknown is used.
- Returns
Deserialized data
New in version 1.0.0.
Changed in version 3.0.0b7: This method returns the deserialized data rather than a
(data, errors)
duple. AValidationError
is raised if invalid data are passed.
-
_run_validator
(self, validator_func, output, *, original_data, error_store, many, partial, pass_original, index=None)¶
-
validate
(self, data: typing.Mapping, *, many: bool = None, partial: typing.Union[bool, types.StrSequenceOrSet] = None)¶ Validate data against the schema, returning a dictionary of validation errors.
- Parameters
data – The data to validate.
many – Whether to validate data as a collection. If None, the value for self.many is used.
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.
- Returns
A dictionary of validation errors.
New in version 1.1.0.
-
_do_load
(self, data: typing.Union[typing.Mapping[str, typing.Any], typing.Iterable[typing.Mapping[str, typing.Any]]], *, many: bool = None, partial: typing.Union[bool, types.StrSequenceOrSet] = None, unknown: str = None, postprocess: bool = True)¶ Deserialize data, returning the deserialized result. This method is private API.
- Parameters
data – The data to deserialize.
many – Whether to deserialize data as a collection. If None, the value for self.many is used.
partial – Whether to validate required fields. If its value is an iterable, only fields listed in that iterable will be ignored will be allowed missing. If True, all fields will be allowed missing. If None, the value for self.partial is used.
unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE. If None, the value for self.unknown is used.
postprocess – Whether to run post_load methods..
- Returns
Deserialized data
-
_normalize_nested_options
(self)¶ Apply then flatten nested schema options. This method is private API.
-
__apply_nested_option
(self, option_name, field_names, set_operation)¶ Apply nested options to nested fields
-
_init_fields
(self)¶ Update self.fields, self.load_fields, and self.dump_fields based on schema options. This method is private API.
-
on_bind_field
(self, field_name: str, field_obj: ma_fields.Field)¶ Hook to modify a field when it is bound to the Schema.
No-op by default.
-
_bind_field
(self, field_name: str, field_obj: ma_fields.Field)¶ Bind field to the schema, setting any necessary attributes on the field (e.g. parent and name).
Also set field load_only and dump_only values if field_name was specified in
class Meta
.
-
_has_processors
(self, tag)¶
-
_invoke_dump_processors
(self, tag: str, data, *, many: bool, original_data=None)¶
-
_invoke_load_processors
(self, tag: str, data, *, many: bool, original_data, partial: typing.Union[bool, types.StrSequenceOrSet])¶
-
_invoke_field_validators
(self, *, error_store: ErrorStore, data, many: bool)¶
-
_invoke_schema_validators
(self, *, error_store: ErrorStore, pass_many: bool, data, original_data, many: bool, partial: typing.Union[bool, types.StrSequenceOrSet], field_errors: bool = False)¶
-
_invoke_processors
(self, tag: str, *, pass_many: bool, data, many: bool, original_data=None, **kwargs)¶
-
exception
great_expectations.core.
ValidationError
(message: typing.Union[str, typing.List, typing.Dict], field_name: str = SCHEMA, data: typing.Union[typing.Mapping[str, typing.Any], typing.Iterable[typing.Mapping[str, typing.Any]]] = None, valid_data: typing.Union[typing.List[typing.Dict[str, typing.Any]], typing.Dict[str, typing.Any]] = None, **kwargs)¶ Bases:
great_expectations.marshmallow__shade.exceptions.MarshmallowError
Raised when validation fails on a field or schema.
Validators and custom fields should raise this exception.
- Parameters
message – An error message, list of error messages, or dict of error messages. If a dict, the keys are subitems and the values are error messages.
field_name – Field name to store the error on. If None, the error is stored as schema-level error.
data – Raw input data.
valid_data – Valid (de)serialized data.
-
normalized_messages
(self)¶
-
great_expectations.core.
post_load
(fn=None, pass_many=False, pass_original=False)¶ Register a method to invoke after deserializing an object. The method receives the deserialized data and returns the processed data.
By default it receives a single object at a time, transparently handling the
many
argument passed to the Schema’sload()
call. Ifpass_many=True
, the raw data (which may be a collection) is passed.If
pass_original=True
, the original data (before deserializing) will be passed as an additional argument to the method.Changed in version 3.0.0:
partial
andmany
are always passed as keyword arguments to the decorated method.
-
great_expectations.core.
pre_dump
(fn=None, pass_many=False)¶ Register a method to invoke before serializing an object. The method receives the object to be serialized and returns the processed object.
By default it receives a single object at a time, transparently handling the
many
argument passed to the Schema’sdump()
call. Ifpass_many=True
, the raw data (which may be a collection) is passed.Changed in version 3.0.0:
many
is always passed as a keyword arguments to the decorated method.
-
class
great_expectations.core.
DictDot
¶ Bases:
object
-
__getitem__
(self, item)¶
-
__setitem__
(self, key, value)¶
-
__delitem__
(self, key)¶
-
-
great_expectations.core.
logger
¶
-
great_expectations.core.
RESULT_FORMATS
= ['BOOLEAN_ONLY', 'BASIC', 'COMPLETE', 'SUMMARY']¶
-
great_expectations.core.
EvaluationParameterIdentifier
¶
-
great_expectations.core.
in_jupyter_notebook
()¶
-
great_expectations.core.
get_metric_kwargs_id
(metric_name, metric_kwargs)¶
-
great_expectations.core.
convert_to_json_serializable
(data)¶ Helper function to convert an object to one that is json serializable
- Parameters
data – an object to attempt to convert a corresponding json-serializable object
- Returns
(dict) A converted test_object
Warning
test_obj may also be converted in place.
-
great_expectations.core.
ensure_json_serializable
(data)¶ Helper function to convert an object to one that is json serializable
- Parameters
data – an object to attempt to convert a corresponding json-serializable object
- Returns
(dict) A converted test_object
Warning
test_obj may also be converted in place.
-
class
great_expectations.core.
RunIdentifier
(run_name=None, run_time=None)¶ Bases:
great_expectations.core.data_context_key.DataContextKey
A RunIdentifier identifies a run (collection of validations) by run_name and run_time.
-
property
run_name
(self)¶
-
property
run_time
(self)¶
-
to_tuple
(self)¶
-
to_fixed_length_tuple
(self)¶
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
to_json_dict
(self)¶
-
classmethod
from_tuple
(cls, tuple_)¶
-
classmethod
from_fixed_length_tuple
(cls, tuple_)¶
-
property
-
class
great_expectations.core.
RunIdentifierSchema
(*, only: types.StrSequenceOrSet = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: typing.Dict = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: typing.Union[bool, types.StrSequenceOrSet] = False, unknown: str = None)¶ Bases:
great_expectations.marshmallow__shade.Schema
Base schema class with which to define custom schemas.
Example usage:
import datetime as dt from dataclasses import dataclass from great_expectations.marshmallow__shade import Schema, fields @dataclass class Album: title: str release_date: dt.date class AlbumSchema(Schema): title = fields.Str() release_date = fields.Date() album = Album("Beggars Banquet", dt.date(1968, 12, 6)) schema = AlbumSchema() data = schema.dump(album) data # {'release_date': '1968-12-06', 'title': 'Beggars Banquet'}
- Parameters
only – Whitelist of the declared fields to select when instantiating the Schema. If None, all fields are used. Nested fields can be represented with dot delimiters.
exclude – Blacklist of the declared fields to exclude when instantiating the Schema. If a field appears in both only and exclude, it is not used. Nested fields can be represented with dot delimiters.
many – Should be set to True if
obj
is a collection so that the object will be serialized to a list.context – Optional context passed to
fields.Method
andfields.Function
fields.load_only – Fields to skip during serialization (write-only fields)
dump_only – Fields to skip during deserialization (read-only fields)
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE.
Changed in version 3.0.0: prefix parameter removed.
Changed in version 2.0.0: __validators__, __preprocessors__, and __data_handlers__ are removed in favor of marshmallow.decorators.validates_schema, marshmallow.decorators.pre_load and marshmallow.decorators.post_dump. __accessor__ and __error_handler__ are deprecated. Implement the handle_error and get_attribute methods instead.
-
run_name
¶
-
run_time
¶
-
make_run_identifier
(self, data, **kwargs)¶
-
great_expectations.core.
_deduplicate_evaluation_parameter_dependencies
(dependencies)¶
-
class
great_expectations.core.
ExpectationConfiguration
(expectation_type, kwargs, meta=None, success_on_last_run=None)¶ Bases:
great_expectations.types.DictDot
ExpectationConfiguration defines the parameters and name of a specific expectation.
-
kwarg_lookup_dict
¶
-
runtime_kwargs
= ['result_format', 'include_config', 'catch_exceptions']¶
-
patch
(self, op: str, path: str, value: Any)¶ - Parameters
op – A jsonpatch operation. One of ‘add’, ‘replace’, or ‘remove’
path – A jsonpatch path for the patch operation
value – The value to patch
- Returns
The patched ExpectationConfiguration object
-
property
expectation_type
(self)¶
-
property
kwargs
(self)¶
-
_get_default_custom_kwargs
(self)¶
-
get_domain_kwargs
(self)¶
-
get_success_kwargs
(self)¶
-
get_runtime_kwargs
(self)¶
-
applies_to_same_domain
(self, other_expectation_configuration)¶
-
isEquivalentTo
(self, other, match_type='success')¶ ExpectationConfiguration equivalence does not include meta, and relies on equivalence of kwargs.
-
__eq__
(self, other)¶ ExpectationConfiguration equality does include meta, but ignores instance identity.
-
__ne__
(self, other)¶ Return self!=value.
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
to_json_dict
(self)¶
-
get_evaluation_parameter_dependencies
(self)¶
-
-
class
great_expectations.core.
ExpectationConfigurationSchema
(*, only: types.StrSequenceOrSet = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: typing.Dict = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: typing.Union[bool, types.StrSequenceOrSet] = False, unknown: str = None)¶ Bases:
great_expectations.marshmallow__shade.Schema
Base schema class with which to define custom schemas.
Example usage:
import datetime as dt from dataclasses import dataclass from great_expectations.marshmallow__shade import Schema, fields @dataclass class Album: title: str release_date: dt.date class AlbumSchema(Schema): title = fields.Str() release_date = fields.Date() album = Album("Beggars Banquet", dt.date(1968, 12, 6)) schema = AlbumSchema() data = schema.dump(album) data # {'release_date': '1968-12-06', 'title': 'Beggars Banquet'}
- Parameters
only – Whitelist of the declared fields to select when instantiating the Schema. If None, all fields are used. Nested fields can be represented with dot delimiters.
exclude – Blacklist of the declared fields to exclude when instantiating the Schema. If a field appears in both only and exclude, it is not used. Nested fields can be represented with dot delimiters.
many – Should be set to True if
obj
is a collection so that the object will be serialized to a list.context – Optional context passed to
fields.Method
andfields.Function
fields.load_only – Fields to skip during serialization (write-only fields)
dump_only – Fields to skip during deserialization (read-only fields)
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE.
Changed in version 3.0.0: prefix parameter removed.
Changed in version 2.0.0: __validators__, __preprocessors__, and __data_handlers__ are removed in favor of marshmallow.decorators.validates_schema, marshmallow.decorators.pre_load and marshmallow.decorators.post_dump. __accessor__ and __error_handler__ are deprecated. Implement the handle_error and get_attribute methods instead.
-
expectation_type
¶
-
kwargs
¶
-
meta
¶
-
make_expectation_configuration
(self, data, **kwargs)¶
-
class
great_expectations.core.
ExpectationSuite
(expectation_suite_name, expectations=None, evaluation_parameters=None, data_asset_type=None, meta=None)¶ Bases:
object
This ExpectationSuite object has create, read, update, and delete functionality for its expectations: -create: self.add_expectation() -read: self.find_expectation_indexes() -update: self.add_expectation() or self.patch_expectation() -delete: self.remove_expectation()
-
add_citation
(self, comment, batch_kwargs=None, batch_markers=None, batch_parameters=None, citation_date=None)¶
-
isEquivalentTo
(self, other)¶ ExpectationSuite equivalence relies only on expectations and evaluation parameters. It does not include: - data_asset_name - expectation_suite_name - meta - data_asset_type
-
__eq__
(self, other)¶ ExpectationSuite equality ignores instance identity, relying only on properties.
-
__ne__
(self, other)¶ Return self!=value.
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
to_json_dict
(self)¶
-
get_evaluation_parameter_dependencies
(self)¶
-
get_citations
(self, sort=True, require_batch_kwargs=False)¶
-
get_table_expectations
(self)¶ Return a list of table expectations.
-
get_column_expectations
(self)¶ Return a list of column map expectations.
-
static
_filter_citations
(citations, filter_key)¶
-
static
_sort_citations
(citations)¶
-
append_expectation
(self, expectation_config)¶ Appends an expectation.
- Parameters
expectation_config (ExpectationConfiguration) – The expectation to be added to the list.
Notes
May want to add type-checking in the future.
-
remove_expectation
(self, expectation_configuration: ExpectationConfiguration, match_type: str = 'domain', remove_multiple_matches: bool = False)¶ - Parameters
expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against for for the removal of expectations.
match_type –
This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters
that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters
remove_multiple_matches – If True, will remove multiple matching expectations. If False, will raise a ValueError.
Returns: The list of deleted ExpectationConfigurations
- Raises
No match –
More than 1 match, if remove_multiple_matches = False –
-
remove_all_expectations_of_type
(self, expectation_types: Union[List[str], str])¶
-
find_expectation_indexes
(self, expectation_configuration: ExpectationConfiguration, match_type: str = 'domain')¶ - Parameters
expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the index of any matching Expectation Configurations on the suite.
match_type –
This determines what kwargs to use when matching. Options are ‘domain’ to match based on the data evaluated by that expectation, ‘success’ to match based on all configuration parameters
that influence whether an expectation succeeds based on a given batch of data, and ‘runtime’ to match based on all configuration parameters
Returns: A list of indexes of matching ExpectationConfiguration
- Raises
-
find_expectations
(self, expectation_configuration: ExpectationConfiguration, match_type: str = 'domain')¶
-
patch_expectation
(self, expectation_configuration: ExpectationConfiguration, op: str, path: str, value: Any, match_type: str)¶ - Parameters
expectation_configuration – A potentially incomplete (partial) Expectation Configuration to match against to find the expectation to patch.
op – A jsonpatch operation (one of ‘add’,’update’, or ‘remove’) (see http://jsonpatch.com/)
path – A jsonpatch path for the patch operation (see http://jsonpatch.com/)
value – The value to patch (see http://jsonpatch.com/)
match_type – The match type to use for find_expectation_index()
Returns: The patched ExpectationConfiguration
- Raises
No match –
More than 1 match –
-
add_expectation
(self, expectation_configuration: ExpectationConfiguration, match_type: str = 'domain', overwrite_existing: bool = True)¶ - Parameters
expectation_configuration – The ExpectationConfiguration to add or update
match_type – The criteria used to determine whether the Suite already has an ExpectationConfiguration and so whether we should add or replace.
overwrite_existing – If the expectation already exists, this will overwrite if True and raise an error if False.
- Returns
The ExpectationConfiguration to add or replace.
- Raises
More than one match –
One match if overwrite_existing = False –
-
-
class
great_expectations.core.
ExpectationSuiteSchema
(*, only: types.StrSequenceOrSet = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: typing.Dict = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: typing.Union[bool, types.StrSequenceOrSet] = False, unknown: str = None)¶ Bases:
great_expectations.marshmallow__shade.Schema
Base schema class with which to define custom schemas.
Example usage:
import datetime as dt from dataclasses import dataclass from great_expectations.marshmallow__shade import Schema, fields @dataclass class Album: title: str release_date: dt.date class AlbumSchema(Schema): title = fields.Str() release_date = fields.Date() album = Album("Beggars Banquet", dt.date(1968, 12, 6)) schema = AlbumSchema() data = schema.dump(album) data # {'release_date': '1968-12-06', 'title': 'Beggars Banquet'}
- Parameters
only – Whitelist of the declared fields to select when instantiating the Schema. If None, all fields are used. Nested fields can be represented with dot delimiters.
exclude – Blacklist of the declared fields to exclude when instantiating the Schema. If a field appears in both only and exclude, it is not used. Nested fields can be represented with dot delimiters.
many – Should be set to True if
obj
is a collection so that the object will be serialized to a list.context – Optional context passed to
fields.Method
andfields.Function
fields.load_only – Fields to skip during serialization (write-only fields)
dump_only – Fields to skip during deserialization (read-only fields)
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE.
Changed in version 3.0.0: prefix parameter removed.
Changed in version 2.0.0: __validators__, __preprocessors__, and __data_handlers__ are removed in favor of marshmallow.decorators.validates_schema, marshmallow.decorators.pre_load and marshmallow.decorators.post_dump. __accessor__ and __error_handler__ are deprecated. Implement the handle_error and get_attribute methods instead.
-
expectation_suite_name
¶
-
expectations
¶
-
evaluation_parameters
¶
-
data_asset_type
¶
-
meta
¶
-
clean_empty
(self, data)¶
-
prepare_dump
(self, data, **kwargs)¶
-
make_expectation_suite
(self, data, **kwargs)¶
-
class
great_expectations.core.
ExpectationValidationResult
(success=None, expectation_config=None, result=None, meta=None, exception_info=None)¶ Bases:
object
-
__eq__
(self, other)¶ ExpectationValidationResult equality ignores instance identity, relying only on properties.
-
__ne__
(self, other)¶ Return self!=value.
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
validate_result_dict
(self, result)¶
-
to_json_dict
(self)¶
-
get_metric
(self, metric_name, **kwargs)¶
-
-
class
great_expectations.core.
ExpectationValidationResultSchema
(*, only: types.StrSequenceOrSet = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: typing.Dict = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: typing.Union[bool, types.StrSequenceOrSet] = False, unknown: str = None)¶ Bases:
great_expectations.marshmallow__shade.Schema
Base schema class with which to define custom schemas.
Example usage:
import datetime as dt from dataclasses import dataclass from great_expectations.marshmallow__shade import Schema, fields @dataclass class Album: title: str release_date: dt.date class AlbumSchema(Schema): title = fields.Str() release_date = fields.Date() album = Album("Beggars Banquet", dt.date(1968, 12, 6)) schema = AlbumSchema() data = schema.dump(album) data # {'release_date': '1968-12-06', 'title': 'Beggars Banquet'}
- Parameters
only – Whitelist of the declared fields to select when instantiating the Schema. If None, all fields are used. Nested fields can be represented with dot delimiters.
exclude – Blacklist of the declared fields to exclude when instantiating the Schema. If a field appears in both only and exclude, it is not used. Nested fields can be represented with dot delimiters.
many – Should be set to True if
obj
is a collection so that the object will be serialized to a list.context – Optional context passed to
fields.Method
andfields.Function
fields.load_only – Fields to skip during serialization (write-only fields)
dump_only – Fields to skip during deserialization (read-only fields)
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE.
Changed in version 3.0.0: prefix parameter removed.
Changed in version 2.0.0: __validators__, __preprocessors__, and __data_handlers__ are removed in favor of marshmallow.decorators.validates_schema, marshmallow.decorators.pre_load and marshmallow.decorators.post_dump. __accessor__ and __error_handler__ are deprecated. Implement the handle_error and get_attribute methods instead.
-
success
¶
-
expectation_config
¶
-
result
¶
-
meta
¶
-
exception_info
¶
-
convert_result_to_serializable
(self, data, **kwargs)¶
-
make_expectation_validation_result
(self, data, **kwargs)¶
-
class
great_expectations.core.
ExpectationSuiteValidationResult
(success=None, results=None, evaluation_parameters=None, statistics=None, meta=None)¶ Bases:
great_expectations.types.DictDot
-
__eq__
(self, other)¶ ExpectationSuiteValidationResult equality ignores instance identity, relying only on properties.
-
__repr__
(self)¶ Return repr(self).
-
__str__
(self)¶ Return str(self).
-
to_json_dict
(self)¶
-
get_metric
(self, metric_name, **kwargs)¶
-
-
class
great_expectations.core.
ExpectationSuiteValidationResultSchema
(*, only: types.StrSequenceOrSet = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: typing.Dict = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: typing.Union[bool, types.StrSequenceOrSet] = False, unknown: str = None)¶ Bases:
great_expectations.marshmallow__shade.Schema
Base schema class with which to define custom schemas.
Example usage:
import datetime as dt from dataclasses import dataclass from great_expectations.marshmallow__shade import Schema, fields @dataclass class Album: title: str release_date: dt.date class AlbumSchema(Schema): title = fields.Str() release_date = fields.Date() album = Album("Beggars Banquet", dt.date(1968, 12, 6)) schema = AlbumSchema() data = schema.dump(album) data # {'release_date': '1968-12-06', 'title': 'Beggars Banquet'}
- Parameters
only – Whitelist of the declared fields to select when instantiating the Schema. If None, all fields are used. Nested fields can be represented with dot delimiters.
exclude – Blacklist of the declared fields to exclude when instantiating the Schema. If a field appears in both only and exclude, it is not used. Nested fields can be represented with dot delimiters.
many – Should be set to True if
obj
is a collection so that the object will be serialized to a list.context – Optional context passed to
fields.Method
andfields.Function
fields.load_only – Fields to skip during serialization (write-only fields)
dump_only – Fields to skip during deserialization (read-only fields)
partial – Whether to ignore missing fields and not require any fields declared. Propagates down to
Nested
fields as well. If its value is an iterable, only missing fields listed in that iterable will be ignored. Use dot delimiters to specify nested fields.unknown – Whether to exclude, include, or raise an error for unknown fields in the data. Use EXCLUDE, INCLUDE or RAISE.
Changed in version 3.0.0: prefix parameter removed.
Changed in version 2.0.0: __validators__, __preprocessors__, and __data_handlers__ are removed in favor of marshmallow.decorators.validates_schema, marshmallow.decorators.pre_load and marshmallow.decorators.post_dump. __accessor__ and __error_handler__ are deprecated. Implement the handle_error and get_attribute methods instead.
-
success
¶
-
results
¶
-
evaluation_parameters
¶
-
statistics
¶
-
meta
¶
-
prepare_dump
(self, data, **kwargs)¶
-
make_expectation_suite_validation_result
(self, data, **kwargs)¶
-
great_expectations.core.
expectationConfigurationSchema
¶
-
great_expectations.core.
expectationSuiteSchema
¶
-
great_expectations.core.
expectationValidationResultSchema
¶
-
great_expectations.core.
expectationSuiteValidationResultSchema
¶
-
great_expectations.core.
runIdentifierSchema
¶