great_expectations.datasource.data_connector.configured_asset_s3_data_connector
¶
Module Contents¶
Classes¶
|
Extension of ConfiguredAssetFilePathDataConnector used to connect to S3 |
-
great_expectations.datasource.data_connector.configured_asset_s3_data_connector.
boto3
¶
-
great_expectations.datasource.data_connector.configured_asset_s3_data_connector.
logger
¶
-
class
great_expectations.datasource.data_connector.configured_asset_s3_data_connector.
ConfiguredAssetS3DataConnector
(name: str, datasource_name: str, bucket: str, assets: dict, execution_engine: Optional[ExecutionEngine] = None, default_regex: Optional[dict] = None, sorters: Optional[list] = None, prefix: str = '', delimiter: str = '/', max_keys: int = 1000, boto3_options: Optional[dict] = None, batch_spec_passthrough: Optional[dict] = None)¶ -
Extension of ConfiguredAssetFilePathDataConnector used to connect to S3
DataConnectors produce identifying information, called “batch_spec” that ExecutionEngines can use to get individual batches of data. They add flexibility in how to obtain data such as with time-based partitioning, downsampling, or other techniques appropriate for the Datasource.
The ConfiguredAssetS3DataConnector is one of two classes (InferredAssetS3DataConnector being the other one) designed for connecting to data on S3.
A ConfiguredAssetS3DataConnector requires an explicit listing of each DataAsset you want to connect to. This allows more fine-tuning, but also requires more setup.
-
static
sanitize_prefix_for_s3
(text: str)¶ Takes in a given user-prefix and cleans it to work with file-system traversal methods (i.e. add ‘/’ to the end of a string meant to represent a directory)
Customized for S3 paths, ignoring the path separator used by the host OS
-
build_batch_spec
(self, batch_definition: BatchDefinition)¶ Build BatchSpec from batch_definition by calling DataConnector’s build_batch_spec function.
- Parameters
batch_definition (BatchDefinition) – to be used to build batch_spec
- Returns
BatchSpec built from batch_definition
-
_get_data_reference_list_for_asset
(self, asset: Optional[Asset])¶
-
_get_full_file_path_for_asset
(self, path: str, asset: Optional[Asset] = None)¶
-
static