great_expectations.core.async_executor

Light weight wrappers to facilitate single code path for both when concurrency is enabled and disabled.

WARNING: This module is experimental.

Module Contents

Classes

AsyncResult(future: Optional[Future] = None, value: Optional[Any] = None)	Wrapper around Future to facilitate single code path
AsyncExecutor(concurrency_config: ConcurrencyConfig, max_workers: int)	Wrapper around ThreadPoolExecutor to facilitate single code path

Functions

patch_https_connection_pool(concurrency_config: ConcurrencyConfig)

Patch urllib3 to enable a higher default max pool size to reduce concurrency bottlenecks.

class great_expectations.core.async_executor.AsyncResult(future: Optional[Future] = None, value: Optional[Any] = None)

Wrapper around Future to facilitate single code path for both when concurrency is enabled and disabled.

WARNING: This class is experimental.

result(self)

Return the value corresponding to the AsyncExecutor.submit() call, blocking if necessary until the execution finishes.

class great_expectations.core.async_executor.AsyncExecutor(concurrency_config: ConcurrencyConfig, max_workers: int)

Bases: contextlib.AbstractContextManager

Wrapper around ThreadPoolExecutor to facilitate single code path for both when concurrency is enabled and disabled.

WARNING: This class is experimental.

__exit__(self, exc_type, exc_value, traceback)

Raise any exception triggered within the runtime context.

submit(self, fn, *args, **kwargs)

Submits a callable to be executed with the given arguments.

Execution occurs either concurrently on a different thread or synchronously (e.g. on the main thread) depending on how the AsyncExecutor instance was initialized.

shutdown(self)

Clean-up the resources associated with the AsyncExecutor and blocks until all running async results finish executing.

It is preferable to not call this method explicitly, and instead use the with statement to ensure shutdown is called.

property execute_concurrently(self)

great_expectations.core.async_executor.patch_https_connection_pool(concurrency_config: ConcurrencyConfig)

Patch urllib3 to enable a higher default max pool size to reduce concurrency bottlenecks.

To have any effect, this method must be called before any database connections are made, e.g. by scripts leveraging Great Expectations before using the Great Expectations library. See example usage in tests/performance/test_bigquery_benchmarks.py.

WARNING: This function is experimental.