great_expectations.core.async_executor
¶
Light weight wrappers to facilitate single code path for both when concurrency is enabled and disabled.
WARNING: This module is experimental.
Module Contents¶
Classes¶
|
Wrapper around Future to facilitate single code path |
|
Wrapper around ThreadPoolExecutor to facilitate single code path |
Functions¶
|
Patch urllib3 to enable a higher default max pool size to reduce concurrency bottlenecks. |
-
class
great_expectations.core.async_executor.
AsyncResult
(future: Optional[Future] = None, value: Optional[Any] = None)¶ Wrapper around Future to facilitate single code path for both when concurrency is enabled and disabled.
WARNING: This class is experimental.
-
result
(self)¶ Return the value corresponding to the AsyncExecutor.submit() call, blocking if necessary until the execution finishes.
-
-
class
great_expectations.core.async_executor.
AsyncExecutor
(concurrency_config: Optional[ConcurrencyConfig], max_workers: int)¶ Bases:
contextlib.AbstractContextManager
Wrapper around ThreadPoolExecutor to facilitate single code path for both when concurrency is enabled and disabled.
WARNING: This class is experimental.
-
__exit__
(self, exc_type, exc_value, traceback)¶ Raise any exception triggered within the runtime context.
-
submit
(self, fn, *args, **kwargs)¶ Submits a callable to be executed with the given arguments.
Execution occurs either concurrently on a different thread or synchronously (e.g. on the main thread) depending on how the AsyncExecutor instance was initialized.
-
shutdown
(self)¶ Clean-up the resources associated with the AsyncExecutor and blocks until all running async results finish executing.
It is preferable to not call this method explicitly, and instead use the with statement to ensure shutdown is called.
-
property
execute_concurrently
(self)¶
-
-
great_expectations.core.async_executor.
patch_https_connection_pool
(concurrency_config: ConcurrencyConfig)¶ Patch urllib3 to enable a higher default max pool size to reduce concurrency bottlenecks.
To have any effect, this method must be called before any database connections are made, e.g. by scripts leveraging Great Expectations before using the Great Expectations library. See example usage in tests/performance/test_bigquery_benchmarks.py.
WARNING: This function is experimental.