You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
InheritableThread is a thread class that pyspark provides to support sync between JVM and python threads.
The ThreadPool class used in this package is python's original multiprocessing.pool.ThreadPool.
What goods/bads will it bring? Dunno, I'll start looking around.
Proposal
Without touching this package's source code, a user can already nicely make their own ThreadPool that uses their own Thread class. If this package were to adopt the pyspark Thread class, then the steps may be:
Create a custom DummyProcess that inherits pyspark.InheritableThread.
Create a custom ThreadPool that uses the custom DummyProcess.
Use the custom ThreadPool in this package's SparkDistributedBackend._get_pool method.
The text was updated successfully, but these errors were encountered:
Context
InheritableThread is a thread class that pyspark provides to support sync between JVM and python threads.
The
ThreadPool
class used in this package is python's originalmultiprocessing.pool.ThreadPool
.What goods/bads will it bring? Dunno, I'll start looking around.
Proposal
Without touching this package's source code, a user can already nicely make their own
ThreadPool
that uses their ownThread
class. If this package were to adopt the pyspark Thread class, then the steps may be:DummyProcess
that inheritspyspark.InheritableThread
.ThreadPool
that uses the customDummyProcess
.ThreadPool
in this package'sSparkDistributedBackend._get_pool
method.The text was updated successfully, but these errors were encountered: