-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Load OGR layers in parallel when opening a project #58337
base: master
Are you sure you want to change the base?
Conversation
There was a deadlock between QgsOgrProvider's loadMetadata() and open() functions: - loadMetadata() would first lock data source related mutex (QgsOgrProviderUtils::DatasetWithLayers::mutex) and then lock the global mutex (sGlobalMutex) - the open() function would call QgsOgrProviderUtils::getLayer() where the order of locking is reversed, i.e. first lock the global mutex, then the data source related mutex Fixed by updating loadMetadata() to lock data source related mutex for the minimum time necessary (certainly before the global mutex gets locked).
Until now, creation of GDAL datasets from within OGR provider would get serialized, because QgsOgrProviderUtils::getLayer() starts by locking a global mutex. This is now changed to lock only for the time while shared data structures need to be accessed, and therefore it is possible that multiple threads can open GDAL datasets in parallel.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Wondering if there would be ways of adding tests for multithreaded dataset opening? (perhaps requires C++ test if the GIL is not released in Python)
🪟 Windows buildsDownload Windows builds of this PR for testing. 🪟 Windows Qt6 buildsDownload Windows Qt6 builds of this PR for testing. |
Parallel loading of GDAL and PostgreSQL layers (when opening a QGIS project) has been added in QGIS 3.32 (#53069) - this is a follow up that enables parallel loading of OGR layers.
In my testing scenario with a QGIS project with 7 remote GeoPackages (using
vsicurl
), the time to load map layers went down from ~5.5 seconds to ~1.5 seconds.There are three commits that introduce it: