-
-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scotch mesh partitioning does not work as expected in parallel #210
Comments
As promised before, here are, for comparison, the outputs of
and here are the ones for build that works as expected
The only major difference I could spot between the two environments are the differences in the MPI provider, MPICH (failing) and OpenMPI (working), and some minor differences in the build strings. Perhaps this could be related to https://gitlab.inria.fr/scotch/scotch/-/issues/39 ? But still, I have no idea why it should work with OpenMPI build in that case... |
It seems to me that there is an issue with non-determinism in scotch 7 which was not present in scotch 6. I have raised an issue for their feedstock at conda-forge/scotch-feedstock#87 which discusses this problem. |
Solution to issue cannot be found in the documentation.
Issue
I am facing an issue with the mesh partitioning with scotch 7. I provide a MWE below
I would expect that the meshes that are read follow the same partition and, thus, the code should work in parallel. At least it used to work for older versions, as I have made use of this extensively in the tests for my package cashocs, e.g., here https://github.com/sblauth/cashocs/blob/main/tests/test_shape_optimization.py
However, when I run this MWE in parallel I get the following output
The error is related to scotch as using
changes the behavior and makes everything work as expected. However, I strangely observe this behavior only when using MPICH and not with OpenMPI - which I don't understand. I will post the outputs of
conda list
for both environments below.Further, the geometry is generated with Gmsh using the following .geo file:
but I guess any user created mesh could be problematic. I can, of course, provide the xdmf file I have generated / am using.
Installed packages
# Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge _sysroot_linux-64_curr_repodata_hack 3 h69a702a_16 conda-forge alsa-lib 1.2.12 h4ab18f5_0 conda-forge asttokens 2.4.1 pyhd8ed1ab_0 conda-forge attr 2.5.1 h166bdaf_1 conda-forge binutils_impl_linux-64 2.40 ha1999f0_7 conda-forge binutils_linux-64 2.40 hb3c18ed_9 conda-forge blosc 1.21.6 hef167b5_0 conda-forge brotli 1.1.0 hd590300_1 conda-forge brotli-bin 1.1.0 hd590300_1 conda-forge bzip2 1.0.8 hd590300_5 conda-forge c-ares 1.32.1 h4bc722e_0 conda-forge ca-certificates 2024.7.4 hbcca054_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cairo 1.18.0 hbb29018_2 conda-forge certifi 2024.7.4 pyhd8ed1ab_0 conda-forge cftime 1.6.4 py311h18e1886_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge contourpy 1.2.1 py311h9547e67_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dbus 1.13.6 h5008d03_3 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge eigen 3.4.0 h00ab1b0_0 conda-forge exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge expat 2.6.2 h59595ed_0 conda-forge fenics 2019.1.0 h26b4920_48 conda-forge fenics-dijitso 2019.1.0 pyhd8ed1ab_39 conda-forge fenics-dolfin 2019.1.0 py311h849a80a_48 conda-forge fenics-ffc 2019.1.0 pyhd8ed1ab_39 conda-forge fenics-fiat 2019.1.0 pyhd8ed1ab_39 conda-forge fenics-libdolfin 2019.1.0 h1268e15_48 conda-forge fenics-ufl 2019.1.0 py311h38be061_38 conda-forge fftw 3.3.10 mpi_mpich_hbcf76dd_10 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_2 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.53.1 py311h61187de_0 conda-forge freetype 2.12.1 h267a509_2 conda-forge gcc 12.3.0 h915e2ae_13 conda-forge gcc_impl_linux-64 12.3.0 h58ffeeb_13 conda-forge gcc_linux-64 12.3.0 h9528a6a_9 conda-forge gettext 0.22.5 h59595ed_2 conda-forge gettext-tools 0.22.5 h59595ed_2 conda-forge glib 2.80.3 h8a4344b_1 conda-forge glib-tools 2.80.3 h73ef956_1 conda-forge gmp 6.3.0 hac33072_2 conda-forge gmpy2 2.1.5 py311hc4f1f91_1 conda-forge graphite2 1.3.13 h59595ed_1003 conda-forge gst-plugins-base 1.24.5 hbaaba92_0 conda-forge gstreamer 1.24.5 haf2f30d_0 conda-forge gxx 12.3.0 h915e2ae_13 conda-forge gxx_impl_linux-64 12.3.0 h2a574ab_13 conda-forge gxx_linux-64 12.3.0 ha28b414_9 conda-forge h5py 3.11.0 nompi_py311h439e445_102 conda-forge harfbuzz 8.5.0 hfac3d4d_0 conda-forge hdf4 4.2.15 h2a13503_7 conda-forge hdf5 1.14.3 mpi_mpich_h0f54ddc_5 conda-forge hypre 2.31.0 mpi_mpich_hd1da18f_1 conda-forge icu 73.2 h59595ed_0 conda-forge importlib-metadata 8.0.0 pyha770c72_0 conda-forge importlib_metadata 8.0.0 hd8ed1ab_0 conda-forge iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge ipython 8.26.0 pyh707e725_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge kernel-headers_linux-64 3.10.0 h4a8ded7_16 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.5 py311h9547e67_1 conda-forge krb5 1.21.3 h659f571_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lcms2 2.16 hb7c19ff_0 conda-forge ld_impl_linux-64 2.40 hf3520f5_7 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libaec 1.1.3 h59595ed_0 conda-forge libasprintf 0.22.5 h661eb56_2 conda-forge libasprintf-devel 0.22.5 h661eb56_2 conda-forge libblas 3.9.0 22_linux64_openblas conda-forge libboost 1.84.0 hba137d9_3 conda-forge libboost-headers 1.85.0 ha770c72_2 conda-forge libbrotlicommon 1.1.0 hd590300_1 conda-forge libbrotlidec 1.1.0 hd590300_1 conda-forge libbrotlienc 1.1.0 hd590300_1 conda-forge libcap 2.69 h0f662aa_0 conda-forge libcblas 3.9.0 22_linux64_openblas conda-forge libclang-cpp15 15.0.7 default_h127d8a8_5 conda-forge libclang13 18.1.8 default_h6ae225f_0 conda-forge libcups 2.3.3 h4637d8d_4 conda-forge libcurl 8.8.0 hca28451_1 conda-forge libdeflate 1.20 hd590300_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 hd590300_2 conda-forge libevent 2.1.12 hf998b51_1 conda-forge libexpat 2.6.2 h59595ed_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libflac 1.4.3 h59595ed_0 conda-forge libgcc-devel_linux-64 12.3.0 h6b66f73_113 conda-forge libgcc-ng 14.1.0 h77fa898_0 conda-forge libgcrypt 1.11.0 h4ab18f5_0 conda-forge libgettextpo 0.22.5 h59595ed_2 conda-forge libgettextpo-devel 0.22.5 h59595ed_2 conda-forge libgfortran-ng 14.1.0 h69a702a_0 conda-forge libgfortran5 14.1.0 hc5f4f2c_0 conda-forge libglib 2.80.3 h8a4344b_1 conda-forge libgomp 14.1.0 h77fa898_0 conda-forge libgpg-error 1.50 h4f305b6_0 conda-forge libhwloc 2.11.1 default_hecaa2ac_1000 conda-forge libiconv 1.17 hd590300_2 conda-forge libjpeg-turbo 3.0.0 hd590300_1 conda-forge liblapack 3.9.0 22_linux64_openblas conda-forge libllvm15 15.0.7 hb3ce162_4 conda-forge libllvm18 18.1.8 hc9dba70_0 conda-forge libnetcdf 4.9.2 nompi_h135f659_114 conda-forge libnghttp2 1.58.0 h47da74e_1 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libogg 1.3.5 h4ab18f5_0 conda-forge libopenblas 0.3.27 pthreads_hac2b453_1 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.43 h2797004_0 conda-forge libpq 16.3 ha72fbe1_0 conda-forge libptscotch 7.0.4 h2376d02_5 conda-forge libsanitizer 12.3.0 hb8811af_13 conda-forge libscotch 7.0.4 h3055ed5_5 conda-forge libsndfile 1.2.2 hc60ed4a_1 conda-forge libsqlite 3.46.0 hde9e2c9_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-devel_linux-64 12.3.0 h6b66f73_113 conda-forge libstdcxx-ng 14.1.0 hc0a3c3a_0 conda-forge libsystemd0 255 h3516f8a_1 conda-forge libtiff 4.6.0 h1dd3fc0_3 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.4.0 hd590300_0 conda-forge libxcb 1.16 hd590300_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libxkbcommon 1.7.0 h2c5496b_1 conda-forge libxml2 2.12.7 h4c95cb1_3 conda-forge libzip 1.10.1 h2629f0a_3 conda-forge libzlib 1.3.1 h4ab18f5_1 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge matplotlib 3.9.1 py311h38be061_0 conda-forge matplotlib-base 3.9.1 py311hffb96ce_0 conda-forge matplotlib-inline 0.1.7 pyhd8ed1ab_0 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge meshio 5.3.5 pyhd8ed1ab_0 conda-forge metis 5.1.0 h59595ed_1007 conda-forge mpc 1.3.1 hfe3b2da_0 conda-forge mpfr 4.2.1 h9458935_1 conda-forge mpg123 1.32.6 h59595ed_0 conda-forge mpi 1.0 mpich conda-forge mpi4py 3.1.6 py311h0b14fea_1 conda-forge mpich 4.2.2 h4a7f18d_100 conda-forge mpmath 1.3.0 pyhd8ed1ab_0 conda-forge mumps-include 5.7.2 ha770c72_0 conda-forge mumps-mpi 5.7.2 h09c71e5_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.3.0 hf1915f5_4 conda-forge mysql-libs 8.3.0 hca2cd23_4 conda-forge ncurses 6.5 h59595ed_0 conda-forge netcdf4 1.7.1 nompi_py311h25b3b55_101 conda-forge nspr 4.35 h27087fc_0 conda-forge nss 3.102 h593d115_0 conda-forge numpy 1.26.4 py311h64a7726_0 conda-forge openjpeg 2.5.2 h488ebb8_0 conda-forge openssl 3.3.1 h4ab18f5_1 conda-forge packaging 24.1 pyhd8ed1ab_0 conda-forge parmetis 4.0.3 h583469f_1006 conda-forge parso 0.8.4 pyhd8ed1ab_0 conda-forge pcre2 10.44 h0f59acf_0 conda-forge petsc 3.21.3 real_h7906ff3_100 conda-forge petsc4py 3.21.3 py311h3fe0c0c_1 conda-forge pexpect 4.9.0 pyhd8ed1ab_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.4.0 py311h82a398c_0 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pixman 0.43.2 h59595ed_0 conda-forge pkg-config 0.29.2 h4bc722e_1009 conda-forge pkgconfig 1.5.5 pyhd8ed1ab_4 conda-forge pluggy 1.5.0 pyhd8ed1ab_0 conda-forge ply 3.11 pyhd8ed1ab_2 conda-forge prompt-toolkit 3.0.47 pyha770c72_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pulseaudio-client 17.0 hb77b528_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pybind11 2.12.0 py311h9547e67_0 conda-forge pybind11-global 2.12.0 py311h9547e67_0 conda-forge pygments 2.18.0 pyhd8ed1ab_0 conda-forge pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge pyqt 5.15.9 py311hf0fb5b6_5 conda-forge pyqt5-sip 12.12.2 py311hb755f60_5 conda-forge pytest 8.2.2 pyhd8ed1ab_0 conda-forge python 3.11.9 hb806964_0_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python_abi 3.11 4_cp311 conda-forge qhull 2020.2 h434a139_5 conda-forge qt-main 5.15.8 ha2b5568_22 conda-forge readline 8.2 h8228510_1 conda-forge rich 13.7.1 pyhd8ed1ab_0 conda-forge scalapack 2.2.0 h417d24c_2 conda-forge scipy 1.14.0 py311h517d4fd_1 conda-forge setuptools 70.3.0 pyhd8ed1ab_0 conda-forge sip 6.7.12 py311hb755f60_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge slepc 3.21.1 real_h97ad6bc_302 conda-forge slepc4py 3.21.1 py311h3888e7a_103 conda-forge snappy 1.2.1 ha2e4443_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge suitesparse 7.7.0 hf4753ba_1 conda-forge superlu 5.2.2 h00795ac_0 conda-forge superlu_dist 9.0.0 h3feb4ed_1 conda-forge sympy 1.12.1 pypyh2585a3b_103 conda-forge sysroot_linux-64 2.17 h4a8ded7_16 conda-forge tbb 2021.12.0 h434a139_3 conda-forge tk 8.6.13 noxft_h4845f30_101 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tornado 6.4.1 py311h331c9d8_0 conda-forge traitlets 5.14.3 pyhd8ed1ab_0 conda-forge typing_extensions 4.12.2 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge xcb-util 0.4.1 hb711507_2 conda-forge xcb-util-image 0.4.0 hb711507_2 conda-forge xcb-util-keysyms 0.4.1 hb711507_0 conda-forge xcb-util-renderutil 0.3.10 hb711507_0 conda-forge xcb-util-wm 0.4.2 hb711507_0 conda-forge xkeyboard-config 2.42 h4ab18f5_0 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.1.1 hd590300_0 conda-forge xorg-libsm 1.2.4 h7391055_0 conda-forge xorg-libx11 1.8.9 hb711507_1 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxrender 0.9.11 hd590300_0 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge xorg-xf86vidmodeproto 2.3.1 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge zipp 3.19.2 pyhd8ed1ab_0 conda-forge zlib 1.3.1 h4ab18f5_1 conda-forge zstd 1.5.6 ha6fb4c9_0 conda-forge
Environment info
The text was updated successfully, but these errors were encountered: