There is an error in meep simulation using mpi.

[semiuse]

Hello.
After adding FluxRegion to simulation, try to check flux information with sim.get_flux_data.
When using 1 core, the flux value is output correctly, but when using 2 or more cores using mpi, the flux value is output as 0.
Is there a solution related to this?

[smartalecH]

You should get the same exact result regardless of the number of procs (unless you are introducing randomness and each proc is seeded differently...)

So something is wrong with your setup but it's not possible to guess based on the information you've provided.

[semiuse]

Thanks for your answer.

I attach the code I am testing and the result.
I ran the same code but.. what's wrong?

■ Test code
import numpy as np
import meep as mp

Air = mp.Medium(index=1.0)
M =mp.Medium(index=1.59)

design_region_width = [135/2]
design_region_height = [0.68]
design_Z_position = [0]
pml_size = 1.0;pml_layers = [mp.PML(pml_size)]
resolution = 10;
design_region_resolution = 25
na = 0.6
focal_length = 90

nf = 1
frequencies = np.array([1 / 0.405])
fcen = 1 / 0.405

source_center = [design_region_width[0]/2, 0, -1]
source_size = mp.Vector3(design_region_width[0], 0, 0)

src = mp.GaussianSource(wavelength=0.405, width=0.100)

source = [mp.Source(src, component=mp.Er,center=source_center,size=source_size),
mp.Source(src, component=mp.Ep,center=source_center,size=source_size,amplitude=1j),]

Sx = pml_size + design_region_width[0]
Sy = pml_size + design_region_width[0]
Sz = 2 * pml_size + 3
cell_size = mp.Vector3(Sx, 0, Sz)

cell_size_2 = mp.Vector3(68.5,0,200)
airy_disk = 2.44*(405*10**-9)2.5/10*-6

sim = mp.Simulation(
cell_size=cell_size_2,
boundary_layers=pml_layers,
resolution=resolution ,
sources=source,
default_material=Air,
dimensions=mp.CYLINDRICAL,
m=-1,
split_chunks_evenly = False,
)
sim.reset_meep()
monitor_size = mp.Vector3(cell_size_2.x - (pml_size), 0, 0)

tran_fr = mp.FluxRegion(center=mp.Vector3((cell_size_2.x-pml_size)/2, 0 , design_region_height[0]/2 + 0.81),
size=mp.Vector3(design_region_width[0],0,0))
tran = sim.add_flux(1/0.405, 0.1, 1, tran_fr)
sim.run(until_after_sources = 30)
flux_trans = sim.get_flux_data(tran)

print(flux_trans)

■ result
Case 1 : num of procs 1
mpiexec -n 1 python test.py

FluxData(E=array([-1.98374886e-03+3.29938913e-03j, -5.37068654e-18-4.39608409e-18j,
-3.05436725e-06+9.98191909e-06j, ...,
5.95308908e-16-1.60013740e-15j, -1.14446978e-01-3.96604159e-02j,
9.95946528e-17-1.92573200e-16j]), H=array([-0.09631837+0.18359964j, -0.18176847-0.13408845j,
0.00183353+0.00054361j, ..., 0.01341489-0.03882735j,
-0.0329049 -0.00956358j, 0.01569025-0.03299112j]))

Case 2 : num of procs 30
mpiexec -n 30 python test.py

FluxData(E=array([0.+0.j, 0.+0.j, 0.+0.j, ..., 0.+0.j, 0.+0.j, 0.+0.j]),
H=array([0.+0.j, 0.+0.j, 0.+0.j, ..., 0.+0.j, 0.+0.j, 0.+0.j]))

[smartalecH]

It sounds like there is probably something wrong with your setup (not necessarily your script). How did you install meep? What system are you running on? Are there multiple versions of MPI on your system?

[semiuse]

1. I installed it as per the documentation.
   (Version 1.24 is in use)
   conda create -n pmp -c conda-forge pymeep==mpi_mpich_
   conda activate pmp

2..I'm using it on the Linux system.
3. When I checked with 'which mpirun', only one path came out.