[RegCNET] Nested run problems

[Maurice.McHugh at noaa.gov]

Greetings all,

We are attempting to perform a nested RegCM3 run with a very large domain using the openMPI compiled RegCM.  We are able to perform MPI simulations using a 192x192 domain (50km) without any problems.  We can also perform a nested 10km run using a 144x96 domain.

We are now trying to perform a nested RegCM simulation using a 192x288 domain (15km) nested within the output of a 50-km resolution 144x96 spatial domain.  

While we can process the ICBCs and compile the model with this domain, the model crashes when we try to run it distributed across the processors (nproc=96).  

We’ve tried running the same domain on a single processor and it can run (slowly) without crashing.  

Therefore, we think it’s a combination of using MPI and the domain size.  Are there any hard coded limitations (memory, etc) when trying to use the MPI compiled RegCM?  Or is there anything else that could cause this?  

We're running on 64-bit Debian Linux OS and using a version of RegCM3 tweaked to work with the gfortran compiler and using the openMPI library.

Many thanks for your help.

Maurice