[RegCNET] ERROR: forrtl: severe (174): SIGSEGV, segmentation fault occurred

Athanasios Tsikerdekis atsikerdekis at gmail.com
Tue Jun 30 22:01:51 CEST 2015


Greetings RegCMers,

I am trying to run RegCM with the 12 bin dust option (DU12). But when I 
set my domain over the Sahara region I get the following error. When my 
domain is set in northern latitudes RegCM seems to run fine. Also when I 
lower the number of the total grid points or when I am using the 4 dust 
bin option the simulation is completed without any problem. I am 
including the *.in file and the output of the model+error.

Since the error is occurs in regcmMPI, there must be a problem with 
parallel processing. I am running my experiments on 24 cores. I already 
tried the "ulimit -s unlimited" before running mpirun.

Any hint or suggestions is greatly appreciated,
Best wishes,
Thanos

ps: You have to disable the chem BC to test the experiment in the *.in 
file I am including.

> forrtl: severe (174): SIGSEGV, segmentation fault occurred
> Image              PC                Routine Line        Source
> regcmMPI           0000000000A9ACDE  Unknown Unknown  Unknown
> regcmMPI           0000000000A78FA2  Unknown Unknown  Unknown
> regcmMPI           0000000000A3A2BE  Unknown Unknown  Unknown
> regcmMPI           00000000004427CE  Unknown Unknown  Unknown
> regcmMPI           000000000059930D  Unknown Unknown  Unknown
> regcmMPI           000000000040C239  Unknown Unknown  Unknown
> regcmMPI           0000000000617F53  Unknown Unknown  Unknown
> regcmMPI           000000000040B94C  Unknown Unknown  Unknown
> libc.so.6          000000373FA1D9F4  Unknown Unknown  Unknown
> regcmMPI           000000000040B859  Unknown Unknown  Unknown
>
> ========================================================
> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> =   EXIT CODE: 174
> =   CLEANING UP REMAINING PROCESSES
> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> ========================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ictp.it/pipermail/regcnet/attachments/20150630/f9db609a/attachment-0001.html>
-------------- next part --------------
! QUADIEEMS domain
 &dimparam
 iy     = 102,
 jx     = 150,
 kz     = 18,
 nsg    = 1,
 /
 &geoparam
 iproj = 'ROTMER',
 ds = 60.0,
 ptop = 5.0,
 clat = 27,
 clon = 15,
 plat = 27,
 plon = 15,
 truelatl = 5.0,
 truelath = 45.0,
 i_band = 0,
 /
 &terrainparam
 domname = 'DustRun',
 smthbdy = .false.,
 lakedpth = .false.,
 fudge_lnd   = .false.,
 fudge_lnd_s = .false.,
 fudge_tex   = .false.,
 fudge_tex_s = .false.,
 ltexture = .true.,
 dirter = 'input_DU12',
 inpter = 'DATASETS',
 /
 &ioparam
 ibyte = 4,
 /
 &debugparam
 debug_level = 1,
 dbgfrq = 3,
 /
 &boundaryparam
 nspgx  = 12,
 nspgd  = 12,
 /
 &globdatparam
 ibdyfrq = 6,
 ssttyp = 'ERSST',
 dattyp = 'EIN15',
 gdate1 = 2007093000,
 gdate2 = 2007110200,
 dirglob = 'input_DU12',
 inpglob = 'DATASETS',
 /
 &globwindow
 lat0 = 0.0
 lat1 = 0.0
 lon0 = 0.0
 lon1 = 0.0
 /
 &restartparam
 ifrest  = .false.,
 mdate0  = 2007100100,
 mdate1  = 2007100100,
 mdate2  = 2007110100,
 /
 &timeparam
 dtrad   =    60.,
 dtabem  =    18.,
 dtsrf   =   600.,
 dt      =   120.,
 /
 &outparam
 ifsave  = .true. ,
   savfrq  =    7200.,
 ifatm  = .true. ,
   atmfrq  =     6.,
 ifrad   = .true. ,
   radfrq  =     6.,
 ifsrf   = .true. ,
 ifsub   = .true. ,
   srffrq  =     3.,
 ifchem  = .true.,
 ifopt   = .true.,
   chemfrq =     6.,
 dirout='output_DU12'
 /
 &physicsparam
 iboudy  =          5,
 ibltyp  =          1,
 icup_lnd =         5,
 icup_ocn =         5,
   igcc   =         1,
 ipptls  =          1,
 iocnflx =          2,
 ipgf    =          0,
 iemiss  =          1,
 lakemod =          0,
 ichem   =          1,
 scenario = 'A1B',
 idcsst = 0,
 iseaice = 0,
 idesseas = 1,
 iconvlwp = 1,
 irrtm = 1, ! Use RRTM==1 radiation scheme instead of CCSM
 /
 &subexparam
 qck1land =   .250E-03,
 qck1oce  =   .250E-03,
 cevaplnd   =   .100E-02,
 cevapoce   =   .100E-02,
 caccrlnd  =      3.000,
 caccroce  =      3.000,
 cftotmax =      0.75,
 /
 &grellparam
 /
 &emanparam
 /
 &chemparam
 chemsimtype = 'DU12', ! Which chemical tracers to be activated.
                           ! One in :
                           !   DUST : Activate 4 dust bins scheme
                           !   DU12 : Activate 12 dust bins scheme
                           !   SSLT : Activate 2 bins Sea salt scheme
                           !   CARB : Activate 4 species black/anthropic
                           !          carbon simulations
                           !   SULF : Activate SO2 and SO4 tracers
                           !   SUCA : Activate both SUKF and CARB
                           !   AERO : Activate all DUST, SSLT, CARB and SULF
                           !   CBMZ : Activate gas phase and sulfate
 ichsolver = 1,   ! Activate the chemical solver
 idirect   = 1,   ! enable or not aerosol feedbacks on radiation and
                  ! dynamics (aerosol direct and semi direct effcts):
                  !  0 = no coupling. Aerosol are only transported and
                  !      don't interact with radiation scheme.
                  !  1 = no coupling to dynamic and thermodynamic. However
                  !      the clear sky surface and top of atmosphere 
                  !      aerosol radiative forcings are diagnosed.
                  !  2 = allows aerosol feedbacks on radiative,
                  !      thermodynamic and dynamic fields.
 ichdrdepo = 1,   ! 1 = enable tracer surface dry deposition. For dust,
                  !     it is calculated by a size settling and dry
                  !     deposition scheme. For other aerosol,a dry
                  !     deposition velocity is simply prescribed further.
                  !     Next release will include an improved aerosol dry
                  !     deposition scheme for non dust aerosols.
 ichcumtra = 1,   ! 1 = enable tracer convective transport and mixing.
 ichsursrc=1,     ! Enable the emissions fluxes.
 ichremlsc=1,     ! 1 = enable tracer rainout
 ichdustemd=2,    ! Choice for parametrisation of dust emission size distribution
                  ! 1 = use the standard scheme (Alfaro et al., Zakey et al.)
                  ! 2 = use a revised soil granulometry ref +
                  ! Kok et al emission size distribution : Menut et al.,2012; + Kok et al., 2011
 rdstemfac = 1, ! Dust emission adjustment factor ( soil erodibility) : linearly reduce or increase the dust flux
 ichebdy = 1,     ! Enable reading of chemical boundary conditions, otherwise put 0 in boundary conditions.
 ichdiag = 0,     ! 1 = enable writing of additional tracer tendency diagnostics in the output
/ 
 &clmparam
 dirclm = '/set/this/to/where/your/input/clm/data/are',
 imask  = 1,
 clmfrq = 12.,
 /
-------------- next part --------------
mkdir: cannot create directory `result': File exists
 No snow.nc file to read
TERRAIN Completed!
SST Completed!
ICBC Completed!
CHEM_ICBC completed!

  This is RegCM trunk
   SVN Revision: tag_4.5-rc1.SVN4771 compiled at: data : Feb 26 2015  time: 23:00:07

 : this run start at  : 2015-06-30 22:27:03+0300
 : it is submitted by : 
 : it is running on   : node007
 : it is using        :           24   processors
 : in directory       : /storage05/st6/DustPaper_2007-2012_NEW_DOMAIN
                       
 CPUS DIM1 =            4
 CPUS DIM2 =            6
 
 Reading model namelist file
 Using default Tiedtke parameter.
 Using default Holtslag parameter.
 Using default RRTM parameter.
 DUST 12 bins simulation
 Initial date of the global simulation: 2007-10-01 00:00:00 UTC         
 Initial date of this run             : 2007-10-01 00:00:00 UTC         
 Final date of this run               : 2007-11-01 00:00:00 UTC         
 Total simulation lenght              :          744  hours
 Timestep in seconds =  120.000000
 Split explicit dtau =   30.000000  60.000000
 Reading Domain file : input_DU12/DustRun_DOMAIN000.nc
 Setting IPCC scenario to A1B     
 Create SAV files :  T
 Create ATM files :  T
 Create RAD files :  T
 Create SRF files :  T
 Create STS files :  T
 Create CHE files :  T
 Create OPT files :  T
 Frequency in hours to create SAV : 7200.0
 Frequency in hours to create ATM :    6.0
 Frequency in hours to create RAD :    6.0
 Frequency in hours to create SRF :    3.0
 Frequency in hours to create CHE :    6.0
 Frequency in hours to create OPT :    6.0
 Physical Parameterizations
  Lateral Boundary conditions :  5
  Semi-Lagrangian Advection   :  0
  Land cumulus conv. scheme   :  5
  Ocean cumulus conv. scheme  :  5
  Moisture schem              :  1
  Ocean Flux scheme           :  2
  Zeng roughness formula      :  1
  Coupling with ROMS ocean    :  0
  Pressure gradient force     :  0
  Prescribed LW emissivity    :  1
  Lake model in BATS          :  0
  Simulate diurnal sst cycle  :  0
  Simulate sea ice cover      :  0
  Simulate desert seasons     :  0
  Enable chem/aerosol model   :  1
  Convective LWP scheme       :  1
 Boundary Pameterizations
  Nudge value high range     :  3.000000
  Nudge value medium range   :  2.000000
  Nudge value low range      :  1.000000
 Model Timestep Pameterizations
  time step for dynamical model in seconds :   120.000000
  time step for surface   model in seconds :   600.000000
  time step for radiation model in minutes :    60.000000
  time step for emission  model in hours   :    18.000000
 Domain grid parameters:
  Map Projection        : ROTMER
  Dot Grid Full Extent  :  150x 102x 18
  Model Top Pressure    :    5.000000 cb
  Model Grid Spacing    :   60.000000 km
  Grid Center Latitude  :   27.000000 deg
  Grid Center longitude :   15.000000 deg
  Pole Latitude         :   27.000000 deg
  Pole longitude        :   15.000000 deg
 PBL limit for Holtstag
  Index of highest allowed pbl :  10
 SUBEX large scale precipitation parameters
  # of bottom no cloud model levels :  1
  Auto-conversion rate:         Land =    0.000250 Ocean =    0.000250
  Relative humidity thresholds: Land =    0.800000 Ocean =    0.900000
  Gultepe factors:              Land =    0.400000 Ocean =    0.400000
  Maximum relative humidity         :    1.010000
  RH0 temperature threshold         :  238.000000
  Land Raindrop Evaporation Rate    :    0.001000
  Ocean Raindrop Evaporation Rate   :    0.001000
  Land Raindrop Accretion Rate      :    3.000000
  Ocean Raindrop Accretion Rate     :    3.000000
  Maximum total cloud cover for rad :    0.750000
  Condensation threshold            :    1.000000
 Tiedtke (1986) Convection Scheme ECMWF 38R2 used.
 Convective Cloud Cover parameters after resolution scaling
  Maximum Convective Cloud Cover :    0.398148
  Convective Cloud Water         :    0.000300
 The surface energy budget is used to calculate the ground temperature.
 The radiation is computed every   30 time steps.
 Relaxation boundary conditions (exponential method)
# k       sigma           a      dsigma       twt(1)       twt(2)         qcon
  1      0.0000      0.0250      0.0500       0.0000       0.0000       0.0000
  2      0.0500      0.0750      0.0500       0.5000       0.5000       0.5000
  3      0.1000      0.1300      0.0600       0.4545       0.5455       0.5455
  4      0.1600      0.1950      0.0700       0.4615       0.5385       0.5385
  5      0.2300      0.2700      0.0800       0.4667       0.5333       0.5333
  6      0.3100      0.3500      0.0800       0.5000       0.5000       0.5000
  7      0.3900      0.4300      0.0800       0.5000       0.5000       0.5000
  8      0.4700      0.5100      0.0800       0.5000       0.5000       0.5000
  9      0.5500      0.5900      0.0800       0.5000       0.5000       0.5000
 10      0.6300      0.6700      0.0800       0.5000       0.5000       0.5000
 11      0.7100      0.7450      0.0700       0.5333       0.4667       0.4667
 12      0.7800      0.8100      0.0600       0.5385       0.4615       0.4615
 13      0.8400      0.8650      0.0500       0.5455       0.4545       0.4545
 14      0.8900      0.9100      0.0400       0.5556       0.4444       0.4444
 15      0.9300      0.9450      0.0300       0.5714       0.4286       0.4286
 16      0.9600      0.9700      0.0200       0.6000       0.4000       0.4000
 17      0.9800      0.9850      0.0100       0.6667       0.3333       0.3333
 18      0.9900      0.9950      0.0100       0.5000       0.5000       0.5000
 19      1.0000
  Constant hor. diff. coef. =  0.450000E+05 m^2 s-1
  Maximumt hor. diff. coef. =  0.468750E+06 m^2 s-1
 READY IC DATA for 2007-10-01 00:00:00 UTC         
 SEARCH BC data for   2007100106
 READY  BC from       2007100100  to   2007100106
  # Ozone profile at (3,3) #
    0.496E-07  0.496E-07  0.496E-07  0.496E-07  0.496E-07  0.523E-07  0.527E-07
    0.577E-07  0.627E-07  0.715E-07  0.852E-07  0.106E-06  0.149E-06  0.226E-06
    0.226E-06  0.414E-06  0.799E-06  0.142E-05  0.318E-05   0.00    
 tracer cbmz index molw
 DUST01          -1  0.000000000000000E+000
 DUST02          -1  0.000000000000000E+000
 DUST03          -1  0.000000000000000E+000
 DUST04          -1  0.000000000000000E+000
 DUST05          -1  0.000000000000000E+000
 DUST06          -1  0.000000000000000E+000
 DUST07          -1  0.000000000000000E+000
 DUST08          -1  0.000000000000000E+000
 DUST09          -1  0.000000000000000E+000
 DUST10          -1  0.000000000000000E+000
 DUST11          -1  0.000000000000000E+000
 DUST12          -1  0.000000000000000E+000
 tracer index coreesponding to bdy species 
  # ichbdy2trac #
         1       2       3       4       5       6       7
         8       9      10      11      12       0       0
         0       0       0       0       0       0       0
         0       0       0       0       0       0       0
         0       0       0       0       0       0       0
         0       0       0       0       0       0       0
         0       0       0       0       0       0       0
         0       0
 Calling inidust
 READY ICCH DATA for 2007-10-01 00:00:00 UTC         
 SEARCH CHBC data for   2007100106
 READY  CHBC from       2007100100  to   2007100106
 Calculating Vertical Modes
 Linearization about standard atmosphere
  # sigma    #
     0.00      0.500E-01  0.100E+00  0.160      0.230      0.310      0.390    
    0.470      0.550      0.630      0.710      0.780      0.840      0.890    
    0.890      0.930      0.960      0.980      0.990      1.000    
  # t mean   #
     218.       218.       218.       218.       230.       239.       248.    
     255.       262.       268.       273.       277.       280.       283.    
     283.       285.       286.       287.       287.    
  # ps mean  #
    100.0    
 Vertical mode problem for kz   =  18
 Number of errors (should be 0) =   0
 Split :    1 => aam :     4.0000, fac :    26.6667
 Split :    2 => aam :     2.0000, fac :    48.0000
 Opening new output file output_DU12//DustRun_ATM.2007100100.nc
 Opening new output file output_DU12//DustRun_SRF.2007100100.nc
 Opening new output file output_DU12//DustRun_STS.2007100100.nc
 Opening new output file output_DU12//DustRun_RAD.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST01.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST02.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST03.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST04.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST05.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST06.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST07.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST08.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST09.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST10.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST11.2007100100.nc
 Opening new output file output_DU12//DustRun_DUST12.2007100100.nc
 Opening new output file output_DU12//DustRun_OPT.2007100100.nc
 JDay       273.00 solar declination angle =  -2.42297491 degrees
                   solar TSI irradiance    =    1367.0000 W/m^2
 $$$ 2007-10-01 03:00:00 UTC, ktau   =               90
 $$$ 1st, 2nd time deriv of ps   =  0.27566E-04 0.21781E-06
 $$$  no. of points w/convection =     846
 SRF variables written at 2007-10-01 03:00:00 UTC         
 $$$ 2007-10-01 06:00:00 UTC, ktau   =              180
 $$$ 1st, 2nd time deriv of ps   =  0.22532E-04 0.19863E-06
 $$$  no. of points w/convection =     952
 SEARCH BC data for   2007100112
 READY  BC from       2007100106  to   2007100112
 SEARCH CHBC data for   2007100112
 READY  CHBC from       2007100106  to   2007100112
 ATM variables written at 2007-10-01 06:00:00 UTC         
 SRF variables written at 2007-10-01 06:00:00 UTC         
 OPT variables written at 2007-10-01 06:00:00 UTC         
 CHE variables written at 2007-10-01 06:00:00 UTC         
 RAD variables written at 2007-10-01 06:00:00 UTC         
 $$$ 2007-10-01 09:00:00 UTC, ktau   =              270
 $$$ 1st, 2nd time deriv of ps   =  0.23594E-04 0.25640E-06
 $$$  no. of points w/convection =     842
 SRF variables written at 2007-10-01 09:00:00 UTC         
 $$$ 2007-10-01 12:00:00 UTC, ktau   =              360
 $$$ 1st, 2nd time deriv of ps   =  0.25462E-04 0.31513E-06
 $$$  no. of points w/convection =     954
 SEARCH BC data for   2007100118
 READY  BC from       2007100112  to   2007100118
 SEARCH CHBC data for   2007100118
 READY  CHBC from       2007100112  to   2007100118
 ATM variables written at 2007-10-01 12:00:00 UTC         
 SRF variables written at 2007-10-01 12:00:00 UTC         
 OPT variables written at 2007-10-01 12:00:00 UTC         
 CHE variables written at 2007-10-01 12:00:00 UTC         
 RAD variables written at 2007-10-01 12:00:00 UTC         
 $$$ 2007-10-01 15:00:00 UTC, ktau   =              450
 $$$ 1st, 2nd time deriv of ps   =  0.19236E-04 0.20775E-06
 $$$  no. of points w/convection =     810
 SRF variables written at 2007-10-01 15:00:00 UTC         
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
regcmMPI           0000000000A9ACDE  Unknown               Unknown  Unknown
regcmMPI           0000000000A78FA2  Unknown               Unknown  Unknown
regcmMPI           0000000000A3A2BE  Unknown               Unknown  Unknown
regcmMPI           00000000004427CE  Unknown               Unknown  Unknown
regcmMPI           000000000059930D  Unknown               Unknown  Unknown
regcmMPI           000000000040C239  Unknown               Unknown  Unknown
regcmMPI           0000000000617F53  Unknown               Unknown  Unknown
regcmMPI           000000000040B94C  Unknown               Unknown  Unknown
libc.so.6          000000373FA1D9F4  Unknown               Unknown  Unknown
regcmMPI           000000000040B859  Unknown               Unknown  Unknown

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 174
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================


More information about the RegCNET mailing list