Segmentation fault when combine a file with 7200x5400 grid using mppnccombine? #127

philenso · 2021-10-29T06:55:01Z

philenso
Oct 29, 2021

Hi all,

I'm facing a problem when combine a MOM restart file with 7200x5400 grid using mppnccombine:

#mppnccombine MOM.res.nc MOM.res.nc.????
Segmentation fault

But it work well when combine a history file of MOM.
Could anybody give me some hints?

Best regards,
Bob

ngs333 · 2021-10-29T14:02:48Z

ngs333
Oct 29, 2021

@philenso
Hi
Could you please provide the name of the host you were running on, the path of the files, and also the
version (e.g. fre/bronx-19) of the NCTools software you are using. Thanks

2 replies

ngs333 Oct 29, 2021

@philenso
Also, in the meantime, please type 'mppnccombine' without arguments to see a list of the required arguments plus some instructions. It will print

               output.nc [input ...]```
plus further instructions.

philenso Oct 29, 2021
Author

Hi,

Thanks!
I run mppnccombine on my own host, not in GFDL.
The version I used is 2.2.6:
#$./mppnccombine
mppnccombine 2.2.6 - (written by [email protected])

Usage: mppnccombine [-v] [-V] [-M] [-a] [-r] [-n #] [-k #] [-e #] [-h #] [-64] [-n4 [-d #] [-s]] [-m]
output.nc [input ...]

And below is one of my input file:
#ncdump -h MOM.res.nc.0000
netcdf MOM.res.nc {
dimensions:
lath = 1350 ;
lonh = 1800 ;
Layer = 75 ;
Time = UNLIMITED ; // (1 currently)
variables:
double lath(lath) ;
lath:long_name = "Latitude" ;
lath:units = "degrees_north" ;
lath:cartesian_axis = "Y" ;
lath:domain_decomposition = 1, 5400, 1, 1350 ;
double lonh(lonh) ;
lonh:long_name = "Longitude" ;
lonh:units = "degrees_east" ;
lonh:cartesian_axis = "X" ;
lonh:domain_decomposition = 1, 7200, 1, 1800 ;
double Layer(Layer) ;
Layer:long_name = "Layer z-rho" ;
Layer:units = "meter" ;
Layer:cartesian_axis = "Z" ;
Layer:positive = "up" ;
double Time(Time) ;
Time:long_name = "Time" ;
Time:units = "days" ;
Time:cartesian_axis = "T" ;
double Temp(Time, Layer, lath, lonh) ;
Temp:long_name = "Potential Temperature" ;
Temp:units = "degC" ;
Temp:checksum = " C7D126A36134F7A" ;

// global attributes:
:filename = "RESTART/MOM.res.nc.0000" ;
:NumFilesInSet = 16 ;

ngs333 · 2021-10-29T17:33:12Z

ngs333
Oct 29, 2021

@philenso
The only thing unusual that I can see from the info you provided are that the grid dimensions are unusually large.
I think the best ways to proceed are either of these two:
A) You can give me access to the files, possibly by pushing them to the gfdl ftp site with secure ftp ( sftp [email protected] ), and then I can download them and run things in the debugger. Or
B) You can look at che core file with the gnu debugger and produce the stack trace via:
gdb mppnccombine core

Of course, B) above depends on having a core (normally you have allow you system to do so, e.g. in bash
typing ulimit -s unlimited), and also the software needed to be compiled with the -g option.

3 replies

philenso Oct 29, 2021
Author

Thank you so much!

Below is the debug information:

[Bob@login_a07 RESTART]$ gdb mppnccombine core.197755
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /g7/Bob/FRE-NCtools-master/postprocessing/mppnccombine/mppnccombine...done.
[New LWP 197755]
Core was generated by `mppnccombine -64 -h 16384 -m MOM.res.nc MOM.res.nc.0000 MOM.res.nc.0001 MOM.res'.
Program terminated with signal 11, Segmentation fault.
#0 process_vars (ncinfile=ncinfile@entry=0x2ba81b7e2010, ncoutfile=ncoutfile@entry=0x2ba81956d010, appendnc=0 '\000',
nrecs=nrecs@entry=0x7ffd65bff06c, nblocks=nblocks@entry=0x7ffd65bff074, bf=bf@entry=0x7ffd65bff070, r=r@entry=0,
nfiles=16, f=f@entry=0, verbose=verbose@entry=0 '\000', missing=missing@entry=1 '\001') at mppnccombine.c:1359
1359 *((double *)(varbuf[(r % (*bf))][v])+offset)=
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 libgcc-4.8.5-16.el7.x86_64

ngs333 Oct 29, 2021

Can you also type where after gdb mppnccombine core finishes with loading.

Also, I dont understand why the last argument (MOM.res) is in the command :
mppnccombine -64 -h 16384 -m MOM.res.nc MOM.res.nc.0000 MOM.res.nc.0001 MOM.res
I guess I dont really know what the complete command is that yo uare using, and its possible some info is cut off.
Have you tried just
mppnccombine -64 -h 16384 -m MOM.res.nc MOM.res.nc.0000 MOM.res.nc.0001 and see if that works?

Finally, also try running with additional options. I woul start with -v and see if that gives any hints.

ngs333 Oct 29, 2021

@philenso Its quite likely that there is a bug, or at least that there is insufficient checks when allocating memory.
I will turn this conversation into a a github issue. Do you have a workaround? Do you need this resolved immediately?
How big is MOM.res.nc.0000 if its run though gzip? And finally, can you sftp the MOM.res.nc.* files into the gfdl sftp site?

ngs333 · 2021-10-29T17:39:58Z

ngs333
Oct 29, 2021

@philenso I should add the mppnccombine options -x , -M and (especially) -v may be helpful if the program may run long enough to provide the info. But also look at the other options.

4 replies

philenso Oct 29, 2021
Author

$ mppnccombine -v -64 -h 16384 -m MOM.res.nc MOM.res.nc.0000 MOM.res.nc.0001
block = 0
2 files to go... processing "MOM.res.nc.0000"
Creating output "MOM.res.nc"
blocking factor=1, num. blocks=1, num. records=1
allocating a buffer of 65544 bytes for the multi-dimensional pointer array
Segmentation fault

$mppnccombine -x -v MOM.res.nc.????
This run will estimate peak memory resident size. No output file will be created.
block = 0
15 files to go... processing "MOM.res.nc.0001"
blocking factor=1, num. blocks=1, num. records=1
allocating a buffer of 65544 bytes for the multi-dimensional pointer array
14 files to go... processing "MOM.res.nc.0002"
blocking factor=1, num. blocks=1, num. records=1
13 files to go... processing "MOM.res.nc.0003"
blocking factor=1, num. blocks=1, num. records=1
12 files to go... processing "MOM.res.nc.0004"
blocking factor=1, num. blocks=1, num. records=1
11 files to go... processing "MOM.res.nc.0005"
blocking factor=1, num. blocks=1, num. records=1
10 files to go... processing "MOM.res.nc.0006"
blocking factor=1, num. blocks=1, num. records=1
9 files to go... processing "MOM.res.nc.0007"
blocking factor=1, num. blocks=1, num. records=1
8 files to go... processing "MOM.res.nc.0008"
blocking factor=1, num. blocks=1, num. records=1
7 files to go... processing "MOM.res.nc.0009"
blocking factor=1, num. blocks=1, num. records=1
6 files to go... processing "MOM.res.nc.0010"
blocking factor=1, num. blocks=1, num. records=1
5 files to go... processing "MOM.res.nc.0011"
blocking factor=1, num. blocks=1, num. records=1
4 files to go... processing "MOM.res.nc.0012"
blocking factor=1, num. blocks=1, num. records=1
3 files to go... processing "MOM.res.nc.0013"
blocking factor=1, num. blocks=1, num. records=1
2 files to go... processing "MOM.res.nc.0014"
blocking factor=1, num. blocks=1, num. records=1
1 file to go... processing "MOM.res.nc.0015"
blocking factor=1, num. blocks=1, num. records=1
Estimated peak memory resident size (k=1) : 22247.5 MB
For estimating RSS for a different value of k, multiply above usage by k.

The peak memory seems quite large

philenso Oct 29, 2021
Author

$ mppnccombine -M -v MOM.res.nc MOM.res.nc.????
block = 0
16 files to go... processing "MOM.res.nc.0000"
rss=13200 KB, delta=13200 KB, maxrss=13200 KB
Creating output "MOM.res.nc"
blocking factor=1, num. blocks=1, num. records=1
allocating a buffer of 65544 bytes for the multi-dimensional pointer array
allocating 43200 bytes for full domain of variable lath
allocating 57600 bytes for full domain of variable lonh
allocating 23328000000 bytes for full domain of variable Temp
Segmentation fault

ngs333 Oct 29, 2021

@philenso
Most likely it fails in allocating the 23GB. How much RAM does your system have?
I think there are some GFDL Analysis computers that will have more than enough RAM.

philenso Oct 29, 2021
Author

My system has about 300GB RAM...
And there is no limit for memory usage for users

ngs333 · 2021-10-29T19:54:13Z

ngs333
Oct 29, 2021

@philenso
There is probably a bug in the code, possibly in handling allocated memory or its indexing, and I will turn the discussion into a github issue soon. Related to that, some questions: a) by when do you need this resolved? b) do you have a workaround? c) what is the size of MOM.res.nc.0000 after its gzipped? d) Can you provide the files in a ftp area such as the gfdl sftp site?
Thanks

2 replies

philenso Oct 30, 2021
Author

Thanks！
a）We hope it can be solved in one month; b) NO; c) Each file is about 1.5GB, and we have 16 files: MOM.res.nc.0000-MOM.res.nc.0015; d)Yes. Should we upload all the files? Please tell me how to upload to file to gfdl sftp site.

ngs333 Oct 30, 2021

Do you have a gfdl user account? Is so then you can use sftp
sftp [email protected] and push up the files to your ftp area with mput or put.
Then let me know what your First and Last (gfdl user name) is and I can download it from there.

philenso · 2021-10-30T03:46:56Z

philenso
Oct 30, 2021
Author

Many thanks for your help. I do not have a GFDL user account. I upload the MOM.res.nc.0000 to cloud. Could you please try this link to see if it could be download. link: https://pan.baidu.com/s/164gFQnWVqUBikvidRTd1zw passcode: qd22 Best regards, Bob

…

------------------ Original ------------------ From:  "Miguel R ***@***.***>; Send time: Saturday, Oct 30, 2021 10:35 AM ***@***.***>; ***@***.***>; ***@***.***>; Subject:  Re: [NOAA-GFDL/FRE-NCtools] Segmentation fault when combine a file with 7200x5400 grid using mppnccombine? (Discussion #127) Do you have a gfdl user account? Is so then you can use sftp sftp ***@***.*** and push up the files to your ftp area with mput or put. Then let me know what your First and Last (gfdl user name) is ang I can download it from there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

0 replies

ngs333 · 2021-11-04T18:36:15Z

ngs333
Nov 4, 2021

@philenso
We havent started fixing the likely bug on this yet, but in the mean time you might see is this
helps as an alternative: https://github.com/coecms/mppnccombine-fast

Also, its possible that the suspected bug affecting processing of large files may be present in other NCTools apps. Do you know what other of these apps you will later use in the workflow or to coninue processing? Thanks

1 reply

philenso Nov 10, 2021
Author

Hi Miguel,

Thank you!
I have successfully installed the "mppnccombine-fast" package and it seems work well for the 7200x5400 grid restart files.
One question is that I used "combine_restarts" to combine the restart files from MOM by one step, and since "combine_restarts" internally use "mppncombine", I have to write an alternative script to combine the restart files one by one with "mppnccombine-fast".

Best regards,
Xinyao

philenso · 2021-11-05T17:33:04Z

philenso
Nov 5, 2021
Author

Hi Miguel, Thank you for your kind help. I try to compile the mppnccombine_fast, and I found the error message: /usr/bin/ld: /opt/hdf5/1.8.12/gnu/lib/libhdf5.a(H5Zdeflate.o): undefined reference to symbol 'inflateEnd' /opt/zlib/1.2.11/gnu/lib/libz.so.1: error adding symbols: DSO missing from command line I seems that the zlib library does not link properly. Could you give me some hints to solve this problem? So far in my workflow with the grid of 7200x5400, I only found problem with mppnccombine, and it is also called by combine_restarts. Best regards, philenso

…

------------------ Original ------------------ From:  "Miguel R ***@***.***>; Send time: Friday, Nov 5, 2021 2:36 AM ***@***.***>; ***@***.***>; ***@***.***>; Subject:  Re: [NOAA-GFDL/FRE-NCtools] Segmentation fault when combine a file with 7200x5400 grid using mppnccombine? (Discussion #127) @philenso We havent started fixing the likely bug on this yet, but in the mean time you might see is this helps as an alternative: https://github.com/coecms/mppnccombine-fast Also, its possible that the suspected bug affecting processing of large files may be present in other NCTools apps. Do you know what other of these apps you will later use in the workflow or to coninue processing? Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

1 reply

ngs333 Nov 30, 2021

@philenso
Hi! I am working on this issue this week. Earlier I tried downloading with the link you sent but I could not.
Can you please upload here at least two of the files to be combined? Also, you should gzip them before uploading.

philenso · 2021-12-08T07:59:55Z

philenso
Dec 8, 2021
Author

Hi Miguel,
Thank you!
It seems that my files are too big to upload (each of them exceed 300MB).

1 reply

ngs333 Dec 8, 2021

@philenso . Thanks for trying. I just finished making some huge files myself. Its not exactly what you have, but I shoul be able to make progress today or very soon.

philenso · 2021-12-08T15:33:25Z

philenso
Dec 8, 2021
Author

Hi Miguel,

Thank you!
I'm trying to upload the files to ZENODO.
Hopefully, it can be completed in a few days and I will let you know as long as it done.

Best,
Philenso

2 replies

philenso Dec 10, 2021
Author

Hi Miguel,

The test files have been uploaded to ZENODO.
Please visit the following links to access the data:
https://doi.org/10.5281/zenodo.5768815
https://doi.org/10.5281/zenodo.5771203

Thank you!

ngs333 Dec 10, 2021

@philenso
Thanks. I was able to download the files. They should help.

ngs333 · 2021-12-10T21:53:27Z

ngs333
Dec 10, 2021

@philenso
I think I have fixed the code and I have created a pull request. I do not know
exactly when the system will merge my fixes into the main branch. In the mean time,
you can copy and compile and use my branch, if you like:

https://github.com/ngs333/FRE-NCtools/tree/combi

Finally, when you do run, please let me know is you are successful , and please let me know
if there are other problems, issues in your workflow using NCTools.
Thanks very much for bringing this issue up.

0 replies

ngs333 · 2021-12-10T22:17:47Z

ngs333
Dec 10, 2021

@philenso
This is the PR, which has some details and how I tested and shows the mppnccombine command
args I used:
#138

0 replies

philenso · 2021-12-21T04:12:54Z

philenso
Dec 21, 2021
Author

Hi Miguel, Thank you so much for your work! I compiled this package and try to combine the restart files, it seems not work well: # mppnccombine ice_model.res.nc ice_model.res.nc.???? Error: cannot write variable "zaxis_1"'s values! Error: cannot write variable "0"'s values! Warning: output file may be incomplete! Could you please take a look at it. Best wishes, Xinyao Original From:"Miguel R Zuniga"< ***@***.*** >; Date:2021/12/11 5:53 To:"NOAA-GFDL/FRE-NCtools"< ***@***.*** >; CC:"容大力"< ***@***.*** >;"Mention"< ***@***.*** >; Subject:Re: [NOAA-GFDL/FRE-NCtools] Segmentation fault when combine a filewith 7200x5400 grid using mppnccombine? (Discussion #127) @philenso I think I have fixed the code and I have created a pull request. I do not know exactly when the system will merge my fixes into the main branch. In the mean time, you can copy and compile and use my branch, if you like: https://github.com/ngs333/FRE-NCtools/tree/combi Finally, when you do run, please let me know is you are successful , and please let me know if there are other problems, issues in your workflow using NCTools. Thanks very much for bringing  this issue up. &mdash; You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

1 reply

ngs333 Dec 21, 2021

@philenso
I think you may need the "-n4" format type argument. I ran as :
mppnccombine -vv -n4 .... At least that is what I used (#138 (comment)) The "-vv" argument is purely for debugging purpose. Can you try that? Thanks
PS There are othe formatting options besides n4 that may suite you better, but at least try n4. Thanks again.

philenso · 2021-12-23T03:46:19Z

philenso
Dec 23, 2021
Author

Hi Miguel， It works! Thank you! Best wishes, Xinyao Original From:"Miguel R Zuniga"< ***@***.*** >; Date:2021/12/21 23:03 To:"NOAA-GFDL/FRE-NCtools"< ***@***.*** >; CC:"容大力"< ***@***.*** >;"Mention"< ***@***.*** >; Subject:Re: [NOAA-GFDL/FRE-NCtools] Segmentation fault when combine a filewith 7200x5400 grid using mppnccombine? (Discussion #127) @philenso I think you may need  the "-n4" argument. I ran as : mppnccombine -vv -n4 ....  At least that is what I used (#138 (comment))  The "-vv" argument is purely for debugging purpose. Can you try that? Thanks &mdash; Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: ***@***.***>

1 reply

ngs333 Dec 23, 2021

@philenso
You are welcome. And please let us know if there are any further issues with the workflow handling those huge files
of yours.

Segmentation fault when combine a file with 7200x5400 grid using mppnccombine? #127

Replies: 13 comments · 18 replies

philenso Oct 29, 2021 Author

philenso Oct 29, 2021 Author

philenso Oct 29, 2021 Author

philenso Oct 29, 2021 Author

philenso Oct 29, 2021 Author

philenso Oct 30, 2021 Author

philenso Oct 30, 2021 Author

philenso Nov 10, 2021 Author

philenso Nov 5, 2021 Author

philenso Dec 8, 2021 Author

philenso Dec 8, 2021 Author

philenso Dec 10, 2021 Author

philenso Dec 21, 2021 Author

philenso Dec 23, 2021 Author

Replies: 13 comments 18 replies

philenso Oct 29, 2021
Author

philenso Oct 29, 2021
Author

philenso Oct 29, 2021
Author

philenso Oct 29, 2021
Author

philenso Oct 29, 2021
Author

philenso Oct 30, 2021
Author

philenso
Oct 30, 2021
Author

philenso Nov 10, 2021
Author

philenso
Nov 5, 2021
Author

philenso
Dec 8, 2021
Author

philenso
Dec 8, 2021
Author

philenso Dec 10, 2021
Author

philenso
Dec 21, 2021
Author

philenso
Dec 23, 2021
Author