Segmentation faults

[psathyrella]

When I run on larger samples, IgSCUEAL seems to kick seq faults about ten percent of the time (~95/1000 jobs, where each job has ten sequences). Logs follow:

[SEND] Sequence 04-A-M_2943612 to MPI node 1
[RECEIVE] Sequence 04-A-M_2943612 from node 1 (9 alignments remaining): V4-34*11,J4*02
[SEND] Sequence 04-A-M_1107136 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_1107136","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V1-18*01,J?","SUPPORT":0.1687626935203052,"SCORE":8047.160355358626,"DELTA_IC":-8047.160355358626,"TERMINATOR":0,"SEQUENCE":"GACCACAGACGCATCCACGAGCACTGTTTACATGGCGCTGCGGAGCCTGAAATCTGACGACACGGCCGTGTATTATAGTTCCCGCGATTTCTCATCTGCCCTCTGGGGCCAGGGA","SISTER_SEQUENCES":{"0":"NODE348","1":"NODE209"},"BRANCH_LENGTHS":{"TO_SISTER":{{5.000000025e-09}{0.0001286543421442442}},"TO_PARENT":{{0.1279953855652854}{4485.903776979053}},"TO_QUERY":{{0.2261950337603474}{0.85454963896991}}},"REARRANGEMENTS":{"V1-18*01,J?":0.1687626935203052,"V1-18*01,J5*02":0.1434141888155421,"V1-18*04,J?":0.07873096170526599,"V1-18*02,J?":0.07253781076073025,"V1-18*02,J5*02":0.0697150539934551,"V1-18*04,J5*02":0.0563133354481517,"V1-18*01,J5*01":0.05389071872010095,"V1-18*01,J4*01":0.03007341541002615,"V1-18*02,J5*01":0.02960649717280993,"V1-18*01,J4*02":0.02787880568984089,"V1-18*03,J5*02":0.02450016610967478,"V1-18*04,J5*01":0.02332684532410361,"V1-18*01,J3*01":0.02164179855076386,"V1-18*01,J6*03":0.01949288284065922,"V1-18*01,J2*01":0.01778238543506694,"V1-18*03,J?":0.01187363571830067,"V1-18*02,J4*01":0.0113027372457437,"V1-18*04,J2*01":0.01120192234741949,"V1-18*04,J4*01":0.01089631135708937,"V1-18*04,J6*03":0.0107443750110703,"V1-18*02,J6*03":0.01019296599363384},"MODELS_TRIED":16200,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"NNGACCACAGACGCATCCACGAGCACTGTTTACATGGCGCTGCGGAGCCTGAAATCTGACGACACGGCCGTGTATTATTGGTCGACAGGGCTGAGT","FW3_AA":"?TTDASTSTVYMALRSLKSDDTAVYYWSTGLS","CDR3":"TCCCGC","CDR3_AA":"SR","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"GATTTCTCATCTGCCCTCTGGGGCCAGGGA","J_AA":"DFSSALWGQG","Mapped Read":"---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------NNGACCACAGACGCATCCACGAGCACTGTTTACATGGCGCTGCGGAGCCTGAAATCTGACGACACGGCCGTGTATTATTGGTCGACAGGGCTGAGTTCC---CGC---------GATTTCTCATCTGCCCTCTGGGGCCAGGGA","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{321}{330}{369}}},"BREAKPOINTS":{{76}},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.01404955757345511,0.01409359126603213,0.007159888877939839,0.04189616111431885,0.05905429497046503,0.06170181950948323,0.06713508169240467,0.1654889321992937,0.2310853891229091,0.3409299640085231,1.123540372970069,2.682209327053691,0.7458619817774067,0.02755255857121293,0.1450060289560257,0.4866524126627346,0.05064460134784571,0.1436209425651628,0.009644212957660374,0.002147411334251959,0.01133581575225164,0.0008842481662898203,0.001130882611333214,0.000998782103531933,0.001192719360368263,0.001146212364101266,0.004786997457793554,0.008433031388062354,0.004144938968578325,0.005808816094464929,0.01378854583722991,0.005163536909114533,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"NODE348":0.2646191754622641,"IGHJ1_01_CRF_01":0.02119906189272637,"NODE323":1.639975749263915e-07,"NODE209":0.2399673301615894,"IGHV1_18_01_CRF_1":0.2361599780965632,"NODE215":0.04416736761646182,"NODE222":0.05620579591911656,"IGHJ4_03_CRF_01":0.01087239844141774,"NODE321":5.845226266121082e-07,"IGHJ2_01_CRF_01":0.03645524926902291,"IGHV1_69_05_CRF_1":7.73929145489338e-08,"NODE223":0.0942071725033531,"NODE217":0.0001507980283316502,"NODE210":0.03698425150899463,"IGHJ6_02_CRF_01":0.0001143649879910973,"IGHJ6_01_CRF_01":0.0001201874267519578,"IGHJ4_02_CRF_01":0.04889930069813665,"NODE213":0.03573366424839848,"IGHV1_18_02_CRF_1":0.2171996236272148,"NODE226":0.03798950451401927,"IGHJ6_04_CRF_01":3.891750612592212e-05,"NODE298":5.019366195444026e-08,"NODE263":5.226418026980619e-08,"IGHV1_69_11_CRF_1":5.6426394062391e-09,"IGHV1_69_07_CRF_1":5.642639894449284e-09,"IGHV1_18_04_CRF_1":0.2144841863311469,"NODE218":0.0001067079974052938,"IGHV1_18_03_CRF_1":0.06753355033212066,"IGHJ5_01_CRF_01":0.02140559030268662,"IGHJ6_03_CRF_01":0.0001470857212962572,"IGHJ3_01_CRF_01":0.003013246881429785,"IGHJ5_02_CRF_01":0.2939438580854014,"IGHJ3_02_CRF_01":0.002288994280585856,"IGHJ4_01_CRF_01":0.01598915200875662,"IGHV1_46_02_CRF_1":2.593138547387373e-07,"NODE331":5.685450136126517e-09,"IGHV1_69_03_CRF_1":2.129486249304418e-08,"IGHV1_69_13_CRF_1":5.6426394062391e-09,"IGHV1_69D_01_CRF_1":5.6426394062391e-09,"IGHV1_69_12_CRF_1":5.6426394062391e-09,"NODE264":4.957367864672318e-08,"NODE273":2.816412020031636e-08,"IGHV1_38_4_01_CRF_1":2.86286733191622e-08,"NODE296":2.79534666414367e-08,"NODE310":2.566421342337107e-07,"IGHV1_46_01_CRF_1":2.544756072700783e-07,"IGHV1_46_03_CRF_1":6.162694363348138e-07,"NODE297":3.594941481501865e-08,"NODE305":3.064921138626917e-08,"NODE274":2.842627980530069e-08,"NODE322":8.865403437273759e-07},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-477/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096285 queued and waiting for resources
srun: job 26096285 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof86:04550] *** Process received signal ***
[gizmof86:04550] Signal: Segmentation fault (11)
[gizmof86:04550] Signal code: Address not mapped (1)
[gizmof86:04550] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 4550 on node gizmof86 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof86:04546] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof86:04546] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof86: task 0: Exited with exit code 139

-----

proc 516
out----

Read 10 sequences

[SEND] Sequence 04-A-M_2161043 to MPI node 1
[RECEIVE] Sequence 04-A-M_2161043 from node 1 (9 alignments remaining): V4-34*03,J3*02
[SEND] Sequence 04-A-M_3089759 to MPI node 1
[RECEIVE] Sequence 04-A-M_3089759 from node 1 (8 alignments remaining): Error/ alignment failed
[SEND] Sequence 04-A-M_3291667 to MPI node 1
[RECEIVE] Sequence 04-A-M_3291667 from node 1 (7 alignments remaining): V1-3*01,J3*02
[SEND] Sequence 04-A-M_0222347 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_0222347","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V3-30*01,J4*02","SUPPORT":0.1318769985697806,"SCORE":8970.486911234248,"DELTA_IC":-8970.486911234248,"TERMINATOR":0,"SEQUENCE":"TGAAGGGCCGATTCACCTTCTCCAGAGACAATTCCAAGAAAACGCTGTCCCCGCAAATGGACAGCCTGAGAGTTGAGGACCCCTTTTGGGGGACTCATCGTTGTTAAACTTTGACTATTGGGGCCAGGGA","SISTER_SEQUENCES":{"0":"NODE445","1":"IGHJ4_02_CRF_01"},"BRANCH_LENGTHS":{"TO_SISTER":{{5.000000025e-09}{5.000000025e-09}},"TO_PARENT":{{5.000000025e-09}{0.1441022137585326}},"TO_QUERY":{{0.244364006254017}{1.056723261281907}}},"REARRANGEMENTS":{"V3-30*01,J4*02":0.1318769985697806,"V3,J4*02":0.05959158972785008,"V3-30*07,J4*02":0.04334413364776475,"V3-30-3*03,J4*02":0.04257894983591554,"V3-30*01,J4*01":0.04083935715882302,"V3-30*01,J1*01":0.03468297271933957,"V3-30*14,J4*02":0.03465891864148216,"V3-30*05,J4*02":0.0329914375145054,"V3-30*18,J4*02":0.03096750646788657,"V3/OR16-12*01,J4*02":0.02989218121743175,"V3-30*16,J4*02":0.02813022613848904,"V3-30*11,J4*02":0.0278765885638487,"V3-30*10,J4*02":0.02729311683091139,"V3-30*09,J4*02":0.02587858386162856,"V3-30,J4*02":0.02562609043142575,"V3-30*01,J4*03":0.02410732739740173,"V3-30*17,J4*02":0.0226624577590648,"V3-30*12,J4*02":0.01862398710181084,"V3-33*06,J4*02":0.01861692389660067,"V3-30*03,J4*02":0.01817059358104309,"V3-30*02,J4*02":0.01816533761737234,"V3-30*01,J?":0.01767391190707603,"V3,J4*01":0.0115024134862091,"V?,J4*02":0.01118804517162647,"V3-30*07,J1*01":0.01116205717370535,"V3-30*07,J4*01":0.01101613442150444},"MODELS_TRIED":16021,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"NTGAAGGGCCGATTCACCTTCTCCAGAGACAATTCCAAGAAAACGCTGTCCCCGCAAATGGACAGCCTGAGAGTTGAGGACNCCCTTTTGGGGGACTCA","FW3_AA":"?KGRFTFSRDNSKKTLSPQMDSLRVED?LLGDS","CDR3":"TCGTTG","CDR3_AA":"SL","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"TTAAACTTTGACTATTGGGGCCAGGGA","J_AA":"LNFDYWGQG","Mapped Read":"---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------NTGAAGGGCCGATTCACCTTCTCCAGAGACAATTCCAAGAAAACGCTGTCCCCGCAAATGGACAGCCTGAGAGTTGAGGACNCCCTTTTGGGGGACTCATCG---TTG------------TTAAACTTTGACTATTGGGGCCAGGGA","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{306}{315}{354}}},"BREAKPOINTS":{{83}},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.45085322801466,0.3239677038796874,0.1039743749822445,0.3983229139633723,0.11048868421681,0.1717966130400485,0.2701874641679753,0.07904994863133302,0.030659030254975,0.01109208300837162,0.004470296702856308,0.005923269800824802,0.008388010296701019,0.01219993057775807,0.005064975658881993,0.001847121142520358,0.0009436823877044408,0.001237688870629853,0.001808122429970612,0.0008075902286256045,0.0002609028030346997,0.000182203421367615,0.0005467039958071922,0.002253320622669284,0.009749967964236744,0.1094482394993599,0.07637551933978209,0.04907221015653911,0.03213942740386044,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"IGHV3_30_09_CRF_1":0.0418357377075905,"IGHJ4_02_CRF_01":0.6654178139224658,"IGHV3_30_14_CRF_1":0.04924272726252444,"NODE405":0.001100439026870182,"NODE446":0.05804446820979609,"NODE445":0.1768126677358203,"NODE210":0.000544047685451084,"IGHJ4_01_CRF_01":0.05330462476131995,"NODE223":0.01346021442031723,"NODE443":0.0424260762268793,"NODE215":0.001272415735901916,"IGHJ5_01_CRF_01":0.0003722296893956261,"IGHJ4_03_CRF_01":0.06742210051954192,"IGHJ6_03_CRF_01":2.014643087862353e-05,"IGHJ5_02_CRF_01":0.0005193961156858213,"NODE222":0.02074372542143424,"NODE442":0.04368984874892554,"IGHJ6_02_CRF_01":7.689422072550709e-05,"NODE218":5.531303088728891e-05,"NODE226":0.05963511749538448,"IGHV3_30_03_CRF_1":0.01825697471294562,"IGHJ6_01_CRF_01":8.776227344553492e-05,"IGHJ3_01_CRF_01":8.225318454738479e-05,"NODE213":0.001015617243123346,"IGHJ6_04_CRF_01":3.099794391245247e-05,"IGHV3_30_3_03_CRF_1":0.05678762449951918,"NODE217":2.298467650015281e-05,"NODE432":0.0181675255558069,"NODE209":0.03114668457902806,"NODE477":2.95962246791252e-07,"IGHJ1_01_CRF_01":0.08238852147012608,"IGHJ2_01_CRF_01":0.002281250284622224,"IGHV3_30_16_CRF_1":0.02831671585759509,"IGHJ3_02_CRF_01":9.988889530493847e-05,"IGHV3_30_18_CRF_1":1.04747781641843e-05,"NODE397":0.0004683579052664735,"IGHV3_30_05_CRF_1":0.04801207561819386,"IGHV3_30_07_CRF_1":0.08100821053301585,"IGHV3_30_11_CRF_1":0.03654020227051658,"IGHV3_30_3_01_CRF_1":0.004750306899467009,"IGHV3_33_02_CRF_1":0.001688540822861517,"IGHV3_30_10_CRF_1":0.03755985731665253,"IGHV3_OR16_13_01_CRF_1":4.309566965681246e-06,"IGHV3_30_01_CRF_1":0.07914541408135536,"NODE526":0.001024041447440272,"IGHV3_30_13_CRF_1":0.02675418061396495,"NODE417":4.820763147547731e-07,"IGHV3_30_06_CRF_1":3.556473041838075e-05,"NODE494":4.54682205116259e-08,"IGHV3_30_17_CRF_1":0.03425437161251904,"IGHV3_74_02_CRF_1":1.115542163444193e-06,"NODE518":0.001172054801356104,"IGHV3_30_3_02_CRF_1":9.66770734162313e-05,"IGHV3_53_04_CRF_1":0.0004680781504087257,"NODE475":8.094093751063798e-07,"NODE406":0.001013536067927482,"IGHV3_66_01_CRF_1":1.224186327114287e-07,"IGHV3_64_02_CRF_1":1.59175529149771e-06,"IGHV3_NL1_01_CRF_1":3.742390834084115e-07,"NODE383":0.01198324443455625,"NODE391":2.041353442387813e-07,"NODE522":1.538782674320324e-07,"NODE508":9.9981192688366e-05,"IGHV3_23_02_CRF_1":0.0001024471764599964,"NODE425":0.001270728974432172,"NODE433":0.01861692389660067,"IGHV3_74_01_CRF_1":3.315937397687933e-07,"IGHV3_11_05_CRF_1":4.961933676719082e-09,"NODE377":1.096273236672124e-08,"IGHV3_30_02_CRF_1":0.01816533761737234,"NODE416":0.001270729051070364,"NODE394":0.0009764401450062344,"IGHV3_30_12_CRF_1":0.01862470711595869,"NODE382":0.00789518743169971,"IGHV3_23_04_CRF_1":9.9981192688366e-05,"IGHV3_7_03_CRF_1":1.097957851862692e-08,"IGHV3_74_03_CRF_1":3.315937397687933e-07,"NODE233":2.801935368317744e-09,"IGHV3_38_01_CRF_1":8.516398060677478e-09,"NODE411":0.0009322235420425684,"IGHV3_23_01_CRF_1":9.9981192688366e-05,"IGHV3_33_04_CRF_1":0.001270371979061775,"IGHV3_38_03_CRF_1":3.857007339184241e-09,"IGHV3_OR16_12_01_CRF_1":0.02989218121743175,"IGHV3_9_03_CRF_1":2.00687390636967e-08,"NODE356":1.911205514786855e-06,"IGHV3_30_19_CRF_1":1.482465693224328e-06,"IGHV3_33_01_CRF_1":8.0962743485018e-07,"IGHV3_33_03_CRF_1":5.585777342890001e-08,"IGHV3_66_02_CRF_1":3.162075686491005e-07,"IGHV3_64_04_CRF_1":1.23848121005456e-07,"IGHV3_33_05_CRF_1":1.121161272414383e-08,"IGHV3_33_06_CRF_1":5.585618576527674e-08,"IGHV3_30_08_CRF_1":7.512558618149833e-07,"IGHV3_66_03_CRF_1":3.162306775946821e-07,"IGHV3_30_15_CRF_1":7.200188706932809e-07},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-516/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096290 queued and waiting for resources
srun: job 26096290 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof43:25765] *** Process received signal ***
[gizmof43:25765] Signal: Segmentation fault (11)
[gizmof43:25765] Signal code: Address not mapped (1)
[gizmof43:25765] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 25765 on node gizmof43 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof43:25761] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof43:25761] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof43: task 0: Exited with exit code 139

-----

proc 538
out----

Read 10 sequences

[SEND] Sequence 04-A-M_3178587 to MPI node 1
[RECEIVE] Sequence 04-A-M_3178587 from node 1 (9 alignments remaining): V3/OR16-13*01,J4*02
[SEND] Sequence 04-A-M_2506680 to MPI node 1
[RECEIVE] Sequence 04-A-M_2506680 from node 1 (8 alignments remaining): V4-30-4*01,J3*02
[SEND] Sequence 04-A-M_4342144 to MPI node 1
[RECEIVE] Sequence 04-A-M_4342144 from node 1 (7 alignments remaining): V3-30*07,J4*02
[SEND] Sequence 04-A-M_3635145 to MPI node 1
[RECEIVE] Sequence 04-A-M_3635145 from node 1 (6 alignments remaining): V3-30*18,J2*01
[SEND] Sequence 04-A-M_2785297 to MPI node 1
[RECEIVE] Sequence 04-A-M_2785297 from node 1 (5 alignments remaining): Error/ alignment failed
[SEND] Sequence 04-A-M_1469223 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_1469223","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V3-30*18,J4*02","SUPPORT":0.3378138701142167,"SCORE":7927.979199338595,"DELTA_IC":-7927.979199338595,"TERMINATOR":0,"SEQUENCE":"CTCCAGAGACAATTCCAAGAGCACACTATATTTGCAAATGAACAGCCTGAGACCTGAGGACACGGCTGTGTATTACCGTGCGAAAACCGTATTTGACTACTGGGGCCAGGGA","SISTER_SEQUENCES":{"0":"IGHV3_30_18_CRF_1","1":"IGHJ4_02_CRF_01"},"BRANCH_LENGTHS":{"TO_SISTER":{{5.000000025e-09}{5.000000025e-09}},"TO_PARENT":{{4.961990153036845e-10}{0.1073009080021175}},"TO_QUERY":{{0.1860012726121573}{5.000000025e-09}}},"REARRANGEMENTS":{"V3-30*18,J4*02":0.3378138701142167,"V3-33*06,J4*02":0.2152630412953875,"V3-30-3*02,J4*02":0.2152624041370635,"V3-30*02,J4*02":0.1293265895035895,"V3-30*13,J4*02":0.01293759429003399,"V3-30*17,J4*02":0.01276951294822532,"V3-30-3*02,J1*01":0.01123691227049751},"MODELS_TRIED":9614,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"NNCTCCAGAGACAATTCCAAGAGCACACTATATTTGCAAATGAACAGCCTGAGACCTGAGGACACGGCTGTGTATTACCGT","FW3_AA":"?SRDNSKSTLYLQMNSLRPEDTAVYYR","CDR3":"GCGAAA","CDR3_AA":"AK","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"GCTTCGGGAGGTGGGAACACCGTATTTGACTACTGGGGCCAGGGA","J_AA":"ASGGGNTVFDYWGQG","Mapped Read":"---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------NNCTCCAGAGACAATTCCAAGAGCACACTATATTTGCAAATGAACAGCCTGAGACCTGAGGACACGGCTGTGTATTACCGTGCG---AAA------------GCTTCGGGAGGTGGGAACACCGTATTTGACTACTGGGGCCAGGGA","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{306}{315}{372}}},"BREAKPOINTS":{{92}},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.076030555379751e-06,2.748992949263751e-06,3.542874186142279e-06,8.574521508178401e-06,1.107386759693333e-05,1.457579276727761e-05,3.809427152299367e-05,2.540461640592983e-05,6.385152141580877e-05,4.318166786413456e-05,0.0001177740077341401,7.538640803196779e-05,0.0001031297162404811,2.850717842642918e-05,7.663173680207677e-05,0.0001054168498051063,7.216205985238165e-05,9.996250425526663e-05,0.0003145036359217945,0.0002083748042786132,0.001149788010279971,0.001520386594143639,0.002054445954448222,0.002495628285802151,0.003858233064940489,0.006409060360584689,0.04525769831388146,1.999999861086513,0.7014918190327262,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"NODE508":3.555938886890516e-05,"IGHJ4_02_CRF_01":0.9628740568170007,"NODE433":0.2242167321390007,"NODE223":0.002685706420816987,"NODE437":0.008113169872696585,"IGHJ4_03_CRF_01":0.008215342349966036,"IGHV3_30_02_CRF_1":0.000176291911696993,"NODE383":0.0005686456701152556,"IGHJ4_01_CRF_01":0.002894013467423028,"IGHV3_33_01_CRF_1":0.0005689884508633321,"NODE215":1.582369972794782e-06,"NODE210":3.788888520726455e-06,"IGHJ5_01_CRF_01":2.70151130072692e-05,"IGHV3_53_01_CRF_1":1.181844566460966e-06,"NODE407":6.035556745458032e-07,"IGHV3_30_13_CRF_1":0.01391621172373319,"NODE209":1.282602746812234e-06,"IGHJ1_01_CRF_01":0.01942310938868808,"NODE476":1.722382282253248e-08,"IGHJ5_02_CRF_01":3.609154099443355e-05,"IGHV3_33_05_CRF_1":0.008537757576121231,"NODE213":1.003969897716897e-06,"IGHV3_30_17_CRF_1":0.01304435374238977,"IGHV3_30_3_02_CRF_1":0.2355561668375718,"NODE226":0.002735924785503192,"IGHV3_23_02_CRF_1":3.983220988030587e-05,"IGHV3_30_12_CRF_1":0.0001024977387329176,"IGHV3_30_06_CRF_1":0.008520144438959659,"NODE222":0.001099985786557851,"IGHJ3_01_CRF_01":1.523054651330249e-07,"IGHV3_21_01_CRF_1":1.932315804810481e-07,"IGHV3_66_03_CRF_1":0.0002183533571122063,"IGHV3_OR16_9_01_CRF_1":2.987646761832251e-08,"IGHV3_23_01_CRF_1":3.540890441595655e-05,"IGHV3_30_03_CRF_1":0.00509837317984531,"IGHV3_30_01_CRF_1":0.0007393497971517163,"IGHV3_21_04_CRF_1":3.193895064766043e-07,"IGHV3_30_05_CRF_1":0.00742237166643112,"IGHV3_30_11_CRF_1":1.201802970644892e-06,"IGHV3_33_02_CRF_1":4.698009041882146e-07,"IGHV3_30_16_CRF_1":7.271222772340245e-05,"IGHV3_30_10_CRF_1":1.117942078066154e-07,"IGHV3_30_19_CRF_1":7.817251479831898e-06,"IGHV3_30_09_CRF_1":8.707519697085026e-07,"IGHJ3_02_CRF_01":2.378211124279575e-07,"IGHJ2_01_CRF_01":7.063723262195003e-07,"IGHV3_21_03_CRF_1":1.932318969466501e-07,"NODE450":1.014435056703202e-05,"IGHV3_35_01_CRF_1":5.120239943432175e-07,"IGHV3_30_08_CRF_1":7.162329079420878e-06,"IGHV3_30_15_CRF_1":1.073823669689719e-06,"IGHV3_66_02_CRF_1":1.759262137442652e-08,"IGHV3_30_07_CRF_1":1.118004895223714e-07,"NODE358":0.0003682858501004715,"NODE442":0.1239159933667703,"NODE443":0.00101827035297441,"IGHV3_30_18_CRF_1":0.2185075482744162,"NODE430":0.1291510368609973,"NODE391":2.244314512750805e-05,"NODE397":2.029053521263485e-08,"IGHV3_OR16_12_01_CRF_1":1.848966376538812e-08,"IGHV3_33_04_CRF_1":4.180952474339489e-07,"IGHV3_38_3_01_CRF_1":1.321251096329132e-07,"IGHV3_64_04_CRF_1":8.579373117887829e-07,"IGHV3_NL1_01_CRF_1":2.270296576541295e-08},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-538/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096293 queued and waiting for resources
srun: job 26096293 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof43:25803] *** Process received signal ***
[gizmof43:25803] Signal: Segmentation fault (11)
[gizmof43:25803] Signal code: Address not mapped (1)
[gizmof43:25803] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 25803 on node gizmof43 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof43:25797] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof43:25797] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof43: task 0: Exited with exit code 139

-----

proc 567
out----

Read 10 sequences

[SEND] Sequence 04-A-M_2218609 to MPI node 1
[RECEIVE] Sequence 04-A-M_2218609 from node 1 (9 alignments remaining): V4-30-2*04,J4*02
[SEND] Sequence 04-A-M_2776497 to MPI node 1
[RECEIVE] Sequence 04-A-M_2776497 from node 1 (8 alignments remaining): V3-9*01,J5*01
[SEND] Sequence 04-A-M_0704231 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_0704231","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V3-74*02,J5*02","SUPPORT":0.2237602737093812,"SCORE":8810.557207400598,"DELTA_IC":-8810.557207400598,"TERMINATOR":0,"SEQUENCE":"GGGCCGATTCACCATCTCCAGGGACAACGCCAAGAACACGCTGTATCTGCAAATGAACAGTCGGAGAGCCGACGCCACGGCTCTCCAGTTCGTATCAAGGAATTGGCTTCATTTTTGGGGCCAGGGA","SISTER_SEQUENCES":{"0":"NODE418","1":"IGHJ5_02_CRF_01"},"BRANCH_LENGTHS":{"TO_SISTER":{{5.000000025e-09}{0.0715855535074412}},"TO_PARENT":{{0.08700002006736011}{0.1267496625856585}},"TO_QUERY":{{0.1180370649855118}{0.988171615254068}}},"REARRANGEMENTS":{"V3-74*02,J5*02":0.2237602737093812,"V3-74*02,J5*01":0.2111168893622409,"V3-74*01,J5*01":0.1325379685020027,"V3/OR16-13*01,J5*02":0.1283132612908074,"V3-74*01,J5*02":0.1214005658395974,"V3/OR16-13*01,J5*01":0.09518954993552102,"V3-74*03,J5*02":0.03131190471035731,"V3-74*02,J?":0.01028342087418143},"MODELS_TRIED":14769,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"NNGGGCCGATTCACCATCTCCAGGGACAACGCCAAGAACACGCTGTATCTGCAAATGAACAGTCGGAGAGCCGACGCCACGGCTCTCCAGTTCTAAGTA","FW3_AA":"?GRFTISRDNAKNTLYLQMNSRRADATALQFXV","CDR3":"TCAAGG","CDR3_AA":"SR","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"AATTGGCTTCATTTTTGGGGCCAGGGA","J_AA":"NWLHFWGQG","Mapped Read":"------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------NNGGGCCGATTCACCATCTCCAGGGACAACGCCAAGAACACGCTGTATCTGCAAATGAACAGTCGGAGAGCCGACGCCACGGCTCTCCAGTTCTAAGTATCA---AGG------------AATTGGCTTCATTTTTGGGGCCAGGGA","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{309}{318}{357}}},"BREAKPOINTS":{{83}},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.1456930636871952,0.2703689890077469,0.4998401229266012,1.989057611030079,0.1060029818613149,0.2003780448756747,0.05065952770098886,0.01039759491221152,0.01695390692930668,0.002792155360233929,0.005016239937116403,0.0006589674772842443,0.001122935525891838,0.0002356821939785753,5.581693287826438e-05,5.761665779748163e-06,3.30598232412334e-06,2.697791988406645e-06,1.558849211755474e-05,1.146899474055181e-05,1.831144527194329e-05,6.874526493384023e-06,3.897117281356305e-06,1.799020120012361e-06,2.957274409543752e-06,5.23041815475654e-06,7.741882881598341e-07,1.116849384987805e-07,1.613979262227906e-07,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"IGHV3_OR16_13_01_CRF_1":0.2355769565450195,"IGHJ5_02_CRF_01":0.505197630560812,"NODE418":0.2685987492996635,"NODE215":0.0006646798602217707,"NODE223":0.2747536601683747,"IGHJ4_01_CRF_01":0.002036010173577946,"IGHV3_7_03_CRF_1":2.11990314494291e-06,"IGHJ4_02_CRF_01":0.007872780349331679,"IGHV3_74_01_CRF_1":0.2663617158075827,"NODE213":0.001169021672724645,"IGHV3_7_01_CRF_1":1.88172605079868e-07,"IGHJ1_01_CRF_01":0.002978418270986093,"NODE218":9.621083301398733e-06,"IGHV3_11_03_CRF_1":1.377037426536239e-07,"IGHJ5_01_CRF_01":0.1723445890638802,"NODE222":0.005023733867614126,"IGHJ6_02_CRF_01":1.098134971540044e-05,"IGHJ2_01_CRF_01":0.0005193745003022194,"IGHV3_33_01_CRF_1":7.217116947125665e-06,"IGHJ6_01_CRF_01":9.974348263729893e-06,"IGHJ4_03_CRF_01":0.001719786938219417,"NODE426":9.318921841915733e-05,"NODE210":0.001144857939214975,"IGHJ3_01_CRF_01":0.0002560899245278436,"NODE209":0.01955209979253981,"NODE413":5.223131880025091e-05,"IGHV3_30_11_CRF_1":4.519107406008061e-07,"IGHV3_74_02_CRF_1":0.1874488922176568,"NODE383":4.344635723758386e-07,"IGHJ3_02_CRF_01":0.0003603656309201829,"NODE217":3.955360387340337e-06,"NODE226":0.0043673811723453,"IGHJ6_03_CRF_01":3.324376589818147e-06,"NODE474":1.763468152859612e-06,"NODE529":1.001940803517348e-07,"NODE445":8.399571690048901e-08,"IGHV3_30_15_CRF_1":3.729288354108478e-08,"IGHV3_30_03_CRF_1":3.665557320488754e-08,"IGHV3_30_10_CRF_1":3.665557320488754e-08,"IGHV3_21_03_CRF_1":1.649234682777622e-07,"NODE362":1.621809308835362e-07,"NODE417":2.985150774309586e-06,"NODE443":3.665557320488754e-08,"IGHV3_20_01_CRF_1":2.032417135315628e-07,"IGHV3_30_01_CRF_1":1.843819333586962e-07,"IGHV3_30_05_CRF_1":4.541270587174071e-08,"NODE360":1.621809309794129e-07,"NODE411":4.733616245074128e-05,"NODE359":0.0002926829837711942,"IGHV3_48_01_CRF_1":1.406929185064368e-08,"IGHJ6_04_CRF_01":1.663596148482191e-06,"IGHV3_74_03_CRF_1":0.04149946926596937,"NODE475":6.232104550923576e-09,"IGHV3_66_01_CRF_1":1.438865568447491e-08,"NODE433":5.709593674570826e-07,"IGHV3_30_17_CRF_1":2.613509056183398e-07,"NODE358":4.751834529420488e-06,"IGHV3_66_03_CRF_1":2.426522623244322e-08,"IGHV3_7_02_CRF_1":2.781840291921345e-08,"NODE450":1.269030583128618e-07,"NODE518":1.02050722187214e-06,"NODE405":3.97676965278125e-06,"NODE437":6.525763531629453e-07,"IGHV3_OR16_12_01_CRF_1":6.534431786251058e-07,"NODE373":9.65616275673788e-08,"IGHV3_30_16_CRF_1":2.784032698171134e-08},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-567/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096298 queued and waiting for resources
srun: job 26096298 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof32:26131] *** Process received signal ***
[gizmof32:26131] Signal: Segmentation fault (11)
[gizmof32:26131] Signal code: Address not mapped (1)
[gizmof32:26131] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 26131 on node gizmof32 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof32:26125] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof32:26125] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof32: task 0: Exited with exit code 139

-----

proc 739
out----

Read 10 sequences

[SEND] Sequence 04-A-M_1552639 to MPI node 1
[RECEIVE] Sequence 04-A-M_1552639 from node 1 (9 alignments remaining): V3-20*01,J3*02
[SEND] Sequence 04-A-M_0296250 to MPI node 1
[RECEIVE] Sequence 04-A-M_0296250 from node 1 (8 alignments remaining): V4-4*07,J5*02
[SEND] Sequence 04-A-M_1646678 to MPI node 1
[RECEIVE] Sequence 04-A-M_1646678 from node 1 (7 alignments remaining): V3-48*02,J5*02
[SEND] Sequence 04-A-M_0621520 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_0621520","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V1-38-4*01,J5*01","SUPPORT":0.4308637519637646,"SCORE":8382.398066699145,"DELTA_IC":-8382.398066699145,"TERMINATOR":0,"SEQUENCE":"GTCACCGTGACCAGGGACATGTCCAAAACCACAGCCTACACAGACCTGACCGGCCTGACATCTGAGGACATGGCTGTTTATTACTATGCATTGGATCGGCAGTACGACTCCTGGGGTCAGGGA","SISTER_SEQUENCES":{"0":"IGHV1_38_4_01_CRF_1","1":"NODE223"},"BRANCH_LENGTHS":{"TO_SISTER":{{0.02421824621219878}{0.06695779922190059}},"TO_PARENT":{{0.3139389687983632}{0.3549767255936095}},"TO_QUERY":{{0.1397280685416624}{1.058491458942536}}},"REARRANGEMENTS":{"V1-38-4*01,J5*01":0.4308637519637646,"V1-38-4*01,J5*02":0.4269675648157789,"V1-38-4*01,J?":0.05203935769589174,"V1-38-4*01,J4*01":0.03212512562639346,"V1-38-4*01,J4*02":0.02437258790848295,"V1-38-4*01,J1*01":0.01345667990319248},"MODELS_TRIED":16221,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"GTCACCGTGACCAGGGACATGTCCAAAACCACAGCCTACACAGACCTGACCGGCCTGACATCTGAGGACATGGCTGTTTATTACTAT","FW3_AA":"VTVTRDMSKTTAYTDLTGLTSEDMAVYYY","CDR3":"GCATTG","CDR3_AA":"AL","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"GATCGGCAGTACTAGGAAGACTCCTGGGGTCAGGGA","J_AA":"DRQYXEDSWGQG","Mapped Read":"---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------GTCACCGTGACCAGGGACATGTCCAAAACCACAGCCTACACAGACCTGACCGGCCTGACATCTGAGGACATGGCTGTTTATTACTATGCA---TTG---------GATCGGCAGTACTAGGAAGACTCCTGGGGTCAGGGA","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{306}{315}{360}}},"BREAKPOINTS":{{91}},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.000102643871589471,0.0001637797429837243,0.0002575945840991782,0.001568693877394063,0.00158429532408744,0.004320384121405998,0.0005844502879769086,0.0005787815646683817,0.000844335782330171,0.00240628407440603,0.004135725472794098,0.006404766884057242,0.01105270629626681,0.01999216573525903,0.06844337501210934,0.1231110584776146,0.2290800902923942,0.4162012139592099,1.267665428357383,0.1061751074428939,0.01672124721255037,0.003880915540585277,0.001444485897468202,0.001316367873300754,0.001930838364837214,0.009845557946803958,0.01604055953920919,0.03376995216898496,0.07278529410980344,0.4387125288275071,0.09332232776955285,0.04313896808210355,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"IGHV1_38_4_01_CRF_1":1,"IGHJ4_03_CRF_01":0.008960243005382156,"NODE215":0.001982776334721728,"IGHJ6_02_CRF_01":3.154305567308635e-05,"IGHJ5_01_CRF_01":0.1949676784801738,"IGHJ4_01_CRF_01":0.006601546436270827,"NODE213":0.001757688736531886,"NODE217":2.065825952975476e-05,"NODE226":0.02552357919012263,"NODE222":0.03489406877949004,"IGHJ5_02_CRF_01":0.4269675648157789,"IGHJ3_01_CRF_01":7.3416843958388e-05,"IGHJ3_02_CRF_01":7.086121200081458e-05,"NODE218":3.246261395686035e-05,"IGHJ4_02_CRF_01":0.02437258790848295,"NODE209":0.01538760017986984,"IGHJ6_01_CRF_01":3.286188420632628e-05,"IGHJ2_01_CRF_01":0.007405295128649624,"NODE210":0.001522463687933825,"IGHJ6_04_CRF_01":9.902711975388886e-06,"NODE223":0.2358960734835908,"IGHJ1_01_CRF_01":0.01345667990319248,"IGHJ6_03_CRF_01":3.244734850748162e-05},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-739/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096312 queued and waiting for resources
srun: job 26096312 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof35:15940] *** Process received signal ***
[gizmof35:15940] Signal: Segmentation fault (11)
[gizmof35:15940] Signal code: Address not mapped (1)
[gizmof35:15940] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 15940 on node gizmof35 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof35:15934] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof35:15934] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof35: task 0: Exited with exit code 139

-----

proc 783
out----

Read 10 sequences

[SEND] Sequence 04-A-M_3225224 to MPI node 1
[RECEIVE] Sequence 04-A-M_3225224 from node 1 (9 alignments remaining): V5-51*01,J4*02
[SEND] Sequence 04-A-M_0010237 to MPI node 1
[RECEIVE] Sequence 04-A-M_0010237 from node 1 (8 alignments remaining): Error/ alignment failed
[SEND] Sequence 04-A-M_0820911 to MPI node 1
[RECEIVE] Sequence 04-A-M_0820911 from node 1 (7 alignments remaining): V4-61*06,J6*03
[SEND] Sequence 04-A-M_1331287 to MPI node 1
[RECEIVE] Sequence 04-A-M_1331287 from node 1 (6 alignments remaining): Error/ alignment failed
[SEND] Sequence 04-A-M_0641793 to MPI node 1
[RECEIVE] Sequence 04-A-M_0641793 from node 1 (5 alignments remaining): V3-48*03,J6*03
[SEND] Sequence 04-A-M_0340093 to MPI node 1
[RECEIVE] Sequence 04-A-M_0340093 from node 1 (4 alignments remaining): Error/ alignment failed
[SEND] Sequence 04-A-M_0362177 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_0362177","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V3-7*01","SUPPORT":0.09524867792594588,"SCORE":7938.291781241467,"DELTA_IC":-7938.291781241467,"TERMINATOR":0,"SEQUENCE":"TCACCACCTCCAGAGACAACGCCAAGAACTCACTGTATCTGCAGATGAACAGCCTGAGAGCCGAAGACACGGCTGTGTATTACTGTGCGAGAGATTCGGCGGGAGA","SISTER_SEQUENCES":{"0":"NODE426"},"BRANCH_LENGTHS":{"TO_SISTER":{{5.000000025e-09}},"TO_PARENT":{{0.1330022910594508}},"TO_QUERY":{{0.09623785515861556}}},"REARRANGEMENTS":{"V3-7*01":0.09524867792594588,"V3-21*01":0.09115831069832338,"V3-48*04":0.09115830737698748,"V3-11*06":0.0666642937226508,"V3":0.04973686368060667,"V3-11*04":0.04726634065758437,"V3-11*05":0.04720650698848655,"V3-48*02":0.04557930412018402,"V3-21*04":0.04557922692734276,"V3-21*02":0.04557915617948531,"V3-11":0.04557915127435799,"V3-48":0.04557915119136709,"V3-21*03":0.04557914925791128,"V3-48*01":0.04557914768750963,"V3-48*03":0.04557914709007468,"V3-7*03":0.04551152734912011,"V3-7*02":0.04551149859769659,"V3-21*01,J?":0.01255207892805976},"MODELS_TRIED":25486,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"NTCACCACCTCCAGAGACAACGCCAAGAACTCACTGTATCTGCAGATGAACAGCCTGAGAGCCGAAGACACGGCTGTGTATTACTGT","FW3_AA":"?TTSRDNAKNSLYLQMNSLRAEDTAVYYC","CDR3":"GCGAGA","CDR3_AA":"AR","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"GATTCGGCGGGACAGTAACTACCCCCGGGGGCCAGGGA","J_AA":"DSAGQXLPPGAR?","Mapped Read":"---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------NTCACCACCTCCAGAGACAACGCCAAGAACTCACTGTATCTGCAGATGAACAGCCTGAGAGCCGAAGACACGGCTGTGTATTACTGTGCG---AGA---------GATTCGGCGGGACAGTAACTACCCCCGGGGGCCAGGGA","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{306}{315}{362}}},"BREAKPOINTS":{},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"IGHV3_11_03_CRF_1":0.001883663113886366,"NODE209":0.05242175556074671,"NODE373":0.05265642347906777,"NODE359":0.05717391785665003,"IGHJ4_03_CRF_01":1.443545230823916e-06,"NODE210":2.425495642073759e-07,"NODE213":1.32259864675077e-06,"NODE217":2.146206208572208e-06,"NODE226":1.130512383065938e-07,"IGHJ6_03_CRF_01":2.284393338724968e-07,"IGHV3_11_01_CRF_1":0.001850373019909466,"NODE222":5.207892302091906e-07,"NODE426":0.04977033271343621,"IGHV3_21_01_CRF_1":0.05105484224343071,"IGHV3_74_01_CRF_1":6.081971969180779e-08,"IGHV3_48_02_CRF_1":0.04724559323151421,"IGHJ2_01_CRF_01":1.074972020894901e-06,"IGHV3_48_03_CRF_1":0.04557916414753405,"IGHV3_11_04_CRF_1":0.00170365482860882,"IGHV3_48_01_CRF_1":0.04557983955583472,"IGHJ6_04_CRF_01":7.22491230459313e-06,"IGHV3_11_05_CRF_1":0.001627370529779283,"NODE362":0.04611310283750515,"IGHJ3_02_CRF_01":3.846954188549019e-07,"IGHJ5_01_CRF_01":7.830105786410569e-08,"NODE215":3.602172499794586e-06,"IGHJ4_02_CRF_01":5.94981970755335e-07,"IGHJ5_02_CRF_01":2.820773960137988e-07,"IGHJ4_01_CRF_01":1.003720863621954e-06,"NODE223":7.905687050284397e-07,"IGHJ1_01_CRF_01":4.979590545016634e-08,"IGHV3_48_04_CRF_1":0.04773294633778306,"NODE363":0.05265831963514057,"IGHV3_21_03_CRF_1":0.04557915626504374,"IGHV3_21_04_CRF_1":0.04740737088739835,"IGHV3_21_02_CRF_1":0.04557916312465859,"NODE360":0.04943182654497387,"IGHV3_OR16_9_01_CRF_1":2.032409669381561e-05,"IGHV3_11_06_CRF_1":0.06998814791741476,"NODE374":0.05274001757674431,"IGHJ6_01_CRF_01":4.598703718644364e-07,"IGHJ3_01_CRF_01":1.255667570452723e-07,"NODE377":0.04611016898899611,"IGHV3_7_02_CRF_1":0.04934984138955492,"NODE356":4.341233847537849e-09,"NODE416":4.165072740486585e-09,"NODE474":4.16507216425803e-09,"NODE518":4.165097914708248e-09,"NODE475":4.165072314161164e-09,"NODE507":4.165143190593753e-09,"NODE476":4.655904871965033e-06,"NODE492":1.428623075663451e-06,"NODE495":2.381485407235793e-08,"IGHV3_20_02_CRF_1":1.169452495352359e-08,"IGHV3_20_01_CRF_1":1.169452495352359e-08,"NODE477":1.305823260168116e-06,"NODE417":3.390335854848945e-08,"NODE425":4.165077223673607e-09,"NODE430":4.165079801422719e-09,"IGHV3_30_07_CRF_1":8.247190479237974e-09,"IGHV3_33_05_CRF_1":8.247190980754878e-09,"IGHV3_30_12_CRF_1":8.247190980762379e-09,"NODE437":8.254486777588145e-09,"IGHV3_33_02_CRF_1":8.140472552196335e-09,"IGHV3_33_01_CRF_1":8.14044505478729e-09,"IGHV3_33_04_CRF_1":8.14044505478729e-09,"IGHV3_33_03_CRF_1":5.807166296107867e-09,"IGHV3_7_01_CRF_1":0.04551149859769659,"IGHV3_7_03_CRF_1":0.04551152734912011,"NODE418":3.604000764441901e-07,"IGHV3_OR16_13_01_CRF_1":2.186813473966455e-08,"IGHV3_74_02_CRF_1":2.186791163587924e-08,"IGHV3_74_03_CRF_1":2.186791163587924e-08,"NODE405":1.249959616617895e-07,"NODE411":0.0001281620944227646,"NODE413":5.024017157015633e-06,"NODE406":3.929026961581791e-08,"NODE358":4.16507261239917e-09,"NODE382":4.165078786014021e-09},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-783/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096316 queued and waiting for resources
srun: job 26096316 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof36:14467] *** Process received signal ***
[gizmof36:14467] Signal: Segmentation fault (11)
[gizmof36:14467] Signal code: Address not mapped (1)
[gizmof36:14467] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 14467 on node gizmof36 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof36:14461] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof36:14461] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof36: task 0: Exited with exit code 139

-----

proc 830
out----

Read 10 sequences

[SEND] Sequence 04-A-M_1068194 to MPI node 1
[RECEIVE] Sequence 04-A-M_1068194 from node 1 (9 alignments remaining): Error/ alignment failed
[SEND] Sequence 04-A-M_2644068 to MPI node 1
[RECEIVE] Sequence 04-A-M_2644068 from node 1 (8 alignments remaining): V3-49*03,J6*02
[SEND] Sequence 04-A-M_1626214 to MPI node 1
[RECEIVE] Sequence 04-A-M_1626214 from node 1 (7 alignments remaining): V5-51*01,J4*02
[SEND] Sequence 04-A-M_4565883 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_4565883","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V1-8*02,J3*01","SUPPORT":0.1675642805239802,"SCORE":8006.016851524969,"DELTA_IC":-8006.016851524969,"TERMINATOR":0,"SEQUENCE":"ACCATGACCAGGAACACCTCCGTAAGCACAGCCTACATGGAGCTGAGCAGCCTGAGACTTGCGGACACGGCCGTGTATTCCTGTGCGAGAGCCCGGGGAACGGTGACTACTGGACGTCTG","SISTER_SEQUENCES":{"0":"NODE275","1":"NODE210"},"BRANCH_LENGTHS":{"TO_SISTER":{{5.000000025e-09}{5.000000025e-09}},"TO_PARENT":{{0.07485338105370724}{0.33195194982656}},"TO_QUERY":{{0.1352749331276989}{2.562125831919472}}},"REARRANGEMENTS":{"V1-8*02,J3*01":0.1675642805239802,"V1-8*02,J3*02":0.109901035217694,"V1-8*02,J5*02":0.06470990285832555,"V1-8*01,J3*01":0.05968932045974096,"V1-8*02,J4*02":0.05747331145518882,"V1-8*01,J5*02":0.05580803453988294,"V1-8*02,J?":0.05054132609041564,"V1-8*02,J5*01":0.03733789651726765,"V1-8*02,J4*01":0.03362656369221666,"V1-8*01,J3*02":0.0321223569392235,"V1-8*02,J2*01":0.0286430906600452,"V1-8*02,J1*01":0.02715374985914839,"V1-8*01,J4*02":0.0265511464060391,"V1-8*02,J6*03":0.02505848361058418,"V1-8*02,J4*03":0.02317094775408541,"V1-8*01,J5*01":0.02218766501609468,"V1-8*02,J6*02":0.02215862003646322,"V1-8*01,J4*03":0.01898020013553821,"V1-8*02,J6*01":0.01854211376532585,"V1-8*01,J4*01":0.01827797374738222,"V1-8*01,J?":0.01656374314606726,"V1-8*01,J6*02":0.01411034847734025,"V1-8*01,J6*03":0.0140279043968909,"V1-8*01,J1*01":0.01399493904332097,"V1-8*01,J2*01":0.01313370898919676,"V1-8*02,J6*04":0.01160772044645099},"MODELS_TRIED":16903,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"ACCATGACCAGGAACACCTCCGTAAGCACAGCCTACATGGAGCTGAGCAGCCTGAGACTTGCGGACACGGCCGTGTATTCCTGT","FW3_AA":"TMTRNTSVSTAYMELSSLRLADTAVYSC","CDR3":"GCGAGA","CDR3_AA":"AR","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"GCCCGGGGAACGGTGACTACTGGACGTCTG","J_AA":"ARGTVTTGRL","Mapped Read":"------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ACCATGACCAGGAACACCTCCGTAAGCACAGCCTACATGGAGCTGAGCAGCCTGAGACTTGCGGACACGGCCGTGTATTCCTGTGCG---AGA------------------------------GCCCGGGGAACGGTGACTACTGGACGTCTG","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{306}{315}{375}}},"BREAKPOINTS":{{101}},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2.897300916944107e-05,4.556180016092084e-05,7.041407155913674e-05,0.0001144616296506973,0.0003480762854720991,0.0005834465989088492,0.0009963806687823536,0.0009017688448823036,0.001388438011298971,0.002456525808931469,0.00442991484846933,0.0004817758678238635,0.001592370046670961,0.001475167205144382,0.002725288593089371,0.005126486036278713,0.009922206981694248,0.01864157179234921,0.08059635810945689,0.07139280918492044,0.1520415959764563,0.2804172506770611,0.4037132220612455,0.2649224912791938,0.370979375074005,0.5372401696356475,0.7872047353201881,0.5965501815607246,0.4736203441982177,1.426844604154726,1.223481807450559,2.822574129690265,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"IGHV1_8_02_CRF_1":0.5867080644320249,"NODE222":0.03808733007657278,"IGHJ5_02_CRF_01":0.1205187880200728,"NODE218":0.02383587943092867,"IGHJ4_01_CRF_01":0.04258464263073224,"NODE297":1.391718519111695e-07,"NODE305":8.253829990766924e-07,"IGHJ1_01_CRF_01":0.04114869186695105,"NODE213":0.01368085719250226,"NODE226":0.009320741807578983,"IGHJ6_01_CRF_01":0.02766877134572506,"IGHJ4_02_CRF_01":0.08402543420619966,"NODE209":0.01533697564703212,"IGHV1_8_01_CRF_1":0.3224899351331709,"IGHV1_46_01_CRF_1":9.7821320509553e-09,"NODE286":7.264520582960659e-07,"NODE298":2.274102949006495e-07,"NODE274":9.212886956118884e-06,"IGHV1_38_4_01_CRF_1":2.538733039789863e-07,"IGHJ2_01_CRF_01":0.0417768416555036,"NODE215":0.03233320131225051,"IGHJ3_01_CRF_01":0.04847303019046937,"IGHJ5_01_CRF_01":0.03994981560702184,"NODE210":0.1787896804486472,"NODE217":0.009348181506037476,"NODE223":0.01957597968100644,"IGHJ4_03_CRF_01":0.04215115084781694,"NODE273":1.249360373002448e-07,"NODE296":1.242218908000579e-07,"NODE310":1.249087575313444e-07,"NODE278":8.54586634048558e-06,"NODE279":6.666179200871117e-07,"NODE275":0.09078097805516665,"IGHJ6_04_CRF_01":0.010175511881231,"IGHJ6_02_CRF_01":0.0124331055627459,"IGHJ6_03_CRF_01":0.006753204053262773,"NODE264":1.249933228199587e-09,"NODE272":1.249909854287576e-09,"NODE316":3.176533451514625e-09,"IGHV1_46_02_CRF_1":8.439560160113187e-09,"IGHV1_46_03_CRF_1":8.439564607719365e-09,"NODE281":1.105489110268879e-09,"IGHV1_OR15_1_04_CRF_1":1.105487753556839e-09,"NODE265":1.249909855383438e-09,"NODE269":8.780527984614766e-09,"IGHV1_3_01_CRF_1":1.254844095594349e-09,"IGHV1_3_02_CRF_1":1.255157200015511e-09,"NODE266":3.562177369430081e-09,"IGHJ3_02_CRF_01":0.1420321850297108},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-830/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096321 queued and waiting for resources
srun: job 26096321 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof38:11884] *** Process received signal ***
[gizmof38:11884] Signal: Segmentation fault (11)
[gizmof38:11884] Signal code: Address not mapped (1)
[gizmof38:11884] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 11884 on node gizmof38 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof38:11878] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof38:11878] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof38: task 0: Exited with exit code 139

-----

proc 868
out----

Read 10 sequences

[SEND] Sequence 04-A-M_1948220 to MPI node 1
[RECEIVE] Sequence 04-A-M_1948220 from node 1 (9 alignments remaining): V1-69*11,J4*02
[SEND] Sequence 04-A-M_4775500 to MPI node 1
[RECEIVE] Sequence 04-A-M_4775500 from node 1 (8 alignments remaining): V3-15*01,J4*02
[SEND] Sequence 04-A-M_2305425 to MPI node 1
[RECEIVE] Sequence 04-A-M_2305425 from node 1 (7 alignments remaining): V3-48*03,J4*02
[SEND] Sequence 04-A-M_1400046 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_1400046","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V1-69*05,J3*01","SUPPORT":0.1758418012064633,"SCORE":7907.580825235807,"DELTA_IC":-7907.580825235807,"TERMINATOR":0,"SEQUENCE":"CTCCATTACTACCGCCGAATCCACGAGCACAGCCTACATGGAACTCAGCAGCCTGAGATCTGAGGACACGGCCGTCTATTATTGTGCGACAGAGGGGGTAACTGGAGCGACTTCCCCC","SISTER_SEQUENCES":{"0":"NODE322","1":"NODE210"},"BRANCH_LENGTHS":{"TO_SISTER":{{0.02694049973111257}{5.000000025e-09}},"TO_PARENT":{{0.05652609867914878}{0.2096178561836136}},"TO_QUERY":{{0.23586117168111}{3.043381161855358}}},"REARRANGEMENTS":{"V1-69*05,J3*01":0.1758418012064633,"V1-69*05,J6*03":0.1336256377164861,"V1-69*05,J6*04":0.1308370702227491,"V1-69*05,J6*01":0.1210484290771364,"V1-69*05,J6*02":0.1121928276155664,"V1-69*05,J?":0.08429540474790131,"V1-69*05,J3*02":0.06904563314271567,"V1-69*05,J2*01":0.03146559288814196,"V1-69*05,J5*01":0.02466425599359217,"V1-69*05,J4*01":0.02058787850133421,"V1-69*05,J4*02":0.01702846571858076,"V1-69*05,J4*03":0.01690159805327694,"V1-69*05,J1*01":0.01459928080460862,"V1-69*12,J3*01":0.01375335389987834,"V1-69*05,J5*02":0.01093467388074798},"MODELS_TRIED":12065,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"NNCTCCATTACTACCGCCGAATCCACGAGCACAGCCTACATGGAACTCAGCAGCCTGAGATCTGAGGACACGGCCGTCTATTATTGT","FW3_AA":"?SITTAESTSTAYMELSSLRSEDTAVYYC","CDR3":"GCG","CDR3_AA":"A","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"ACAGAGGGGGTAACTGGAGCGACTTCCCCC","J_AA":"TEGVTGATSP","Mapped Read":"---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------NNCTCCATTACTACCGCCGAATCCACGAGCACAGCCTACATGGAACTCAGCAGCCTGAGATCTGAGGACACGGCCGTCTATTATTGTGCG------------------------------------ACAGAGGGGGTAACTGGAGCGACTTCCCCC","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{306}{309}{375}}},"BREAKPOINTS":{{89}},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.0001310710454037902,0.0001932042364155736,0.0003063744857198927,0.0005141436689568915,0.001547513432603232,0.001486479044453484,0.002619878381723223,0.004835340293618706,0.009314888772649035,0.001528058560204877,0.002898418429262686,0.004931448275673128,0.009825367785066885,0.0201336895408048,0.0366153054001051,0.008689015492664379,0.01842362807318606,0.04621376718842608,0.09029793732388793,0.2092523928019326,0.8454242456139121,2.361512681271983,1.528403321864571,0.5524799525113536,0.5370934036821691,0.2021702732890599,0.2473530471972929,0.1895229625434973,0.5259617649093626,0.5094177481796935,0.8398868136104161,0.6297830095206495,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"NODE322":0.8335985563987657,"IGHJ6_03_CRF_01":0.07143119227571236,"IGHV1_18_03_CRF_1":8.61032854074848e-08,"IGHJ6_01_CRF_01":0.125659515974051,"NODE223":0.0126513046159699,"NODE274":7.096917416365591e-08,"IGHV1_69_05_CRF_1":0.1294699931705351,"IGHJ3_02_CRF_01":0.06957267301808906,"IGHJ5_02_CRF_01":0.0110075778465735,"NODE305":1.813988210385695e-07,"IGHJ6_04_CRF_01":0.06017948844571726,"NODE338":0.005069707746932834,"IGHJ3_01_CRF_01":0.05371153508347072,"IGHV1_69_09_CRF_1":4.214389615870958e-05,"IGHJ4_03_CRF_01":0.01695228291172191,"NODE209":0.02892700460467768,"IGHV1_18_04_CRF_1":9.002894146303091e-08,"NODE226":0.00915111857025192,"IGHJ6_02_CRF_01":0.04211695159493954,"NODE323":0.02302057919346618,"NODE213":0.03657567618648833,"IGHJ1_01_CRF_01":0.01467738367667886,"NODE217":0.07149690493042513,"NODE298":9.12328346501519e-08,"NODE222":0.02068628026260964,"NODE215":0.0628117531990124,"NODE336":0.001818115714339123,"IGHJ4_01_CRF_01":0.012444084372294,"IGHJ5_01_CRF_01":0.01228356593378083,"IGHJ2_01_CRF_01":0.03290078658573792,"IGHJ4_02_CRF_01":0.02041413015303647,"NODE218":0.0732652756715185,"IGHV1_18_02_CRF_1":2.661192261128038e-08,"NODE210":0.141083514087244,"IGHV1_18_01_CRF_1":3.986135260054165e-09,"IGHV1_69_02_CRF_1":7.630818264848599e-05,"IGHV1_69_11_CRF_1":0.001063383273092001,"IGHV1_69_14_CRF_1":8.717120733411641e-05,"NODE316":2.246920604246586e-06,"NODE266":2.294209084951067e-06,"IGHV1_58_01_CRF_1":1.165379673416276e-08,"NODE334":0.0002712339516051824,"NODE342":2.949102257951889e-05,"NODE264":0.0001401590046693688,"NODE348":1.847118161222567e-05,"IGHV1_3_01_CRF_1":2.927171080487451e-07,"NODE297":8.103906842486713e-08,"NODE340":2.175681135646649e-05,"IGHV1_69_04_CRF_1":6.700584519460484e-05,"IGHV1_69_10_CRF_1":4.962484354145754e-05,"IGHV1_69_08_CRF_1":7.046068308623292e-05,"NODE331":0.0008896434556657609,"IGHV1_69_06_CRF_1":8.806862973898626e-05,"IGHV1_69_03_CRF_1":0.0006369940453127153,"IGHV1_69_13_CRF_1":0.001097588545016872,"IGHV1_69D_01_CRF_1":0.001364572492187601,"IGHV1_69_12_CRF_1":0.0009898658877978213,"NODE272":2.086748744436206e-06,"NODE273":2.087561152901495e-06,"IGHV1_38_4_01_CRF_1":6.366488362518788e-08,"NODE296":5.972623215372523e-08,"NODE310":5.972814504913622e-08,"NODE265":2.086748494613254e-06,"NODE269":4.242838044699463e-06,"IGHV1_3_02_CRF_1":3.116258254765393e-07,"IGHV1_46_02_CRF_1":6.947403435521872e-09,"IGHV1_46_01_CRF_1":6.947403435521872e-09,"IGHV1_46_03_CRF_1":1.24199464335613e-08,"IGHV1_58_02_CRF_1":1.197695538296431e-08,"NODE263":2.591013355461415e-06},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-868/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096328 queued and waiting for resources
srun: job 26096328 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof39:14335] *** Process received signal ***
[gizmof39:14335] Signal: Segmentation fault (11)
[gizmof39:14335] Signal code: Address not mapped (1)
[gizmof39:14335] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 14335 on node gizmof39 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof39:14331] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof39:14331] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof39: task 0: Exited with exit code 139

-----

proc 893
out----

Read 10 sequences

[SEND] Sequence 04-A-M_3703765 to MPI node 1
[RECEIVE] Sequence 04-A-M_3703765 from node 1 (9 alignments remaining): V3-48*03,J5*01
[SEND] Sequence 04-A-M_3877474 to MPI node 1
Error:

Master node received an error:Bad binary operator placement  in the following context: '<<ERROR HERE>HyPhy Base Object>'

Function call stack
1 : returnAVL={"SEQUENCE_NAME":"04-A-M_3877474","REFERENCE":"Human IgG (heavy chain) with all unique F+ORF alleles, based on IMGT from Dec 2014","BEST_REARRANGEMENT":"V3-30*02","SUPPORT":0.1260836271203857,"SCORE":9693.712033866159,"DELTA_IC":-9693.712033866159,"TERMINATOR":0,"SEQUENCE":"ATGCAGACTCCGTGAAGGCCCGATTCACCATCTCCAGAGACAACTCCAAGAACACCCTGTATCTCCACATGAACAGCCTGAGAGCTGAGGACACGGCTCTGTATTGCTGTGCGACAACTGGGGCCGGGGA","SISTER_SEQUENCES":{"0":"NODE430"},"BRANCH_LENGTHS":{"TO_SISTER":{{5.000000025e-09}},"TO_PARENT":{{0.05033329049390001}},"TO_QUERY":{{0.1858917164453578}}},"REARRANGEMENTS":{"V3-30*02":0.1260836271203857,"V3-30*18":0.1251384771749349,"V3-30*03":0.1166580656857181,"V3-NL1*01":0.1137003297770463,"V3":0.07710518456420976,"V3-33*06":0.06623470564284771,"V3-30":0.06256922690908076,"V3-33*03":0.05518271748886507,"V3-33*05":0.03206693761300343,"V3-66*02":0.0288804205491709,"V3-66*03":0.01783918315106901,"V3-30*14":0.01770020882202635,"V3-66*01":0.01440334357988199,"V3-33*01":0.01114230909587544,"V3-30*01":0.01114209614399057,"V3-30*06":0.01111523601168212},"MODELS_TRIED":25365,"EXTRA":{"FW1":"","FW1_AA":"","CDR1":"","CDR1_AA":"","FW2":"","FW2_AA":"","CDR2":"","CDR2_AA":"","FW3":"NATGCAGACTCCGTGAAGGCCCGATTCACCATCTCCAGAGACAACTCCAAGAACACCCTGTATCTCCACATGAACAGCCTGAGAGCTGAGGACACGGCTCTGTATTGCTGT","FW3_AA":"?ADSVKARFTISRDNSKNTLYLHMNSLRAEDTALYCC","CDR3":"GCGACA","CDR3_AA":"AT","JUNCTION":"","JUNCTION_AA":"","D_ALLELE":<HyPhy Base Object>,"D":<HyPhy Base Object>,"D_AA":<HyPhy Base Object>,"D_SCORE":<HyPhy Base Object>,"D/INV_ALLELE":<HyPhy Base Object>,"D/INV":<HyPhy Base Object>,"D/INV_AA":<HyPhy Base Object>,"D/INV_SCORE":<HyPhy Base Object>,"J":"ACTGGGGCCGGGGA","J_AA":"TGAG?","Mapped Read":"---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------NATGCAGACTCCGTGAAGGCCCGATTCACCATCTCCAGAGACAACTCCAAGAACACCCTGTATCTCCACATGAACAGCCTGAGAGCTGAGGACACGGCTCTGTATTGCTGTGCG---ACA---------ACTGGGGCCGGGGA","ALIGNMENT_SPANS":{{0}{0}{0}{0}{0}{306}{315}{338}}},"BREAKPOINTS":{},"BREAKPOINT_SUPPORT":{{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},"BRANCH_SUPPORT":{"NODE425":0.004342727496289033,"NODE222":2.920474088606741e-06,"NODE209":0.001237038754967854,"NODE430":0.06404425979972835,"IGHJ6_04_CRF_01":4.46131157145119e-06,"IGHV3_35_01_CRF_1":6.634336355381047e-06,"NODE223":2.285695875041109e-06,"IGHV3_30_08_CRF_1":0.005565676057067111,"IGHJ4_03_CRF_01":5.632942462503304e-06,"IGHV3_53_03_CRF_1":5.652661625049139e-05,"IGHV3_30_18_CRF_1":0.06297926043458058,"NODE215":2.5314618786227e-06,"NODE443":0.06264509815259728,"IGHJ2_01_CRF_01":2.393043240714896e-06,"NODE218":3.720894935545659e-07,"NODE226":2.765954714160783e-06,"IGHV3_NL1_01_CRF_1":0.1137284617160054,"NODE432":0.0626808177842537,"NODE213":2.263576695290638e-06,"NODE217":3.963893877393722e-06,"IGHJ5_01_CRF_01":9.69855000420834e-07,"IGHJ3_02_CRF_01":1.53333516492202e-05,"IGHV3_30_06_CRF_1":0.005557666703895695,"IGHJ4_02_CRF_01":1.749262778086634e-06,"IGHJ1_01_CRF_01":6.549537259229358e-06,"IGHV3_33_05_CRF_1":0.03208758365985277,"IGHV3_30_12_CRF_1":0.005563711499444642,"NODE417":0.0001917789494875712,"IGHV3_66_03_CRF_1":0.0178610956989318,"NODE442":0.06268251992118176,"NODE477":2.489381920581276e-05,"IGHJ3_01_CRF_01":5.606849588804475e-05,"IGHV3_30_3_03_CRF_1":0.005557660777050328,"IGHV3_33_03_CRF_1":0.05518344152561689,"NODE475":0.0001749395008496655,"NODE476":0.005736153882511684,"NODE384":0.001768851952959304,"IGHV3_30_3_01_CRF_1":0.005557667438498375,"IGHV3_30_13_CRF_1":0.005557659894202696,"NODE446":0.005557660777050328,"IGHV3_30_3_02_CRF_1":0.003665441696591134,"NODE394":0.01447723630631767,"IGHV3_66_02_CRF_1":0.01440905087550038,"IGHV3_30_01_CRF_1":0.005557658315979299,"IGHV3_30_11_CRF_1":0.005557660777050328,"NODE433":0.06257612115615195,"IGHV3_OR16_12_01_CRF_1":8.330757358011954e-05,"NODE382":0.0001145707761322435,"NODE397":0.002023649491398694,"NODE391":0.001771894952321696,"NODE383":0.001784745869398201,"NODE406":0.0001870357012162168,"IGHJ5_02_CRF_01":1.126792748544727e-06,"IGHV3_66_01_CRF_1":0.01441378353652223,"IGHJ4_01_CRF_01":2.257556762725675e-06,"IGHJ6_02_CRF_01":3.486445671743047e-07,"IGHJ6_03_CRF_01":2.250670329363065e-06,"NODE210":1.397446392973075e-05,"IGHJ6_01_CRF_01":2.41523696091405e-06,"IGHV3_53_04_CRF_1":0.00201451474516879,"NODE445":0.005584585708139527,"IGHV3_30_09_CRF_1":0.005557659632465241,"NODE355":9.316122711809691e-06,"NODE529":7.951240699609057e-06,"NODE531":1.980840922569316e-08,"IGHV3_25_04_CRF_1":1.502345003422824e-08,"NODE356":5.196446868872397e-05,"NODE416":0.0001939220136587237,"NODE474":0.0001749261139593156,"NODE518":0.000747003392985857,"NODE526":0.0002075632704231022,"IGHV3_64_02_CRF_1":3.127763504568954e-07,"IGHV3_64_01_CRF_1":3.1277637133375e-07,"NODE519":6.952591711155332e-05,"NODE521":2.317430998767057e-06,"IGHV3_64D_06_CRF_1":2.460497717582961e-07,"NODE522":9.231801914281842e-09,"IGHV3_64_04_CRF_1":2.834769974934213e-05,"NODE507":0.0001751202858448405,"IGHV3_23_04_CRF_1":7.858115467986893e-08,"NODE508":7.858118988371888e-08,"IGHV3_23_02_CRF_1":7.858130763777916e-08,"IGHV3_23_01_CRF_1":7.858117486397272e-08,"NODE509":1.08888735033599e-06,"NODE511":1.084770909569322e-06,"IGHV3_23_03_CRF_1":1.084770761923658e-06,"IGHV3_23_05_CRF_1":1.084770920085421e-06,"NODE492":9.409699690578357e-05,"NODE494":1.536317866485956e-06,"NODE498":1.547321749443437e-06,"IGHV3_9_01_CRF_1":1.323393993163571e-07,"IGHV3_9_03_CRF_1":1.323394353987179e-07,"IGHV3_9_02_CRF_1":1.323394620386861e-07,"NODE495":5.046133289410468e-06,"IGHV3_20_02_CRF_1":2.2609572940756e-07,"IGHV3_20_01_CRF_1":2.2609572940756e-07,"NODE483":7.307799341332711e-07,"NODE478":7.383590841753349e-07,"NODE480":8.736681973593393e-08,"IGHV3_OR16_10_02_CRF_1":8.736682145806315e-08,"IGHV3_OR16_10_01_CRF_1":8.736682145806315e-08,"IGHV3_OR16_10_03_CRF_1":8.736681973593393e-08,"IGHV3_30_14_CRF_1":0.01770020882202635,"IGHV3_30_16_CRF_1":0.005557619034838671,"IGHV3_30_17_CRF_1":0.005557619034838671,"IGHV3_30_07_CRF_1":0.005557617981298213,"IGHV3_30_15_CRF_1":0.005557622488124207,"IGHV3_30_10_CRF_1":0.005557616591959131,"NODE450":0.005557619169195662,"IGHV3_30_03_CRF_1":0.1166580656857181,"IGHV3_30_19_CRF_1":0.0055576193035476,"IGHV3_30_05_CRF_1":0.005557619454847702,"NODE437":0.005584612757893979,"IGHV3_33_02_CRF_1":0.00555769759927469,"IGHV3_33_01_CRF_1":0.005557696337981465,"IGHV3_33_04_CRF_1":0.005557698799077805,"IGHV3_33_06_CRF_1":0.003665482847984288,"IGHV3_30_02_CRF_1":0.06256924976291492,"NODE426":0.00537410195848496,"IGHV3_7_02_CRF_1":8.40592886890368e-09,"IGHV3_7_01_CRF_1":8.40592886890368e-09,"IGHV3_7_03_CRF_1":8.405929362352113e-09,"NODE418":0.0002745950725491216,"IGHV3_OR16_13_01_CRF_1":1.495864988358199e-07,"IGHV3_74_02_CRF_1":3.418050986771827e-08,"IGHV3_74_01_CRF_1":3.418050986771827e-08,"IGHV3_74_03_CRF_1":3.418050986771827e-08,"NODE405":7.471560404087536e-05,"NODE411":3.637564188945651e-05,"NODE407":3.715813417177525e-06,"NODE358":5.077495003384471e-05,"IGHV3_53_01_CRF_1":5.649734687869786e-05,"IGHV3_53_02_CRF_1":5.649737202024508e-05,"NODE386":2.486223784188187e-07,"IGHV3_38_3_01_CRF_1":1.944811115317316e-07,"NODE359":8.961945697585266e-05,"NODE373":5.110365519775789e-08,"NODE377":5.110368823668753e-08,"IGHV3_11_06_CRF_1":6.098269507793912e-08,"NODE374":5.11036447818798e-08,"NODE360":5.110365508063206e-08,"NODE362":5.110365626165085e-08,"IGHV3_48_02_CRF_1":5.1103647440961e-08,"IGHV3_48_01_CRF_1":5.110364644808736e-08,"IGHV3_48_04_CRF_1":5.110363458133866e-08,"NODE363":5.110364360625278e-08,"IGHV3_21_01_CRF_1":1.435055793363471e-08,"IGHV3_21_03_CRF_1":1.435058181652851e-08,"IGHV3_21_04_CRF_1":1.435055936825774e-08,"IGHV3_21_02_CRF_1":1.435056098167696e-08,"IGHV3_48_03_CRF_1":5.110363266191671e-08},"FRAME":0}
-------
2 : ExecuteCommands in string "returnAVL = "+returnValue using basepath /fh/fast/matsen_e/dralph/work/igscueal/A/data/istartstop-0-10000/igs-893/.
-------
3 : mpiNode=ReceiveAJob(0)
-------
4 : SendAJob(seqID)
-------

-----
err----
srun: job 26096329 queued and waiting for resources
srun: job 26096329 has been allocated resources
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gizmof40:19304] *** Process received signal ***
[gizmof40:19304] Signal: Segmentation fault (11)
[gizmof40:19304] Signal code: Address not mapped (1)
[gizmof40:19304] Failing at address: 0x31
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 19304 on node gizmof40 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gizmof40:19299] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gizmof40:19299] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
srun: error: gizmof40: task 0: Exited with exit code 139

-----