Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release #18

joeflack4 · 2024-03-22T23:33:44Z

addresses #16

Changes

SSSOM outputs
- Add: Code for 2 new outputs: hpo-umls.sssom.tsv and hpo-mesh.sssom.tsv

General
- Add: Python requirement: PyYaml
- Update: ORCID in SSSOM config
- Bug fix: Edge case for missing intermediate file in pipeline
- Remove: Intentionally duplicative, no longer needed 'MEDGENCUI' xrefs
- Add: Refactor to load mapping set using function that handles lots of repetitive stuff: get_mapping_set()

joeflack4 · 2024-03-23T20:03:41Z

CC @souzadevinicius Just adding some SSSOM outputs for the MedGen ingest.

src/utils.py

makefile

joeflack4 · 2024-03-24T04:35:09Z

src/utils.py

+    os.remove(temp_filtered_config_path)
+
+
+def write_tsv_with_comments(df: pd.DataFrame, comments_file: Union[Path, str], outpath: Union[Path, str]):


General: write_tsv_with_comments()

@matentzn Let me know what the existing best practice is supposed to be for this.
I have this function (and several others it's used with) in the repos for GARD, MedGen, ICD11, a TIMS repo, and possibly elsewhere. Maybe there's a function in sssom-py that does this like I want? I have a dataframe and a path to a sssom-metadata.yml, and I want to write that to a file and include the comment at the top.

yep,

instantiate a MappingSetDataFrame() object with the dataframe

use write_table() method to frite the result

See example: https://github.com/mapping-commons/sssom-py/blob/be8a77b5a2ef42cd2b92b74d892d7626dec27471/src/sssom/cli.py#L280

I recall this a bit now, thanks. Will do.

Done~

Now my write_sssom() function has been rewritten to utilize sssom-py:

def write_sssom(df: pd.DataFrame, config_path: Union[Path, str], outpath: Union[Path, str]): """Writes a SSSOM file""" with open(config_path, 'r') as yaml_file: metadata: Dict = yaml.load(yaml_file, Loader=yaml.FullLoader) converter = curies.Converter.from_prefix_map(metadata['curie_map']) msdf: MappingSetDataFrame = MappingSetDataFrame(converter=converter, df=df, metadata=metadata) with open(outpath, 'w') as f: write_table(msdf, f)

As with my icd11 PR which had the same refactor, while working on this I also encountered some possible bugs and UX issues, and made:

MappingSetDataFrame issues when no converter passed mapping-commons/sssom-py#513

UX: Extra entries in curie_map of written TSV mapping-commons/sssom-py#514

UX: Second empty line break at bottom of file mapping-commons/sssom-py#515

Suggestion: write_table: pass outpath alternative to TextIO mapping-commons/sssom-py#516

Suggestion: MappingSetDataFrame: .to_csv() and read_csv() mapping-commons/sssom-py#517

src/utils.py

joeflack4 · 2024-03-24T04:54:07Z

.github/workflows/buid_and_release.yml

@@ -39,3 +39,5 @@ jobs:
          output/release/medgen.obo
          output/release/medgen-disease-extract.obo
          output/release/medgen-xrefs.robot.template.tsv
+          output/release/hpo-umls.sssom.tsv
+          output/release/hpo-mesh.sssom.tsv


SSSOM Outputs

@matentzn @twhetzel review

@matentzn See question(s)

I ran a new build and created a release which includes these. Here are some sample rows.

Questions

Double checking that you 100% want only UMLS mappings
Should MedGen-derived prefixes just be "UMLS", or one of ("UMLS", "MEDGENCUI", or "MEDGEN") if/where applicable?
UMLS: Compose 100% of the subjects in hpo-umls.sssom.tsv currently.
MEDGENCUI: Note that there were no MEDGENCUI<->HPO mappings in this set in MedGenIDMappings.txt. Also note that I did see your instruction in Finalise MedGen xref table #15: Remove all rows with MEDGENCUI in it (we only need xrefs to MEDGEN:123 and UMLS:123) (please see & respond to my related question in that issue). However, applied our previous logic discussed in "Refactoring > Prefix assignment", and verified afterwards that there was no MEDGENCUI.
MEDGEN: These are IDs that have neither CN or C at the beginning; they are UIDs. I did not check to see if we could actually get HPO<->MEDGEN mappings. Do you want them?

Subject/object order & filename
I think I renamed hpo-umls.sssom.tsv incorrectly. Just realizing this now. I have the subjects as UMLS. But HPO comes first in the filename; so perhaps I should make it so that the position is consistent? Does it matter to you which is the subject and which is the object?

Are my columns good?
I could include mapping_justification, but I don't know their process, and I don't know if we can say if there is or isn't variability in how the do their mappings.

hpo-umls.sssom.tsv

subject_id subject_label predicate_id object_id

UMLS:C0000727 Acute abdomen skos:exactMatch HP:0033400

UMLS:C0000729 Abdominal cramps skos:exactMatch HP:0032155

UMLS:C0000731 Abdominal distention skos:exactMatch HP:0003270

hpo-mesh.sssom.tsv

subject_id predicate_id object_id

HP:0000003 skos:exactMatch MESH:D021782

HP:0000005 skos:exactMatch MESH:D040582

HP:0000011 skos:exactMatch MESH:D001750

This one doesn't have subject_label or object_label because I couldn't guarantee that the label in MedGenIDMappings.txt was an accurate reflection of either source in 100% of cases. I imagine these labels are coming either from UMLS or from MedGen.

hpo-mesh-no-matches.sssom.tsv

hpo-mesh-no-matches.sssom.tsv.zip

subject_id predicate_id object_id umls_id umls_label

skos:exactMatch MESH:C000591739 UMLS:C1970109 Aromatase excess syndrome

skos:exactMatch MESH:C000596385 UMLS:C3495589 Jalili syndrome

skos:exactMatch MESH:C000597569 UMLS:C4042185 Teratoid Rhabdoid Tumor

Not included in the release. This is for analysis. I just wanted you to see the cases where there were no matches. There were only about 2,300 HPO<->MeSH mappings that could be derived out of the ~16,000 MeSH terms that have UMLS mappings. I included umls_id and umls_label in this set for review. It also includes all of the rows for all of the matches; not just the non-matches. I sorted the non-matches to the top.

Double checking that you 100% want only UMLS mappings

For now yes. Maybe in the future (but not now) we will look for medgen mappings as well, but for now, we only want HPO-UMLS mappings.

@joeflack4 Ensure this is done (probably already so)

I think I renamed hpo-umls.sssom.tsv incorrectly. Just realizing this now. I have the subjects as UMLS. But HPO comes first in the filename; so perhaps I should make it so that the position is consistent? Does it matter to you which is the subject and which is the object?

Its cosmetic, ask your own inner sense of style. I like the order, but wont insist on it if it takes a lot of time to fix

@joeflack4 flip the order in the file name

I could include mapping_justification, but I don't know their process, and I don't know if we can say if there is or isn't variability in how the do their mappings.

Please use sssom toolkit as you want to release a valid sssom file. mapping_justification is mandatory. Since its MedGen I would tend to use semapv:ManualMappingCuration.

@joeflack4 add mapping_justification

I edited your comment w/ checkboxes. Will do.

.github/workflows/buid_and_release.yml

- Add: Code for 2 new outputs: hpo-umls.sssom.tsv and hpo-mesh.sssom.tsv General - Add: Python requirement: PyYaml - Update: ORCID in SSSOM config - Bug fix: Edge case for missing intermediate file in pipeline - Remove: Intentionally duplicative, no longer needed 'MEDGENCUI' xrefs - Add: Refactor to load mapping set using function that handles lots of repetitive stuff: get_mapping_set()

- Update: Utilizing standard sssom-py functionality - Add: sssom validate - Add: mapping_justification - Update: Subject/object order & filename - Add: Mandatory filtering of UMLS matches only in HPO-UMLS SSSOM.

joeflack4 · 2024-04-07T02:23:04Z

makefile

+#medgen.sssom.tsv: medgen.obographs.json
+#	sssom parse medgen.obographs.json -I obographs-json -m config/medgen.sssom-metadata.yml -o $@
+sssom: umls-hpo.sssom.tsv
+	sssom validate umls-hpo.sssom.tsv


SSSOM validation

As with the ICD11 PR, I also added SSSOM validation here.

joeflack4 · 2024-04-07T02:24:50Z

@matentzn @twhetzel All's good with this one now as well, I believe.

matentzn

I did not do a detailed code review, but what I saw looks good! Thanks!

joeflack4 · 2024-04-08T22:27:01Z

@matentzn Roger.
@twhetzel This has been amply reviewed; merging now.

joeflack4 changed the title ~~SSSOM~~ SSSOM outputs: HPO & MeSH Mar 23, 2024

joeflack4 requested review from matentzn and twhetzel March 23, 2024 20:03

joeflack4 self-assigned this Mar 23, 2024

joeflack4 added the enhancement New feature or request label Mar 23, 2024

joeflack4 linked an issue Mar 23, 2024 that may be closed by this pull request

Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release #16

Closed

joeflack4 changed the title ~~SSSOM outputs: HPO & MeSH~~ Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release #16 Mar 23, 2024

joeflack4 changed the title ~~Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release #16~~ Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release Mar 23, 2024

joeflack4 commented Mar 23, 2024

View reviewed changes

src/utils.py Show resolved Hide resolved

joeflack4 changed the title ~~Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release~~ SSSOM outputs && robot template tweaks Mar 23, 2024

joeflack4 force-pushed the sssom branch 3 times, most recently from fe3fecb to 4ac6825 Compare March 24, 2024 04:00

joeflack4 commented Mar 24, 2024

View reviewed changes

makefile Show resolved Hide resolved

joeflack4 force-pushed the sssom branch from 4ac6825 to 4ab08d4 Compare March 24, 2024 04:29

joeflack4 changed the title ~~SSSOM outputs && robot template tweaks~~ SSSOM outputs Mar 24, 2024

joeflack4 commented Mar 24, 2024

View reviewed changes

.github/workflows/buid_and_release.yml Show resolved Hide resolved

joeflack4 commented Mar 24, 2024

View reviewed changes

.github/workflows/buid_and_release.yml Show resolved Hide resolved

joeflack4 changed the title ~~SSSOM outputs~~ Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release Mar 24, 2024

joeflack4 commented Mar 24, 2024

View reviewed changes

.github/workflows/buid_and_release.yml Show resolved Hide resolved

joeflack4 force-pushed the sssom branch 6 times, most recently from 91e9e37 to bff4784 Compare March 30, 2024 00:09

joeflack4 force-pushed the sssom branch from bff4784 to bd2ff8a Compare March 30, 2024 20:17

joeflack4 mentioned this pull request Mar 30, 2024

ICD11 --> ORDO mappings monarch-initiative/icd11#12

Merged

joeflack4 mentioned this pull request Apr 7, 2024

MEDGENCUI possible future edge case handling #20

Closed

joeflack4 force-pushed the sssom branch from e1009f6 to 0dd37ea Compare April 7, 2024 02:19

SSSOM outputs

d640dad

- Update: Utilizing standard sssom-py functionality - Add: sssom validate - Add: mapping_justification - Update: Subject/object order & filename - Add: Mandatory filtering of UMLS matches only in HPO-UMLS SSSOM.

joeflack4 force-pushed the sssom branch from 0dd37ea to d640dad Compare April 7, 2024 02:22

joeflack4 commented Apr 7, 2024

View reviewed changes

matentzn approved these changes Apr 8, 2024

View reviewed changes

joeflack4 merged commit 0b8e063 into main Apr 8, 2024

joeflack4 deleted the sssom branch April 8, 2024 22:27

This was referenced Apr 15, 2024

MedGen build failure #22

Open

MedGen build error #23

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release #18

Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release #18

joeflack4 commented Mar 22, 2024 •

edited

Loading

joeflack4 commented Mar 23, 2024

joeflack4 Mar 24, 2024 •

edited

Loading

matentzn Apr 4, 2024

joeflack4 Apr 4, 2024 •

edited

Loading

joeflack4 Apr 7, 2024

joeflack4 Mar 24, 2024 •

edited

Loading

matentzn Apr 4, 2024 •

edited by joeflack4

Loading

joeflack4 Apr 4, 2024

joeflack4 Apr 7, 2024

joeflack4 Apr 7, 2024

joeflack4 commented Apr 7, 2024

matentzn left a comment

joeflack4 commented Apr 8, 2024

		os.remove(temp_filtered_config_path)


		def write_tsv_with_comments(df: pd.DataFrame, comments_file: Union[Path, str], outpath: Union[Path, str]):

subject_id	subject_label	predicate_id	object_id
UMLS:C0000727	Acute abdomen	skos:exactMatch	HP:0033400
UMLS:C0000729	Abdominal cramps	skos:exactMatch	HP:0032155
UMLS:C0000731	Abdominal distention	skos:exactMatch	HP:0003270

subject_id	predicate_id	object_id
HP:0000003	skos:exactMatch	MESH:D021782
HP:0000005	skos:exactMatch	MESH:D040582
HP:0000011	skos:exactMatch	MESH:D001750

predicate_id	object_id	umls_id	umls_label
skos:exactMatch	MESH:C000591739	UMLS:C1970109	Aromatase excess syndrome
skos:exactMatch	MESH:C000596385	UMLS:C3495589	Jalili syndrome
skos:exactMatch	MESH:C000597569	UMLS:C4042185	Teratoid Rhabdoid Tumor

Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release #18

Add HPO<->UMLS and HPO<->MeSH mapping to MedGen release #18

Conversation

joeflack4 commented Mar 22, 2024 • edited Loading

Changes

joeflack4 commented Mar 23, 2024

joeflack4 Mar 24, 2024 • edited Loading

Choose a reason for hiding this comment

General: write_tsv_with_comments()

matentzn Apr 4, 2024

Choose a reason for hiding this comment

joeflack4 Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

joeflack4 Apr 7, 2024

Choose a reason for hiding this comment

joeflack4 Mar 24, 2024 • edited Loading

Choose a reason for hiding this comment

SSSOM Outputs

Questions

hpo-umls.sssom.tsv

hpo-mesh.sssom.tsv

hpo-mesh-no-matches.sssom.tsv

matentzn Apr 4, 2024 • edited by joeflack4 Loading

Choose a reason for hiding this comment

joeflack4 Apr 4, 2024

Choose a reason for hiding this comment

joeflack4 Apr 7, 2024

Choose a reason for hiding this comment

joeflack4 Apr 7, 2024

Choose a reason for hiding this comment

SSSOM validation

joeflack4 commented Apr 7, 2024

matentzn left a comment

Choose a reason for hiding this comment

joeflack4 commented Apr 8, 2024

joeflack4 commented Mar 22, 2024 •

edited

Loading

joeflack4 Mar 24, 2024 •

edited

Loading

General: `write_tsv_with_comments()`

joeflack4 Apr 4, 2024 •

edited

Loading

joeflack4 Mar 24, 2024 •

edited

Loading

`hpo-umls.sssom.tsv`

`hpo-mesh.sssom.tsv`

`hpo-mesh-no-matches.sssom.tsv`

matentzn Apr 4, 2024 •

edited by joeflack4

Loading