Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Readme mimic-iv postgres build and concepts links #1685

Merged
merged 3 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions mimic-iv/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@

The repository consists of a number of Structured Query Language (SQL) scripts which build the MIMIC-IV database in a number of systems and extract useful concepts from the raw data. Subfolders include:

* [buildmimic](/mimic-iv/buildmimic) - Scripts to build MIMIC-IV in various relational database management system (RDMS), in particular [postgres](/buildmimic/postgres) is a popular open source option
* [buildmimic](/mimic-iv/buildmimic) - Scripts to build MIMIC-IV in various relational database management system (RDMS), in particular [postgres](/mimic-iv/buildmimic/postgres) is a popular open source option
* [concepts](/mimic-iv/concepts) - Useful views/summaries of the data in MIMIC-IV, e.g. demographics, organ failure scores, severity of illness scores, durations of treatment, easier to analyze views, etc. The paper above describes these in detail, and a README in the subfolder lists concepts generated.

### Concepts

The [MIMIC-IV concepts](/mimic-iv/concepts) are written in an SQL syntax compatible with BigQuery. These scripts have been converted to PostgreSQL by a script. To generate the concepts in PostgreSQL, see the [MIMIC-IV postgresql concepts subfolder](/mimic-iv/concepts/postgres).
The [MIMIC-IV concepts](/mimic-iv/concepts) are written in an SQL syntax compatible with BigQuery. These scripts have been converted to PostgreSQL by a script. To generate the concepts in PostgreSQL, see the [MIMIC-IV postgresql concepts subfolder](/mimic-iv/concepts_postgres).

Tables in the BigQuery `physionet-data.mimic_derived` dataset are generated using the concepts made available in this folder. These tables are generated using the code in the [latest release on GitHub](https://github.com/MIT-LCP/mimic-code/releases).
10 changes: 5 additions & 5 deletions mimic-iv/concepts/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# MIMIC-IV Concepts

This folder contains scripts to generate useful abstractions of raw MIMIC-IV data ("concepts").
The scripts are written using the **BigQuery Standard SQL Dialect**. Concepts are categorized into folders if possible, otherwise they remain in the top-level directory. The [postgres](/mimic-iv/concepts/postgres) subfolder contains automatically generated PostgreSQL versions of these scripts; [see below for how these were generated](#postgresql-concepts). Concepts are categorized into folders if possible, otherwise they remain in the top-level directory.
The scripts are written using the **BigQuery Standard SQL Dialect**. Concepts are categorized into folders if possible, otherwise they remain in the top-level directory. The [postgres](/mimic-iv/concepts_postgres) subfolder contains automatically generated PostgreSQL versions of these scripts; [see below for how these were generated](#postgresql-concepts). Concepts are categorized into folders if possible, otherwise they remain in the top-level directory.

The concepts are organized into individual SQL scripts, with each script generating a table. The BigQuery `mimiciv_derived` dataset under `physionet-data` contains the concepts pregenerated. Access to this dataset is available to MIMIC-IV approved users: see the [cloud instructions](https://mimic.mit.edu/docs/gettingstarted/cloud/) on how to access MIMIC-IV on BigQuery (which includes the derived concepts).

Expand All @@ -26,9 +26,9 @@ In general the concepts may be generated in any order, except for the *first_day

## Generating the concepts on PostgreSQL

These instructions are used to regenerate the [postgres](/mimic-iv/concepts/postgres) scripts from the BigQuery dialect scripts in the concepts folder.
These instructions are used to regenerate the [postgres](/mimic-iv/concepts_postgres) scripts from the BigQuery dialect scripts in the concepts folder.

* **If you just want to create PostgreSQL concepts for your installation of MIMIC-IV, go to the [postgres](/mimic-iv/concepts/postgres) subfolder**
* **If you just want to create PostgreSQL concepts for your installation of MIMIC-IV, go to the [postgres](/mimic-iv/concepts_postgres) subfolder**
* If you would like to understand the process better, and possibly improve upon it, read on

Analogously to [MIMIC-III Concepts](https://github.com/MIT-LCP/mimic-code/tree/master/concepts), the SQL scripts here are written in BigQuery's Standard SQL syntax. The concepts have been carefully written to allow conversion to PostgreSQL, so that only the following changes are necessary to make them compaible with PostgreSQL:
Expand All @@ -37,13 +37,13 @@ Analogously to [MIMIC-III Concepts](https://github.com/MIT-LCP/mimic-code/tree/m
* modify SQL scripts for incompatible syntax
* run the modified SQL scripts and direct the output into tables in the PostgreSQL database

To do this, we have created a (*nix/Mac OS X) compatible shell script which performs regular expression replacements for each script. To simplify the process for users, we output these automatically generated scripts to the [postgres](/mimic-iv/concepts/postgres) folder.
To do this, we have created a (*nix/Mac OS X) compatible shell script which performs regular expression replacements for each script. To simplify the process for users, we output these automatically generated scripts to the [postgres](/mimic-iv/concepts_postgres) folder.
Re-running this shell script can be done as follows:

1. Open a terminal in the `concepts` folder.
2. Run [convert_bigquery_to_postgres.sh](convert_bigquery_to_postgres.sh).
* e.g. `bash convert_bigquery_to_postgres.sh`
* This file outputs the scripts to the [postgres](/mimic-iv/concepts/postgres) subfolder after applying a few changes.
* This file outputs the scripts to the [postgres](/mimic-iv/concepts_postgres) subfolder after applying a few changes.
* This also creates the `postgres_make_concepts.sql` script in the postgres subfolder.

### Known Problems
Expand Down