pandasData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 0 : Introduction\n",
    "_____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will use pandas to analyze the csv file and prepare it for a seq2seq model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using pandas we can sort the data suitable for our needs. To achieve this we need to understand how the data is interpreted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Current dataset in this example is results of radiology reports and the diagnosis results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Pandas Basics\n",
    "_____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "below you can see how to import the data presented in csv to your python code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataFrame=pd.read_csv(\"ReportsDATASET.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets see details about your data, describe is much more useful for numeric data but still you can see that there are 1984 text values and 1982 unique values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>1982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>\\nSIGNATURE\\nXXXX\\n\\nRADIOLOGY REPORT\\nchest pain CHEST 2V FRONTAL/LATERAL XXXX, XXXX XXXX XXXX Comparison: XXXX, XXXX. The heart and lungs have XXXX XXXX in the interval. Both lungs are clear and expanded. Heart and mediastinum normal. Transcribed by - PSCB Transcription Date - XXXX\\n\\nIMPRESSION\\nNo active disease. DICTATED BY : Dr. XXXX XXXX XXXX XXXX XXXX ELECTRONICALLY SIGNED XXXX. XXXX XXXX XXXX XXXX XXXX TRANSCRIBED XXXX XXXX\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                            Text\n",
       "count                                                                                                                                                                                                                                                                                                                                                                                                                                                       1984\n",
       "unique                                                                                                                                                                                                                                                                                                                                                                                                                                                      1982\n",
       "top     \\nSIGNATURE\\nXXXX\\n\\nRADIOLOGY REPORT\\nchest pain CHEST 2V FRONTAL/LATERAL XXXX, XXXX XXXX XXXX Comparison: XXXX, XXXX. The heart and lungs have XXXX XXXX in the interval. Both lungs are clear and expanded. Heart and mediastinum normal. Transcribed by - PSCB Transcription Date - XXXX\\n\\nIMPRESSION\\nNo active disease. DICTATED BY : Dr. XXXX XXXX XXXX XXXX XXXX ELECTRONICALLY SIGNED XXXX. XXXX XXXX XXXX XXXX XXXX TRANSCRIBED XXXX XXXX\\n\\n\n",
       "freq                                                                                                                                                                                                                                                                                                                                                                                                                                                           2"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataFrame.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "with looking at the shape of dataframe we can see row and columns size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1984, 1)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataFrame.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "with head() method you can see first lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\nExam: 2 views of the chest XXXX/XXXX.\\n \\nComparison: None.\\n \\nIndication: Positive TB test\\n \\nFindings:\\nThe cardiac silhouette and mediastinum size are within normal limits.\\nThere is no pulmonary edema. There is no focal consolidation. There\\nare no XXXX of a pleural effusion. There is no evidence of\\npneumothorax.\\n \\nImpression:\\nNormal chest x-XXXX. \\nThis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EXAM(S): Chest, 2 views, frontal and lateral\\n\\nDATE\\nXXXX\\n\\nCOMPARISON\\nNone.\\n\\nINDICATION\\nPreop bariatric surgery.\\n\\nFINDINGS\\nBorderline cardiomegaly. Midline sternotomy XXXX. Enlarged pulmonary arteries. Clear lungs. Inferior XXXX XXXX XXXX.\\n\\nIMPRESSION\\nNo acute pulmonary findings. \\n XXXX XXXX for the opportunity to care for your patient. If XXXX have any questions regarding this report, please XXXX the radiologist, Dr. XXXX XXXX, at XXXX.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nExam\\nXray Chest PA and Lateral\\n\\nDate\\nXXXX\\n\\nHistory\\nrib pain after a XXXX, XXXX XXXX steps this XXXX. Pain to R back, R elbow and R rib XXXX, no previous heart or lung hx, non-XXXX, no hx ca\\n\\nImpression\\nNo displaced rib fractures, pneumothorax, or pleural effusion identified. Well-expanded and clear lungs. Mediastinal contour within normal limits. No acute cardiopulmonary abnormality identified.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nRADIOLOGY REPORT\\n\\nExamination\\nPA and lateral views of the chest XXXX, XXXX at XXXX hours History: XXXX-year-old XXXX with XXXX. Comparison: None available Findings: There are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. There are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.There are streaky opacities in the right upper lobe, XXXX scarring. The cardiomediastinal silhouette is normal in size and contour. There is no pneumothorax or large pleural effusion. Transcribed by - PSC Transcription Date - XXXX\\n\\nIMPRESSION\\n1. Bullous emphysema and interstitial fibrosis. 2. Probably scarring in the left apex, although difficult to exclude a cavitary lesion. 3. Opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or CT thorax to document resolution.\\n\\nSIGNATURE\\nXXXX\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\nEXAMINATION: CHEST ( FRONTAL AND LATERAL): XXXX, XXXX XXXX PM \\n \\nCLINICAL INDICATION: Chest and nasal congestion.\\n \\nCOMPARISXXXX/XXXX.\\n \\nFINDINGS:\\nThe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. There is no pneumothorax or pleural effusion. There\\nare no focal areas of consolidation. Cholecystectomy clips are\\npresent. Small T-spine osteophytes. There is biapical pleural\\nthickening, unchanged from prior. Mildly hyperexpanded lungs.\\n \\nIMPRESSION:\\nNo acute cardiopulmonary abnormality.\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Text\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\nExam: 2 views of the chest XXXX/XXXX.\\n \\nComparison: None.\\n \\nIndication: Positive TB test\\n \\nFindings:\\nThe cardiac silhouette and mediastinum size are within normal limits.\\nThere is no pulmonary edema. There is no focal consolidation. There\\nare no XXXX of a pleural effusion. There is no evidence of\\npneumothorax.\\n \\nImpression:\\nNormal chest x-XXXX. \\nThis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n\n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               EXAM(S): Chest, 2 views, frontal and lateral\\n\\nDATE\\nXXXX\\n\\nCOMPARISON\\nNone.\\n\\nINDICATION\\nPreop bariatric surgery.\\n\\nFINDINGS\\nBorderline cardiomegaly. Midline sternotomy XXXX. Enlarged pulmonary arteries. Clear lungs. Inferior XXXX XXXX XXXX.\\n\\nIMPRESSION\\nNo acute pulmonary findings. \\n XXXX XXXX for the opportunity to care for your patient. If XXXX have any questions regarding this report, please XXXX the radiologist, Dr. XXXX XXXX, at XXXX.\\n\n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             \\nExam\\nXray Chest PA and Lateral\\n\\nDate\\nXXXX\\n\\nHistory\\nrib pain after a XXXX, XXXX XXXX steps this XXXX. Pain to R back, R elbow and R rib XXXX, no previous heart or lung hx, non-XXXX, no hx ca\\n\\nImpression\\nNo displaced rib fractures, pneumothorax, or pleural effusion identified. Well-expanded and clear lungs. Mediastinal contour within normal limits. No acute cardiopulmonary abnormality identified.\\n\n",
       "3  \\nRADIOLOGY REPORT\\n\\nExamination\\nPA and lateral views of the chest XXXX, XXXX at XXXX hours History: XXXX-year-old XXXX with XXXX. Comparison: None available Findings: There are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. There are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.There are streaky opacities in the right upper lobe, XXXX scarring. The cardiomediastinal silhouette is normal in size and contour. There is no pneumothorax or large pleural effusion. Transcribed by - PSC Transcription Date - XXXX\\n\\nIMPRESSION\\n1. Bullous emphysema and interstitial fibrosis. 2. Probably scarring in the left apex, although difficult to exclude a cavitary lesion. 3. Opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or CT thorax to document resolution.\\n\\nSIGNATURE\\nXXXX\\n\\n\n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                             \\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\nEXAMINATION: CHEST ( FRONTAL AND LATERAL): XXXX, XXXX XXXX PM \\n \\nCLINICAL INDICATION: Chest and nasal congestion.\\n \\nCOMPARISXXXX/XXXX.\\n \\nFINDINGS:\\nThe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. There is no pneumothorax or pleural effusion. There\\nare no focal areas of consolidation. Cholecystectomy clips are\\npresent. Small T-spine osteophytes. There is biapical pleural\\nthickening, unchanged from prior. Mildly hyperexpanded lungs.\\n \\nIMPRESSION:\\nNo acute cardiopulmonary abnormality.\\n\\n"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataFrame.head() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets unlimit displayed width"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_colwidth', None)\n",
    "dataFrame.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are quite long lines but we have very important key words\n",
    "\n",
    "Findings and Impressions \n",
    "\n",
    "where findigs tells us what is happened \n",
    "\n",
    "and \n",
    "\n",
    "Impressions the diagnosis by looking at what happened\n",
    "\n",
    "lets search for keywords and see how many lines have does keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of total rows 1984 \n",
      "Number of rows containing the keywords : 1982\n"
     ]
    }
   ],
   "source": [
    "# List you keywords\n",
    "keywords = ['IMPRESSION', 'FINDINGS']\n",
    "\n",
    "# dataFrame[\"Text\"] is column selection\n",
    "# str.contains checsk if keywords exits\n",
    "# '|'.join(keywords) joins keywords with or so if one of them exits we count\n",
    "# case=False makes it case in sensitive\n",
    "# na=False makes NaN values counts as not includings\n",
    "\n",
    "mask = dataFrame['Text'].str.contains('|'.join(keywords), case=False, na=False)\n",
    "\n",
    "# Count the number of rows containing the keywords\n",
    "count = mask.sum()\n",
    "\n",
    "print(f'Number of total rows {dataFrame.shape[0]} ')\n",
    "print(f'Number of rows containing the keywords : {count}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['IMPRESSION', 'FINDINGS']\n"
     ]
    }
   ],
   "source": [
    "print(keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "all lines except  2, lets see what are those lines. We already created a mask lets invert it and apply it "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                                   Text\n",
      "15    \\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\n\\n\n",
      "1501    \\nChest PA-Lat XR\\n\\nImaging Study\\nXR Chest PA and Lateral\\n\\n\n"
     ]
    }
   ],
   "source": [
    "# Invert the mask to get rows that do not contain the keywords\n",
    "inverse_mask = ~mask\n",
    "\n",
    "# Filter the DataFrame using the inverted mask\n",
    "rows_without_keywords = dataFrame[inverse_mask]\n",
    "\n",
    "print(rows_without_keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like those two lines are not containing important data.   \n",
    "\n",
    "Lets see if is ther lines with only one of our keywords\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a mask is created for word  impression\n",
    "mask_keyword1 = dataFrame['Text'].str.contains(keywords[0], case=False, na=False)\n",
    "\n",
    "# a mask is created for word findings\n",
    "mask_keyword2 = dataFrame['Text'].str.contains(keywords[1], case=False, na=False)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text    319\n",
      "dtype: int64\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3452/1681464399.py:7: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  print(key1df[nokey2].count())\n",
      "/tmp/ipykernel_3452/1681464399.py:9: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  key1df[nokey2].head()\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nExam\\nXray Chest PA and Lateral\\n\\nDate\\nXXXX\\n\\nHistory\\nrib pain after a XXXX, XXXX XXXX steps this XXXX. Pain to R back, R elbow and R rib XXXX, no previous heart or lung hx, non-XXXX, no hx ca\\n\\nImpression\\nNo displaced rib fractures, pneumothorax, or pleural effusion identified. Well-expanded and clear lungs. Mediastinal contour within normal limits. No acute cardiopulmonary abnormality identified.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>\\nEXAM\\nPA and LAT view CHEST XXXX, XXXX XXXX PM\\n\\nIndication\\nChronic XXXX XXXX\\n\\nComparisons\\nXXXX\\n\\nDiscussion\\nLungs are overall hyperexpanded with flattening of the diaphragms. No focal consolidation. No pleural effusions or pneumothoraces. Heart and mediastinum of normal size and contour. Degenerative changes in the thoracic spine.\\n\\nImpression\\nHyperexpanded but clear lungs.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>EXAM(S): Chest, 2 views, frontal and lateral\\n\\nDATE\\nXXXX\\n\\nCOMPARISON\\nXXXX, XXXX\\n\\nINDICATION\\nXXXX, hypoxia.\\n\\nIMPRESSION\\nBorderline heart size. Elevated left diaphragm. Clear right lung. Tracheostomy tube tip above the carina. Extensive airspace disease in the left base. No large effusion or pneumothorax. \\n XXXX XXXX for the opportunity to care for your patient. If XXXX have any questions regarding this report, please XXXX the radiologist, Dr. XXXX XXXX, at XXXX.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>\\nRADIOLOGY REPORT\\n\\nExam\\nChest x-XXXX XXXX and lateral, XXXX Indication: XXXX-year-old male with chest pain. Comparison: None Discussion: Lungs are clear without focal consolidation, effusion, or pneumothorax. Normal heart size. Negative for pneumoperitoneum. Bony thorax and soft tissue grossly unremarkable Transcribed by - PSC Transcription Date - XXXX\\n\\nIMPRESSION\\nNegative acute cardiopulmonary abnormality.\\n\\nSIGNATURE\\nXXXX\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>\\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\nExam: Xray Chest PA and Lateral \\n \\nDate: XXXX, XXXX XXXX PM \\n \\nHistory: XXXX DYSPNEA \\n \\nImpression:\\n \\nComparison XXXX, XXXX.\\n \\nSuggestion of slightly more prominent interstitial markings, which\\nmay represent some bronchitic/bronchiolitis changes. No suspicious\\nnodules, pneumonia, effusions, or CHF. Stable mediastinal contour.\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Text\n",
       "2                                                                       \\nExam\\nXray Chest PA and Lateral\\n\\nDate\\nXXXX\\n\\nHistory\\nrib pain after a XXXX, XXXX XXXX steps this XXXX. Pain to R back, R elbow and R rib XXXX, no previous heart or lung hx, non-XXXX, no hx ca\\n\\nImpression\\nNo displaced rib fractures, pneumothorax, or pleural effusion identified. Well-expanded and clear lungs. Mediastinal contour within normal limits. No acute cardiopulmonary abnormality identified.\\n\n",
       "26                                                                                           \\nEXAM\\nPA and LAT view CHEST XXXX, XXXX XXXX PM\\n\\nIndication\\nChronic XXXX XXXX\\n\\nComparisons\\nXXXX\\n\\nDiscussion\\nLungs are overall hyperexpanded with flattening of the diaphragms. No focal consolidation. No pleural effusions or pneumothoraces. Heart and mediastinum of normal size and contour. Degenerative changes in the thoracic spine.\\n\\nImpression\\nHyperexpanded but clear lungs.\\n\n",
       "28  EXAM(S): Chest, 2 views, frontal and lateral\\n\\nDATE\\nXXXX\\n\\nCOMPARISON\\nXXXX, XXXX\\n\\nINDICATION\\nXXXX, hypoxia.\\n\\nIMPRESSION\\nBorderline heart size. Elevated left diaphragm. Clear right lung. Tracheostomy tube tip above the carina. Extensive airspace disease in the left base. No large effusion or pneumothorax. \\n XXXX XXXX for the opportunity to care for your patient. If XXXX have any questions regarding this report, please XXXX the radiologist, Dr. XXXX XXXX, at XXXX.\\n\n",
       "29                                         \\nRADIOLOGY REPORT\\n\\nExam\\nChest x-XXXX XXXX and lateral, XXXX Indication: XXXX-year-old male with chest pain. Comparison: None Discussion: Lungs are clear without focal consolidation, effusion, or pneumothorax. Normal heart size. Negative for pneumoperitoneum. Bony thorax and soft tissue grossly unremarkable Transcribed by - PSC Transcription Date - XXXX\\n\\nIMPRESSION\\nNegative acute cardiopulmonary abnormality.\\n\\nSIGNATURE\\nXXXX\\n\\n\n",
       "30                                                                           \\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\nExam: Xray Chest PA and Lateral \\n \\nDate: XXXX, XXXX XXXX PM \\n \\nHistory: XXXX DYSPNEA \\n \\nImpression:\\n \\nComparison XXXX, XXXX.\\n \\nSuggestion of slightly more prominent interstitial markings, which\\nmay represent some bronchitic/bronchiolitis changes. No suspicious\\nnodules, pneumonia, effusions, or CHF. Stable mediastinal contour.\\n\\n"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#apply mask that looks for keyword1 impression\n",
    "key1df=dataFrame[mask_keyword1]\n",
    "\n",
    "# reverse mask that looks for keyword2 findings\n",
    "nokey2=~mask_keyword2\n",
    "\n",
    "print(key1df[nokey2].count())\n",
    "# when you apply no findigs on impression df you get one results\n",
    "key1df[nokey2].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as can bee seen discussion is sometimes used instead of findings,\n",
    "lets se how mnay lines that doesnt contains findings, contains discussion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text    319\n",
      "dtype: int64\n",
      "Text    46\n",
      "dtype: int64\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3452/2907338884.py:2: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  print(key1df[nokey2].count())\n",
      "/tmp/ipykernel_3452/2907338884.py:3: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  only_impressions=key1df[nokey2]\n",
      "/tmp/ipykernel_3452/2907338884.py:4: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  print(only_impressions[mask_discus].count())\n"
     ]
    }
   ],
   "source": [
    "mask_discus = dataFrame['Text'].str.contains(\"Discussion\", case=False, na=False)\n",
    "print(key1df[nokey2].count())\n",
    "only_impressions=key1df[nokey2]\n",
    "print(only_impressions[mask_discus].count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets now filter for the lines that contains findings but not impression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text    1\n",
      "dtype: int64\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3452/1212262148.py:7: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  print(key2df[nokey1].count())\n",
      "/tmp/ipykernel_3452/1212262148.py:9: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  key2df[nokey1].head(10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>809</th>\n",
       "      <td>\\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\nExamination: PA lateral views of the chest dated XXXX.\\n \\nComparison: None\\n \\nHistory: XXXX-year-old female, tobacco use, preop.\\n \\nFindings: There are no focal areas of consolidation. No pleural\\neffusions. No pneumothorax. Heart size within normal limits.\\nCalcified granulomas. Degenerative changes thoracic spine.\\n \\nFindings:\\nNo acute cardiopulmonary abnormality. \\nThis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Text\n",
       "809  \\nChest PA-Lat XR\\n\\nImaging Study\\nXray Chest PA and Lateral\\nExamination: PA lateral views of the chest dated XXXX.\\n \\nComparison: None\\n \\nHistory: XXXX-year-old female, tobacco use, preop.\\n \\nFindings: There are no focal areas of consolidation. No pleural\\neffusions. No pneumothorax. Heart size within normal limits.\\nCalcified granulomas. Degenerative changes thoracic spine.\\n \\nFindings:\\nNo acute cardiopulmonary abnormality. \\nThis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#apply mask that looks for keyword2 findings\n",
    "key2df=dataFrame[mask_keyword2]\n",
    "\n",
    "# reverse mask that looks for keyword1 impression\n",
    "nokey1=~mask_keyword1\n",
    "\n",
    "print(key2df[nokey1].count())\n",
    "# when you apply no findigs on impression df you get one results\n",
    "key2df[nokey1].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so in here we have no impressions but the first finding should be named as impressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "key1df is lines with impression  \n",
    "key2df is lines with findings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text    1981\n",
      "dtype: int64\n",
      "Text    1663\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(key1df.count())\n",
    "print(key2df.count())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3452/2725145516.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  print(key1df[mask_keyword2].count())\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text    1662\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(key1df[mask_keyword2].count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we have 1662 lines that include both keywords which is good for us"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2 : Clearing Data\n",
    "-----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part we will start to work on invasive moves to start changin data.  \n",
    "\n",
    "Dont forget that all our moves will be made on dataFrame variable and not on .csv file. \n",
    "\n",
    "So if we want to keep results we need to save them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets keep letters comma full stop and empty space, regex will help us here   \n",
    "\n",
    "a-z is for lower case\n",
    "A-Z is for upper case  \n",
    ". is for dot  \n",
    ", is comma  \n",
    "\\s is for full stop\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a regular expression pattern to match desired characters\n",
    "pattern = r'[^a-zA-Z.,\\s]'\n",
    " # Matches anything that is not a letter, full stop, comma, or whitespace\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we mostly get rid of semi colons, we want \\ beacuse it helps us detect new lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Apply the pattern to each element in the DataFrame and replace non-matching characters with an empty string\n",
    "df = dataFrame.replace(to_replace=pattern, value='', regex=True)\n",
    "\n",
    "# Save the cleaned DataFrame back to a CSV file\n",
    "df.to_csv('cleaned_file.csv', index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and now we will make all lower case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3452/2281763407.py:5: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",
      "  df = df.applymap(lambda x: x.lower() if isinstance(x, str) else x)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest xxxxxxxx.\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no xxxx of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest xxxxx. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>exams chest,  views, frontal and lateral\\n\\ndate\\nxxxx\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy xxxx. enlarged pulmonary arteries. clear lungs. inferior xxxx xxxx xxxx.\\n\\nimpression\\nno acute pulmonary findings. \\n xxxx xxxx for the opportunity to care for your patient. if xxxx have any questions regarding this report, please xxxx the radiologist, dr. xxxx xxxx, at xxxx.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nexam\\nxray chest pa and lateral\\n\\ndate\\nxxxx\\n\\nhistory\\nrib pain after a xxxx, xxxx xxxx steps this xxxx. pain to r back, r elbow and r rib xxxx, no previous heart or lung hx, nonxxxx, no hx ca\\n\\nimpression\\nno displaced rib fractures, pneumothorax, or pleural effusion identified. wellexpanded and clear lungs. mediastinal contour within normal limits. no acute cardiopulmonary abnormality identified.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with xxxx. comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe, xxxx scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral xxxx, xxxx xxxx pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparisxxxxxxxx.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           Text\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest xxxxxxxx.\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no xxxx of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest xxxxx. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n\n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         exams chest,  views, frontal and lateral\\n\\ndate\\nxxxx\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy xxxx. enlarged pulmonary arteries. clear lungs. inferior xxxx xxxx xxxx.\\n\\nimpression\\nno acute pulmonary findings. \\n xxxx xxxx for the opportunity to care for your patient. if xxxx have any questions regarding this report, please xxxx the radiologist, dr. xxxx xxxx, at xxxx.\\n\n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \\nexam\\nxray chest pa and lateral\\n\\ndate\\nxxxx\\n\\nhistory\\nrib pain after a xxxx, xxxx xxxx steps this xxxx. pain to r back, r elbow and r rib xxxx, no previous heart or lung hx, nonxxxx, no hx ca\\n\\nimpression\\nno displaced rib fractures, pneumothorax, or pleural effusion identified. wellexpanded and clear lungs. mediastinal contour within normal limits. no acute cardiopulmonary abnormality identified.\\n\n",
       "3  \\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with xxxx. comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe, xxxx scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\nxxxx\\n\\n\n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                             \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral xxxx, xxxx xxxx pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparisxxxxxxxx.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# Lets read our cleaned file\n",
    "df = pd.read_csv('cleaned_file.csv')\n",
    "\n",
    "# Convert all string columns to lowercase\n",
    "df = df.applymap(lambda x: x.lower() if isinstance(x, str) else x)\n",
    "\n",
    "\n",
    "# Save the cleaned DataFrame back to a CSV file\n",
    "df.to_csv('cleaned_file.csv', index=False)\n",
    "\n",
    "df.head() # to see\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "since this data is anonymized with usage of \"xxxx\" it is nice to remove them. If you train an aı with such tokens it might create un wanted effects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest .\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no  of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest x. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>exams chest,  views, frontal and lateral\\n\\ndate\\n\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy . enlarged pulmonary arteries. clear lungs. inferior   .\\n\\nimpression\\nno acute pulmonary findings. \\n   for the opportunity to care for your patient. if  have any questions regarding this report, please  the radiologist, dr.  , at .\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nexam\\nxray chest pa and lateral\\n\\ndate\\n\\n\\nhistory\\nrib pain after a ,   steps this . pain to r back, r elbow and r rib , no previous heart or lung hx, non, no hx ca\\n\\nimpression\\nno displaced rib fractures, pneumothorax, or pleural effusion identified. wellexpanded and clear lungs. mediastinal contour within normal limits. no acute cardiopulmonary abnormality identified.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral views of the chest ,  at  hours history yearold  with . comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe,  scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  \\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\n\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral ,   pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparis.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       Text\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest .\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no  of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest x. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n\n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     exams chest,  views, frontal and lateral\\n\\ndate\\n\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy . enlarged pulmonary arteries. clear lungs. inferior   .\\n\\nimpression\\nno acute pulmonary findings. \\n   for the opportunity to care for your patient. if  have any questions regarding this report, please  the radiologist, dr.  , at .\\n\n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             \\nexam\\nxray chest pa and lateral\\n\\ndate\\n\\n\\nhistory\\nrib pain after a ,   steps this . pain to r back, r elbow and r rib , no previous heart or lung hx, non, no hx ca\\n\\nimpression\\nno displaced rib fractures, pneumothorax, or pleural effusion identified. wellexpanded and clear lungs. mediastinal contour within normal limits. no acute cardiopulmonary abnormality identified.\\n\n",
       "3  \\nradiology report\\n\\nexamination\\npa and lateral views of the chest ,  at  hours history yearold  with . comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe,  scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  \\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\n\\n\\n\n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                             \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral ,   pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparis.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('cleaned_file.csv')\n",
    "\n",
    "df = df.replace(to_replace=\"xxxx\", value='', regex=True)\n",
    "\n",
    "\n",
    "\n",
    "df.to_csv('cleaned_file.csv', index=False)\n",
    "\n",
    "df.head() # to see"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Choosing where to split\n",
    "_____________________________"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All our data is one column but we need one input as text and one output as summary.  \n",
    "\n",
    "So lets look at how we can split it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest xxxxxxxx.\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no xxxx of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest xxxxx. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>exams chest,  views, frontal and lateral\\n\\ndate\\nxxxx\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy xxxx. enlarged pulmonary arteries. clear lungs. inferior xxxx xxxx xxxx.\\n\\nimpression\\nno acute pulmonary findings. \\n xxxx xxxx for the opportunity to care for your patient. if xxxx have any questions regarding this report, please xxxx the radiologist, dr. xxxx xxxx, at xxxx.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nexam\\nxray chest pa and lateral\\n\\ndate\\nxxxx\\n\\nhistory\\nrib pain after a xxxx, xxxx xxxx steps this xxxx. pain to r back, r elbow and r rib xxxx, no previous heart or lung hx, nonxxxx, no hx ca\\n\\nimpression\\nno displaced rib fractures, pneumothorax, or pleural effusion identified. wellexpanded and clear lungs. mediastinal contour within normal limits. no acute cardiopulmonary abnormality identified.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with xxxx. comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe, xxxx scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral xxxx, xxxx xxxx pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparisxxxxxxxx.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           Text\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest xxxxxxxx.\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no xxxx of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest xxxxx. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n\n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         exams chest,  views, frontal and lateral\\n\\ndate\\nxxxx\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy xxxx. enlarged pulmonary arteries. clear lungs. inferior xxxx xxxx xxxx.\\n\\nimpression\\nno acute pulmonary findings. \\n xxxx xxxx for the opportunity to care for your patient. if xxxx have any questions regarding this report, please xxxx the radiologist, dr. xxxx xxxx, at xxxx.\\n\n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \\nexam\\nxray chest pa and lateral\\n\\ndate\\nxxxx\\n\\nhistory\\nrib pain after a xxxx, xxxx xxxx steps this xxxx. pain to r back, r elbow and r rib xxxx, no previous heart or lung hx, nonxxxx, no hx ca\\n\\nimpression\\nno displaced rib fractures, pneumothorax, or pleural effusion identified. wellexpanded and clear lungs. mediastinal contour within normal limits. no acute cardiopulmonary abnormality identified.\\n\n",
       "3  \\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with xxxx. comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe, xxxx scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\nxxxx\\n\\n\n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                             \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral xxxx, xxxx xxxx pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparisxxxxxxxx.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "structure of each report is as follows \n",
    "\n",
    "header   \n",
    "\n",
    "inner text  \n",
    "\n",
    "\n",
    "header \n",
    "\n",
    "\n",
    "inner text\n",
    "\n",
    "\n",
    "which is described as \\nheader\\ninnertext\\n\n",
    "\n",
    "\n",
    "so we can use \\n as a splitter "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look for impression\\n and findings\\n to see when those words used as headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask_inw = df['Text'].str.contains(\"impression\\\\n\", case=False, na=False)\n",
    "mask_fnw = df['Text'].str.contains(\"findings\\\\n\", case=False, na=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text    443\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[mask_fnw].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text    1549\n",
       "dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[mask_inw].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest xxxxxxxx.\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no xxxx of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest xxxxx. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>exams chest,  views, frontal and lateral\\n\\ndate\\nxxxx\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy xxxx. enlarged pulmonary arteries. clear lungs. inferior xxxx xxxx xxxx.\\n\\nimpression\\nno acute pulmonary findings. \\n xxxx xxxx for the opportunity to care for your patient. if xxxx have any questions regarding this report, please xxxx the radiologist, dr. xxxx xxxx, at xxxx.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral xxxx, xxxx xxxx pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparisxxxxxxxx.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\npa and lateral chest xxxx\\n \\nindication xxxxyearold female, chest pain\\n \\ncomparisxxxxxxxx\\n \\nimpression no acute findings\\n \\nfindings heart size within normal limits, stable mediastinal and\\nhilar contours. mild hyperinflation appears similar to prior. no\\nfocal alveolar consolidation, no definite pleural effusion seen.\\nscattered chronic appearing irregular interstitial markings, no\\ntypical findings of pulmonary edema.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\npa and lateral chest xxxx\\n \\nindication xxxxyearold male, pain\\n \\ncomparison none\\n \\nimpression no acute cardiopulmonary findings\\n \\nfindings heart size within normal limits. no focal alveolar\\nconsolidation, no definite pleural effusion seen. no typical\\nfindings of pulmonary edema. no pneumothorax.\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 Text\n",
       "0                                                                                    \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest xxxxxxxx.\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no xxxx of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest xxxxx. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n\n",
       "1                                                                                                                                               exams chest,  views, frontal and lateral\\n\\ndate\\nxxxx\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy xxxx. enlarged pulmonary arteries. clear lungs. inferior xxxx xxxx xxxx.\\n\\nimpression\\nno acute pulmonary findings. \\n xxxx xxxx for the opportunity to care for your patient. if xxxx have any questions regarding this report, please xxxx the radiologist, dr. xxxx xxxx, at xxxx.\\n\n",
       "4   \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral xxxx, xxxx xxxx pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparisxxxxxxxx.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n\n",
       "13                                                                                                    \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\npa and lateral chest xxxx\\n \\nindication xxxxyearold female, chest pain\\n \\ncomparisxxxxxxxx\\n \\nimpression no acute findings\\n \\nfindings heart size within normal limits, stable mediastinal and\\nhilar contours. mild hyperinflation appears similar to prior. no\\nfocal alveolar consolidation, no definite pleural effusion seen.\\nscattered chronic appearing irregular interstitial markings, no\\ntypical findings of pulmonary edema.\\n\\n\n",
       "17                                                                                                                                                                                                                                \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\npa and lateral chest xxxx\\n \\nindication xxxxyearold male, pain\\n \\ncomparison none\\n \\nimpression no acute cardiopulmonary findings\\n \\nfindings heart size within normal limits. no focal alveolar\\nconsolidation, no definite pleural effusion seen. no typical\\nfindings of pulmonary edema. no pneumothorax.\\n\\n"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[mask_fnw].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turns out most of \"findings\" words is used inside text and it is header in very few lines.\n",
    "\n",
    "When looked as \"findings\\n\" rows 2 and 3 doesnt shows up "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest xxxxxxxx.\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no xxxx of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest xxxxx. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>exams chest,  views, frontal and lateral\\n\\ndate\\nxxxx\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy xxxx. enlarged pulmonary arteries. clear lungs. inferior xxxx xxxx xxxx.\\n\\nimpression\\nno acute pulmonary findings. \\n xxxx xxxx for the opportunity to care for your patient. if xxxx have any questions regarding this report, please xxxx the radiologist, dr. xxxx xxxx, at xxxx.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nexam\\nxray chest pa and lateral\\n\\ndate\\nxxxx\\n\\nhistory\\nrib pain after a xxxx, xxxx xxxx steps this xxxx. pain to r back, r elbow and r rib xxxx, no previous heart or lung hx, nonxxxx, no hx ca\\n\\nimpression\\nno displaced rib fractures, pneumothorax, or pleural effusion identified. wellexpanded and clear lungs. mediastinal contour within normal limits. no acute cardiopulmonary abnormality identified.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with xxxx. comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe, xxxx scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral xxxx, xxxx xxxx pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparisxxxxxxxx.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           Text\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexam  views of the chest xxxxxxxx.\\n \\ncomparison none.\\n \\nindication positive tb test\\n \\nfindings\\nthe cardiac silhouette and mediastinum size are within normal limits.\\nthere is no pulmonary edema. there is no focal consolidation. there\\nare no xxxx of a pleural effusion. there is no evidence of\\npneumothorax.\\n \\nimpression\\nnormal chest xxxxx. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n\n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         exams chest,  views, frontal and lateral\\n\\ndate\\nxxxx\\n\\ncomparison\\nnone.\\n\\nindication\\npreop bariatric surgery.\\n\\nfindings\\nborderline cardiomegaly. midline sternotomy xxxx. enlarged pulmonary arteries. clear lungs. inferior xxxx xxxx xxxx.\\n\\nimpression\\nno acute pulmonary findings. \\n xxxx xxxx for the opportunity to care for your patient. if xxxx have any questions regarding this report, please xxxx the radiologist, dr. xxxx xxxx, at xxxx.\\n\n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \\nexam\\nxray chest pa and lateral\\n\\ndate\\nxxxx\\n\\nhistory\\nrib pain after a xxxx, xxxx xxxx steps this xxxx. pain to r back, r elbow and r rib xxxx, no previous heart or lung hx, nonxxxx, no hx ca\\n\\nimpression\\nno displaced rib fractures, pneumothorax, or pleural effusion identified. wellexpanded and clear lungs. mediastinal contour within normal limits. no acute cardiopulmonary abnormality identified.\\n\n",
       "3  \\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with xxxx. comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe, xxxx scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\nxxxx\\n\\n\n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                             \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination chest  frontal and lateral xxxx, xxxx xxxx pm \\n \\nclinical indication chest and nasal congestion.\\n \\ncomparisxxxxxxxx.\\n \\nfindings\\nthe cardiomediastinal silhouette and pulmonary vasculature are within\\nnormal limits. there is no pneumothorax or pleural effusion. there\\nare no focal areas of consolidation. cholecystectomy clips are\\npresent. small tspine osteophytes. there is biapical pleural\\nthickening, unchanged from prior. mildly hyperexpanded lungs.\\n \\nimpression\\nno acute cardiopulmonary abnormality.\\n\\n"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text    285\n",
       "dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask_ex=df[\"Text\"].str.contains(\"examination\\\\n\", case=False, na=False)\n",
    "df[mask_ex].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "turns out one other important header is examination. So notice that we first thought that there are two headers at first examination yet when we use special characters like newline(\\n) to ensure our understanding of formatting is correct we saw that in 1463 lines of data where \"findings\" is found, only 443 of them are headers. And we have another header called examination. and it is used as header in 285 line which is considerable amount of data. so lets see if examination part works as we expect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with xxxx. comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe, xxxx scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral views of the chest dated xxxx. comparison xxxx films of the chest dated xxxx. history xxxxyearold female, chest pain. findings no focal areas of consolidation. no suspicious pulmonary opacities. heart size within normal limits. no pleural effusions. no evidence of pneumothorax. osseous structures intact. transcribed by  psc transcription date  xxxx\\n\\nimpression\\nno acute cardiopulmonary abnormality.\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold woman with xxxx for  weeks. comparison none available findings the lungs are clear, and without focal air space opacity. the cardiomediastinal silhouette is normal in size and contour, and stable. there is no pneumothorax large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\nno acute cardiopulmonary abnormality.\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral chest radiographs xxxx at xxxx hours. history xxxxyearold female with breast mass and smoking history. comparison pa and lateral chest redressed xxxx findings the heart size and cardiomediastinal silhouette are normal. there is hyperexpansion of the lungs with flattening of the hemidiaphragms. there is no focal airspace opacity, pleural effusion, or pneumothorax. there multilevel degenerative changes of thoracic spine. transcribed by  psc transcription date  xxxx\\n\\nimpression\\nemphysema, however no acute cardiopulmonary finding.\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral chest xxxxx dated xxxx, xxxx at xxxx p.m.. indication xxxxyearold female with chest pain, rule out pneumonia.. comparison twoview chest radiograph dated xxxx, xxxx.. findings the lungs are clear bilaterally. specifically, no evidence of focal consolidation, pneumothorax, or pleural effusion.. cardio mediastinal silhouette is unremarkable. visualized osseous structures of the thorax are without acute abnormality. transcribed by  psc transcription date  xxxx\\n\\nimpression\\nno acute cardiopulmonary abnormality..\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>\\nexamination\\npa and lateral chest radiographs dated xxxx at xxxx hours.\\n\\ncomparison\\nnone.\\n\\nhistory\\nxxxxyearold with osteoarthritis of the hip scheduled for total hip replacement. preoperative evaluation.\\n\\nfindings\\nthe heart, pulmonary xxxx and mediastinum are within normal limits. there is no pleural effusion or pneumothorax. there is no focal air space opacity to suggest a pneumonia. there are degenerative changes of the thoracic spine. there is a calcified granuloma identified in the right suprahilar region. the aorta is mildly tortuous and ectatic. there is asymmetric right apical smooth pleural thickening. there are severe degenerative changes of the xxxx.\\n\\nimpression\\nno acute cardiopulmonary disease.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral chest xxxxx dated xxxx, xxxx at xxxx p.m.. indication xxxxyearold woman, prior to enbrel therapy.. comparison none. findings the lungs are clear bilaterally. specifically, no evidence of focal consolidation, pneumothorax, or pleural effusion.. minimal right basilar subsegmental atelectasis noted. cardio mediastinal silhouette is unremarkable. tortuosity of the thoracic aorta noted. scattered calcified granulomas are seen without evidence of active granulomatoustuberculous process. visualized osseous structures of the thorax are without acute abnormality. transcribed by  pscb transcription date  xxxx\\n\\nimpression\\nno acute cardiopulmonary abnormality.. xxxx xxxx, m.d. xxxx xxxx xxxx electronically xxxx xxxx, m.d. xxxx xxxx xxxx transcribed xxxx  xxxx xxxx  radres xxxx\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>\\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination\\nfrontal and lateral views of the chest dated xxxx \\n \\ncomparisxxxxxxxx\\n \\nhistory chest pain\\n \\nfindings status post xxxx sternotomy and cabg. heart size is\\nnormal. coronary vascular stent. the lungs are clear. there are no\\nfocal air space consolidations. no pleural effusions or\\npneumothoraces. the hilar and mediastinal contours are stable.\\ncalcified mediastinal lymph xxxx. normal pulmonary vascularity. \\ndegenerative changes of the spine.\\n \\nimpression no acute abnormality. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>\\nexamination\\nchest  views dated xxxx, xxxx.\\n\\nhistory and indication\\nchest pain.\\n\\ncomparison\\nxxxx.\\n\\nfindings\\nthe xxxx examination consists of frontal and lateral radiographs of the chest. the cardiomediastinal contours are within normal limits. pulmonary vascularity is within normal limits. no focal consolidation, pleural effusion, or pneumothorax identified. deformity of the right clavicle related to remote xxxx is again seen. visualized upper abdomen grossly unremarkable.\\n\\nimpression\\nno evidence of acute cardiopulmonary process.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>\\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with chest pain. comparison xxxx, xxxx findings the heart size is stable. the aorta is ectatic and atherosclerotic but stable. xxxx sternotomy xxxx are again noted. the scarring in the left lower lobe is again noted and unchanged from prior exam. there are mild bilateral prominent lung interstitial opacities consistent with emphysematous disease. the calcified granulomas are stable. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. changes of emphysema and left lower lobe scarring, both stable. . unchanged degenerative and atherosclerotic changes of the thoracic aorta.\\n\\nsignature\\nxxxx\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Text\n",
       "3   \\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with xxxx. comparison none available findings there are diffuse bilateral interstitial and alveolar opacities consistent with chronic obstructive lung disease and bullous emphysema. there are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.there are streaky opacities in the right upper lobe, xxxx scarring. the cardiomediastinal silhouette is normal in size and contour. there is no pneumothorax or large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. bullous emphysema and interstitial fibrosis. . probably scarring in the left apex, although difficult to exclude a cavitary lesion. . opacities in the bilateral upper lobes could represent scarring, however the absence of comparison exam, recommend short interval followup radiograph or ct thorax to document resolution.\\n\\nsignature\\nxxxx\\n\\n\n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \\nradiology report\\n\\nexamination\\npa and lateral views of the chest dated xxxx. comparison xxxx films of the chest dated xxxx. history xxxxyearold female, chest pain. findings no focal areas of consolidation. no suspicious pulmonary opacities. heart size within normal limits. no pleural effusions. no evidence of pneumothorax. osseous structures intact. transcribed by  psc transcription date  xxxx\\n\\nimpression\\nno acute cardiopulmonary abnormality.\\n\\nsignature\\nxxxx\\n\\n\n",
       "21                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              \\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold woman with xxxx for  weeks. comparison none available findings the lungs are clear, and without focal air space opacity. the cardiomediastinal silhouette is normal in size and contour, and stable. there is no pneumothorax large pleural effusion. transcribed by  psc transcription date  xxxx\\n\\nimpression\\nno acute cardiopulmonary abnormality.\\n\\nsignature\\nxxxx\\n\\n\n",
       "34                                                                                                                                                                                                                                                                                                                                                                                                              \\nradiology report\\n\\nexamination\\npa and lateral chest radiographs xxxx at xxxx hours. history xxxxyearold female with breast mass and smoking history. comparison pa and lateral chest redressed xxxx findings the heart size and cardiomediastinal silhouette are normal. there is hyperexpansion of the lungs with flattening of the hemidiaphragms. there is no focal airspace opacity, pleural effusion, or pneumothorax. there multilevel degenerative changes of thoracic spine. transcribed by  psc transcription date  xxxx\\n\\nimpression\\nemphysema, however no acute cardiopulmonary finding.\\n\\nsignature\\nxxxx\\n\\n\n",
       "35                                                                                                                                                                                                                                                                                                                                                                                                                                   \\nradiology report\\n\\nexamination\\npa and lateral chest xxxxx dated xxxx, xxxx at xxxx p.m.. indication xxxxyearold female with chest pain, rule out pneumonia.. comparison twoview chest radiograph dated xxxx, xxxx.. findings the lungs are clear bilaterally. specifically, no evidence of focal consolidation, pneumothorax, or pleural effusion.. cardio mediastinal silhouette is unremarkable. visualized osseous structures of the thorax are without acute abnormality. transcribed by  psc transcription date  xxxx\\n\\nimpression\\nno acute cardiopulmonary abnormality..\\n\\nsignature\\nxxxx\\n\\n\n",
       "48                                                                                                                                                                                                                                                                                    \\nexamination\\npa and lateral chest radiographs dated xxxx at xxxx hours.\\n\\ncomparison\\nnone.\\n\\nhistory\\nxxxxyearold with osteoarthritis of the hip scheduled for total hip replacement. preoperative evaluation.\\n\\nfindings\\nthe heart, pulmonary xxxx and mediastinum are within normal limits. there is no pleural effusion or pneumothorax. there is no focal air space opacity to suggest a pneumonia. there are degenerative changes of the thoracic spine. there is a calcified granuloma identified in the right suprahilar region. the aorta is mildly tortuous and ectatic. there is asymmetric right apical smooth pleural thickening. there are severe degenerative changes of the xxxx.\\n\\nimpression\\nno acute cardiopulmonary disease.\\n\n",
       "53                                                                                                                                                           \\nradiology report\\n\\nexamination\\npa and lateral chest xxxxx dated xxxx, xxxx at xxxx p.m.. indication xxxxyearold woman, prior to enbrel therapy.. comparison none. findings the lungs are clear bilaterally. specifically, no evidence of focal consolidation, pneumothorax, or pleural effusion.. minimal right basilar subsegmental atelectasis noted. cardio mediastinal silhouette is unremarkable. tortuosity of the thoracic aorta noted. scattered calcified granulomas are seen without evidence of active granulomatoustuberculous process. visualized osseous structures of the thorax are without acute abnormality. transcribed by  pscb transcription date  xxxx\\n\\nimpression\\nno acute cardiopulmonary abnormality.. xxxx xxxx, m.d. xxxx xxxx xxxx electronically xxxx xxxx, m.d. xxxx xxxx xxxx transcribed xxxx  xxxx xxxx  radres xxxx\\n\\nsignature\\nxxxx\\n\\n\n",
       "61                                                                                                                                                                                                                                                                                                                                                         \\nchest palat xr\\n\\nimaging study\\nxray chest pa and lateral\\nexamination\\nfrontal and lateral views of the chest dated xxxx \\n \\ncomparisxxxxxxxx\\n \\nhistory chest pain\\n \\nfindings status post xxxx sternotomy and cabg. heart size is\\nnormal. coronary vascular stent. the lungs are clear. there are no\\nfocal air space consolidations. no pleural effusions or\\npneumothoraces. the hilar and mediastinal contours are stable.\\ncalcified mediastinal lymph xxxx. normal pulmonary vascularity. \\ndegenerative changes of the spine.\\n \\nimpression no acute abnormality. \\nthis examination and reported findings have been reviewed and\\nconfirmed by the undersigned.\\n\\n\n",
       "64                                                                                                                                                                                                                                                                                                                                                                                                                                                                       \\nexamination\\nchest  views dated xxxx, xxxx.\\n\\nhistory and indication\\nchest pain.\\n\\ncomparison\\nxxxx.\\n\\nfindings\\nthe xxxx examination consists of frontal and lateral radiographs of the chest. the cardiomediastinal contours are within normal limits. pulmonary vascularity is within normal limits. no focal consolidation, pleural effusion, or pneumothorax identified. deformity of the right clavicle related to remote xxxx is again seen. visualized upper abdomen grossly unremarkable.\\n\\nimpression\\nno evidence of acute cardiopulmonary process.\\n\n",
       "74                                                                                                                                                                                                                                                                                     \\nradiology report\\n\\nexamination\\npa and lateral views of the chest xxxx, xxxx at xxxx hours history xxxxyearold xxxx with chest pain. comparison xxxx, xxxx findings the heart size is stable. the aorta is ectatic and atherosclerotic but stable. xxxx sternotomy xxxx are again noted. the scarring in the left lower lobe is again noted and unchanged from prior exam. there are mild bilateral prominent lung interstitial opacities consistent with emphysematous disease. the calcified granulomas are stable. transcribed by  psc transcription date  xxxx\\n\\nimpression\\n. changes of emphysema and left lower lobe scarring, both stable. . unchanged degenerative and atherosclerotic changes of the thoracic aorta.\\n\\nsignature\\nxxxx\\n\\n"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[mask_ex].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3452/3284326336.py:2: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  exdf[mask_fnw].head()\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>\\nexamination\\npa and lateral chest radiographs dated xxxx at xxxx hours.\\n\\ncomparison\\nnone.\\n\\nhistory\\nxxxxyearold with osteoarthritis of the hip scheduled for total hip replacement. preoperative evaluation.\\n\\nfindings\\nthe heart, pulmonary xxxx and mediastinum are within normal limits. there is no pleural effusion or pneumothorax. there is no focal air space opacity to suggest a pneumonia. there are degenerative changes of the thoracic spine. there is a calcified granuloma identified in the right suprahilar region. the aorta is mildly tortuous and ectatic. there is asymmetric right apical smooth pleural thickening. there are severe degenerative changes of the xxxx.\\n\\nimpression\\nno acute cardiopulmonary disease.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>\\nexamination\\nchest  views dated xxxx, xxxx.\\n\\nhistory and indication\\nchest pain.\\n\\ncomparison\\nxxxx.\\n\\nfindings\\nthe xxxx examination consists of frontal and lateral radiographs of the chest. the cardiomediastinal contours are within normal limits. pulmonary vascularity is within normal limits. no focal consolidation, pleural effusion, or pneumothorax identified. deformity of the right clavicle related to remote xxxx is again seen. visualized upper abdomen grossly unremarkable.\\n\\nimpression\\nno evidence of acute cardiopulmonary process.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>\\nexamination\\nxray chest pa and lateral\\n\\nexamination date\\nxxxx\\n\\ncomparison\\nchest xxxxx xxxx\\n\\nrelevant clinical information\\npain in thoracic spine pain started in leg area two weeks ago now having severe pain in upper xxxx back area rt side. hf\\n\\nfindings\\nno airspace disease, effusion or noncalcified nodule. normal heart size and mediastinum. left axillary surgical clips unchanged visualized xxxx of the chest xxxx are within normal limits.\\n\\nimpression\\nno acute cardiopulmonary abnormality. \\n if xxxx have questions regarding this report, please xxxx xxxx on  xxxx or xxxx xxxx. \\nthis examination and reported findings have been reviewed and confirmed by the undersigned.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>\\nexamination\\nchest  views xxxx, xxxx\\n\\nclinical history\\nchest pain\\n\\ncomparison\\nnone\\n\\nfindings\\nthe lungs are grossly clear without focal pneumonic consolidation, large effusion or pneumothorax. heart size is within normal limits.\\n\\nimpression\\nclear lungs\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>\\nexamination\\nchest radiograph, frontal and lateral views\\n\\ncomparison\\nxxxx\\n\\nfindings\\ncardiomediastinal silhouette is normal. pulmonary vasculature and xxxx are normal. no consolidation, pneumothorax or large pleural effusion. osseous structures and soft tissues are normal.\\n\\nimpression\\nno acute cardiopulmonary disease.\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           Text\n",
       "48   \\nexamination\\npa and lateral chest radiographs dated xxxx at xxxx hours.\\n\\ncomparison\\nnone.\\n\\nhistory\\nxxxxyearold with osteoarthritis of the hip scheduled for total hip replacement. preoperative evaluation.\\n\\nfindings\\nthe heart, pulmonary xxxx and mediastinum are within normal limits. there is no pleural effusion or pneumothorax. there is no focal air space opacity to suggest a pneumonia. there are degenerative changes of the thoracic spine. there is a calcified granuloma identified in the right suprahilar region. the aorta is mildly tortuous and ectatic. there is asymmetric right apical smooth pleural thickening. there are severe degenerative changes of the xxxx.\\n\\nimpression\\nno acute cardiopulmonary disease.\\n\n",
       "64                                                                                                                                                                                      \\nexamination\\nchest  views dated xxxx, xxxx.\\n\\nhistory and indication\\nchest pain.\\n\\ncomparison\\nxxxx.\\n\\nfindings\\nthe xxxx examination consists of frontal and lateral radiographs of the chest. the cardiomediastinal contours are within normal limits. pulmonary vascularity is within normal limits. no focal consolidation, pleural effusion, or pneumothorax identified. deformity of the right clavicle related to remote xxxx is again seen. visualized upper abdomen grossly unremarkable.\\n\\nimpression\\nno evidence of acute cardiopulmonary process.\\n\n",
       "81                                         \\nexamination\\nxray chest pa and lateral\\n\\nexamination date\\nxxxx\\n\\ncomparison\\nchest xxxxx xxxx\\n\\nrelevant clinical information\\npain in thoracic spine pain started in leg area two weeks ago now having severe pain in upper xxxx back area rt side. hf\\n\\nfindings\\nno airspace disease, effusion or noncalcified nodule. normal heart size and mediastinum. left axillary surgical clips unchanged visualized xxxx of the chest xxxx are within normal limits.\\n\\nimpression\\nno acute cardiopulmonary abnormality. \\n if xxxx have questions regarding this report, please xxxx xxxx on  xxxx or xxxx xxxx. \\nthis examination and reported findings have been reviewed and confirmed by the undersigned.\\n\n",
       "133                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \\nexamination\\nchest  views xxxx, xxxx\\n\\nclinical history\\nchest pain\\n\\ncomparison\\nnone\\n\\nfindings\\nthe lungs are grossly clear without focal pneumonic consolidation, large effusion or pneumothorax. heart size is within normal limits.\\n\\nimpression\\nclear lungs\\n\n",
       "150                                                                                                                                                                                                                                                                                                                                                                                                                 \\nexamination\\nchest radiograph, frontal and lateral views\\n\\ncomparison\\nxxxx\\n\\nfindings\\ncardiomediastinal silhouette is normal. pulmonary vasculature and xxxx are normal. no consolidation, pneumothorax or large pleural effusion. osseous structures and soft tissues are normal.\\n\\nimpression\\nno acute cardiopulmonary disease.\\n"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exdf=df[mask_ex]   #  lets create a dat frame with mask that finds examination\\n is applied\n",
    "exdf[mask_fnw].head() # lets look if is there lines with findings\\n on lines with examination\\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turns out examination is not as realiable as we tought. Some rows include examination header with findings header. Not all examination lines include all the data. So this is not a very well dataset. An formatting is not realiable across. Best thing is to use the data that we can rely on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: Actually splitting the data\n",
    "____________"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets create a new csv file only with our keywords to work on them. Note that we are now looking for rows that contains both of our keywords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3452/553152395.py:9: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  filtered_df=filtered_df[mask2]\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv('cleaned_file.csv')\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "mask1=df[\"Text\"].str.contains(\"findings\\\\n\", case=False, na=False)\n",
    "mask2=df[\"Text\"].str.contains(\"impression\\\\n\", case=False, na=False)\n",
    "filtered_df=df[mask1]\n",
    "filtered_df=filtered_df[mask2]\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Save the filtered DataFrame to a new CSV file\n",
    "filtered_df.to_csv('keywordFiltered.csv', index=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text    409\n",
       "dtype: int64"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now we are down to 409 rows, lets see if our headers always occurs in same order or not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('keywordFiltered.csv')\n",
    "\n",
    "# Define your keywords (case insensitive)\n",
    "keyword1 = 'Findings'.lower()\n",
    "keyword2 = 'Impression'.lower()\n",
    "\n",
    "# Initialize a list to store the type values\n",
    "type_values = []\n",
    "\n",
    "# Iterate over the rows\n",
    "for index, row in df.iterrows():\n",
    "    text = row['Text'].lower()  # Convert text to lowercase for case insensitivity\n",
    "    index1 = text.find(keyword1)  # Find the index of keyword1\n",
    "    index2 = text.find(keyword2)  # Find the index of keyword2\n",
    "    if index1 != -1 and index2 != -1:  # Both keywords are found\n",
    "        if index1 < index2:  # Keyword1 occurs first\n",
    "            type_values.append(1)\n",
    "        else:  # Keyword2 occurs first\n",
    "            type_values.append(2)\n",
    "    elif index1 != -1:  # Only keyword1 is found\n",
    "        type_values.append(1)\n",
    "    elif index2 != -1:  # Only keyword2 is found\n",
    "        type_values.append(2)\n",
    "    else:  # Neither keyword is found\n",
    "        type_values.append(0)\n",
    "\n",
    "# Add the type values to the DataFrame\n",
    "df['type'] = type_values\n",
    "\n",
    "# Save the DataFrame back to a CSV file\n",
    "df.to_csv('typed.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23\n"
     ]
    }
   ],
   "source": [
    "count_ones = df['type'].value_counts().get(2, 0)\n",
    "\n",
    "print(count_ones)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Text  \\\n",
      "14                                                                                                                                                                                                                                                                                                                                                                                                                   \\npa and lateral chest\\n\\n\\nindication\\nyearold female, pain, short of breath\\n\\ncomparison\\n\\n\\nimpression\\nchronic changes as described, no acute findings\\n\\nfindings\\nstable flattening of the posterior diaphragm and scattered chronic appearing irregular interstitial markings with no focal alveolar consolidation. stable cardiomediastinal silhouette with normal heart size and aortic ectasiatortuosity, stable mediastinal contours. no definite pleural effusion seen, no typical findings of pulmonary edema. following spine ossifications and marginal osteophytes again noted.\\n   \n",
      "34                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \\nsupine ap and lateral chest\\n\\n\\nindication\\nyearold female,  not otherwise specified\\n\\ncomparison\\n\\n\\nimpression\\nno acute findings.\\n\\nfindings\\nstable cardiomediastinal silhouette with mild cardiomegaly and aortic ectasia and tortuosity. no alveolar consolidation, no findings of pleural effusion. chronic appearing bilateral rib contour deformities compatible with old fractures. no pneumothorax.\\n   \n",
      "45                                                                                                                                                                                                  \\npa and lateral chest\\n\\n\\nindication\\nyearold female, post  closure\\n\\ncomparison\\nnone\\n\\nimpression\\n.  densities in left base may be compatible with scarring or subsegmental atelectasis . abnormal opacity in the right base  due at  in part to atelectasis with right hemidiaphragm eventration, question small right pleural effusion\\n\\nfindings\\natrial septal occluder artifact. rotated frontal position, overall heart size within normal limits, no typical findings of pulmonary edema.  densities in the left base, small focal  opacity in the right base with focal posterior right hemidiaphragm elevation and obscured right costophrenic . biapical pleuroparenchymal irregularities most compatible with scarring, chronic appearing right th rib contour deformity. no pneumothorax seen.\\n   \n",
      "49                                                                                                                                                                                                                                                                                                                                                                                                                                       \\npa and lateral chest\\n\\n\\nindication\\nyearold female, chest pain\\n\\ncomparison\\nnone\\n\\nimpression\\n. cardiomegaly . indistinct vascular margination which may be secondary to bronchovascular crowding however differential diagnosis includes mild pulmonary edema, atypical infection, inflammation\\n\\nfindings\\nheart size mildly to moderately enlarged. mild right hemidiaphragm elevation with mild bronchovascular crowding, right greater than left, indistinct vascular margination. no definite focal alveolar consolidation, no pleural effusion  demonstrated.\\n   \n",
      "126                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \\npa and lateral chest\\n, \\n\\nindication\\nchest pain\\n\\ncomparison\\n, \\n\\nimpression\\nno acute cardiopulmonary abnormality.\\n\\nfindings\\nno focal consolidation, suspicious pulmonary opacity or definite pleural effusion. heart size and pulmonary vascularity within normal limits. stable mediastinal contour. calcified hilar lymph . visualized osseous structures unremarkable. \\nthis examination and reported findings have been reviewed and confirmed by the undersigned.\\n   \n",
      "136                                                                                                                                                                                                                                                                                                                                                                                \\nap and lateral chest\\n\\n\\nindication\\nyearold male, chest pain\\n\\ncomparison\\n\\n\\nimpression\\ncardiomegaly and increased interstitial opacities which may be compatible with mild pulmonary edema, differential diagnosis includes infection, inflammation, aspiration\\n\\nfindings\\nstable enlargement of the cardiac silhouette, stable mediastinal contours. increased interstitial markings in the central lungs and bases, right greater than left.  opacity on the lateral view over the heart also present on the previous exam suggesting chronic subsegmental atelectasis or scarring. no definite pleural effusion seen.\\n   \n",
      "141                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \\npa and lateral chest\\n\\n\\nindication\\nyearold female, pain\\n\\ncomparison\\nnone\\n\\nimpression\\nno acute cardiopulmonary findings\\n\\nfindings\\nheart size within normal limits. no focal alveolar consolidation, no definite pleural effusion seen. no typical findings of pulmonary edema. no pneumothorax.\\n\\nimaging study\\nxray chest pa and lateral\\n   \n",
      "161  \\nap and lateral chest\\n\\n\\nindication\\nyearold female, pain\\n\\ncomparison\\nnone\\n\\nimpression\\nborderline heart size and abnormal interstitial pulmonary pattern which may be compatible with chronic interstitial change, differential diagnosis is  and includes asymmetric pulmonary edema, inflammation, atypical infection, infiltrative process. comparison with previous exams would be of . bilateral hilar fullness may indicate pulmonary hypertension and clinical correlation is recommended, differential diagnosis reactive lymphadenopathy, metastatic disease.\\n\\nfindings\\nheart size near top normal, bilateral hilar fullness nonspecific in appearance, mild aortic ectasiatortuosity. diaphragm flattening and relative apical lucencies suggestive of emphysema,  and irregular interstitial markings, right greater than left. prominent left epicardial fat , no focal alveolar consolidation, no definite pleural effusion seen. atrial septal occluder artifact. mild spine curvature.\\n   \n",
      "164                                                                                                                                                                                                                                                                                                                                                                                                                                                        addendum begins\\n\\naddendum\\nimpression should  no acute preoperative findings. \\naddendum  \\nexams chest,  views, frontal and lateral\\n\\ndate\\n\\n\\ncomparison\\nnone.\\n\\nindication\\npreop  ankle surgery\\n\\nfindings\\ncardiac and mediastinal contours are within normal limits. prior granulomatous disease. the lungs are otherwise clear. bony structures are intact.\\n\\nimpression\\nacute preoperative findings. \\n   for the opportunity to care for your patient. if  have any questions regarding this report, please  the radiologist, dr.  , at .\\n   \n",
      "192                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \\npa and lateral chest\\n\\n\\nindication\\nyearold female, copd exacerbation, short of breath\\n\\ncomparison\\n\\n\\nimpression\\nbibasilar opacities, right greater than left, features suggest a combination of consolidation and atelectasis\\n\\nfindings\\nstreaky and patchy bibasilar opacities, triangular density projected over the heart on the lateral view. no definite pleural effusion seen, no typical findings of pulmonary edema. considering differences in technical factors  stable cardiomediastinal silhouette with normal heart size.\\n   \n",
      "200                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \\npa and lateral chest\\n\\n\\nindication\\nyearold male, , difficulty breathing\\n\\ncomparison\\n\\n\\nimpression\\nno acute findings\\n\\nfindings\\nheart size within normal limits, stable mediastinal and hilar contours. no alveolar consolidation, no findings of pleural effusion or pulmonary edema. chronic appearing contour deformity of the right posterolateral th rib again noted suggestive of old injury.\\n   \n",
      "226                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \\npa and lateral chest\\n\\n\\nindication\\nyearold female, , \\n\\ncomparison\\nnone\\n\\nimpression\\nno acute findings\\n\\nfindings\\nheart size within normal limits. no alveolar consolidation, no findings of pleural effusion or pulmonary edema. no pneumothorax.\\n   \n",
      "233                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       \\npa and lateral chest\\n\\n\\nindication\\nyearold female, mvc, pain\\n\\ncomparison\\nnone\\n\\nimpression\\nno acute  related findings. please note that nondisplaced fractures may not be demonstrated.\\n\\nfindings\\nno focal consolidation, pneumothorax or definite pleural effusion. heart size and pulmonary vascularity within normal limits, no mediastinal widening characteristic in appearance of vascular injury. no acute osseous injury  demonstrated.\\n   \n",
      "256                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \\npa and lateral chest\\n\\n\\nindication\\nyearold female,  distress\\n\\ncomparison\\nnone\\n\\nimpression\\nmildly hyperinflated lungs, air trapping versus inspiratory .\\n\\nfindings\\nmildly hyperinflated lungs with flattened posterior diaphragm. no alveolar consolidation, no findings of pleural effusion or pulmonary edema. heart size within normal limits. no pneumothorax.\\n   \n",
      "261                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \\npa and lateral chest\\n\\n\\nindication\\n year old female, chest pain\\n\\ncomparison\\nnone\\n\\nimpression\\nlow lung volumes with bronchovascular crowding, bibasilar opacities  due at  in part to atelectasis, infiltrates, pulmonary edema difficult to entirely exclude.\\n\\nfindings\\nlow lung volumes with streaky bibasilar opacities, right greater than left. bronchovascular crowding, indistinct central vascular margination. no findings to suggest pleural effusion. accounting for technical factors heart size  within normal limits.\\n   \n",
      "264                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \\npa and lateral chest\\n\\n\\nindication\\nyearold male, \\n\\ncomparison\\n\\n\\nimpression\\nno acute cardiopulmonary findings\\n\\nfindings\\nheart size within normal limits, stable mediastinal and hilar contours. no alveolar consolidation, no findings of pleural effusion or pulmonary edema. no pneumothorax. small rounded bilateral axillary densities not seen on the previous exam most suggestive of artifacts, healed right lateral th rib fracture noted..\\n   \n",
      "291                                                                                                                                                                                                                                                                                                                                                                                                                                                                       \\nap and lateral chest\\n\\n\\nindication\\nyearold male, pain, syncope\\n\\ncomparison\\nnone\\n\\nimpression\\nno acute cardiopulmonary findings\\n\\nfindings\\napical lordotic frontal view. heart size within normal limits, mild aortic ectasiatortuosity. no focal alveolar consolidation, no definite pleural effusion seen. no typical findings of pulmonary edema. chronic appearing contour irregularity of the distal left clavicle and   widening may be posttraumatic or postsurgical, verterbroplasty noted at the thoracolumbar junction.\\n   \n",
      "299                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       \\npa and lateral chest\\n\\n\\nindication\\nyearold female, preop evaluation\\n\\ncomparison\\n\\n\\nimpression\\nno acute findings\\n\\nfindings\\nheart size within normal limits, stable mediastinal contours with aortic ectasiatortuosity. left hilar and left lower lobe calcifications  indicate a previous granulomatous process. no alveolar consolidation, no findings of pleural effusion or pulmonary edema. no pneumothorax.\\n   \n",
      "301                                                                                                                                                                                                                                                   \\npa and lateral chest\\n\\n\\nindication\\nyearold female, chest pain\\n\\ncomparison\\n\\n\\nimpression\\nchronic changes as described, no acute findings\\n\\nfindings\\nheart size within normal limits, stable mediastinal and hilar contours, coronary artery stent artifact,   and clips suggest cabg. mediastinal and hilar calcifications  indicate a previous granulomatous process. stable hyperinflation, bilateral upper lobe pleuroparenchymal near and nodular irregularities, right greater than left,  opacities in the peripheral right lung most compatible with scarring. no  abnormal pulmonary opacities, no definite pleural effusion seen. no typical findings of pulmonary edema. osseous demineralization, stable appearance of t and t  fractures.\\n   \n",
      "318                                                                                                                                                                                                                                                                                                                                                                                                                                                                \\nap and lateral chest\\n\\n\\nindication\\nyearold female, short of breath, pain\\n\\ncomparison\\n\\n\\nimpression\\n. bilateral pleural effusions, right larger than left . abnormal pulmonary opacities which may be due to atelectasis, differential diagnosis includes infection, aspiration, atypical distribution pulmonary edema\\n\\nfindings\\nbilateral pleural effusions, left small, right moderate in size, abnormal opacities in the adjacent lung bases. limited assessment of heart size due to obscured margins, stable mediastinal contours.\\n   \n",
      "332                                                                                                                                                                                                                                                                                                                                                  \\npa and lateral chest\\n\\n\\nindication\\nyearold male, \\n\\ncomparison\\nnone\\n\\nimpression\\nprominent interstitial markings in the central lungs and bases which may be secondary to low lung volumes with bronchovascular crowding, differential considerations include interstitial infiltrates of inflammatory or infectious etiology and mild pulmonary edema. clinical correlation is recommended.\\n\\nfindings\\nmild hypoventilation with bronchovascular crowding and prominent central and basilar interstitial markings. no focal alveolar consolidation, no pleural effusion demonstrated. considering technical factors heart size  within normal limits.\\n   \n",
      "343                                                                                                                                                                                                                                                                                                                                                                                 \\npa and lateral chest\\n\\n\\nindication\\nyearold female, pain, alleged physical assault\\n\\ncomparison\\nnone\\n\\nimpression\\nno acute findings. please note that fractures may not be demonstrated and consider additional imaging as clinically indicated.\\n\\nfindings\\nno focal consolidation, pneumothorax or definite pleural effusion. nodular density projected over the left base with no lateral view correlate  secondary to soft tissue overlay. heart size and pulmonary vascularity within normal limits, no mediastinal widening characteristic in appearance of vascular injury. no acute osseous injury  demonstrated.\\n   \n",
      "404                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \\npa and lateral chest\\n\\n\\nindication\\nchest pain, palpitations\\n\\ncomparison\\nnone\\n\\nimpression\\nhyperinflated lungs, air trapping versus inspiratory .\\n\\nfindings\\nhyperinflated lungs with mildly flattened posterior diaphragm and increased retrosternal airspace. no alveolar consolidation, no findings of pleural effusion or pulmonary edema. heart size within normal limits. no pneumothorax.\\n   \n",
      "\n",
      "     type  \n",
      "14      2  \n",
      "34      2  \n",
      "45      2  \n",
      "49      2  \n",
      "126     2  \n",
      "136     2  \n",
      "141     2  \n",
      "161     2  \n",
      "164     2  \n",
      "192     2  \n",
      "200     2  \n",
      "226     2  \n",
      "233     2  \n",
      "256     2  \n",
      "261     2  \n",
      "264     2  \n",
      "291     2  \n",
      "299     2  \n",
      "301     2  \n",
      "318     2  \n",
      "332     2  \n",
      "343     2  \n",
      "404     2  \n"
     ]
    }
   ],
   "source": [
    "rows_with_type_2 = df[df['type'] == 2]\n",
    "\n",
    "print(rows_with_type_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "some lines have findings first and some lines have impression first   \n",
    "\n",
    " so we and new columns and send data in accordingly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sometimes findings occurs first and some time impressions occur first.\n",
    "\n",
    "We need to act accoringly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import re\n",
    "\n",
    "# create a function to split so we can apply it to rows\n",
    "def extract_sections(report):\n",
    "    # using regex patterns helps use better detection. \n",
    "    findings_pattern = re.compile(r'findings\\s*(.*?)\\s*(impression|$)', re.DOTALL | re.IGNORECASE)\n",
    "    impression_pattern = re.compile(r'impression\\s*(.*?)\\s*$', re.DOTALL | re.IGNORECASE)\n",
    "    # re.compile makes regex into an object\n",
    "    # findings pattern finds data between findings and impression\n",
    "    # re.Dotall makes dot match with anaything\n",
    "    # re.ignorecase makes case unimportant\n",
    "\n",
    "    # Search for findings section\n",
    "    findings_match = findings_pattern.search(report)\n",
    "    findings = findings_match.group(1).strip() if findings_match else \"\"\n",
    "\n",
    "    # Search for impression section\n",
    "    impression_match = impression_pattern.search(report)\n",
    "    impression = impression_match.group(1).strip() if impression_match else \"\"\n",
    "\n",
    "    return findings, impression\n",
    "\n",
    "# Read the original CSV file\n",
    "with open('typed.csv', mode='r', newline='', encoding='utf-8') as infile:\n",
    "    reader = csv.DictReader(infile)\n",
    "    reports = [row['Text'] for row in reader]\n",
    "\n",
    "# Process the reports to extract findings and impressions\n",
    "processed_reports = []\n",
    "for report in reports:\n",
    "    findings, impression = extract_sections(report)\n",
    "    processed_reports.append({'report': report, 'findings': findings, 'impression': impression})\n",
    "\n",
    "# Write the new CSV file with findings and impressions\n",
    "with open('processed_reports.csv', mode='w', newline='', encoding='utf-8') as outfile:\n",
    "    fieldnames = ['report', 'findings', 'impression']\n",
    "    writer = csv.DictWriter(outfile, fieldnames=fieldnames)\n",
    "    \n",
    "    writer.writeheader()\n",
    "    for processed_report in processed_reports:\n",
    "        writer.writerow(processed_report)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some smart people have written findings on impressions and write \" see impressions\" on findings. we cant auto seperate those lines so we have to ignore them.  It is a really badly written dataset. So if you can manage to make it usable you are good at it.\n",
    "\n",
    "\n",
    "we need to remove rows wtih only word \"see\" is on findings row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# simple method to get values without ssee\n",
    "\n",
    "\n",
    "df = pd.read_csv(\"processed_reports.csv\")\n",
    "df_filtered = df[df['findings'] != 'see']\n",
    "df_filtered.to_csv(\"processed_reports\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "report        398\n",
       "findings      398\n",
       "impression    398\n",
       "dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_filtered.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are down to nearly 400 rows. No worries tho we can augment data before training but before lets clear leading empty spaces and dots im impressions column \n",
    "\n",
    "with lstrip we can use \"left strip\"  to get rid of leading part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"processed_reports.csv\")\n",
    "df['impression'] = df['impression'].str.strip(' . ')  # if starts with leading empty space and dot and empty sapce again\n",
    "df['impression'] = df['impression'].str.strip(' .')  # if starts with leading empty space and dot\n",
    "df['impression'] = df['impression'].str.strip('. ')  # if starts with leading empty space and  in reverse order\n",
    "df['impression'] = df['impression'].str.strip('.')  #   if starts with dot\n",
    "df['impression'] = df['impression'].str.strip(' ')  #   if starts with empty\n",
    "\n",
    "\n",
    "\n",
    "df.to_csv(\"final.csv\", index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}