text_scraper.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Run all these blocks at the beginning of a session (if applicable) as they will be continuously used throughout the notebook for each section"
      ],
      "metadata": {
        "id": "vySLC7SpQLTK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# run if connecting to google drive\n",
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "id": "ZFuyi0oeNxDW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# used for writing to multiple sheets in a single Excel workbook\n",
        "!pip install xlsxwriter"
      ],
      "metadata": {
        "id": "MGey82ayQE0X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "metadata": {
        "id": "ad02_DrwQJd-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Script for Scraping Text Articles from Fake News Sites\n",
        "## General notes on webscraping\n",
        "Webscraping: Extracting content from a website using a script or other automated process\n",
        "\n",
        "Before scraping:\n",
        "* Review the website's terms and conditions\n",
        " * Certain websites do not allow scraping of any kind (e.g. Youtube does not allow people to download videos hosted on that platform under their ToS)\n",
        " * Check the robots.txt file of the website (generally accessible by appending '/robots.txt' to the end of the main page's url) - this specifies which web scrapers/bots/APIs are allowed to do what action on their website\n",
        "\n",
        "Choosing a webscraper:\n",
        "* Various Python libraries available depending on usage\n",
        "* With the rise of GenAI that train on scapped data, becoming difficult to scrape as websites are blocking webscrapers to prevent LLMs from training on their data\n",
        "* Building own webscraping script will always be better than using existing webscraping software (for the aforementioned AI reason)\n",
        "\n",
        "## Existing Webscrapers\n",
        "1. BeautifulSoup\n",
        "\n",
        "* Documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/\n",
        "* Tutorial: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#quick-start\n",
        "* Basic description: Python library that extracts and presents data from HTML and XML files in a format that can be easily navigated\n",
        " * Pros: Works with any website, can extract any type of format with some tweaking\n",
        " * Cons: Need precise HTML structure and understanding, code needs to be different for every website\n",
        "\n",
        "2. Newspaper3k\n",
        "\n",
        "* Documentation: https://newspaper.readthedocs.io/en/latest/\n",
        "* Tutorial: https://towardsdatascience.com/the-easy-way-to-web-scrape-articles-online-d28947fc5979\n",
        "* Basic description: Python library that extracts content from news articles online\n",
        " * Pros: Relatively easier to use, decently plug-and-play for any website\n",
        " * Cons: Seems to be text-only, slow\n",
        "\n",
        "3. Newscatcher\n",
        "\n",
        "* Documentation: https://github.com/kotartemiy/newscatcher\n",
        "* Tutorial: https://codarium.substack.com/p/build-your-first-news-data-pipeline-with-python-newscatcher-5fa18b8c5083\n",
        "* Basic description: Python library that extracts metadata from news sites about their articles (titles, summaries, etc.)\n",
        " * Pros: Relatively easy to use\n",
        " * Cons: Works for websites formatted as news websites, does not extract article text"
      ],
      "metadata": {
        "id": "t5oH1bSYw5Ki"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Experimenting with Newscatcher\n",
        "\n",
        "* Read articles from the propaganda website [The Counter Signal](https://thecountersignal.com/)\n",
        "* Save scraped articles in a Pandas dataframe that can be converted to an Excel spreadsheet\n",
        " * Headers: title, author, text, date, url\n",
        "\n",
        "Notes/Observations:\n",
        "* Easy to set up and get going with minimal work\n",
        "* Tend to only be compatible with specifically formatted news sites (it pulls from url/feeds and url/rss which doesn't work for hoax websites that do not follow these conventions)\n",
        " * Would need to look into whether it's possible to specify to the library which page(s) to look at\n",
        "* Need to look into whether or not it's possible to use Newscatcher to download audio, especially for video-hosting platforms like Youtube"
      ],
      "metadata": {
        "id": "CUK6ecJBUwG4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install newspaper3k"
      ],
      "metadata": {
        "id": "Kg2-2O96UvW6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import newspaper\n",
        "from newspaper import Article\n",
        "from newspaper import Source"
      ],
      "metadata": {
        "id": "bSXC7conUBkA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# build newspaper for the news website\n",
        "# setting memoize_articles to False to avoid caching articles from each run\n",
        "pulse = newspaper.build(\"https://thecountersignal.com/\", memoize_articles=False)\n",
        "\n",
        "# create a new dataframe where the articles will be saved\n",
        "news_df = pd.DataFrame()\n",
        "\n",
        "# parse articles for each article found on the webpage\n",
        "for article in pulse.articles:\n",
        "  article.download()\n",
        "  article.parse()\n",
        "\n",
        "  # run this if we want to get a summary of the articles\n",
        "  # using the built-in nlp process of the library\n",
        "  # article.nlp()\n",
        "\n",
        "  # create a temp dataframe to hold info for each article\n",
        "  # and read the articles into the temp dataframe\n",
        "  temp_df = pd.DataFrame(columns = ['title', 'author', 'text', 'url'])\n",
        "\n",
        "  temp_df['title'] = article.title\n",
        "  temp_df['author'] = article.authors\n",
        "  temp_df['text'] = article.text\n",
        "  temp_df['url'] = article.url\n",
        "\n",
        "  # add article to the masterlist dataframe\n",
        "  news_df = pd.concat([news_df, temp_df], ignore_index=True)"
      ],
      "metadata": {
        "id": "iPTJ62evWByo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "news_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "FnXvU8LsbI_C",
        "outputId": "1c857486-0b26-4471-9068-52a81f14f22c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  title         author                                               text  \\\n",
              "0   NaN  Mike Campbell  Alberta Premier Danielle Smith announced that ...   \n",
              "1   NaN  Mike Campbell  With the election four weeks out, the BC Conse...   \n",
              "2   NaN  Mike Campbell  Environment Minister Stephen Guilbeault mostly...   \n",
              "3   NaN  Mike Campbell  Prime Minister Justin Trudeau has become so to...   \n",
              "4   NaN       Tcs Wire  NDP leader Jagmeet Singh has voted to keep Pri...   \n",
              "\n",
              "                                                 url  \n",
              "0  https://thecountersignal.com/breaking-smith-an...  \n",
              "1  https://thecountersignal.com/bc-conservatives-...  \n",
              "2  https://thecountersignal.com/guilbeault-says-t...  \n",
              "3  https://thecountersignal.com/justin-trudeau-ha...  \n",
              "4  https://thecountersignal.com/singh-votes-to-ke...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-45104cd1-1ced-49c7-9fde-737bb8be9add\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>author</th>\n",
              "      <th>text</th>\n",
              "      <th>url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "      <td>Mike Campbell</td>\n",
              "      <td>Alberta Premier Danielle Smith announced that ...</td>\n",
              "      <td>https://thecountersignal.com/breaking-smith-an...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>Mike Campbell</td>\n",
              "      <td>With the election four weeks out, the BC Conse...</td>\n",
              "      <td>https://thecountersignal.com/bc-conservatives-...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>Mike Campbell</td>\n",
              "      <td>Environment Minister Stephen Guilbeault mostly...</td>\n",
              "      <td>https://thecountersignal.com/guilbeault-says-t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>Mike Campbell</td>\n",
              "      <td>Prime Minister Justin Trudeau has become so to...</td>\n",
              "      <td>https://thecountersignal.com/justin-trudeau-ha...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>Tcs Wire</td>\n",
              "      <td>NDP leader Jagmeet Singh has voted to keep Pri...</td>\n",
              "      <td>https://thecountersignal.com/singh-votes-to-ke...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-45104cd1-1ced-49c7-9fde-737bb8be9add')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-45104cd1-1ced-49c7-9fde-737bb8be9add button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-45104cd1-1ced-49c7-9fde-737bb8be9add');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-ac1b76c4-8907-43e7-9a25-e7b95703a6a1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-ac1b76c4-8907-43e7-9a25-e7b95703a6a1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-ac1b76c4-8907-43e7-9a25-e7b95703a6a1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "news_df",
              "repr_error": "Out of range float values are not JSON compliant: nan"
            }
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# export dataframe as an excel sheet\n",
        "output_fp = '/content/drive/My Drive/kickstarter_seed_project/newscatcher_test.xlsx'\n",
        "\n",
        "news_df.to_excel(output_fp, index=False)\n",
        "\n",
        "# use the following code to write to a specific sheet in the excel spreadsheet\n",
        "'''\n",
        "# create excel writer object to initialize new workbook\n",
        "writer = pd.ExcelWriter(output, engine=\"xlsxwriter\")\n",
        "\n",
        "# write dataframes to different worksheets\n",
        "df1.to_excel(writer, sheet_name=source1, index=False)\n",
        "df2.to_excel(writer, sheet_name=source2, index=False)\n",
        "df3.to_excel(writer, sheet_name=source3, index=False)\n",
        "\n",
        "# close the excel writer and output file\n",
        "writer.close()\n",
        "'''"
      ],
      "metadata": {
        "id": "R8RsTxlVcU9I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Experimenting with Beautiful Soup\n",
        "* Read articles from the hoax website [The Pulse](https://www.thepulse.one/)\n",
        "* Save scraped articles in a Pandas dataframe that can be converted to an Excel spreadsheet\n",
        " * Headers: title, author (if possible), text, date (if possible), url\n",
        "\n",
        " Experiment with front page (latest news) only for now\n",
        "\n",
        " Notes/Observations:\n",
        " * More complicated to get started\n",
        " * More versatile (can do it with any website, preliminary research suggests it's possible to use in combination with other libraries to scrape audio files)\n",
        " * Website-dependent (need to manually detect html structure of each website for the code to work)\n",
        " * Output may need to be cleaned up on a case-by-case basis (in the first pass, BeautifulSoup was catching weird 'None' rows in the article text)"
      ],
      "metadata": {
        "id": "eiXDpINIPSJD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from bs4 import BeautifulSoup\n",
        "import os\n",
        "import re\n",
        "import requests"
      ],
      "metadata": {
        "id": "DGDgE2ewP__M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pulse_df = pd.DataFrame(columns = ['title', 'author', 'date', 'text', 'url'])\n",
        "\n",
        "root = requests.get('https://www.thepulse.one/archive?sort=new')\n",
        "site_soup = BeautifulSoup(root.content, 'html.parser')\n",
        "\n",
        "links = site_soup.find_all('a', attrs={'class': 'pencraft pc-reset _color-pub-primary-text_3axfk_204 _font-pub-headings_3axfk_140 _clamp_3axfk_259 _clamp-3_3axfk_271 _reset_3axfk_1'})\n",
        "\n",
        "for link in links:\n",
        "  path = requests.get(link.get('href'))\n",
        "  article_soup = BeautifulSoup(path.content, 'html.parser')\n",
        "\n",
        "  # extract article info from soup\n",
        "  article_title = article_soup.find(attrs={'class': 'post-title unpublished'})\n",
        "  article_author = article_soup.find(attrs={'class': 'pencraft pc-reset _decoration-hover-underline_3axfk_298 _reset_3axfk_1'})\n",
        "  article_date = article_soup.find(attrs={'class': 'pencraft pc-reset _color-pub-secondary-text_3axfk_207 _line-height-20_3axfk_95 _font-meta_3axfk_131 _size-11_3axfk_35 _weight-medium_3axfk_162 _transform-uppercase_3axfk_242 _reset_3axfk_1 _meta_3axfk_442'})\n",
        "  article_text = article_soup.find_all('p')\n",
        "\n",
        "  text = []\n",
        "\n",
        "  for p in article_text:\n",
        "    text.append(str(p.string))\n",
        "\n",
        "  # delete first text index which is None type\n",
        "  text = text[1:]\n",
        "  article_text = '\\n'.join(text) # turn text into newline separated string\n",
        "\n",
        "  # clean up article text to get rid of the weird empty lines\n",
        "  article_text = re.sub(r'None\\n', '', article_text)\n",
        "\n",
        "\n",
        "  # create new row of dataframe\n",
        "  new_row = {\n",
        "      'title': article_title.string,\n",
        "      'author': article_author.string,\n",
        "      'date': article_date.string,\n",
        "      'text': article_text,\n",
        "      'url': link.get('href')\n",
        "  }\n",
        "\n",
        "  # add row to dataframe\n",
        "  pulse_df.loc[len(pulse_df)] = new_row"
      ],
      "metadata": {
        "id": "hWFiUAt_Rqw1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pulse_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "1dpmOry5iCFO",
        "outputId": "420f3899-ef81-4bff-fc16-6bf22e63b5c0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                               title       author  \\\n",
              "0          What The Heck Is Going on With The World?  Joe Martino   \n",
              "1    What If You Heard a Rumor That You Don’t Exist?   Tom Bunzel   \n",
              "2                    Not Another Video About 9/11...  Joe Martino   \n",
              "3                              A Sense of the Sacred   Tom Bunzel   \n",
              "4  The Genetic Modification Of Our Food Compared ...  Arjun Walia   \n",
              "\n",
              "           date                                               text  \\\n",
              "0  Sep 26, 2024  I titled this in jest.\\nIt has been a busy mon...   \n",
              "1  Sep 25, 2024  When I lived in Los Angeles I worked with a te...   \n",
              "2  Sep 09, 2024  It simply states we are at a time where many u...   \n",
              "3  Sep 07, 2024  It made me consider where the sacred might be ...   \n",
              "4  Aug 26, 2024  “As part of the process, they portrayed the va...   \n",
              "\n",
              "                                                 url  \n",
              "0  https://www.thepulse.one/p/what-the-heck-is-go...  \n",
              "1  https://www.thepulse.one/p/what-if-you-heard-a...  \n",
              "2  https://www.thepulse.one/p/not-another-video-a...  \n",
              "3   https://www.thepulse.one/p/a-sense-of-the-sacred  \n",
              "4  https://www.thepulse.one/p/the-genetic-modific...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-804b3440-89f9-462c-b3a5-e56174f91053\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>author</th>\n",
              "      <th>date</th>\n",
              "      <th>text</th>\n",
              "      <th>url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>What The Heck Is Going on With The World?</td>\n",
              "      <td>Joe Martino</td>\n",
              "      <td>Sep 26, 2024</td>\n",
              "      <td>I titled this in jest.\\nIt has been a busy mon...</td>\n",
              "      <td>https://www.thepulse.one/p/what-the-heck-is-go...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>What If You Heard a Rumor That You Don’t Exist?</td>\n",
              "      <td>Tom Bunzel</td>\n",
              "      <td>Sep 25, 2024</td>\n",
              "      <td>When I lived in Los Angeles I worked with a te...</td>\n",
              "      <td>https://www.thepulse.one/p/what-if-you-heard-a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Not Another Video About 9/11...</td>\n",
              "      <td>Joe Martino</td>\n",
              "      <td>Sep 09, 2024</td>\n",
              "      <td>It simply states we are at a time where many u...</td>\n",
              "      <td>https://www.thepulse.one/p/not-another-video-a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A Sense of the Sacred</td>\n",
              "      <td>Tom Bunzel</td>\n",
              "      <td>Sep 07, 2024</td>\n",
              "      <td>It made me consider where the sacred might be ...</td>\n",
              "      <td>https://www.thepulse.one/p/a-sense-of-the-sacred</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The Genetic Modification Of Our Food Compared ...</td>\n",
              "      <td>Arjun Walia</td>\n",
              "      <td>Aug 26, 2024</td>\n",
              "      <td>“As part of the process, they portrayed the va...</td>\n",
              "      <td>https://www.thepulse.one/p/the-genetic-modific...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-804b3440-89f9-462c-b3a5-e56174f91053')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-804b3440-89f9-462c-b3a5-e56174f91053 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-804b3440-89f9-462c-b3a5-e56174f91053');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-386502f2-2edb-46d2-83eb-e354269dfc66\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-386502f2-2edb-46d2-83eb-e354269dfc66')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-386502f2-2edb-46d2-83eb-e354269dfc66 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "pulse_df",
              "summary": "{\n  \"name\": \"pulse_df\",\n  \"rows\": 12,\n  \"fields\": [\n    {\n      \"column\": \"title\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 12,\n        \"samples\": [\n          \"The Lore of Underground \\\"Alien\\\" Bases on Earth Goes Far Beyond Harvard Academics...\",\n          \"Merck Was Injecting Girls With Aluminum But Told Them It Was A Saline Placebo: This Is Why It's Called \\\"Big Pharma\\\"\",\n          \"What The Heck Is Going on With The World?\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"author\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"Joe Martino\",\n          \"Tom Bunzel\",\n          \"Arjun Walia\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"date\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 12,\n        \"samples\": [\n          \"Jul 06, 2024\",\n          \"Jul 17, 2024\",\n          \"Sep 26, 2024\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 12,\n        \"samples\": [\n          \"\\\"Flying saucers are real and they are space ships from another solar system. I think that they possibly are manned by intelligent observers who are members of a race that may have been investigating our Earth for centuries.\\\" \\nHarvard academics recently suggested that a highly advanced civilization could be residing deep underground or, perhaps in a base inside the Moon. \\nThe authors, Tim Lomas (Harvard University), Brendan Case (Harvard University) and Michael Paul Masters (Montana Tech of the University of Montana), make the case for the possibility of so-called \\u201ccrypto-terrestrials\\u201d to explain unidentified and unexplainable observations made around the world each year. \\nThey join Harvard Astronomer Avi Loeb, who has been making similar claims of extraterrestrials living among us for the past few years. Let\\u2019s not forget about Dr. John Mack, a former head of the department of psychiatry who investigated the UFO phenomenon for decades. \\nMasters is a biological anthropologist who said he was asked to help research potential explanations for UFOs. \\nHe said \\\"aliens\\\" may actually just be humans from far in the future who have figured out how to time travel.\\u00a0He is not the first to propose this theory, it\\u2019s based on numerous observations and thousands of corroborating stories from \\u201cexperiencers\\u201d throughout human history. \\nDr. Michael A. Persinger, an American-Canadian professor of psychology at Laurentian University, observed that for thousands of years and within every known human culture, normal individuals have reported brief and often repeated \\u2018visitations\\u2019 by humanoid phenomena that are associated with strange flying objects.\\nIt\\u2019s not hard to make an argument that observations of UFOs and contact by beings from beyond the Earth is one of mankind\\u2019s oldest stories.\\nThe authors speculate that intelligent beings may have gone underground until we're ready for contact, and that we must seem extremely primitive to them based on what we see flying around in the skies. \\nWithin the idea of \\u201ccrypto-terrestrials\\u201d are multiple hypotheses, which you can refer to the paper for more details if interested. \\nIn this article, I wanted to focus on the general idea of an extraterrestrial presence on our planet and the Moon. I also want to explore where exactly this theory comes from. \\nDid it just pop out of thin air? Is it that \\u201cfar out\\u201d there to think this could even be a possibility? If so, where are they? \\nI thought it would be useful given the fact that this recent hypothesis made headlines, making it appear as if it is a new theory among academics, but it\\u2019s not. And I wish these academics would have expressed that in their paper. \\nThe Pulse is a reader-supported publication. To receive new posts and support our work, consider becoming a free or paid subscriber.\\nIt\\u2019s a theme that garners serious attention every few years and has for multiple decades. \\n\\u201cIt is not out of the question that artifacts of these visits still exist, or even that some kind of base is maintained (possibly automatically) within the solar system to provide continuity for successive expeditions. Because of weathering and the possibility of detection and interference by the inhabitants of the Earth, it would be preferable not to erect such a base on the Earth\\u2019s surface. The Moon seems one reasonable alternative. Forthcoming high resolution photographic reconnaissance of the Moon from space vehicles \\u2013 particularly of the back side \\u2013 might bear these possibilities in mind.\\u201d\\nThis highly classified document was somehow obtained by the CIA, who were keeping tabs on the USSR space program at the time. What\\u2019s interesting is that this document is from the 60\\u2019s, while the Wikileaks document is from 1979. Perhaps Korenevsky\\u2019s intelligence was correct? \\nShare\\nRemote viewing is the ability of someone to describe the characteristics of a remote geographical location regardless their present location. \\nFor example, a professionally trained remote viewer by the Central Intelligence Agency and academics at the Stanford Research Institute (SRI), where this type of activity took place for more than 25 years, would successfully and repeatedly be able to describe in-depth details of a distant location without actually being there. It was called the STARGATE program.\\nDespite the fact that the entire remote viewing program was officially declassified in 1995, it appears that this probably wasn\\u2019t true given the fact that many from within the program have \\u2018blown the whistle\\u2019 so to speak. Many claimed that they were tasked to \\u201cview\\u201d information about various groups of extraterrestrials that were visiting our planet.\\nThis document sure does provide some validation to such claims. \\nI also found this interesting because it correlates with yet another remote viewer named Pat Price. Price was said to be one of SRI\\u2019s most successful and skilled remote viewers.\\nA well-known story about Price comes from Captain Frederick H. Atwater, a retired US Army officer who was involved in these remote viewing experiments. I have confirmed the validity of this story through my conversations with Paul H. Smith (mentioned earlier in the article) among various other sources and literature.\\nThe story is that Price had remotely viewed four extraterrestrial bases on Earth, one of which was located under Mount Ziel, which lies approximately 80 miles west-northwest of Pine Gap, Australia. \\nThe other bases were said to be under Mount Perdido in the Pyrenees, Mount Inyangani in Zimbabwe and under Mount Hayes in Alaska. He described the occupants as \\u2018looking like homo sapiens, except for the lungs, heart, blood and eyes.\\u2019\\nHow can so many who have held such positions all be considered lunatics? Or considered to be making false claims? If you tie in all of the data and research that has accompanied this phenomenon, it does make you wonder. \\nShare\\nIndigenous stories of highly advanced civilizations living underground are also quite prevalent. It\\u2019s a theme that spans multiple cultures and centuries. Examples are not hard to find.\\n\\u201cThe old ones tell stories about people from the stars who lived underground near Tanana. There are many stories the old ones told about the Star People who live among them and went underground near Tanana. The Inupiat believe they came to Earth on a spaceship.\\u201d \\nIn her work, \\u201cEncounters with Star People: Untold Stories of American Indians\\u201d she explains,\\n\\u201cBelief in subterranean worlds has been handed down as myths or legends among generations of people from all over the world. For example, Socrates spoke of huge hollows within the Earth that were inhabited and vast caverns where rivers flowed. The Cherokee Indians tell that when they first came to the southeastern United States, they found many well-tended gardens but not the people who cared for them. Eventually, they discovered a group of people who lived underground and came out only at night to tend the gardens. They harvested the food and took it underground to their cities.These people were small, had blue skin and large black eyes. The sun rays were too harsh for them so they built their cities underground and only came out at night using the light of the moon. The Cherokee called them the \\u201cMoon People.\\u201d\\nSo, just to reiterate, the idea of highly advanced life dwelling within our planet is not novel.  But what Have Humans Done Underground?\\n\\u201cIt is ironic that the U.S. would begin a devastating war, allegedly in search of weapons of mass destruction, when the most worrisome developments in this field are occurring in your own backyard. It is ironic that the U.S. should be fighting monstrously expensive wars in Iraq and Afghanistan, allegedly to bring democracy to those countries, when it itself can no longer claim to be called a democracy, when trillions, and I mean thousands of billions of dollars have been spent on projects about which both the Congress and the\\u00a0Commander in\\u00a0Chief\\u00a0have been kept deliberately in the dark.\\u201d\\u00a0\\nTo be perfectly honest, it\\u2019s quite difficult to determine exactly what humans have really accomplished. I believe this is the case because some of the most interesting accomplishments are most likely \\u201cclassified\\u201d and lie within the realm of what are known as Special Access Programs (SAPs).\\nSome of these programs could involve the construction of deep underground military bases, known as (DUMBS). Furthermore, some of this money could also be going towards the construction of under-ocean, and in-bottom (of the ocean) military bases.\\nIn 1987, Deputy Director of Engineering and Construction for the U.S. Army Corps of Engineers,\\u00a0Lloyd A. Duscha,\\u00a0gave a speech at an engineering conference titled \\u201cUnderground Facilities for Defense \\u2013 Experience and Lessons.\\u201d In the first paragraph of his speech he states the following:\\n\\u2013 (New York: Elsevier Science Publishing Company, Inc., 1987, pp. 109-113.)\\u201d\\nHe then went on to discuss the Corps\\u2019 involvement in the construction of the large and elaborate NORAD base in the 1960s, buried deep beneath Cheyenne Mountain, in Colorado.\\nThe Pulse is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.\\nThere are documents available that expose a deep underground command center that was to be built far below areas such as Washington, D.C., and China Lake, California, during the Cold War. Documents show that in 1964 the military was considering building a huge underground cavity 4,000 feet deep beneath China Lake. \\n\\u201cAs it happens, after giving a public talk a couple of years ago, I was approached by a man who had been a uniformed member of the United States Navy. We chatted for a while and when he mentioned that he had spent some time at China Lake my ears perked up. I asked him if there was an underground facility at China Lake. He said that indeed there is, and that it is impressively large and deep. I asked him if he had ever been in it, and he said that he had, though not to the deepest levels. I asked him how deep the deepest part extended. He looked at me soberly and said very quietly, \\u201cIt goes one mile deep.\\u201d I then asked him what the underground base contains. He replied, \\u201cWeapons.\\u201d I responded, \\u201cWhat sort of weaponry?\\u201d And he answered without pausing, \\u201cWeapons more powerful than nuclear weapons.\\u201d\\nThis may all be hearsay, but again, it\\u2019s interesting to explore. \\nAbstract:\\n\\u201cThe construction of thirty manned in-bottom bases within the ocean floors is technically and\\u00a0economically\\u00a0feasible\\u2026.This could take 15 years. The major technology for a land-linked station in-bottom is established now; only adaptations are needed. The remaining experimental phases will require further development of equipment and techniques applicable to remote sea access. There are useful assignments for a succession of three experimental stations other than advancing in-bottom construction techniques. Science and engineering concerned with the oceans and their resources will be furthered and military tests of undersea base functions complimenting deeper operations can be accomplished. \\nPresently, establishing an in-bottom facility and building upon this will present fewer technical difficulties that do the submersibles which would support it and use it. Subsequent to the completion of the third phase of a demonstration program, which would be a remote, deep water station, and the evaluation of it, a multiple base program, could be\\u00a0implemented.\\u201d\\nIt\\u2019s truly hard to know how far along we\\u2019ve come with the technological plans discussed in this article. It\\u2019s now 2024, and these facilities must be extremely advanced, perhaps, in my opinion, 100-1000 years ahead of where the \\u201cmainstream\\u201d world is at, but perhaps I\\u2019m wrong. \\nIt\\u2019s unfortunate that our world is still dominated by secrecy. The United States has a history of government agencies existing in secret for years. The National Security Agency (NSA) was founded in 1952, its existence was hidden until the mid 1960\\u2019s. Even more secretive is the National Reconnaissance Office, which was founded in 1960 but remained completely secret for 30 years. Who knows how far it has gone today in 2024. \\nI hope by reading this article you understand where the lore of extraterrestrial bases on our planet comes from. \\nThe Pulse is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.\\nShare\\nVery well done article, short and concise. One thing I would mention in regards to DUMB's is the patents that exist for atomic boring machines. These are from the 1970's and 80's and are potentially what was used to create the underground tunnel/ base system\\nNo posts\\nReady for more?\",\n          \"I\\u2019ve seen some people scoff at the word \\u201cBig Pharma.\\u201d To someone like me, it refers to the documented criminal activities we\\u2019ve seen take place for decades by large pharmaceutical companies. \\n\\\"A corporation may treat both criminal and civil penalties as simply business expenses, to be weighed against the revenues earned from illegal behaviour. But human beings can be put in jail, and that is a whole other matter. Conceivably, convicting corporate executives of criminal behaviour and sentencing them to terms of imprisonment might be a more effective deterrent to the \\u201crepeat offender\\u201d behaviour demonstrated by Pfizer.\\\"\\nThe story below isn\\u2019t quite \\u2018criminal\\u2019 per se, because Merck and federal health regulatory agencies believe aluminum adjuvants in vaccines are completely safe, but we\\u2019ll get to that later. \\nAAHS is more potent than traditional aluminum vaccine adjuvants. \\nAs you can see in the image below, the trial consent form stated that \\u201cOne half of the participants will receive the active vaccine, while the other half will get the placebo vaccine (meaning a vaccine without active substance)\\u201d leading the participants to falsely believe the placebo was inactive. \\nThe next image shows Merck\\u2019s advertisement informing potential participants that they would be receiving a saline placebo.\\nA number of trial participants experienced chronic disabling symptoms, including some randomized to the adjuvant \\u201cplacebo\\u201d group.\\n\\u201cWe examined the informed consent form and the recruitment brochure for the FUTURE II Gardasil vaccine trial conducted in Denmark; and we interviewed several FUTURE II trial participants and their treating physicians. We also reviewed regulatory documentation related to Gardasil vaccine approval process and the guidelines on evaluation of adjuvants used in human vaccines.\\nThe Pulse is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.\\n\\u201cMerck claims its AAHS formulation is proprietary and, to date, has refused to provide samples to independent researchers for testing, so the full composition of AAHS remains unknown.\\u201d\\nFederal health regulatory agencies have pointed to their belief that aluminum in vaccines is completely safe given it\\u2019s been used for more than 90 years in vaccines. Along with that, as far as HPV vaccine clinical trials go, justification for putting aluminum in the placebo in those trials was to 'assess the safety of HPV virus-like particles.\\u2019\\n\\u201cThe stated rationale of using AAHS control-to characterise the safety of the HPV virus-like particles-lacks clinical relevance. A non-placebo control may have obscured an accurate assessment of safety and the participant consent process of some trials raises ethical concerns.\\u201d\\nSeveral girls who participated in the trials, including some in the placebo group, suffered chronic disabling symptoms, raising questions about the toxic effects of the adjuvant.\\n\\u201cFirst, this practice needlessly exposes trial subjects to risks only, without any possible benefit, and is therefore a clear violation of medical ethics guidelines which require that research involving human subjects be designed so as to minimize harms and maximize benefits.\\nAluminum has no place in biology. It\\u2019s an extremely toxic substance that originates underground. It\\u2019s great for making certain things, like airplanes, but it wreaks havoc on biology. \\nWhile fact checkers have often misled the public stating things like \\u201cthe amount of aluminum in a vaccine is equivalent to canned food\\u201d this is misinformed. The aluminum we ingest through food is discarded via digestion, whereas aluminum injected into the body or bloodstream is treated entirely differently.\\nThis is another example of the lack of investigation and nuance on the part of fact checkers and armchair university scientists. \\nAdjuvants \\u2014 substances added to vaccines to increase immune response \\u2014 can produce adverse effects. Aluminum is an effective adjuvant but scientists have long raised concerns about its safety in vaccines because it is a known neurotoxin.\\nUsing a placebo that can independently produce the adverse effects confounds the trial results and \\u201cnullifies the very concept of a placebo-controlled trial,\\u201d the researchers wrote, making it impossible to evaluate vaccine safety.\\nMerck has conducted multiple clinical trials for Gardasil, first approved in 2006, testing the drug on approximately 30,000 females and males. Claims of vaccine safety based on those trials became the basis for a large international campaign by the World Health Organization to vaccinate 90% of girls globally by 2030 and 80% of all U.S. teenagers by 2030.\\nIn all but one of its placebo-controlled pre-licensure clinical trials for the Gardasil vaccine, Merck used its AAHS adjuvant as the placebo.\\nAnother interesting thing to note about HPV infections is when it comes to women in particular, approximately 70 percent of those who get an infection will clear it all by themselves within the first year, you don\\u2019t even have to detect it. Keep in mind that only a handful of HPV infections can actually lead to cancer. \\nWithin two years, approximately 90 percent of these infections will clear all by themselves. By three years, 10 percent of that original group will still have an HPV infection, and 5 percent of this 10 percent will have progressed into what are known as a precancerous lesion. There are three types of precancerous lesions, CIN1, which requires no treatment, C1N2 and the most severe, CIN3.\\n\\u201cSo now you have that small group (the remaining 5 percent)\\u2026who have precancerous lesions and now let\\u2019s look at that moving into invasive carcinoma. What we know then is that amongst women with CIN3 lesions, it takes five years for about twenty percent of them to become invasive carcinomas. That\\u2019s a pretty slow process. It takes about thirty years for forty percent of them to become invasive cervical carcinomas.\\u201d\\nShare\\n\\u201cThe HPV Vaccine on Trial is a shocking tale, chronicling the global efforts to sell and compel this alleged miracle. The book opens with the vaccine\\u2019s invention, winds through its regulatory labyrinths, details the crushing denial and dismissal of reported harms and deaths, and uncovers the enormous profits pharma and inventors have reaped. Authors Holland, Mack Rosenberg, and Iorio drill down into the clinical trial data, government approvals, advertising, and personal accounts of egregious injuries that have followed in countries as far-flung as Japan, Australia, Colombia, India, Ireland, the U.K. and Denmark. The authors have written an unprecedented expos\\u00e9 about this vaunted vaccine.\\nWritten in plain language, the book is for everyone concerned \\u2013 parents, patients, doctors, nurses, scientists, healthcare organizations, government officials, and schools. Ultimately, this book is not just about the HPV vaccine, but about how industry, government, and medical authorities may be putting the world\\u2019s children in harm\\u2019s way.\\u201d\\nWe\\u2019ve also covered a number of examples of serious injuries in the past decade, before we were subjected to extreme censorship.\\nThe Pulse is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.\\nFor example, a 14-year-old boy named Christopher Bunch passed away a few years ago, and the mother and father claimed that it was as a\\u00a0result of the HPV vaccine. His mother started a petition  claiming that her son \\u201cdied as a direct result of the HPV vaccine.\\u201d\\nThe letter, posted below, states,\\n\\u201cMultiple independent scientific organizations and major regulatory and public health authorities, including the World Health Organization (WHO), the U.S. Centers for Disease Control and Prevention (CDC), and the U.S. Food and Drug Administration (FDA) have repeatedly evaluated the safety of HPV vaccines. The results of these evaluations continue to be reassuring.\\u201d\\nThe Same Old Story Over and Over Again\\nWe\\u2019ve seen it since we began this work in 2009.\\n\\u201cThe medical profession is being bought by the pharmaceutical industry, not only in terms of the practice of medicine, but also in terms of teaching and research. The academic institutions of this country are allowing themselves to be the paid agents of the pharmaceutical industry. I think it\\u2019s disgraceful.\\u201d\\nJust as we have discussions surrounding the benefits of vaccines, a one-sided only discussion is destroying important facts that shed light on the importance of informed consent and choice. \\nShare\\n@Victorforsythe\\n0 seconds ago\\nThe Fluoride Zombies are Ok with a secrete society called the Masons can print all the paper money they want, with no oversight and its completely UNCONSTITUTIONAL! They killed/murdered my spiritual Master for revealing their secretes. Now Facebook has me on lifetime ban for the internet. Can you see this?\\nThank you isn't enough for what you're doing, but a heartfelt thank you. I hope this can help: it's an organized summary of reports and research, always being updated. More than 200 reports at the time of this writing.\\nhttps://birdseyeviewperspective.substack.com/p/vaccines-lies-research-fraud-harms\\nNo posts\\nReady for more?\",\n          \"I titled this in jest.\\nIt has been a busy month, and I was out of town for a stretch of time, yet still, I still found myself having many conversations about the state of the world, what\\u2019s going on with current events, and how we might navigate the uncertainty and wild times we are experiencing.\\nOften people want to know where I stand on topics surrounding RFK Jr, Trump, Kamala, Israel, the state of Canada or the WHO pandemic treaty\\u2026\\nIn most cases, people are looking for a certain answer. But the truth is, there is no certain answer I can provide, nor can anyone.\\nInstead, what is best now is the honest and true ability to make sense on a deep level that includes many layers of our being. \\nWithin that includes accepting we cannot know for sure and that we are partly along for the ride! (Which is meant to be enjoyed by the way.)\\nI don\\u2019t have much time to write these days, but I can produce much more content if I talk to an inanimate electronic device (camera) and send it off to you when I\\u2019m done. So I did just that.\\nBelow are some thoughts on how we might choose to orient to the current state of the world in a wise way. Allowing ourselves to know, make sense, explore, be open, find peace, and engage without losing our minds or hearts.\\nWould love to hear your thoughts in the comments :)\\nShare\\nThe Pulse is a reader-supported publication. To receive new posts and support our work, consider becoming a free or paid subscriber.\\nThis is exactly what I\\u2019ve been working on: maintaining awareness of what\\u2019s going on in the world without losing my head; at times a very difficult process. Thank you Joe and CE\\nMy thoughts?\\nYouTube. No thank you. \\nNo posts\\nReady for more?\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"url\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 12,\n        \"samples\": [\n          \"https://www.thepulse.one/p/the-lore-of-underground-alien-bases\",\n          \"https://www.thepulse.one/p/merck-was-injecting-girls-with-aluminum\",\n          \"https://www.thepulse.one/p/what-the-heck-is-going-on-with-the\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# export dataframe as an excel sheet\n",
        "output_fp = '/content/drive/My Drive/kickstarter_seed_project/beautiful_soup_test.xlsx'\n",
        "\n",
        "pulse_df.to_excel(output_fp, index=False)\n",
        "\n",
        "# use the following code to write to a specific sheet in the excel spreadsheet\n",
        "'''\n",
        "# create excel writer object to initialize new workbook\n",
        "writer = pd.ExcelWriter(output, engine=\"xlsxwriter\")\n",
        "\n",
        "# write dataframes to different worksheets\n",
        "df1.to_excel(writer, sheet_name=source1, index=False)\n",
        "df2.to_excel(writer, sheet_name=source2, index=False)\n",
        "df3.to_excel(writer, sheet_name=source3, index=False)\n",
        "\n",
        "# close the excel writer and output file\n",
        "writer.close()\n",
        "'''"
      ],
      "metadata": {
        "id": "JEy8nTDPf5aF"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}