notebooks/Tutorial5_Summary+Comments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "regional-catalog",
   "metadata": {},
   "source": [
    "#### Summary\n",
    "\n",
    "The notebooks presented here provides a basic guide for how to create an AMPEL science channel. This generally consists of the steps:\n",
    "- Sketch of a scientific idea, hypothesis or goal\n",
    "- Identification of existing or creation of new units which can be produce the test statistic or decisive parameters. This process is identical to what astronomers are used to doing (typically through python tools).\n",
    "- Channel debug based on small data samples.\n",
    "- Version freezing (e.g. through github tags)\n",
    "- Channel merged with live AMPEL instance at computer center for archive or live usage.\n",
    "\n",
    "When a scientific study is to be evaluated all of the necessary data and logs, together with e.g. evaluations of external spectroscopic classificaiton, is exported into a channel specific database. Together with the channel definition this can even be made public to fully detail all steps.\n",
    "\n",
    "The process outlines above \"front-loads\" much development work, in that this is done prior to having access to live data. While potenitally annoying, it is only after this initial development phase that a channel can be seen as fully reproducibily. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "static-cassette",
   "metadata": {},
   "source": [
    "#### What is different running in a full instance?\n",
    "\n",
    "What we define as a \"full\" live AMPEL instance is AMPEL being run on a high-performance computer center, usually in a way that combines multiple channels belonging to multiple users (with different stream access etc). \n",
    "\n",
    "All channel and unit definitions stay the same, but a few things do change. The most obvious is that a local (i.e. laptop) install generally does not have the capacity to parse a full alert stream or access months of real-time data.\n",
    "\n",
    "A second difference is that catalog matching can be done most efficiently - catalogs such as Gaia can only be hosted here. AMPEL uses either `catsHTM` or `extcats` to host such catalog, and make them available either for selecting or vetoing transients. As an example, the tutorials in this notebook uses the `SimpleDecentFilter`, while users of the full instance can use the full `DecentFilter`. The latter can also be tuned to reject likely stars based on Gaia data. \n",
    "\n",
    "A third difference concerns the use of secrets, passwords or access tokens. These should not be kept in potentially open github repositories, nor stored in containers that will eventually be made public. Such `Secrets` should instead either be encrypted or kept outside of the main container. The AMPEL operator will assist with this."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tribal-somalia",
   "metadata": {},
   "source": [
    "#### What about visualization and data access?\n",
    "\n",
    "Visual inspection of data is still critical, especially in the development phase of a channel. As difference science users do this through different means, there is no \"central\" single AMPEL front end. Instead, results can for example be viewed locally (as in the first notebook), exported to a Slack channel or distributed through an API to a front-end such as a SkyPortal instance. Units already exists for these tasks.\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wanted-buddy",
   "metadata": {},
   "source": [
    "#### The AMPEL live instance hosted at DESY\n",
    "\n",
    "One full AMPEL instance is currently hosted by the DESY computer center in Zeuthen. This parses public and partnership ZTF data, which is later combined with other MM real-time streams. The instance is open for public use (contact the maintainers). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}