jupyter/InternalDependenciesJava.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2f0eabc4",
   "metadata": {},
   "source": [
    "# Internal Dependencies\n",
    "<br>  \n",
    "\n",
    "### References\n",
    "- [Analyze java package metrics in a graph database](https://joht.github.io/johtizen/data/2023/04/21/java-package-metrics-analysis.html)\n",
    "- [Calculate metrics](https://101.jqassistant.org/calculate-metrics/index.html)\n",
    "- [Neo4j Python Driver](https://neo4j.com/docs/api/python-driver/current)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4191f259",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plot\n",
    "from neo4j import GraphDatabase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c5dab37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please set the environment variable \"NEO4J_INITIAL_PASSWORD\" in your shell \n",
    "# before starting jupyter notebook to provide the password for the user \"neo4j\". \n",
    "# It is not recommended to hardcode the password into jupyter notebook for security reasons.\n",
    "\n",
    "driver = GraphDatabase.driver(uri=\"bolt://localhost:7687\", auth=(\"neo4j\", os.environ.get(\"NEO4J_INITIAL_PASSWORD\")))\n",
    "driver.verify_connectivity()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1db254b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cypher_query_from_file(cypherFileName):\n",
    "    with open(cypherFileName) as file:\n",
    "        return ' '.join(file.readlines())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59310f6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def query_cypher_to_data_frame(filename : str, limit: int = 10_000):\n",
    "    cypher_query_template = \"{query}\\nLIMIT {row_limit}\"\n",
    "    cypher_query = get_cypher_query_from_file(filename)\n",
    "    cypher_query = cypher_query_template.format(query = cypher_query, row_limit = limit)\n",
    "    records, summary, keys = driver.execute_query(cypher_query)\n",
    "    return pd.DataFrame([r.values() for r in records], columns=keys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c09da482",
   "metadata": {},
   "outputs": [],
   "source": [
    "def query_first_non_empty_cypher_to_data_frame(*filenames : str, limit: int = 10_000):\n",
    "    \"\"\"\n",
    "    Executes the Cypher queries of the given files and returns the first result that is not empty.\n",
    "    If all given file names result in empty results, the last (empty) result will be returned.\n",
    "    By additionally specifying \"limit=\" the \"LIMIT\" keyword will appended to query so that only the first results get returned.\n",
    "    \"\"\"    \n",
    "    result=pd.DataFrame()\n",
    "    for filename in filenames:\n",
    "        result=query_cypher_to_data_frame(filename, limit)\n",
    "        if not result.empty:\n",
    "            return result\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a56670c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The following cell uses the build-in %html \"magic\" to override the CSS style for tables to a much smaller size.\n",
    "#This is especially needed for PDF export of tables with multiple columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "006b9dc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* CSS style for smaller dataframe tables. */\n",
    ".dataframe th {\n",
    "    font-size: 8px;\n",
    "}\n",
    ".dataframe td {\n",
    "    font-size: 8px;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6323e85e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pandas DataFrame Display Configuration\n",
    "pd.set_option('display.max_colwidth', 300)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "91d80bf7",
   "metadata": {},
   "source": [
    "## Artifacts\n",
    "\n",
    "List the artifacts this notebook is based on. Different sorting variations help finding artifacts by their features and support larger code bases where the list of all artifacts gets too long.\n",
    "\n",
    "Only the top 30 entries are shown. The whole table can be found in the following CSV report:  \n",
    "`List_all_Java_artifacts`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc682db6",
   "metadata": {},
   "outputs": [],
   "source": [
    "artifacts = query_cypher_to_data_frame(\"../cypher/Internal_Dependencies/List_all_Java_artifacts.cypher\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e48ead1b",
   "metadata": {},
   "source": [
    "### Table 1a - Top 30 artifacts with the highest package count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f822ddca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of packages descending\n",
    "artifacts.sort_values(by=['packages','artifactName'], ascending=[False, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f4661b3",
   "metadata": {},
   "source": [
    "### Table 1b - Top 30 artifacts with the highest type count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61ed6b36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of types descending\n",
    "artifacts.sort_values(by=['types','artifactName'], ascending=[False, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e508a21",
   "metadata": {},
   "source": [
    "### Table 1c - Top 30 artifacts with the highest number of incoming dependencies\n",
    "\n",
    "The following table lists the top 30 artifacts that are used the most by other artifacts (highest count of incoming dependencies, highest in-degree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "107c0a1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of incoming dependencies descending\n",
    "artifacts.sort_values(by=['incomingDependencies','artifactName'], ascending=[False, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52593397",
   "metadata": {},
   "source": [
    "### Table 1d - Top 30 artifacts with the highest number of outgoing dependencies\n",
    "\n",
    "The following table lists the top 30 artifacts that are depending on the highest number of other artifacts (highest count of outgoing dependencies, highest out-degree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eee13bd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of outgoing dependencies descending\n",
    "artifacts.sort_values(by=['outgoingDependencies','artifactName'], ascending=[False, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c928800",
   "metadata": {},
   "source": [
    "### Table 1e - Top 30 artifacts with the lowest package count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18d73c15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of packages ascending\n",
    "artifacts.sort_values(by=['packages','artifactName'], ascending=[True, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bdd5de0",
   "metadata": {},
   "source": [
    "### Table 1f - Top 30 artifacts with the lowest type count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0078e92e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of types ascending\n",
    "artifacts.sort_values(by=['types','artifactName'], ascending=[True, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f628024",
   "metadata": {},
   "source": [
    "### Table 1g - Top 30 artifacts with the lowest number of incoming dependencies\n",
    "\n",
    "The following table lists the top 30 artifacts that are used the least by other artifacts (lowest count of incoming dependencies, lowest in-degree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6100d597",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of incoming dependencies ascending\n",
    "artifacts.sort_values(by=['incomingDependencies','artifactName'], ascending=[True, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad07b654",
   "metadata": {},
   "source": [
    "### Table 1h - Top 30 artifacts with the lowest number of outgoing dependencies\n",
    "\n",
    "The following table lists the top 30 artifacts that are depending on the lowest number of other artifacts (lowest count of outgoing dependencies, lowest out-degree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd096a38",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort by number of outgoing dependencies ascending\n",
    "artifacts.sort_values(by=['outgoingDependencies','artifactName'], ascending=[True, True]).reset_index(drop=True).head(30)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d7cfd862",
   "metadata": {},
   "source": [
    "## Cyclic Dependencies\n",
    "\n",
    "Cyclic dependencies occur when one package uses a class of another package and vice versa. \n",
    "These dependencies can lead to problems when one of these packages needs to be changed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9112f590",
   "metadata": {},
   "source": [
    "## Table 2a - Cyclic Dependencies Overview\n",
    "\n",
    "Show the top 40 cyclic dependencies sorted by the most promising to resolve first. This is done by calculating the number of forward dependencies (first cycle participant to second cycle participant) in relation to backward dependencies (second cycle participant back to first cycle participant). The higher this rate (approaching 1), the easier it should be to resolve the cycle by focussing on the few backward dependencies.\n",
    "\n",
    "Only the top 40 entries are shown. The whole table can be found in the following CSV report:  \n",
    "`Cyclic_Dependencies`\n",
    "\n",
    "**Columns:**\n",
    "- *artifactName* identifies the artifact of the first participant of the cycle\n",
    "- *packageName* identifies the package of the first participant of the cycle\n",
    "- *dependentArtifactName* identifies the artifact of the second participant of the cycle\n",
    "- *dependentPackageName* identifies the package of the second participant of the cycle\n",
    "- *forwardToBackwardBalance* is between 0 and 1. High for many forward and few backward dependencies.\n",
    "- *numberForward* contains the number of dependencies from the first participant of the cycle to the second one\n",
    "- *numberBackward* contains the number of dependencies from the second participant of the cycle back to the first one\n",
    "- *someForwardDependencies* lists some forward dependencies in the text format \"type1 -> type2\"\n",
    "- *backwardDependencies* lists the backward dependencies in the format \"type1 <- type2\" that are recommended to get resolved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa634a4e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cyclic_dependencies = query_cypher_to_data_frame(\"../cypher/Cyclic_Dependencies/Cyclic_Dependencies.cypher\")\n",
    "cyclic_dependencies.head(40)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "02146c4d",
   "metadata": {},
   "source": [
    "### Table 2b - Cyclic Dependencies Break Down\n",
    "\n",
    "Lists packages with cyclic dependencies with every dependency in a separate row sorted by the most promising  dependency first.\n",
    "\n",
    "Only the top 40 entries are shown. The whole table can be found in the following CSV report:  \n",
    "`Cyclic_Dependencies_Breakdown`\n",
    "\n",
    "**Columns in addition to Table 2a:**\n",
    "- *dependency* shows the cycle dependency in the text format \"type1 -> type2\" (forward) or \"type2<-type1\" (backward)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf49a164",
   "metadata": {},
   "outputs": [],
   "source": [
    "cyclic_dependencies_breakdown = query_cypher_to_data_frame(\"../cypher/Cyclic_Dependencies/Cyclic_Dependencies_Breakdown.cypher\",limit=40)\n",
    "cyclic_dependencies_breakdown"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52f46a62",
   "metadata": {},
   "source": [
    "### Table 2c - Cyclic Dependencies Break Down - Backward Dependencies Only\n",
    "\n",
    "Lists packages with cyclic dependencies with every dependency in a separate row sorted by the most promising  dependency first. This table only contains the backward dependencies from the second participant of the cycle back to the first one that are the most promising to resolve.\n",
    "\n",
    "Only the top 40 entries are shown. The whole table can be found in the following CSV report:  \n",
    "`Cyclic_Dependencies_Breakdown_BackwardOnly`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61521c88",
   "metadata": {},
   "outputs": [],
   "source": [
    "cyclic_dependencies_breakdown_backward = query_cypher_to_data_frame(\"../cypher/Cyclic_Dependencies/Cyclic_Dependencies_Breakdown_Backward_Only.cypher\",limit=40)\n",
    "cyclic_dependencies_breakdown_backward"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3f59da8d",
   "metadata": {},
   "source": [
    "## Interface Segregation Candidates\n",
    "\n",
    "Well known from [Design Principles and Design Patterns by Robert C. Martin](http://staff.cs.utu.fi/~jounsmed/doos_06/material/DesignPrinciplesAndPatterns.pdf), the *Interface Segregation Principle* suggests that software components should have narrow, focused interfaces rather than large, general-purpose ones. The goal is to minimize the dependencies between components and increase modularity, flexibility, and maintainability.\n",
    "\n",
    "Smaller, focused and purpose-driven interfaces\n",
    "\n",
    "- make it easier to modify individual components without affecting the rest of the system.\n",
    "- make it clearer which client is affected by which change.\n",
    "- don’t force their clients to depend on methods they don’t need.\n",
    "- reduce the scope of changes since a change to one component doesn’t affect others.\n",
    "- lead to a more loosely coupled architecture that is easier to understand and maintain.\n",
    "\n",
    "Reference: [Analyze java package metrics in a graph database](https://joht.github.io/johtizen/data/2023/04/21/java-package-metrics-analysis.html#interface-segregation)\n",
    "\n",
    "### How to apply the results\n",
    "\n",
    "If just one method of a type is used, especially in many places, then the result of this method can be used to call e.g. a method or constuct an object instead of using the whole object and then just calling that single method.\n",
    "\n",
    "If there are a couple of methods that are used for a distinct purpose, those could be factored out into a separate interface. The original type can extended/implement the new interface so that there are no breaking changes. Then all the callers, that use only this group of methods, can be changed to the new interface.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a966e4c3",
   "metadata": {},
   "source": [
    "### Table 4 - Top 40 most used combinations of methods\n",
    "\n",
    "The following table shows the top 40 most used combinations of methods of larger types that might benefit from applying the *Interface Segregation Principle*. The whole table can be found in the CSV report `Candidates_for_Interface_Segregation`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63fb6ba9",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "interface_segregation_candidates=query_cypher_to_data_frame(\"../cypher/Internal_Dependencies/Candidates_for_Interface_Segregation.cypher\", limit=40)\n",
    "interface_segregation_candidates"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4be116e8",
   "metadata": {},
   "source": [
    "## Package Usage"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bb185191",
   "metadata": {},
   "source": [
    "### Table 5 - Types that are used by multiple packages\n",
    "\n",
    "This table shows the top 40 packages that are used by the highest number of different packages. The whole table can be found in the CSV report `List_types_that_are_used_by_many_different_packages`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36a34924",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "types_used_by_many_packages=query_cypher_to_data_frame(\"../cypher/Internal_Dependencies/List_types_that_are_used_by_many_different_packages.cypher\", limit=40)\n",
    "types_used_by_many_packages"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "73dd1b13",
   "metadata": {},
   "source": [
    "### Table 6 - Packages that are used by multiple artifacts\n",
    "\n",
    "This table shows the top 30 artifacts that only use a few (compared to all existing) packages of another artifact.\n",
    "The whole table can be found in the CSV report `ArtifactPackageUsage`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf3d43c5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "used_packages_of_dependent_artifact=query_cypher_to_data_frame(\"../cypher/Internal_Dependencies/How_many_packages_compared_to_all_existing_are_used_by_dependent_artifacts.cypher\",limit=30)\n",
    "used_packages_of_dependent_artifact"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d37ba7ae",
   "metadata": {},
   "source": [
    "### Table 7 - Types that are used by multiple artifacts\n",
    "\n",
    "This table shows the top 30 types that only use a few (compared to all existing) types of another artifact. The whole table can be found in the CSV report `ClassesPerPackageUsageAcrossArtifacts`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74f2a5c8",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "used_types_of_dependent_artifact=query_cypher_to_data_frame(\"../cypher/Internal_Dependencies/How_many_classes_compared_to_all_existing_in_the_same_package_are_used_by_dependent_packages_across_different_artifacts.cypher\", limit=30)\n",
    "used_types_of_dependent_artifact"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd8a40e9",
   "metadata": {},
   "source": [
    "### Table 8 - Duplicate package names across artifacts\n",
    "\n",
    "This table shows the top 30 duplicate package names across artifacts. They are ordered by the number of duplicates descending.\n",
    "\n",
    "This might lead to confusion, makes importing more error prone and might even lead to duplicate classes where only one of them will be loaded by the class loader. If a package is named the same way in two or more artifacts this even allows another artifact to access package protected classes, methods or members which might not be intended. \n",
    "\n",
    "The whole table can be found in the CSV report `DuplicatePackageNamesAcrossArtifacts`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2547f52e",
   "metadata": {},
   "outputs": [],
   "source": [
    "duplicate_package_names_across_artifacts=query_cypher_to_data_frame(\"../cypher/Artifact_Dependencies/Artifacts_with_duplicate_packages.cypher\", limit=30)\n",
    "duplicate_package_names_across_artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fead8222",
   "metadata": {},
   "source": [
    "### Table 9 - Annotated elements\n",
    "\n",
    "This table shows 30 most used Java Annotations including some examples where they are used.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9055e352",
   "metadata": {},
   "outputs": [],
   "source": [
    "annotated_elements=query_cypher_to_data_frame(\"../cypher/Java/Annotated_code_elements.cypher\", limit=30)\n",
    "annotated_elements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bf903e2",
   "metadata": {},
   "source": [
    "### Table 10 - Distance distribution between dependent files\n",
    "\n",
    "This table shows the file directory distance distribution between dependent files. Intuitively, the distance is given by the fewest number of change directory commands needed to navigate between a file and a dependency it uses. Those are aggregate to see how many dependent files are in the same directory, how many are just one change directory command apart, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b86a804",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_first_non_empty_cypher_to_data_frame(\"../cypher/Internal_Dependencies/Get_file_distance_as_shortest_contains_path_for_dependencies.cypher\",\n",
    "                                           \"../cypher/Internal_Dependencies/Set_file_distance_as_shortest_contains_path_for_dependencies.cypher\", limit=20)"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "JohT"
   }
  ],
  "code_graph_analysis_pipeline_data_validation": "ValidateJavaInternalDependencies",
  "kernelspec": {
   "display_name": "codegraph",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "title": "Object Oriented Design Quality Metrics for Java with Neo4j"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}