Merge pull request #37 from IBM/12-generate-pip-package

12 generate pip package

.gitignore

@@ -0,0 +1,13 @@
+.git
+venv
+.idea
+__pycache__
+test/data/javacores/output/export.xml
+test/data/javacores/output/export.xsl
+tmp
+test/data/javacores-WAIT2/report.xml
+test/data/javacores-WAIT2/report.xsl
+.vscode
+dist
+src/javacore_analyser.egg-info
+reports

.travis.yml

@@ -3,5 +3,8 @@ python:
  - "3.9"
 install:
  - pip install -r requirements.txt
+env:
+ global:
+  - PYTHONPATH=src:test
 script:
- - python -m unittest test.test_javacore test.test_thread_snapshot test.test_javacore_set test.test_java_thread test.test_stack_trace test.test_javacore_analyzer test.test_tips test.test_verbose_gc_parser test.test_gc_collection test.test_code_snapshot_collection
+ - python -m unittest test.test_javacore test.test_thread_snapshot test.test_javacore_set test.test_java_thread test.test_stack_trace test.test_javacore_analyser test.test_tips test.test_verbose_gc_parser test.test_gc_collection test.test_code_snapshot_collection

CONTRIBUTING.md

@@ -81,16 +81,21 @@ To run the tool with sample data perform the following steps:
 When the window appears, add the following commandline to **run parameters**  
 `test/data/javacores /tmp/javacoreanalyser_output`  
 Change the second parameter to the directory where you want the output report be created.
-2. Right click again on **javacore_analyzer.py** and select **Run** or **Debug**.
+2. Right click again on **javacore_analyser.py** and select **Run** or **Debug**.
 
 To run web application:
-1. Right click on **javacore_analyzer_web.py** directory in **Project** view and select **Modify Run Configuration...**.
+1. Right click on **javacore_analyser_web.py** directory in **Project** view and select **Modify Run Configuration...**.
 2. Add the following **Environmental variables:**
    * **DEBUG:True**
    * **REPORTS_DIR:/tmp/web_reports**  
    You can change the report dir to the location when you want to store the report. 
    The application will start on http://localhost:5000
 
+
+## Build pip package 
+Follow the steps from [Packaging projects](https://packaging.python.org/en/latest/tutorials/packaging-projects/).
+Currently Chris has an API keys for test and production pypi
+
 ## Testing
 As default the tests in Pycharm are ran in the current selected directory. However we want to run them in main 
 directory of the tool (**javacore-analyser** directory, not **test** directory). 

README.md

@@ -16,41 +16,57 @@ The tool analyzes Javacores and verbose gc logs and provides some reports like c
 ## Installation and usage
 
 ### Installation:
-The tool requires Python 3.9 or higher plus some packages - see more in [REQUIREMENTS](REQUIREMENTS.md). Despite it is not mandatory, it is recommended in Python to use virtual environment to manage packages.
+The tool requires Python 3.9 or higher plus some packages - see more in [REQUIREMENTS](REQUIREMENTS.md). 
+Despite it is not mandatory, it is recommended in Python to use virtual environment to manage packages.
+
+#### Installing from pip
+This is most common option which you will need in 99% of situations  
 
 Steps:
 1. Download and install Python. Usually the latest version is supported. Search for supported versions in 
 [REQUIREMENTS](REQUIREMENTS.md)
-2. [Download](https://github.com/IBM/javacore-analyser/releases) and unpack the tool.
-3. Create and activate Virtual Environment according to [Creating virtual environments](https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments).
-4. Install all required pip packages. Navigate in command line to unpacked tool and type
-   `pip install -r requirements.txt`
+2. Create and activate Virtual Environment according to [Creating virtual environments](https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments).
+3. Run the following command:
+   `pip install javacore-analyser`  
+    OR
+   `pip install --extra-index-url https://test.pypi.org/simple/ javacore-analyser` - if you want an experimental version
+
+#### Installing from sources
+This is recommended for geeks only:
+1. Repeat steps 1 and 2 from above
+2. Download the project files either from [Releases](https://github.com/IBM/javacore-analyser/releases) or from [main](https://github.com/IBM/javacore-analyser/archive/refs/heads/main.zip)
+3. Extract the code and from code directory execute
+   `pip install .`
 
 ### Running the tool:
 
 #### Running cmd application: 
-1. Activate your created virtual environment according to activate Virtual Environment according to [Creating virtual environments](https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments)
-2. Install the requirements using pip: 
+1. Install application if not done yet
+2. Activate your created virtual environment according to activate Virtual Environment according to [Creating virtual environments](https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments)
+3. Install the requirements using pip: 
  `pip install requirements.txt` 
-3. Navigate in command line to unpacked tool and run the following command:  
-  `python javacore_analyzer.py <input-data> <output-dir>`  
+4. Run the following command from cmd: `javacore-analyser-batch <input-data> <generated-reports-dir>`
 Where `<input-data>` is one of the following:
 * The directory containing javacores and optionally verbose gc
 * Archive (7z, zip, tar.gz, tar.bz2) containing the same
-* List of the javacores separated by `'` character. Optionally you can add `--separator` option to define your own separator.
+* List of the javacores separated by `;` character. Optionally you can add `--separator` option to define your own separator.
 You can type the following command to obtain the help:  
-`python javacore_analyzer.py --help`
+`javacore-analyser-batch --help`
 
 #### Running web application:
-1. Repeat steps 1 and 2 from cmd application
-2. Navigate to unpacked tool and run the following commands: 
+1. Repeat steps 1-3 from cmd application
+2. OPTIONAL: set the following variables:
   ```
   export REPORTS_DIR=/tmp/reports  
-  flask --app javacore_analyser_web run
+  export PORT=5000
   ``` 
- The first command sets the location of the generated reports stored.  
- The second command starts the server which can be accessed by navigating to http://localhost:5000
+ The first parameter sets where the reports need to be stored. If not set, then the `reports` dir will be created in current location.  
+ The first parameter set the port to use by application. If not specified, 5000 will be used.  
+3. Execute the following command from cmd:  
+  `javacore_analyser_web`
 
+  Now you can type (http://localhost:5000/).  
+
 <!-- The following are OPTIONAL, but strongly suggested to have in your repository. -->
 <!--
 * [dco.yml](.github/dco.yml) - This enables DCO bot for you, please take a look https://github.com/probot/dco for more details.

pyproject.toml

@@ -0,0 +1,68 @@
+#
+# Copyright IBM Corp. 2024 - 2024
+# SPDX-License-Identifier: Apache-2.0
+#
+
+[build-system]
+requires = ["hatchling", "versioningit"]
+build-backend = "hatchling.build"
+
+[project]
+dynamic = ["version"]
+name = "javacore_analyser"
+#version = "2.0rc1"
+authors = [
+    { name="Krzysztof Kazmierczyk", email="[email protected]" },
+    { name="Piotr Aniola", email="[email protected]" },
+    { name="Tadeusz Janasiewicz", email="[email protected]" },
+]
+description = "The tool to review IBM Javacore files"
+readme = "README.md"
+license = {file = "LICENSE"}
+requires-python = ">=3.9"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3 :: Only",
+    "License :: OSI Approved :: Apache Software License",
+    "Operating System :: OS Independent",
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "Environment :: Web Environment",
+    "Framework :: Flask",
+    "Intended Audience :: Customer Service",
+    "Intended Audience :: System Administrators",
+    "Natural Language :: English",
+]
+dependencies = [
+    "py7zr",
+    "lxml",
+    "pyana",
+    "importlib-resources",
+    "flask",
+    "waitress",
+    "tqdm"
+]
+
+[project.scripts]
+javacore_analyser_batch="javacore_analyser.javacore_analyser_batch:main"
+javacore_analyser_web="javacore_analyser.javacore_analyser_web:main"
+
+[project.urls]
+Homepage = "https://github.com/IBM/javacore-analyser"
+Issues = "https://github.com/IBM/javacore-analyser/issues"
+Changelog = "https://github.com/IBM/javacore-analyser/CHANGELOG.md"
+Source = "https://github.com/IBM/javacore-analyser"
+
+[tool.hatch.version]
+source = "versioningit"
+
+[tool.versioningit.format]
+distance = "{next_version}.dev{distance}"
+dirty = "{version}"
+distance-dirty = "{next_version}.dev{distance}"

requirements.txt

@@ -1,5 +1,7 @@
-dicttoxml
 py7zr
 lxml
 pyana
+importlib-resources
+waitress # Production WSGI server
+flask # WSGI server for development the code
 tqdm

src/javacore_analyser/__init__.py

@@ -0,0 +1,4 @@
+#
+# Copyright IBM Corp. 2024 - 2024
+# SPDX-License-Identifier: Apache-2.0
+#

code_snapshot_collection.py → src/javacore_analyser/code_snapshot_collection.py

@@ -3,9 +3,9 @@
 # SPDX-License-Identifier: Apache-2.0
 #
 
-from abstract_snapshot_collection import AbstractSnapshotCollection
-from stack_trace import StackTrace
-from thread_snapshot import ThreadSnapshot
+from javacore_analyser.abstract_snapshot_collection import AbstractSnapshotCollection
+from javacore_analyser.stack_trace import StackTrace
+from javacore_analyser.thread_snapshot import ThreadSnapshot
 
 
 class CodeSnapshotCollection(AbstractSnapshotCollection):

java_thread.py → src/javacore_analyser/java_thread.py

@@ -3,7 +3,7 @@
 # SPDX-License-Identifier: Apache-2.0
 #
 
-from abstract_snapshot_collection import AbstractSnapshotCollection
+from javacore_analyser.abstract_snapshot_collection import AbstractSnapshotCollection
 
 
 class Thread(AbstractSnapshotCollection):

javacore.py → src/javacore_analyser/javacore.py

@@ -8,8 +8,8 @@
 import logging
 import os.path
 
-from constants import THREAD_INFO, DATETIME, SIGINFO, ENCODING
-from thread_snapshot import ThreadSnapshot
+from javacore_analyser.constants import THREAD_INFO, DATETIME, SIGINFO, ENCODING
+from javacore_analyser.thread_snapshot import ThreadSnapshot
 
 
 class CorruptedJavacoreException(Exception):

javacore_analyzer.py → src/javacore_analyser/javacore_analyser_batch.py

@@ -17,16 +17,13 @@
 
 import py7zr
 
-import logging_utils
-from constants import DEFAULT_FILE_DELIMITER
-from javacore_set import JavacoreSet
+from javacore_analyser import logging_utils
+from javacore_analyser.constants import DEFAULT_FILE_DELIMITER
+from javacore_analyser.javacore_set import JavacoreSet
 
 SUPPORTED_ARCHIVES_FORMATS = {"zip", "gz", "tgz", "bz2", "lzma", "7z"}
 
 
-
-
-
 def extract_archive(input_archive_filename, output_path):
     """
 
@@ -56,8 +53,8 @@ def extract_archive(input_archive_filename, output_path):
         logging.info("Processing 7z file")
     else:
         raise Exception("The format of file is not supported. "
-                      "Currently we support only zip, tar.gz, tgz, tar.bz2 and 7z. "
-                      "Cannot proceed.")
+                        "Currently we support only zip, tar.gz, tgz, tar.bz2 and 7z. "
+                        "Cannot proceed.")
 
     file.extractall(path=output_path)
     file.close()
@@ -67,15 +64,15 @@ def extract_archive(input_archive_filename, output_path):
 
 def main():
     logging_utils.create_console_logging()
-    logging.info("Wait2 tool")
+    logging.info("IBM Javacore analyser")
     logging.info("Python version: " + sys.version)
     logging.info("Preferred encoding: " + locale.getpreferredencoding())
 
     parser = argparse.ArgumentParser()
     parser.add_argument("input_param", help="Input file(s) or directory")
     parser.add_argument("output_param", help="Report output directory")
     parser.add_argument("--separator",
-                        help='Input files separator (default "' + DEFAULT_FILE_DELIMITER + '"',
+                        help='Input files separator (default "' + DEFAULT_FILE_DELIMITER + '")',
                         default=DEFAULT_FILE_DELIMITER)
     args = parser.parse_args()
 
@@ -119,8 +116,9 @@ def generate_javecore_set_data(files):
     - JavacoreSet: Generated JavacoreSet object containing the processed data.
     """
 
-    # Location when we store extracted archive or copied javacores files
+
     try:
+        # Location when we store extracted archive or copied javacores files
         javacores_temp_dir = tempfile.TemporaryDirectory()
 
         javacores_temp_dir_name = javacores_temp_dir.name
@@ -139,7 +137,6 @@ def generate_javecore_set_data(files):
         javacores_temp_dir.cleanup()
 
 
-
 # Assisted by WCA@IBM
 # Latest GenAI contribution: ibm/granite-8b-code-instruct
 def process_javacores_and_generate_report_data(input_files, output_dir):

javacore_analyser_web.py → src/javacore_analyser/javacore_analyser_web.py

@@ -13,10 +13,11 @@
 from pathlib import Path
 
 from flask import Flask, render_template, request, send_from_directory, redirect
+from waitress import serve
 
-import javacore_analyzer
-import logging_utils
-from constants import DEFAULT_REPORTS_DIR, DEFAULT_PORT
+import javacore_analyser.javacore_analyser_batch
+from javacore_analyser.constants import DEFAULT_REPORTS_DIR, DEFAULT_PORT
+from javacore_analyser.logging_utils import create_console_logging, create_file_logging
 
 """
 To run the application from cmd type:
@@ -25,13 +26,13 @@
 """
 app = Flask(__name__)
 with app.app_context():
-    logging_utils.create_console_logging()
+    create_console_logging()
     logging.info("Javacore analyser")
     logging.info("Python version: " + sys.version)
     logging.info("Preferred encoding: " + locale.getpreferredencoding())
     reports_dir = os.getenv("REPORTS_DIR", DEFAULT_REPORTS_DIR)
     logging.info("Reports directory: " + reports_dir)
-    logging_utils.create_file_logging(reports_dir)
+    create_file_logging(reports_dir)
 
 
 @app.route('/')
@@ -101,12 +102,20 @@ def upload_file():
 
         # Process the uploaded file
         report_output_dir = reports_dir + '/' + report_name
-        javacore_analyzer.process_javacores_and_generate_report_data(input_files, report_output_dir)
+        javacore_analyser.javacore_analyser_batch.process_javacores_and_generate_report_data(input_files,
+                                                                                             report_output_dir)
 
         return redirect("/reports/" + report_name + "/index.html")
     finally:
         javacores_temp_dir.cleanup()
 
+def main():
+    debug = os.getenv("DEBUG", False)
+    port = os.getenv("PORT", DEFAULT_PORT)
+    if debug:
+        app.run(debug=True, port=port)  # Run Flask for development
+    else:
+        serve(app, port=port)  # Run Waitress in production
 
 if __name__ == '__main__':
     """
@@ -115,4 +124,4 @@ def upload_file():
     PORT - application port
     REPORTS_DIR - the directory when the reports are stored as default
     """
-    app.run(debug=os.getenv("DEBUG", False), port=os.getenv("PORT", DEFAULT_PORT))
+    main()