Chatbot parser generic output 2

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_1.txt

@@ -0,0 +1,41 @@
+MATLAB
+ note
+    To run a MATLAB program on the HPC-UGent infrastructure you must compile it first,
+    because the MATLAB license server is not accessible from cluster workernodes
+    (except for the interactive debug cluster).
+    Compiling MATLAB programs is only possible on the interactive debug cluster,
+    not on the HPC-UGent login nodes where resource limits w.r.t. memory and max. number of progress are too strict.
+Why is the MATLAB compiler required?
+The main reason behind this alternative way of using MATLAB is
+licensing: only a limited number of MATLAB sessions can be active at the
+same time. However, once the MATLAB program is compiled using the MATLAB
+compiler, the resulting stand-alone executable can be run without
+needing to contact the license server.
+Note that a license is required for the MATLAB Compiler, see
+https://nl.mathworks.com/help/compiler/index.html. If the mcc
+command is provided by the MATLAB installation you are using, the MATLAB
+compiler can be used as explained below.
+Only a limited amount of MATLAB sessions can be active at the same time
+because there are only a limited amount of MATLAB research licenses
+available on the UGent MATLAB license server. If each job would need a
+license, licenses would quickly run out.
+How to compile MATLAB code
+Compiling MATLAB code can only be done from the login nodes, because
+only login nodes can access the MATLAB license server, workernodes on
+clusters cannot.
+To access the MATLAB compiler, the MATLAB module should be loaded
+first. Make sure you are using the same MATLAB version to compile and
+to run the compiled MATLAB program.
+$ module avail MATLAB/
+----------------------/apps/gent/RHEL8/zen2-ib/modules/all----------------------
+   MATLAB/2021b    MATLAB/2022b-r5 (D)
+$ module load MATLAB/2021b
+After loading the MATLAB module, the mcc command can be used. To get
+help on mcc, you can run mcc -?.
+To compile a standalone application, the -m flag is used (the -v
+flag means verbose output). To show how mcc can be used, we use the
+magicsquare example that comes with MATLAB.
+First, we copy the magicsquare.m example that comes with MATLAB to
+example.m:
+cp $EBROOTMATLAB/extern/examples/compiler/magicsquare.m example.m
+To compile a MATLAB program, use mcc -mv:

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_1_metadata.json

@@ -0,0 +1,15 @@
+{
+    "main_title": "MATLAB",
+    "subtitle": "How-to-compile-MATLAB-code",
+    "source_file": "../../mkdocs/docs/HPC/MATLAB.md",
+    "title_depth": 2,
+    "directory": "MATLAB",
+    "links": {
+        "0": "https://docs.hpc.ugent.be/interactive_debug"
+    },
+    "parent_title": "",
+    "previous_title": null,
+    "next_title": "MATLAB_paragraph_2",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/MATLAB/#how-to-compile-matlab-code"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_2.txt

@@ -0,0 +1,47 @@
+mcc -mv example.m
+Opening log file:  /user/home/gent/vsc400/vsc40000/java.log.34090
+Compiler version: 8.3 (R2021b)
+Dependency analysis by REQUIREMENTS.
+Parsing file "/user/home/gent/vsc400/vsc40000/example.m"
+	(Referenced from: "Compiler Command Line").
+Deleting 0 temporary MEX authorization files.
+Generating file "/user/home/gent/vsc400/vsc40000/readme.txt".
+Generating file "run\_example.sh".
+Libraries
+To compile a MATLAB program that needs a library, you can use the
+-I library_path flag. This will tell the compiler to also look for
+files in library_path.
+It's also possible to use the -a path flag. That will result in all
+files under the path getting added to the final executable.
+For example, the command mcc -mv example.m -I examplelib -a datafiles
+will compile example.m with the MATLAB files in examplelib, and will
+include all files in the datafiles directory in the binary it
+produces.
+Memory issues during compilation
+If you are seeing Java memory issues during the compilation of your
+MATLAB program on the login nodes, consider tweaking the default maximum
+heap size (128M) of Java using the _JAVA_OPTIONS environment variable
+with:
+export _JAVA_OPTIONS="-Xmx64M"
+The MATLAB compiler spawns multiple Java processes. Because of the
+default memory limits that are in effect on the login nodes, this might
+lead to a crash of the compiler if it's trying to create to many Java
+processes. If we lower the heap size, more Java processes will be able
+to fit in memory.
+Another possible issue is that the heap size is too small. This could
+result in errors like:
+Error: Out of memory
+A possible solution to this is by setting the maximum heap size to be
+bigger:
+export _JAVA_OPTIONS="-Xmx512M"
+Multithreading
+MATLAB can only use the cores in a single workernode (unless the
+Distributed Computing toolbox is used, see
+https://nl.mathworks.com/products/distriben.html).
+The amount of workers used by MATLAB for the parallel toolbox can be
+controlled via the parpool function: parpool(16) will use 16
+workers. It's best to specify the amount of workers, because otherwise
+you might not harness the full compute power available (if you have too
+few workers), or you might negatively impact performance (if you have
+too many workers). By default, MATLAB uses a fixed number of workers
+(12).

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_2_metadata.json

@@ -0,0 +1,12 @@
+{
+    "main_title": "MATLAB",
+    "subtitle": "Multithreading",
+    "source_file": "../../mkdocs/docs/HPC/MATLAB.md",
+    "title_depth": 2,
+    "directory": "MATLAB",
+    "parent_title": "",
+    "previous_title": "MATLAB_paragraph_1",
+    "next_title": "MATLAB_paragraph_3",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/MATLAB/#multithreading"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_3.txt

@@ -0,0 +1,55 @@
+You should use a number of workers that is equal to the number of cores
+you requested when submitting your job script (the ppn value, see Generic resource requirements).
+You can determine the right number of workers to use via the following
+code snippet in your MATLAB program:
+% specify the right number of workers (as many as there are cores available in the job) when creating the parpool
+c = parcluster('local')
+pool = parpool(c.NumWorkers)
+See also [the parpool
+documentation](https://nl.mathworks.com/help/distcomp/parpool.html).
+Java output logs
+Each time MATLAB is executed, it generates a Java log file in the users
+home directory. The output log directory can be changed using:
+MATLAB_LOG_DIR=<OUTPUT_DIR>
+where <OUTPUT_DIR> is the name of the desired output directory. To
+create and use a temporary directory for these logs:
+# create unique temporary directory in $TMPDIR (or /tmp/$USER if
+$TMPDIR is not defined)
+# instruct MATLAB to use this directory for log files by setting $MATLAB_LOG_DIR
+$  export MATLAB_LOG_DIR=$ (mktemp -d -p $TMPDIR:-/tmp/$USER)
+You should remove the directory at the end of your job script:
+rm -rf $MATLAB_LOG_DIR
+Cache location
+When running, MATLAB will use a cache for performance reasons. This
+location and size of this cache can be changed through the
+MCR_CACHE_ROOT and MCR_CACHE_SIZE environment variables.
+The snippet below would set the maximum cache size to 1024MB and the
+location to /tmp/testdirectory.
+export MATLAB_CACHE_ROOT=/tmp/testdirectory 
+export MATLAB_CACHE_SIZE=1024M 
+So when MATLAB is running, it can fill up to 1024MB of cache in
+/tmp/testdirectory.
+MATLAB job script
+All of the tweaks needed to get MATLAB working have been implemented in
+an example job script. This job script is also available on the HPC.
+
+#!/bin/bash
+#PBS -l nodes=1:ppn=1
+#PBS -l walltime=1:0:0
+#
+# Example (single-core) MATLAB job script
+#
+# make sure the MATLAB version matches with the one used to compile the MATLAB program!
+module load MATLAB/2021b
+# use temporary directory (not $HOME) for (mostly useless) MATLAB log files
+# subdir in $TMPDIR (if defined, or /tmp otherwise)
+export MATLAB_LOG_DIR=$(mktemp -d -p  ${TMPDIR:-/tmp})
+# configure MATLAB Compiler Runtime cache location & size (1GB)
+# use a temporary directory in /dev/shm (i.e. in memory) for performance reasons
+export MCR_CACHE_ROOT=$(mktemp -d -p /dev/shm)
+export MCR_CACHE_SIZE=1024MB
+# change to directory where job script was submitted from
+cd $PBS_O_WORKDIR
+# run compiled example MATLAB program 'example', provide '5' as input argument to the program
+# $EBROOTMATLAB points to MATLAB installation directory
+./run_example.sh $EBROOTMATLAB 5

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/MATLAB/MATLAB_paragraph_3_metadata.json

@@ -0,0 +1,15 @@
+{
+    "main_title": "MATLAB",
+    "subtitle": "MATLAB-job-script",
+    "source_file": "../../mkdocs/docs/HPC/MATLAB.md",
+    "title_depth": 2,
+    "directory": "MATLAB",
+    "links": {
+        "0": "https://docs.hpc.ugent.be/running_batch_jobs/#generic-resource-requirements"
+    },
+    "parent_title": "",
+    "previous_title": "MATLAB_paragraph_2",
+    "next_title": null,
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/MATLAB/#matlab-job-script"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/index/index_paragraph_1.txt

@@ -0,0 +1,17 @@
+---
+hide: "toc"
+---
+ Welcome to the HPC-UGent documentation
+---
+Use the menu on the left to navigate, or use the search box on the top right.
+You are viewing documentation intended for people using *.
+Use the OS dropdown in the top bar to switch to a different operating system.
+Quick links
+- Getting Started  | Getting Access
+- Recording of HPC-UGent intro
+- Linux Tutorial
+- Hardware overview
+- FAQ | Troubleshooting | Best practices | Known issues
+
+If you find any problems in this documentation, please report them by mail to <[email protected]> or open a pull request.
+If you still have any questions, you can contact the HPC-UGent team.

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/index/index_paragraph_1_metadata.json

@@ -0,0 +1,25 @@
+{
+    "main_title": "index",
+    "subtitle": "Welcome-to-the-HPC-UGent-documentation",
+    "source_file": "../../mkdocs/docs/HPC/index.md",
+    "title_depth": 1,
+    "directory": "index",
+    "links": {
+        "0": "https://docs.hpc.ugent.be/getting_started",
+        "1": "https://docs.hpc.ugent.be/account",
+        "2": "https://www.ugent.be/hpc/en/training/introhpcugent-recording",
+        "3": "https://docs.hpc.ugent.be/linux-tutorial",
+        "4": "https://www.ugent.be/hpc/en/infrastructure",
+        "5": "https://docs.hpc.ugent.be/FAQ",
+        "6": "https://docs.hpc.ugent.be/troubleshooting",
+        "7": "https://docs.hpc.ugent.be/best_practices",
+        "8": "https://docs.hpc.ugent.be/known_issues",
+        "9": "https://github.com/hpcugent/vsc_user_docs",
+        "10": "https://www.ugent.be/hpc/en/support"
+    },
+    "parent_title": "",
+    "previous_title": null,
+    "next_title": null,
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_1.txt

@@ -0,0 +1,44 @@
+Interactive and debug cluster
+Purpose
+The purpose of this cluster is to give the user an environment where
+there should be no waiting in the queue to get access to a limited
+number of resources. This environment allows a user to immediately
+start working, and is the ideal place for interactive work such as
+development, debugging and light production workloads (typically
+sufficient for training and/or courses).
+This environment should be seen as an extension or even replacement of the login nodes,
+instead of a dedicated compute resource. The interactive cluster is
+overcommitted, which means that more CPU cores can be requested for
+jobs than physically exist in the cluster. Obviously, the performance of
+this cluster heavily depends on the workloads and the actual overcommit
+usage. Be aware that jobs can slow down or speed up during their
+execution.
+Due to the restrictions and sharing of the CPU resources (see
+section Restrictions and overcommit factor) jobs on this cluster
+should normally start more or less immediately. The tradeoff is that
+performance must not be an issue for the submitted jobs. This means that
+typical workloads for this cluster should be limited to:
+-   Interactive jobs (see
+    chapter Running interactive jobs)
+-   Cluster desktop sessions (see
+    chapter Using the HPC-UGent web portal)
+-   Jobs requiring few resources
+-   Debugging programs
+-   Testing and debugging job scripts
+Submitting jobs
+To submit jobs to the HPC-UGent interactive and debug cluster nicknamed
+donphan, first use:
+module swap cluster/donphan
+Then use the familiar qsub, qstat, etc. commands (see
+chapter Running batch jobs).
+Restrictions and overcommit factor
+Some limits are in place for this cluster:
+-   each user may have at most 5 jobs in the queue (both running and
+    waiting to run);
+-   at most 3 jobs per user can be running at the same time;
+-   running jobs may allocate no more than 8 CPU cores and no more than
+    27200 MiB of memory in total, per user;
+In addition, the cluster has an overcommit factor of 6. This means that
+6 times more cores can be allocated than physically exist.
+Simultaneously, the default memory per core is 6 times less than what
+would be available on a non-overcommitted cluster.

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_1_metadata.json

@@ -0,0 +1,18 @@
+{
+    "main_title": "interactive_debug",
+    "subtitle": "Restrictions-and-overcommit-factor",
+    "source_file": "../../mkdocs/docs/HPC/interactive_debug.md",
+    "title_depth": 3,
+    "directory": "interactive_debug",
+    "links": {
+        "0": "https://docs.hpc.ugent.be/interactive_debug/#restrictions-and-overcommit-factor",
+        "1": "https://docs.hpc.ugent.be/running_interactive_jobs/#running-interactive-jobs",
+        "2": "https://docs.hpc.ugent.be/web_portal/#using-the-hpc-ugent-web-portal",
+        "3": "https://docs.hpc.ugent.be/running_batch_jobs/#running-batch-jobs"
+    },
+    "parent_title": "",
+    "previous_title": null,
+    "next_title": "interactive_debug_paragraph_2",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/interactive_debug/#restrictions-and-overcommit-factor"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_2.txt

@@ -0,0 +1,11 @@
+Please note that based on the (historical) workload of the interactive
+and debug cluster, the above restrictions and the overcommitment ratio
+might change without prior notice.
+Shared GPUs
+Each node in the donphan cluster has a relatively small GPU that is shared between all jobs.
+This means that you don't need to reserve it and thus possibly wait for it.
+But this also has a downside for performance and security: jobs might be competing for the same GPU resources (cores, memory or encoders) without
+any preset fairshare and there is no guarantee one job cannot access another job's memory
+(as opposed to having reserved GPUs in the GPU clusters).
+All software should behave the same as on the dedicated GPU clusters (e.g. using CUDA or OpenGL acceleration
+from a cluster desktop via the webportal).

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/interactive_debug/interactive_debug_paragraph_2_metadata.json

@@ -0,0 +1,12 @@
+{
+    "main_title": "interactive_debug",
+    "subtitle": "Shared-GPUs",
+    "source_file": "../../mkdocs/docs/HPC/interactive_debug.md",
+    "title_depth": 3,
+    "directory": "interactive_debug",
+    "parent_title": "",
+    "previous_title": "interactive_debug_paragraph_1",
+    "next_title": null,
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/interactive_debug/#shared-gpus"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_1.txt

@@ -0,0 +1,42 @@
+Introduction to HPC
+What is HPC?
+"High Performance Computing" (HPC) is computing on a "Supercomputer", a computer with at the
+frontline of contemporary processing capacity -- particularly speed of
+calculation and available memory.
+While the supercomputers in the early days (around 1970) used only a few
+processors, in the 1990s machines with thousands of processors began to
+appear and, by the end of the 20th century, massively parallel
+supercomputers with tens of thousands of "off-the-shelf" processors were
+the norm. A large number of dedicated processors are placed in close
+proximity to each other in a computer cluster.
+A computer cluster consists of a set of loosely or tightly connected computers that work
+together so that in many respects they can be viewed as a single system.
+The components of a cluster are usually connected to each other through
+fast local area networks ("LAN") with each node (computer used as a
+server) running its own instance of an operating system. Computer
+clusters emerged as a result of convergence of a number of computing
+trends including the availability of low cost microprocessors,
+high-speed networks, and software for high performance distributed
+computing.
+Compute clusters are usually deployed to improve performance and
+availability over that of a single computer, while typically being more
+cost-effective than single computers of comparable speed or
+availability.
+Supercomputers play an important role in the field of computational
+science, and are used for a wide range of computationally intensive
+tasks in various fields, including quantum mechanics, weather
+forecasting, climate research, oil and gas exploration, molecular
+modelling (computing the structures and properties of chemical
+compounds, biological macromolecules, polymers, and crystals), and
+physical simulations (such as simulations of the early moments of the
+universe, airplane and spacecraft aerodynamics, the detonation of
+nuclear weapons, and nuclear fusion). [^1]
+What is the HPC-UGent infrastructure?
+The HPC is a collection of computers with AMD and/or Intel CPUs, running a
+Linux operating system, shaped like pizza boxes and stored above and
+next to each other in racks, interconnected with copper and fiber
+cables. Their number crunching power is (presently) measured in hundreds
+of billions of floating point operations (gigaflops) and even in
+teraflops.
+The HPC-UGent infrastructure relies on parallel-processing technology to offer UGent researchers an
+extremely fast solution for all their data processing needs.

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/introduction/introduction_paragraph_1_metadata.json

@@ -0,0 +1,12 @@
+{
+    "main_title": "introduction",
+    "subtitle": "What-is-the-HPC-UGent-infrastructure",
+    "source_file": "../../mkdocs/docs/HPC/introduction.md",
+    "title_depth": 2,
+    "directory": "introduction",
+    "parent_title": "",
+    "previous_title": null,
+    "next_title": "introduction_paragraph_2",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/introduction/#what-is-the-hpc-ugent-infrastructure"
+}