parsed mds by for chatbot parser script for account.md, connecting.md, compiling_your_software.md (FOR REVIEW ONLY, NO MERGE!) #664

EwDa291 · 2024-08-08T09:33:15Z

The scripts are still very much a messy work in progress. They need some extra features and need to be cleaned before they can be merged.

EwDa291 · 2024-08-08T15:12:10Z

This last commit made the amount of changed files very large since I also included the output of the script. The important scripts can be found in main.py and jinja_parser.py. I will make some more changes to the file structure in the future to clean it up a bit.

… successfully yet

This file is just used to test some things locally and not part of the parser

…utorial are both read and saved properly

…docs into chatbot_parser

…down

stdweird · 2024-08-29T15:34:12Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/account/account_paragraph_8.txt

+and "permanently".
+Click "Confirm"
+You will now be taken to the authentication page of your institute.
+You will now have to log in with CAS using your UGent account.


with CAS remove it? or explain it

CAS -> CAS (Central Authentication Service)?

stdweird · 2024-08-29T15:35:15Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/account/account_paragraph_8.txt

+You will now have to log in with CAS using your UGent account.
+You either have a login name of maximum 8 characters, or a (non-UGent)
+email address if you are an external user. In case of problems with your
+UGent password, please visit: <https://password.ugent.be/>. After


@EwDa291 remove the < and > . not sure what the : is for

stdweird · 2024-08-29T15:37:36Z

...processor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1.txt

@@ -0,0 +1,10 @@
+Compiling and testing your software on the HPC
+All nodes in the HPC cluster are running the "RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty)" 


what happened with the double quotes?, not needed to include the clusters i think

stdweird · 2024-08-29T15:38:04Z

...processor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1.txt

@@ -0,0 +1,10 @@
+Compiling and testing your software on the HPC
+All nodes in the HPC cluster are running the "RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty)" 
+Operating system, which is a specific version of Red Hat Enterprise Linux. This means that all the 


no capital O i guess (or add one to System)?

stdweird · 2024-08-29T15:38:27Z

...processor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1.txt

+All nodes in the HPC cluster are running the "RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty)" 
+Operating system, which is a specific version of Red Hat Enterprise Linux. This means that all the 
+software programs
+(executable) that the end-user wants to run on the HPC first must be


executables

stdweird · 2024-08-29T15:39:12Z

...processor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1.txt

+Operating system, which is a specific version of Red Hat Enterprise Linux. This means that all the 
+software programs
+(executable) that the end-user wants to run on the HPC first must be
+compiled for RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty). It also means that you first have to install all the


is there a macro for rhel without the clusters? or have value RHEL 8.8 (on clusters ...)

stdweird · 2024-08-29T15:42:27Z

...rocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_12.txt

+Note: The Intel Parallel Studio Cluster Edition contains equivalent
+compilers for all GNU compilers. Hereafter the overview for C, C++ and
+Fortran compilers.
+|             | Sequential Program |           | **Parallel Program (with MPI)** |           |


this table needs some explanation if you want to keep it around

stdweird · 2024-08-29T15:48:16Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1.txt

+Connecting to the HPC infrastructure
+Before you can really start using the HPC clusters, there are several things
+you need to do or know:
+1.  You need to log on to the cluster using an SSH client to one of


put the webportal before the ssh client.

stdweird · 2024-08-29T15:48:55Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1.txt

+    that you need from your desktop computer to the cluster. At the end
+    of a job, you might want to transfer some files back.
+3.  Optionally, if you wish to use programs with a **graphical user
+    interface**, you will need an X-server on your client system and log


wait, do we still recommend this for GUI usage? @boegel @hajgato

stdweird · 2024-08-29T15:50:05Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1.txt

+3.  Optionally, if you wish to use programs with a **graphical user
+    interface**, you will need an X-server on your client system and log
+    in to the login nodes with X-forwarding enabled.
+4.  Often several versions of software packages and libraries are


Extra starting line to explain that this is about the software you want to use

stdweird · 2024-08-29T15:52:15Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_10.txt

+$ exit
+logout
+Connection to login.hpc.ugent.be closed.
+ tip "tip: Setting your Language right"


tip tip? maybe add the word locale here? this is not only about language. and one or lines introduction what a locale is

stdweird · 2024-08-29T15:53:29Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_15.txt

@@ -0,0 +1,7 @@
+Fast file transfer for large datasets
+See the section on rsync in chapter 5 of the Linux intro manual.


chapter 5 -> "file uploading" chapter

what is the "intro manual"?

stdweird · 2024-08-29T15:56:54Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_15.txt

+Fast file transfer for large datasets
+See the section on rsync in chapter 5 of the Linux intro manual.
+Changing login nodes
+It can be useful to have control over which login node you are on. However, when you connect to the HPC (High-Performance Computing) system, you are directed to a random login node, which might not be the one where you already have an active session. To address this, there is a way to manually switch your active login node.


HPC system? consistent terminology please. also not sure you still need to explain what HPC means at this point in the manual

stdweird · 2024-08-29T15:57:53Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_16.txt

+$ hostname
+gligar08.gastly.os
+Rather than always starting a new session on the HPC, you can also use a terminal multiplexer like screen or tmux.
+These can make sessions that 'survives' across disconnects.


maybe we should also promote or recommend the webportal equivalnets?

stdweird · 2024-08-29T15:58:59Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_2.txt

@@ -0,0 +1,18 @@
+Connection restrictions
+Since March 20th 2020, restrictions are in place that limit from where


this can be temoved i guess? not sure you make footnotes, but imho this is not relevant anymore

stdweird · 2024-08-29T16:00:19Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_2.txt

+All other IP domains are blocked by default. If you are connecting from
+an IP address that is not allowed direct access, you have the following
+options to get access to VSC login nodes:
+-   Use an VPN connection to connect to UGent the network (recommended). See <https://helpdesk.ugent.be/vpn/en/> for more information. 


@EwDa291 more <>

stdweird · 2024-08-29T16:00:47Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_2.txt

+an IP address that is not allowed direct access, you have the following
+options to get access to VSC login nodes:
+-   Use an VPN connection to connect to UGent the network (recommended). See <https://helpdesk.ugent.be/vpn/en/> for more information. 
+-   Whitelist your IP address automatically by accessing


mention that this is streamlined when the webportal is used

stdweird · 2024-08-29T16:03:41Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_3.txt

+connect, with an error message like:
+ssh_exchange_identification: read: Connection reset by peer
+First Time connection to the HPC infrastructure
+The remaining content in this chapter is primarily focused for people utilizing a terminal with SSH.


for consitency, and if this truly only for ssh, make a ssh chapter like there is one for the webportal.

stdweird · 2024-08-29T16:05:11Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_8.txt

+cd examples
+ tip
+    Typing cd ex followed by tab (the Tab-key) will generate the cd examples
+    command. Command-line completion (also tab completion) is a common feature of the bash command


also tab -> also known as tab

stdweird · 2024-08-29T16:06:49Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_8.txt

+The first action is to copy the contents of the HPC examples directory to
+your home directory, so that you have your own personal copy and that
+you can start using the examples. The "-r" option of the copy command
+will also copy the contents of the sub-directories "recursively".


add (which is not done by default)

stdweird · 2024-08-29T16:08:20Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_9.txt

@@ -0,0 +1,19 @@
+cp -r /apps/gent/tutorials/Intro-HPC/examples ~/
+Go to your home directory, check your own private examples directory, ... and start working.
+cd


missing a cd experiment1 or so?

stdweird · 2024-08-29T16:09:11Z

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_9.txt

+         accelgor           4      3      2       9        18        1
+          donphan           0      0     16      16        16       13
+          gallade           2      0      5      16        19      136
+For a full view of the current loads and queues see:


queues are not shown here. they also don't exist anymore ;)

EwDa291 · 2024-08-30T07:28:14Z

@EwDa291 counting tokens is with something like
import tiktoken

def num_tokens_from_string(string: str, encoding_name: str) -> int:
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

print(num_tokens_from_string("Hello world, let's test tiktoken.", "cl100k_base"))
it's not clear what the dependencies of tiktoken are (it's odd there are so few). you when you do a pip isnstall, capture the output and maybe paste a list here.

The output i got when doing pip install is this:

Collecting tiktoken
  Downloading tiktoken-0.7.0-cp312-cp312-win_amd64.whl.metadata (6.8 kB)
Collecting regex>=2022.1.18 (from tiktoken)
  Downloading regex-2024.7.24-cp312-cp312-win_amd64.whl.metadata (41 kB)
Collecting requests>=2.26.0 (from tiktoken)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting charset-normalizer<4,>=2 (from requests>=2.26.0->tiktoken)
  Using cached charset_normalizer-3.3.2-cp312-cp312-win_amd64.whl.metadata (34 kB)
Collecting idna<4,>=2.5 (from requests>=2.26.0->tiktoken)
  Using cached idna-3.8-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests>=2.26.0->tiktoken)
  Using cached urllib3-2.2.2-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests>=2.26.0->tiktoken)
  Downloading certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
Downloading tiktoken-0.7.0-cp312-cp312-win_amd64.whl (799 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 799.3/799.3 kB 1.9 MB/s eta 0:00:00
Downloading regex-2024.7.24-cp312-cp312-win_amd64.whl (269 kB)
Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Downloading certifi-2024.8.30-py3-none-any.whl (167 kB)
Using cached charset_normalizer-3.3.2-cp312-cp312-win_amd64.whl (100 kB)
Using cached idna-3.8-py3-none-any.whl (66 kB)
Using cached urllib3-2.2.2-py3-none-any.whl (121 kB)
Installing collected packages: urllib3, regex, idna, charset-normalizer, certifi, requests, tiktoken
Successfully installed certifi-2024.8.30 charset-normalizer-3.3.2 idna-3.8 regex-2024.7.24 requests-2.32.3 tiktoken-0.7.0 urllib3-2.2.2

…of paragraphs

EwDa291 and others added 3 commits August 8, 2024 11:27

initial commit

1ebc363

Merge branch 'hpcugent:main' into chatbot_parser

34df842

some cleanup

10edb20

EwDa291 and others added 26 commits August 9, 2024 10:39

used jinja to replace macros

85a93ec

adapt if-mangler to accommodate for nested if-clauses

dfff5fa

adapt the parser to take all files as input, not all files get parsed…

649ddec

… successfully yet

adapt the parser to take all files as input, not all files get parsed…

2116d6e

… successfully yet

small update, not important

159aa62

change to the templates

75765e5

change to accommodate for more nested if-clauses

57d9cfe

Delete scripts/HPC chatbot preprocessor/start_checker.py

75d345b

This file is just used to test some things locally and not part of the parser

make sure files with duplicate names between normal files and linux-t…

ff7a9fc

…utorial are both read and saved properly

Merge branch 'chatbot_parser' of https://github.com/EwDa291/vsc_user_…

47a33b7

…docs into chatbot_parser

fixed the problem of some files being written in reST instead of mark…

7d279d6

…down

some small fixes

8047572

remove try-except-structure

7d1c5ed

collapse all code into one file

984b0cd

Rename file

8f5eeaa

cleanup repository

2b97b7a

Rename directory

b595301

add a main function

90c8ab7

make file paths non os-specific

b8ae706

use docstrings to document the functions

b751497

rewrite the if-mangler to make it more readable

0f8eb5d

got rid of most global variables

9938e92

fixed some issues with if statements

508b22c

fixed some issues with if statements

a25ce2d

got rid of all global variables

80d0535

small changes to make file more readable

9163a75

stdweird reviewed Aug 29, 2024

View reviewed changes

dropped <> around links and started new function to calculate length …

170a10c

…of paragraphs

boegel marked this pull request as draft August 30, 2024 08:12

boegel changed the title ~~Parser script for the chatbot input~~ Parser script for the chatbot input (DO NOT REVIEW/MERGE, SEE PR #675) Aug 30, 2024

strip out code & co so only parsed MDs for selected pages remain

f279701

boegel force-pushed the chatbot_parser branch from 97003d5 to f279701 Compare August 30, 2024 08:16

boegel changed the title ~~Parser script for the chatbot input (DO NOT REVIEW/MERGE, SEE PR #675)~~ parsed mds by for chatbot parser script for account.md, connecting.md, compiling_your_software.md (FOR REVIEW ONLY, NO MERGE!) Aug 30, 2024

stdweird mentioned this pull request Sep 8, 2024

Chatbot parser generic output part 1 #676

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parsed mds by for chatbot parser script for account.md, connecting.md, compiling_your_software.md (FOR REVIEW ONLY, NO MERGE!) #664

parsed mds by for chatbot parser script for account.md, connecting.md, compiling_your_software.md (FOR REVIEW ONLY, NO MERGE!) #664

EwDa291 commented Aug 8, 2024

EwDa291 commented Aug 8, 2024 •

edited

Loading

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

stdweird Aug 29, 2024

EwDa291 commented Aug 30, 2024

		@@ -0,0 +1,10 @@
		Compiling and testing your software on the HPC
		All nodes in the HPC cluster are running the "RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty)"

		@@ -0,0 +1,7 @@
		Fast file transfer for large datasets
		See the section on rsync in chapter 5 of the Linux intro manual.

		@@ -0,0 +1,18 @@
		Connection restrictions
		Since March 20th 2020, restrictions are in place that limit from where

parsed mds by for chatbot parser script for account.md, connecting.md, compiling_your_software.md (FOR REVIEW ONLY, NO MERGE!) #664

Are you sure you want to change the base?

parsed mds by for chatbot parser script for account.md, connecting.md, compiling_your_software.md (FOR REVIEW ONLY, NO MERGE!) #664

Conversation

EwDa291 commented Aug 8, 2024

EwDa291 commented Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EwDa291 commented Aug 30, 2024

EwDa291 commented Aug 8, 2024 •

edited

Loading