Support for JOR files #141

lefessan · 2024-03-20T10:00:24Z

JOR files are journals of COBOL executions. This commit implements
minimalist support for JOR files with the following features:

extensible: the format can be easily extended in the future
compact: the format is binary and does not take too much space on disk

Currently, the JOR file contains the arguments, the execution start
and stop times and per-file statistics of opens, reads, writes, starts
and closes operations.

Initial work by Emilien Lemaire <[email protected]> Additional features: * Do not check for enter/exit section: instead, sum time from paragraphs * Support for modules, CALL and ENTRY points * Support for recursive calls * Allocate virtual stack on demand instead of statically * Correct handling of EXIT PARAGRAPH code with 'goto' * Prevent CANCEL from dlclose a module during profiling * Customize CSV result file with COB_PROF_FORMAT * Customize CSV filename using $b/$f/$d/$t * Add some tests for RECURSIVE on PROGRAM-ID

GitMensch

obviously this has one commit too much in currently; for the first draft see some notes in the review; the biggest question is (likely answered in NEWS and gnucobol.texi): why should a user want that, what is its practical purpose?

Should there be binaries under /bin that take an existing jor file and convert it to xml/json/text?

GitMensch · 2024-03-20T10:05:46Z

libcob/common.c

@@ -3050,6 +3056,7 @@ cob_stop_run (const int status)
 		longjmp (return_jmp_buf, 1);
 	}
 #endif
+	cob_jor_exit (status, "STOP RUN reached");


you'd like to log the status here as well

GitMensch · 2024-03-20T10:07:37Z

libcob/common.h

+enum cob_jor_file_operation {
+/* order must match field_file_op_name[] in jor.c */
+	COB_JOR_WRITE_TRY = 0,
+	COB_JOR_WRITE_OK,
+	COB_JOR_READ_TRY,
+	COB_JOR_READ_OK,
+	COB_JOR_START_TRY,
+	COB_JOR_START_OK,
+	COB_JOR_OPEN_TRY,
+	COB_JOR_OPEN_OK,
+	COB_JOR_CLOSE_TRY,
+	COB_JOR_CLOSE_OK,
+	COB_JOR_AFTER_LAST_OPERATION
+};


put that above, add a trailing "_MAX" entry and use that max entry as index instead of COB_JOR_FILE_OPERATIONS

GitMensch · 2024-03-20T10:08:28Z

libcob/common.h

+COB_EXPIMP void cob_jor_exit (int, const char*, ...);
+COB_EXPIMP void cob_jor_file_operation (cob_file*, enum cob_jor_file_operation);
+
+/* For extensions to update the way JOR are allocated and saved/sent. */
+struct cob_jor_funcs {
+	void (*save) (const char* filename, char* buffer, int len);
+	char* (*allocate)(int *size);
+};
+COB_EXPIMP void cob_jor_set_funcs (struct cob_jor_funcs *);


that's not external, so move tgat part to coblocal

GitMensch · 2024-03-20T10:10:31Z

libcob/jor.c

+   Copyright (C) 2023 Free Software Foundation, Inc.
+   Written by Emilien Lemaire and Fabrice Le Fessant.


So you did start that last year already, together with Emilien?

GitMensch · 2024-03-20T10:11:52Z

libcob/jor.c

+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <stdarg.h>
+
+#include "config.h"
+
+/* include internal and external libcob definitions, forcing exports */
+#define COB_LIB_EXPIMP
+#include "coblocal.h"
+
+#include "tarstamp.h"
+#include "common.h"
+
+#include <sys/types.h>
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+#ifdef	_WIN32
+#define WIN32_LEAN_AND_MEAN
+#include <windows.h>
+#undef MOUSE_MOVED
+#include <process.h>
+#include <io.h>
+#include <fcntl.h>	/* for _O_BINARY only */
+#endif
+
+#include <sys/time.h>


as clangd and some compilers will tell you - these are not all used in the source, please cleanup

GitMensch · 2024-03-20T10:18:27Z

libcob/jor.c

+	if (COB_JOR_FILE_OPERATIONS != COB_JOR_AFTER_LAST_OPERATION){
+		fprintf (stderr, "COB_JOR_FILE_OPERATIONS = %d\n",
+			 COB_JOR_FILE_OPERATIONS);
+		fprintf (stderr, "COB_JOR_AFTER_LAST_OPERATION = %d\n",
+			 COB_JOR_AFTER_LAST_OPERATION);
+		exit (2);
+	}
+	if ( field_file_op_name[COB_JOR_FILE_OPERATIONS-1] == NULL ){
+		fprintf (stderr,
+			 "field_file_op_name[%d] not initialized\n",
+			 COB_JOR_FILE_OPERATIONS-1);
+		exit (2);
+	}


use the common abort and message functions here or drop that part

GitMensch · 2024-03-20T10:21:29Z

libcob/jor.c

+	if (!cobsetptr->cob_jor_enable &&
+	    /* testsuite clears COB_JOR_ENABLE... */
+	    !getenv ("COB_JOR_ENABLED")) return ;


that will be already be handled in the settings pointer, so this part can be removed

GitMensch · 2024-03-20T10:23:06Z

libcob/fileio.c

cob_jor_file_operation is called often, so I think we may check !cobsetptr->cob_jor_enable in this file instead (saving the function call and its resulting stack/register changes on in and out), possibly wrap that in a macro so you keep the single lines here. But I've done no perf tracing on this...

GitMensch · 2024-03-20T10:30:31Z

libcob/jor.c

+{
+	int i;
+
+	if (!!cobsetptr) return ;


that would be an error, I therefore would suggest to drop that

GitMensch · 2024-03-20T10:33:31Z

libcob/jor.c

the outline at the start is well done; please add doxygen comments to each of the functions as well

GitMensch · 2024-03-20T11:13:22Z

What do you think of moving the stats to be only/mostly set on save_status as per Ron's implementation OCamlPro/gnucobol4@4a61f7c OCamlPro/gnucobol4@24a3751?

What about merging the fileio stats (only active for programs that are compiled with tracing) to GC 3.3 (that would also include OCamlPro/gnucobol4@980d0b4)?

lefessan · 2024-03-20T14:55:19Z

The idea behind this PR is to be able to get a minimal trace of the execution, in production mode. This is not possible with standard tracing options that are too expensive. You can imagine a batch failing during the night, people might want to understand what happened exactly during the run, and maybe to preceding steps, without running them again in trace mode. Also, you get information like the time spent, the IO stats, and probably more in the future.

There are maybe other solutions, but I think it's worth investigating this mode.

And yes, there needs to be some program that will translate these binary files into json or something like that...

By the way, I am not really targeting 3.3, ideally, it's something that could land in 4.0 once 3.3 has been released.

GitMensch · 2024-03-20T16:22:50Z

By the way, I am not really targeting 3.3, ideally, it's something that could land in 4.0 once 3.3 has been released.

Ah, then please edit the PR to have that based on master then.
Still, it would be useful to check the review comments, including the question to do it at the same place: save_status and in 4.x consider to provide the option to disable/enable the file recording by file (using the IO_ options, see referenced commits).

This is not possible with standard tracing options that are too expensive.

We still can consider to use these as basis -> check for a new trace value "binary" and in this case do the tracing you currently do.

In any case I suggest to write a plan (which includes your binary format), then send that to the developer mailing list for additional input - especially when you target trunk.

JOR files are journals of COBOL executions. This commit implements minimalist support for JOR files with the following features: * extensible: the format can be easily extended in the future * compact: the format is binary and does not take too much space on disk Currently, the JOR file contains the arguments, the execution start and stop times and per-file statistics of opens, reads, writes, starts and closes operations.

lefessan marked this pull request as draft March 20, 2024 10:19

GitMensch requested changes Mar 20, 2024

View reviewed changes

lefessan force-pushed the z-2024-03-18-jor branch from 5dda38e to 1501659 Compare March 25, 2024 16:53

Add bin/jorread to translate JOR files to JSON

8298eea

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for JOR files #141

Support for JOR files #141

lefessan commented Mar 20, 2024

GitMensch left a comment

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch Mar 20, 2024

GitMensch commented Mar 20, 2024

lefessan commented Mar 20, 2024

GitMensch commented Mar 20, 2024

		Copyright (C) 2023 Free Software Foundation, Inc.
		Written by Emilien Lemaire and Fabrice Le Fessant.

Support for JOR files #141

Are you sure you want to change the base?

Support for JOR files #141

Conversation

lefessan commented Mar 20, 2024

GitMensch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GitMensch commented Mar 20, 2024

lefessan commented Mar 20, 2024

GitMensch commented Mar 20, 2024