Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User tracking #3419

Closed
wants to merge 26 commits into from
Closed

User tracking #3419

wants to merge 26 commits into from

Conversation

sjanssen2
Copy link
Contributor

This PR adds the ability to define some JavaScript code for visitor tracking of Qiita. Raw code, i.e. contents of variable JS_CODE in config section user_tracking is injected as top element in <body> of the sitebase.html template.

We, i.e. NFDI4Microbiota will use this mechanism to measure performance indicators.

sjanssen2 and others added 26 commits March 21, 2024 09:31
…ots#3384)

* Update CHANGELOG.md

* make help email address configurable instead of hard coding

* codestyle

* don't default with email addresses but raise Errors

* set mail addresses in example configuration, which is also used for
testing

* addessing Antonios issues + switch to a non deprecated read fct.

* updating dummy help email address in tests

---------

Co-authored-by: Antonio Gonzalez <[email protected]>
* Initial commmit to resource allocation plots

* Update to util.py and tests

* Update test_util.py

* revert 072023.ipynb

* Debug test_util

* testing plot based on № failures

* Addressed Antonio's comments

* Updates according to @charles-cowart comments

* define functions as const @charles-cowart
* Update CHANGELOG.md

* add trailing/separating whitespace

got the error `Error running SQL: SYNTAX_ERROR. MSG: trailing junk after numeric literal at or near "1A" LINE 6: WHERE study_id = 1AND artifact_type = '... ^ ` on my test instance

log says
```
==> bcf_qiita/Logs/qiita_21174.log <==
[E 240322 13:35:00 web:1669] Uncaught exception GET /study/description/1 (127.0.0.1)
    HTTPServerRequest(protocol='https', host='qiita.jlab.bio', method='GET', uri='/study/description/1', version='HTTP/1.0', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 256, in _execute
        cur.execute(sql, sql_args)
      File "/homes/sjanssen/bcf_qiita/envs/qiita/lib/python3.9/site-packages/psycopg2/extras.py", line 146, in execute
        return super().execute(query, vars)
    psycopg2.errors.SyntaxError: trailing junk after numeric literal at or near "1A"
    LINE 6:                      WHERE study_id = 1AND artifact_type = '...
                                                  ^
    
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 199, in _raise_execution_error
        raise ValueError(
    ValueError: Error running SQL: SYNTAX_ERROR. MSG: trailing junk after numeric literal at or near "1A"
    LINE 6:                      WHERE study_id = 1AND artifact_type = '...
                                                  ^
    
    
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/envs/qiita/lib/python3.9/site-packages/tornado/web.py", line 1590, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/homes/sjanssen/bcf_qiita/envs/qiita/lib/python3.9/site-packages/tornado/web.py", line 3006, in wrapper
        return method(self, *args, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_pet/handlers/study_handlers/base.py", line 28, in get
        study_info = study_get_req(study, self.current_user.id)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_pet/handlers/api_proxy/studies.py", line 120, in study_get_req
        study.artifacts(artifact_type='BIOM')) != 0
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/study.py", line 1108, in artifacts
        for aid in qdb.sql_connection.TRN.execute_fetchflatten()]
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 49, in wrapper
        return func(self, *args, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 389, in execute_fetchflatten
        return list(chain.from_iterable(self.execute()[idx]))
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 49, in wrapper
        return func(self, *args, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 313, in execute
        return self._execute()
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 260, in _execute
        self._raise_execution_error(sql, sql_args, e)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 205, in _raise_execution_error
        raise ValueError("Error running SQL query: %s" % str(error))
    ValueError: Error running SQL query: Error running SQL: SYNTAX_ERROR. MSG: trailing junk after numeric literal at or near "1A"
    LINE 6:                      WHERE study_id = 1AND artifact_type = '...
                                                  ^
    
    
[E 240322 13:35:00 web:2161] 500 GET /study/description/1 (127.0.0.1) 159.63ms
[E 240322 13:35:00 web:1114] Uncaught exception in write_error
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 256, in _execute
        cur.execute(sql, sql_args)
      File "/homes/sjanssen/bcf_qiita/envs/qiita/lib/python3.9/site-packages/psycopg2/extras.py", line 146, in execute
        return super().execute(query, vars)
    psycopg2.errors.SyntaxError: trailing junk after numeric literal at or near "1A"
    LINE 6:                      WHERE study_id = 1AND artifact_type = '...
                                                  ^
    
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 199, in _raise_execution_error
        raise ValueError(
    ValueError: Error running SQL: SYNTAX_ERROR. MSG: trailing junk after numeric literal at or near "1A"
    LINE 6:                      WHERE study_id = 1AND artifact_type = '...
                                                  ^
    
    
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/envs/qiita/lib/python3.9/site-packages/tornado/web.py", line 1590, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/homes/sjanssen/bcf_qiita/envs/qiita/lib/python3.9/site-packages/tornado/web.py", line 3006, in wrapper
        return method(self, *args, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_pet/handlers/study_handlers/base.py", line 28, in get
        study_info = study_get_req(study, self.current_user.id)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_pet/handlers/api_proxy/studies.py", line 120, in study_get_req
        study.artifacts(artifact_type='BIOM')) != 0
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/study.py", line 1108, in artifacts
        for aid in qdb.sql_connection.TRN.execute_fetchflatten()]
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 49, in wrapper
        return func(self, *args, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 389, in execute_fetchflatten
        return list(chain.from_iterable(self.execute()[idx]))
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 49, in wrapper
        return func(self, *args, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 313, in execute
        return self._execute()
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 260, in _execute
        self._raise_execution_error(sql, sql_args, e)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 205, in _raise_execution_error
        raise ValueError("Error running SQL query: %s" % str(error))
    ValueError: Error running SQL query: Error running SQL: SYNTAX_ERROR. MSG: trailing junk after numeric literal at or near "1A"
    LINE 6:                      WHERE study_id = 1AND artifact_type = '...
                                                  ^
    
    
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 256, in _execute
        cur.execute(sql, sql_args)
      File "/homes/sjanssen/bcf_qiita/envs/qiita/lib/python3.9/site-packages/psycopg2/extras.py", line 146, in execute
        return super().execute(query, vars)
    psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pk_logging"
    DETAIL:  Key (logging_id)=(294) already exists.
    
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 199, in _raise_execution_error
        raise ValueError(
    ValueError: Error running SQL: UNIQUE_VIOLATION. MSG: duplicate key value violates unique constraint "pk_logging"
    DETAIL:  Key (logging_id)=(294) already exists.
    
    
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/homes/sjanssen/bcf_qiita/envs/qiita/lib/python3.9/site-packages/tornado/web.py", line 1112, in send_error
        self.write_error(status_code, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_pet/handlers/base_handlers.py", line 69, in write_error
        LogEntry.create(
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/logger.py", line 103, in create
        return cls(qdb.sql_connection.TRN.execute_fetchlast())
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 49, in wrapper
        return func(self, *args, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 336, in execute_fetchlast
        return self.execute()[-1][0][0]
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 49, in wrapper
        return func(self, *args, **kwargs)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 313, in execute
        return self._execute()
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 260, in _execute
        self._raise_execution_error(sql, sql_args, e)
      File "/homes/sjanssen/bcf_qiita/qiita-spots/qiita/qiita_db/sql_connection.py", line 205, in _raise_execution_error
        raise ValueError("Error running SQL query: %s" % str(error))
    ValueError: Error running SQL query: Error running SQL: UNIQUE_VIOLATION. MSG: duplicate key value violates unique constraint "pk_logging"
    DETAIL:  Key (logging_id)=(294) already exists.
```

---------

Co-authored-by: Antonio Gonzalez <[email protected]>
* Update CHANGELOG.md

* let user configure the center point for the stats world map

---------

Co-authored-by: Antonio Gonzalez <[email protected]>
* Update CHANGELOG.md

* add db changes including fix qiita-spots#3368
* Update CHANGELOG.md

* PacBio_SMRT new instruments
* Update CHANGELOG.md

* adding current_human_filtering to PrepTemplate
* Update CHANGELOG.md

* fix qiita-spots#3352
* Update CHANGELOG.md

* fix qiita-spots#3387
* Update CHANGELOG.md

* fix qiita-spots#3376
* Update CHANGELOG.md

* updating QiitaPrepTemp06182020.xlsx
* Update CHANGELOG.md

* add reprocess_job_id to PrepTemplate
* Update CHANGELOG.md

* clean plugins

* patch 54.sql

* fix Internal Qiita jobs

* sample_template.generate_files

* rm extra filepath

* qiita.filepath_filepath_id_seq

* basedir_len

* basedir_len * 2

* basedir_len * 3

* get_db_files_base_dir

* address @charles-cowart comment
* Update CHANGELOG.md

* clean plugins

* patch 54.sql

* fix Internal Qiita jobs

* sample_template.generate_files

* rm extra filepath

* qiita.filepath_filepath_id_seq

* basedir_len

* basedir_len * 2

* basedir_len * 3

* get_db_files_base_dir

* adding test_db_sql

* missing `

* echo $QIITA_CONFIG_FP

* -p 32768

* -l localhost

* pgport

* testing 1

* testing 2

* localhost

* testing 3

* -h localhost

* -W

* PGPASSWORD

* doc

* flake8

* mods after chatting with @charles-cowart

* exit 1

* "$row_counts"

* #

* `echo $row_counts`

* full tests
* Update CHANGELOG.md

* Note about missing mod_zip

As I wasted too much time in figuring out what's wrong with anonymous link downloads, I thought it would be handy to summarize my lessons learned for others

* add git

"secretly" adding git as an Ubuntu package to be installed in a fresh system. Git later referred to for obtaining the Qiita sources.

---------

Co-authored-by: Antonio Gonzalez <[email protected]>
* Update CHANGELOG.md

* fix qiita-spots#3381

* fallback

* rm qiimp from test
* Update to DB qiita.slurm_resource_allocations

* connected tests to database

* Update util.py

* debugging changes to test

* Update test_util.py

* Update test_util.py

* Tests update

* Update test_meta_util.py

* Updates to @antgonza comments

* Updates to @charles-cowart comments
* Update CHANGELOG.md

* fix admin processing_job msg None

* update keys
* user profile can now also store social handles from google scholar, ORCID and ResearchGate

* set DEFAULT NULL to new qiita.qiita_user columns

* extending existing test

* extending tests

* add tests for validator functions for WTform StringFields

* operate on str instead of field.data
* Update to DB qiita.slurm_resource_allocations

* connected tests to database

* Update util.py

* debugging changes to test

* Update test_util.py

* Update test_util.py

* Tests update

* Update test_meta_util.py

* Updates to @antgonza comments

* Updates to @charles-cowart comments

* Resource Allocation Insertion into DB

- Added functionality of updating resource allocation data into db in qiita_db/util.py
- Added tests for the added functionality in qiita_db/test/test_util.py
- Moved MaxRSS_helper function from qiita_core/util.py to qiita_db/util.py.
- Moved MaxRSS_helper test from qiita_core/tests/test_util.py to qiita_db/test/test_util.py

* Debug Memory Model

* Update test_util.py

* Added columns to slurm_resource_allocation table

* Update test_processing_job.py

* Update test_util.py

* Automatic date adjustment in utill.py

* Update util.py
* use SQL's NOW() fct to log creation of a new user account (validated or not).

* at -> on

* add new col

* use patch for DB changes

* forgot to remove addition of comment here

* moving changes to patch 92

* move adding date from populate_test_db.sql to here

* revert changes, see patch

* adding table name :-/

* order in DB changed due to UPDATE of creation_timestamp for user [email protected]

* fix test
* use SQL's NOW() fct to log creation of a new user account (validated or not).

* at -> on

* add new col

* adding a new page for admins to easily list all users not yet validated and offer means to delete those

* flake8

* fix tests

* move DB changes to patch

* revert file

* no change here

* execute merge manually

* moving data insertion to test_db_sql patch

* account for additional test users
@sjanssen2 sjanssen2 closed this Jun 21, 2024
@sjanssen2
Copy link
Contributor Author

sorry, I've used the wrong fork (and branch) for this PR :-/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants