Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[API] Test statistics query effficiency needs improving #7778

Open
2 of 6 tasks
SchrodingersGat opened this issue Jul 31, 2024 · 6 comments · May be fixed by #8487
Open
2 of 6 tasks

[API] Test statistics query effficiency needs improving #7778

SchrodingersGat opened this issue Jul 31, 2024 · 6 comments · May be fixed by #8487
Labels
api Relates to the API bug Identifies a bug which needs to be addressed plugin Plugin ecosystem refactor
Milestone

Comments

@SchrodingersGat
Copy link
Member

Please verify that this bug has NOT been raised before.

  • I checked and didn't find a similar issue

Describe the bug*

New test statistics API endpoint needs optimizing for number of database hits:

Ref: #7164

image

image

Steps to Reproduce

  • Open "test statistics" tab (e.g. for a build order)
  • View output of server logs (with query count debugging turned on)

Expected behaviour

API / database queries should be annotated to prevent duplicate queries

Deployment Method

  • Docker
  • Package
  • Bare metal
  • Other - added info in Steps to Reproduce

Version Information

Version Information:

InvenTree-Version: 0.16.0 dev
Django Version: 4.2.14
Commit Hash: c2c66c5
Commit Date: 2024-07-31
Commit Branch: build-test-results
Database: postgresql
Debug-Mode: True
Deployed using Docker: True
Platform: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with
Installer: DOC

Active plugins: [{'name': 'InvenTreeBarcode', 'slug': 'inventreebarcode', 'version': '2.1.0'}, {'name': 'InvenTreeCoreNotificationsPlugin', 'slug': 'inventreecorenotificationsplugin', 'version': '1.0.0'}, {'name': 'InvenTreeCurrencyExchange', 'slug': 'inventreecurrencyexchange', 'version': '1.0.0'}, {'name': 'InvenTreeLabel', 'slug': 'inventreelabel', 'version': '1.1.0'}, {'name': 'InvenTreeLabelMachine', 'slug': 'inventreelabelmachine', 'version': '1.0.0'}, {'name': 'InvenTreeLabelSheet', 'slug': 'inventreelabelsheet', 'version': '1.0.0'}, {'name': 'DigiKeyPlugin', 'slug': 'digikeyplugin', 'version': '1.0.0'}, {'name': 'LCSCPlugin', 'slug': 'lcscplugin', 'version': '1.0.0'}, {'name': 'MouserPlugin', 'slug': 'mouserplugin', 'version': '1.0.0'}, {'name': 'TMEPlugin', 'slug': 'tmeplugin', 'version': '1.0.0'}]

Please verify if you can reproduce this bug on the demo site.

  • I can reproduce this bug on the demo site.

Relevant log output

http://localhost:8000/api/test-statistics/by-build/1/
|------|-----------|----------|----------|----------|------------|
| Type | Database  |   Reads  |  Writes  |  Totals  | Duplicates |
|------|-----------|----------|----------|----------|------------|
| REQU |  default  |    1     |    0     |    1     |     0      |
|------|-----------|----------|----------|----------|------------|
| RESP |  default  |   129    |    0     |   129    |    122     |
|------|-----------|----------|----------|----------|------------|
Total queries: 130 in 0.1329s 


Executed 60 time(s).
SELECT COUNT(*) AS "__count" FROM "stock_stockitemtestresult" WHERE
("stock_stockitemtestresult"."stock_item_id" = #number# AND
"stock_stockitemtestresult"."template_id" = #number# AND
"stock_stockitemtestresult"."result")
@SchrodingersGat SchrodingersGat added bug Identifies a bug which needs to be addressed question This is a question triage:not-checked Item was not checked by the core team api Relates to the API refactor and removed triage:not-checked Item was not checked by the core team labels Jul 31, 2024
@SchrodingersGat SchrodingersGat added this to the 0.16.0 milestone Jul 31, 2024
@SchrodingersGat
Copy link
Member Author

@martonmiklos FYI

@SchrodingersGat
Copy link
Member Author

When viewing stats by part it gets even worse:

image

@SchrodingersGat SchrodingersGat removed the question This is a question label Jul 31, 2024
@martonmiklos
Copy link
Contributor

Uh, I will look into it. How did you activated this sql debug output?

@SchrodingersGat
Copy link
Member Author

Use the env var INVENTREE_DEBUG_QUERYCOUNT=True or in your config file, debug_querycount=True

@martonmiklos
Copy link
Contributor

martonmiklos commented Aug 1, 2024

Use the env var INVENTREE_DEBUG_QUERYCOUNT=True or in your config file, debug_querycount=True

I have reworked the queries to use joins, however I have not been able to use the debug output as I wanted.
I set the env var (tried both the config.yaml too):
image

I tried to send the statistic requests (which were server properly) right after the server started, however the SQL debug output was not activated. However when I logged in the first (notification list query I think) got debugged.

Is there any way to set the SQL debug for each query? Or how can I specify which query should it debug?

@wolflu05
Copy link
Contributor

wolflu05 commented Aug 1, 2024

There are certain thresholds defined (>25 queries or >0.1s). Set them to a lower value (1 for to log for all requests). You can find them in the settings.py right after the implementation for the previously mentioned querycount enable var. just search with your code editor.

martonmiklos added a commit to martonmiklos/InvenTree that referenced this issue Aug 1, 2024
@SchrodingersGat SchrodingersGat modified the milestones: 0.16.0, 0.17.0 Aug 10, 2024
@SchrodingersGat SchrodingersGat added the plugin Plugin ecosystem label Nov 12, 2024
@SchrodingersGat SchrodingersGat modified the milestones: 0.17.0, 1.0.0 Nov 12, 2024
@SchrodingersGat SchrodingersGat linked a pull request Nov 15, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api Relates to the API bug Identifies a bug which needs to be addressed plugin Plugin ecosystem refactor
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants