[pydoclint] Add support for Sphinx-style docstrings in the pydoclint rules.
#13286
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
| code | total | + violation | - violation | + fix | - fix |
|---|---|---|---|---|---|
| D411 | 16226 | 16225 | 1 | 0 | 0 |
| D410 | 16172 | 16172 | 0 | 0 | 0 |
| D413 | 4037 | 4025 | 12 | 0 | 0 |
| D412 | 205 | 204 | 1 | 0 | 0 |
| D214 | 12 | 12 | 0 | 0 | 0 |
| D405 | 2 | 0 | 2 | 0 | 0 |
| D409 | 1 | 0 | 1 | 0 | 0 |
Linter (preview)
ℹ️ ecosystem check detected linter changes. (+36758 -1124 violations, +0 -0 fixes in 7 projects; 47 projects unchanged)
apache/airflow (+34666 -928 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL
+ airflow/api/common/delete_dag.py:47:6: D410 [*] Missing blank line after section ("param") + airflow/api/common/delete_dag.py:48:6: D410 [*] Missing blank line after section ("param") + airflow/api/common/delete_dag.py:48:6: D411 [*] Missing blank line before section ("param") + airflow/api/common/delete_dag.py:51:6: D411 [*] Missing blank line before section ("param") + airflow/api/common/delete_dag.py:51:6: D413 [*] Missing blank line after last section ("param") + airflow/api/common/mark_tasks.py:107:6: D410 [*] Missing blank line after section ("param") + airflow/api/common/mark_tasks.py:109:6: D410 [*] Missing blank line after section ("param") + airflow/api/common/mark_tasks.py:109:6: D411 [*] Missing blank line before section ("param") + airflow/api/common/mark_tasks.py:110:6: D410 [*] Missing blank line after section ("param") + airflow/api/common/mark_tasks.py:110:6: D411 [*] Missing blank line before section ("param") + airflow/api/common/mark_tasks.py:111:6: D410 [*] Missing blank line after section ("param") ... 15388 additional changes omitted for rule D410 + airflow/api/common/mark_tasks.py:111:6: D411 [*] Missing blank line before section ("param") + airflow/api/common/mark_tasks.py:112:6: D411 [*] Missing blank line before section ("param") ... 15404 additional changes omitted for rule D411 + airflow/api/common/mark_tasks.py:119:6: D413 [*] Missing blank line after last section ("return") - airflow/api/common/mark_tasks.py:122:9: DOC201 `return` is not documented in docstring + airflow/api/common/mark_tasks.py:269:6: D413 [*] Missing blank line after last section ("param") + airflow/api/common/mark_tasks.py:299:6: D413 [*] Missing blank line after last section ("raises") - airflow/api/common/mark_tasks.py:302:9: DOC201 `return` is not documented in docstring + airflow/api/common/mark_tasks.py:353:6: D413 [*] Missing blank line after last section ("raises") - airflow/api/common/mark_tasks.py:356:9: DOC201 `return` is not documented in docstring + airflow/api/common/mark_tasks.py:440:6: D413 [*] Missing blank line after last section ("return") ... 3610 additional changes omitted for rule D413 - airflow/api/common/mark_tasks.py:446:9: DOC201 `return` is not documented in docstring - airflow/api/common/mark_tasks.py:82:5: DOC201 `return` is not documented in docstring - airflow/api/common/trigger_dag.py:107:5: DOC201 `return` is not documented in docstring ... 887 additional changes omitted for rule DOC201 + airflow/api_connexion/security.py:69:6: D412 [*] No blank lines allowed between a section header and its content ("param") - airflow/configuration.py:1245:23: DOC501 Raised exception `AirflowConfigException` missing from docstring + airflow/configuration.py:1276:1: DOC202 Docstring should not have a returns section because the function doesn't return anything + airflow/configuration.py:1575:1: DOC202 Docstring should not have a returns section because the function doesn't return anything + airflow/dag_processing/manager.py:1113:1: DOC202 Docstring should not have a returns section because the function doesn't return anything + airflow/dag_processing/manager.py:919:1: DOC202 Docstring should not have a returns section because the function doesn't return anything + airflow/dag_processing/processor.py:164:1: DOC202 Docstring should not have a returns section because the function doesn't return anything + airflow/dag_processing/processor.py:744:1: DOC202 Docstring should not have a returns section because the function doesn't return anything ... 67 additional changes omitted for rule DOC202 + airflow/decorators/bash.py:95:6: D412 [*] No blank lines allowed between a section header and its content ("param") + airflow/io/path.py:279:1: DOC502 Raised exception is not explicitly raised: `NotImplementedError` + airflow/io/path.py:310:10: D412 [*] No blank lines allowed between a section header and its content ("param") + airflow/jobs/job.py:444:6: D412 [*] No blank lines allowed between a section header and its content ("param") + airflow/macros/__init__.py:42:6: D412 [*] No blank lines allowed between a section header and its content ("param") + airflow/macros/__init__.py:61:6: D412 [*] No blank lines allowed between a section header and its content ("param") ... 168 additional changes omitted for rule D412 - airflow/providers/amazon/aws/hooks/datasync.py:108:19: DOC501 Raised exception `AirflowBadRequest` missing from docstring - airflow/providers/amazon/aws/hooks/datasync.py:202:19: DOC501 Raised exception `AirflowBadRequest` missing from docstring + airflow/providers/amazon/aws/hooks/datasync.py:229:1: DOC502 Raised exception is not explicitly raised: `ClientError` - airflow/providers/amazon/aws/hooks/datasync.py:233:19: DOC501 Raised exception `AirflowBadRequest` missing from docstring - airflow/providers/amazon/aws/hooks/datasync.py:248:19: DOC501 Raised exception `AirflowBadRequest` missing from docstring - airflow/providers/amazon/aws/hooks/datasync.py:263:19: DOC501 Raised exception `AirflowBadRequest` missing from docstring ... 17 additional changes omitted for rule DOC501 + airflow/providers/amazon/aws/hooks/datasync.py:275:1: DOC502 Raised exception is not explicitly raised: `AirflowBadRequest` - airflow/providers/fab/auth_manager/security_manager/override.py:1534:9: D405 [*] Section name should be properly capitalized ("NOTE") ... 35548 additional changes omitted for project
apache/superset (+1841 -188 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL
+ RELEASING/generate_email.py:47:6: D410 [*] Missing blank line after section ("param") + RELEASING/generate_email.py:49:6: D411 [*] Missing blank line before section ("return") + RELEASING/generate_email.py:49:6: D413 [*] Missing blank line after last section ("return") - RELEASING/generate_email.py:52:5: DOC201 `return` is not documented in docstring + superset/charts/data/api.py:448:10: D410 [*] Missing blank line after section ("param") + superset/charts/data/api.py:450:10: D411 [*] Missing blank line before section ("raises") + superset/charts/data/api.py:450:10: D413 [*] Missing blank line after last section ("raises") - superset/charts/data/api.py:456:19: DOC501 Raised exception `ValidationError` missing from docstring + superset/cli/lib.py:45:6: D410 [*] Missing blank line after section ("param") + superset/cli/lib.py:46:6: D411 [*] Missing blank line before section ("return") ... 2019 additional changes omitted for project
bokeh/bokeh (+3 -3 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL
+ src/bokeh/command/subcommands/info.py:86:6: D410 [*] Missing blank line after section ("param") + src/bokeh/command/subcommands/info.py:87:6: D411 [*] Missing blank line before section ("return") + src/bokeh/command/subcommands/info.py:87:6: D413 [*] Missing blank line after last section ("return") - src/bokeh/command/subcommands/info.py:89:5: DOC201 `return` is not documented in docstring - src/bokeh/document/document.py:767:9: DOC201 `return` is not documented in docstring - src/bokeh/document/models.py:160:9: DOC201 `return` is not documented in docstring
fronzbot/blinkpy (+182 -0 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview
+ blinkpy/api.py:117:6: D410 [*] Missing blank line after section ("param") + blinkpy/api.py:118:6: D411 [*] Missing blank line before section ("param") + blinkpy/api.py:118:6: D413 [*] Missing blank line after last section ("param") + blinkpy/api.py:128:6: D410 [*] Missing blank line after section ("param") + blinkpy/api.py:129:6: D411 [*] Missing blank line before section ("param") + blinkpy/api.py:129:6: D413 [*] Missing blank line after last section ("param") + blinkpy/api.py:140:6: D410 [*] Missing blank line after section ("param") + blinkpy/api.py:141:6: D411 [*] Missing blank line before section ("param") + blinkpy/api.py:141:6: D413 [*] Missing blank line after last section ("param") + blinkpy/api.py:157:6: D410 [*] Missing blank line after section ("param") ... 172 additional changes omitted for project
spruceid/siwe-py (+20 -0 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview
+ siwe/siwe.py:295:10: D410 [*] Missing blank line after section ("param") + siwe/siwe.py:296:10: D410 [*] Missing blank line after section ("param") + siwe/siwe.py:296:10: D411 [*] Missing blank line before section ("param") + siwe/siwe.py:297:10: D410 [*] Missing blank line after section ("param") + siwe/siwe.py:297:10: D411 [*] Missing blank line before section ("param") + siwe/siwe.py:298:10: D410 [*] Missing blank line after section ("param") + siwe/siwe.py:298:10: D411 [*] Missing blank line before section ("param") + siwe/siwe.py:300:10: D410 [*] Missing blank line after section ("param") + siwe/siwe.py:300:10: D411 [*] Missing blank line before section ("param") + siwe/siwe.py:304:10: D411 [*] Missing blank line before section ("return") ... 10 additional changes omitted for project
zulip/zulip (+20 -5 violations, +0 -0 fixes)
ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL
+ zerver/webhooks/raygun/view.py:105:6: D413 [*] Missing blank line after last section ("param") + zerver/webhooks/raygun/view.py:116:6: D410 [*] Missing blank line after section ("param") + zerver/webhooks/raygun/view.py:117:6: D411 [*] Missing blank line before section ("return") + zerver/webhooks/raygun/view.py:117:6: D413 [*] Missing blank line after last section ("return") - zerver/webhooks/raygun/view.py:144:5: DOC201 `return` is not documented in docstring + zerver/webhooks/raygun/view.py:150:6: D410 [*] Missing blank line after section ("param") + zerver/webhooks/raygun/view.py:151:6: D411 [*] Missing blank line before section ("return") + zerver/webhooks/raygun/view.py:151:6: D413 [*] Missing blank line after last section ("return") - zerver/webhooks/raygun/view.py:202:5: DOC201 `return` is not documented in docstring + zerver/webhooks/raygun/view.py:208:6: D410 [*] Missing blank line after section ("param") ... 15 additional changes omitted for project
... Truncated remaining completed project reports due to GitHub comment length restrictions
Changes by rule (12 rules affected)
| code | total | + violation | - violation | + fix | - fix |
|---|---|---|---|---|---|
| D411 | 16226 | 16225 | 1 | 0 | 0 |
| D410 | 16172 | 16172 | 0 | 0 | 0 |
| D413 | 4037 | 4025 | 12 | 0 | 0 |
| DOC201 | 1062 | 0 | 1062 | 0 | 0 |
| D412 | 205 | 204 | 1 | 0 | 0 |
| DOC202 | 93 | 93 | 0 | 0 | 0 |
| DOC501 | 45 | 0 | 45 | 0 | 0 |
| DOC502 | 26 | 26 | 0 | 0 | 0 |
| D214 | 12 | 12 | 0 | 0 | 0 |
| D405 | 2 | 0 | 2 | 0 | 0 |
| RUF100 | 1 | 1 | 0 | 0 | 0 |
| D409 | 1 | 0 | 1 | 0 | 0 |
Formatter (stable)
✅ ecosystem check detected no format changes.
Formatter (preview)
✅ ecosystem check detected no format changes.
|
Wow thanks. This is great. There are a lot of new ecosystem reports. Some of the superset violations look somewhat suspicious:
Like why are all the names empty? |
|
It might just be that the docstyle is a bit too aggressive right now. Should this be considered a sphinx docstring? Before I try to figure it out from the code. Did you gate the new code style behind preview mode? I think we have to because this change greatly increases the scope of the rule and it's probably good to get some feedback before rolling it out to everyone |
|
All the |
augustelalande
force-pushed
the
sphinx
branch
from
45ab7bf
to
c790e03
Compare
|
Looking at the airflow result I'm unsure if the parsing is correct. Could you take a look at them? Why is
Why is
The other thing I'm unsure about is whether the blank line rule between parameters should be disabled by default. All the examples I found don't require blank lines between parameters. What do you think? |
It is over indented relative to the docstring as a whole, since it is indented within the
Sphinx doesn't have sections per se, so to make the sphinxs docstrings fit into the mold in use for google and numpy docstrings, I consider each Note: It's not possible to combine all the param entries into a single param sections, since as far as I know there is no obligation that params must follow each other, although in practice they usually would. |
That makes sense but it seems confusing to users because I wouldn't consider those sections and it seems to cause many false positives in other rules. I would have to think this through more carefully but would we emit different diagnostics if we designed this from scratch? |
|
Everything sphinx can parse successfully with default settings should be fine, no? And sphinx definitely doesn't require blank lines between Here is a docstring that uses a lot of sphinx features and which is parsed correctly by sphinx. class SAM(Optimizer):
"""Sharpness Aware Minimization
Implements the 'Sharpness Aware Minimization' (SAM) algorithm introducued in
`Sharpness Aware Minimization`_) and the adaptive variant of it introduced in `ASAM`_.
We can use inline math: :math:`\\alpha`. You don't need the double backslash if
the python docstring is a raw string (r-string).
Block math:
.. math::
(a + b)^2 = a^2 + 2ab + b^2
.. note::
Implementation based on: https://github.com/cybertronai/pytorch-lamb
.. seealso:: :class:`LAMB`
.. _Sharpness Aware Minimization:
https://arxiv.org/abs/2010.01412
.. _ASAM:
https://arxiv.org/abs/2102.11600
:param base_optimizer: Base optimizer for SAM. (If the explanation goes
over two lines, the second line needs to be indented.
:param rho: Neighborhood size.
:raises ValueError: if ``rho`` is negative.
:example:
Everything associated with the example section needs to be indented, so that Sphinx
knows what belongs to the example.
.. code-block:: python
# Use AdamW as the base optimizer.
base_optimizer = AdamW(model.parameters())
# Wrap the base optimizer in SAM.
optimizer = SAM(base_optimizer)
# Closure required for recomputing the loss after computing epsilon(w).
def _closure():
return loss_function(logits=model(input), targets=targets)
loss = _closure()
loss.backward()
optimizer.step(closure=_closure)
optimizer.zero_grad()
You can also write it in this syntax:
>>> import template
>>> a = template.MainClass1()
>>> a.function1(1,1,1)
2
""" |
We probably would not emit D410 among others for sphinx style docstrings. And in fact if the user specifies sphinx as their docstring convention then those rules get disabled (each convention runs a different subset of D). The problem is when the convention is unspecified we still run it. One solution would be to disable innapropriate rules if sphinx is detected. There is some precendence for that, and I have already done it for some of them. I don't know what a "proper" implementation of "D410" would look like for sphinx, since there are no "sections" per se |
|
One of the problems is that |
MichaReiser
added
rule
|
So, is the problem here that pydocstyle and pydoclint are incompatible? I see a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add support for Sphinx-style docstrings in the
pydoclintrules. This also defines (out of necessity) the set of pydocstyle rules enforced when the sphinx style is selected. For lack of a good reference I just copied the exclusions from the pep257 selection, minusRule::SectionNotOverIndented.Part of #12434.
Resolves #12520.
Test Plan
Test cases added.