add function and variable annotations.

[alchemistcai]

Hello there!

I have added some tentative function annotations into haddock's python code for better type-hints and auto-completion supports from IDEs.

Before I continue to check them carefully,I'd like to seek your opinions on some questions:

1. function signature

Many functions have type specifications in their doc-strings.Their type specifications may be narrow.

For example,most functions' implementations accept str and Path both but they are just Path in some doc-strings;

some functions accept Iterator or Sequence but they are list in doc-strings.

Which one is better?Annotating functions in a minimum generic way to be compatible with code implementations, or same as the doc-string in a "Programming by Contract" way?

2. Literal types

I add some Literal types and Typedicts definations to narrow possibities.Some are unlikely to change in the short term but others may be too hardcode,say:

from type import Literal, TypedDict

# useful and stable
LogLevel = Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
ImgFormat = Literal["png", "pdf", "svg", "jpeg", "webp"]

# useful but may be too hardcode
class TensorPaths(TypedDict):
    """A dict contains all tensor paths."""
    tensor_psf: Path
    tensor_pdb: Path
    tensor_para_psf: Path
    tensor_para_pdb: Path
    tensor_dani_psf: Path
    tensor_dani_pdb: Path
# haddock.core.cns_path.get_tensors
def get_tensors(path: Union[str, Path]) -> TensorPaths:
    """Generate paths for tensors."""
    tensors = {
        "tensor_psf": Path(path, "tensor.psf"),
        "tensor_pdb": Path(path, "tensor.pdb"),
        "tensor_para_psf": Path(path, "tensor_para.psf"),
        "tensor_para_pdb": Path(path, "tensor_para.pdb"),
        "tensor_dani_psf": Path(path, "tensor_dani.psf"),
        "tensor_dani_pdb": Path(path, "tensor_dani.pdb"),
        }
    return tensors

As there are many funtions converting yaml,toml files or accepting **keywords,narrowing their parameters help type-hints in current codes,but maybe too strict.I perfer Any type in this case.What's your suggestions?

3. typing_extensions module

There are some useful types in typing module of python 3.10+,say Self, ParamSpec,and Concatenate.

Python use typing_extensions module to support backport.I'd like to import typing_extensions but it's not belong to python standard libriary.

Is it neccesory to add typing_extensions into requirements?

4. iteration on dicts

Not related to this topic,but I found that there are many statements are:

d = dict(...) #some dict
for key in d.keys():
    ...

xxx = list(d.keys())

I prefer for key in d and list(d) because they are conciser. d.keys() is more explict.Are there any other benefits I don't know?

[joaomcteixeira]

Hi @alchemistcai

Thanks so much for this post. I really love these discussions. ☺️

Let's go step-by-step:

Many functions have type specifications in their doc-strings.Their type specifications may be narrow.
For example,most functions' implementations accept str and Path both but they are just Path in some doc-strings;

Type specifications in docstrings are important mostly for users of the functions (that is, programmers importing these functions for their applications). Indeed, we have not revisited all functions to homogenize type specifications. This can be improved. Some functions work with both str and Path, others only with Path. All could work with both if we add a Path(str) in the beginning. When we require only Path, it is also as an effort to start treating paths as such and steady but slowly avoid using strings to represent paths. But this is more of an ideal than a requirement.

some functions accept Iterator or Sequence but they are list in doc-strings.

You are right. In these cases, we should update the docstring to Iterator, or whatever is the more general case.

Which one is better?Annotating functions in a minimum generic way to be compatible with code implementations, or same as the doc-string in a "Programming by Contract" way?

In my personal opinion, the first for sure. This is a very deep conversation, a topic full of controversy, and many things need to be explained to make that decision. Python is a dynamic-typed language. Therefore, it relies entirely on polymorphism. That's the power of Python. And that's a conscient decision. You gain tons of flexibility while developing, but you lose speed because the compiler needs to identify the types line by line. Python was designed that way.

On the other hand, you have to test your Python programs entirely on runtime. There are no compiler checks here. That's why our test cases are so heavy (and could be much more). That's why libraries like Hypothesis were developed, and test-driven development in Python is so necessary.

You can also do program-by-contract in Python. I have worked that way. The PROS: you may identify bugs faster. CONS: yet another layer to maintain. Most libraries out there addressing P-b-C are unfunctionally slow. The best is to use a decorator with assert statements. It's the least overhead. Below is an example:

def type_check(ordered_types, named_types):

    def decorator(func):

        def wrapper(*args, **kwargs):

            for arg, type_ in zip(args, ordered_types):
                assert isinstance(arg, type_)

            for arg, type_ in zip(kwargs.values(), named_types):
                assert isinstance(arg, type_)

            return func(*args, **kwargs)

        return wrapper

    return decorator


@type_check((str, float), (int, ))
def func1(a, b, *, c=0):  # the '*' is mandatory
    print(a, b, c)


func1('a', 1.0, c=3)
func1('a', 1.0, c='a')  # assertion error

Surely it can be expanded. You can also use if not... followed by raise AssertionError to ensure -O won't disable the assert statements. I like the asserts.

As there are many funtions converting yaml,toml files or accepting **keywords,narrowing their parameters help type-hints in current codes,but maybe too strict.I perfer Any type in this case.What's your suggestions?

I have no idea. Let's discuss the use of type hints instead. First, consider that haddock3 is a fast-developing code. Therefore, "enforcing" types at this moment might be adding unnecessary stress on the developers. Think about the str, Path example. I understand type hints help developers and IDEs. But, the time it takes to write a type-hint, you can write or improve the type spec in the docstring and have a double gain because that information will also be available for users of the function. I am aware there are two streams of thought in the Python world: one in favor and another against type-hints. I am a bit of the latter because I don't see any benefit in type-hints. Again, Python is a dynamic-type language. Transforming Python into a static-type language is anti-nature. By doing so, you have the worst of both worlds: the slow speed of Python and the burden of static types. If static types are that important for the program, it's best to work with a static-type language from scratch (and you gain its speed). We do "enforce" some types in the parameter files .yaml most to facilitate parameter validation and integration with the i-VRESSE project.

typing_extensions module

When I was the lead developer of haddock3, I usually opposed adding dependencies to the project. Sometimes I was too strict, but I continue having the same opinion. For a project like HADDOCK3, adding these dependencies to the run time adds a burden. But if the dependency is on the developer environment, that's okay (see our tox implementations). If you want to improve haddock3 stability, work on tests. Do tests. That's my suggestion. 😉

Is it neccesory to add typing_extensions into requirements?

If it is a dependency, I would say yes.

iteration on dicts

I always prefer to use .keys() because it is explicit. Who knows in future versions if the behavior of for k in d: changes... The .keys() method will likely remain stable over time. Obviously, if you want to benefit the most from polymorphism, you would avoid using the .keys() method.

In conclusion:

What I wrote here is mostly my personal view on using type hints. I don't see any benefit on type-hints that docstrings won't provide. Now, should you contribute with type-hints to haddock3? Currently, I am no longer the lead developer of haddock3. I keep contributing to the project actively as an external collaborator, though. Implementing type-hints adds another layer of maintenance, which is not required for the science being developed or for the correct behavior of the software. I think it depends a lot on who is maintaining the software and the time available.

However, I would appreciate it so much if you could contribute to the correct documentation of type specifications or improve polymorphism behavior wherever it is beneficial. That's my opinion.

The idea of type-hints is not totally bad. But I think hinters should parse information from well-established docstring formats, like the numpy-doc, instead of requiring yet another layer of maintenance.

My comments here were very bold. This topic is full of controversy. I wrote here my personal opinion.

Cheers,
I loved this discussion ☺️

[alchemistcai]

Transforming Python into a static-type language is anti-nature. By doing so, you have the worst of both worlds: the slow speed of Python and the burden of static types.

This point of view is apropos.

I'd like to push a request of type-annotation-added version anyway,just as another flavor.Maybe I can find some bugs by type checks too.

[tomschr]

I found this discussion by accident. Although I'm not really connected to this project, maybe I can share some experiences you might find interesting. 😊

I'm the maintainer of python-semver and I had the same considerations. This is what I learned and observed:

• The current trend in Python is towards type hints. Although this per se not an advantage in itself, but it shows that many(?) developers see a benefit. They see this as an advantage that justifies the additional effort.
• More and more project add type hints in their projects. See first item. The more projects use it, the more experiences we gain. Type hints are still "new" to Python and we are all learning.
• Yes, type hints can be annoying. But I also see the benefits. For example, hints in IDEs, being "forced"/reminded to think about what a function expects and returns etc. I have the impression, this improves my programming skills and avoid bugs in my code.
• Types can be checked in the GitHub Actions and gives me more confidence in the code and test suite.
• I can use different type checkers. I like choice, so if something stops working I can switch to something different. Also interesting to see which checker reports what error.
• Type hints are still optional. As far as I can tell, I haven't heard anyone wanting to change this in Python. That was my (positive) take away, as I'm not forced to use them when I write a little throwaway script.

There are certainly pros and cons. Everybody is free to decide if this is worth the effort. I don't have any insights for this project, so I can't give any recommendations for or against it. For a more elaborate analysis, see https://realpython.com/lessons/pros-and-cons-type-hints/.

From my own experiences working with maintaining python-semver, I solved the type hints vs. docstring issue with this approach:

• Every function or method signatures contain type hints.
• The docstrings contain no type information, only descriptions of the arguments and about their purpose.
• The documentation uses the sphinx-autodoc-typehints extension. This adds the types automatically to the output. There is no mismatch.

Here is the example how this looks: https://python-semver.readthedocs.io/en/latest/api.html#module-semver.version

For my project it worked out and the additional effort was and is still manageable. The types are not only checked, but also documented. I like this approach. 😉 For others, the situation might be different.

Thanks for reading.

[alchemistcai]

The documentation uses the sphinx-autodoc-typehints extension. This adds the types automatically to the output. There is no mismatch.

I think this is a great suggestion.Auto-generated documents including type-hints can make the api references more explict for programmers.

[joaomcteixeira]

Hi @tomschr

Very nice reply. You placed the stake right in the middle 😉. Indeed, both tendencies have their reasons, and I like how you approached type hints in your project. I did not know about the existence of sphinx-autodoc-typehints, but its existence supports the argument of the extra burden of keeping type documentation in docstrings and type hints. Which was one of the main reasons why I never adopted type hints. And to be honest, knowing about sphinx-autodoc-typehints makes me consider adopting typehints (eventually). But it's not straightforward as descriptions like this are still a bit cryptic: Union[int, str, None, Tuple[Union[int, str], ...]]. At the end of the day, it depends on the workforce of the project developers and the scope of the project, obviously.

Very nice discussion. Thanks to both of you 👏

[tomschr]

Thank you for your kind comment. 🙂 Glad that I could contribute a little bit to your understanding.

One thing I'd like to amend your comment:

But it's not straightforward as descriptions like this are still a bit cryptic: Union[int, str, None, Tuple[Union[int, str], ...]].

That's very true. It's not very readable. Fortunately, you can define some types yourself. I did that in the _types.py file.

For example, a single unit of a version can be an integer, None, or a string. This can be expressed with this type definition:

VersionPart = Union[int, Optional[str]]

If you use a more modern Python version, you could simplify it to an even more readable line:

VersionPart = int | Optional[str]

Somewhere else you could use this definition in your function signature.

The only downside I currently see is, that the mentioned Sphinx extension resolve the defined types to their original ones. It could be helpful if it wouldn't. Or make it a configuration option. I have not yet found a way to break the habit of the extension.

Probably, the notation could be solved in the Sphinx extension. I don't like this Union[...] thing either. I guess, it's just a pull request away. 😉

[joaomcteixeira]

Hehe, thanks, @tomschr, for this reply. I am very happy to have learned this from you and see you use type hints yet with a good amount of criticism 😉 The only way to learn and improve is to discuss things in length. Going back to your Python semver project, I will look at it carefully. I personally maintain a repository template of good practices for creating a Python project. There, I gather different strategies. Not saying they are perfect, but it reflects the setup I like the most so far and is the one I used to configure haddock3 back in 2021.

https://github.com/joaomcteixeira/python-project-skeleton

Cheers,

[tomschr]

Thanks, that's nice. I agree with you! I also learn from you and others, so it's not a one-way street. 😄

Haha, I hope you won't find too many bugs. 😉 I don't even claim it's the perfect example to study type hints in Python. As I was a bit lazy, it was the one project that was in my mind and that I know.

Thanks for the link! 👍 Something to read later.

Enjoy your journey with type hints.