👌Improve: Spruce up orm.Entity.fields interface

This commit first ensures that all query operations are supported in the
new syntax, including negation. The commit also separates the operations
by field type via subclassing. It introduces a uniform `orm.fields.add_field`
API as an abstraction of the type-dependent allocation of
`orm.fields.QbField` flavors.

A distinction is made in the API between database columns and attribute
fields via the `is_attribute` parameter. If set to `True` (the default), the field
is considered an attribute field and as such, `QbField().backend_key` will
return `'attribute.key'` in accordance with the `QueryBuilder` mechanics.
Note that _only_ in the case of attribute fields, `backend_key` may be
aliased via the `alias` parameter. This is done to avoid restrictions on
database field naming while maintaining a pythonic interface in the
frontend via the `key` parameter, which is validated as pythonic
(supporting dot notation).

docs/source/howto/query.rst

@@ -99,7 +99,6 @@ For example, you can iterate over the results of your query in a for loop:
     This avoids loading the entire query result into memory, and it also delays committing changes made to AiiDA objects inside the loop until the end of the loop is reached.
     If an exception is raised before the loop ends, all changes are reverted.
 
-
 .. _how-to:query:filters:
 
 Filters
@@ -160,6 +159,59 @@ In case you want all calculation jobs with state ``finished`` or ``excepted``, y
         },
     )
 
+.. _how-to:query:filters:programmatic:
+
+Programmatic syntax for filters
+-------------------------------
+
+.. versionadded:: 2.6
+
+Filter keys may be defined programmatically, providing in modern IDEs (including AiiDA's ``verdi shell``) autocompletion of fields and operators.
+For example, the above query may be given as
+
+.. code-block:: python
+
+    qb = QueryBuilder()
+    qb.append(
+        CalcJobNode,
+        filters={
+            CalcJobNode.fields.process_state: {'in': ['finished', 'excepted']},
+        },
+    )
+
+In this approach, ``CalcJobNode.fields.`` will suggest (autocomplete) the queryable fields of ``CalcJobNode`` allowing the user to explore the node's attributes directly while constructing the query.
+
+Alternatively, the entire filtering expression may be provided programmatically as logical expressions:
+
+.. code-block:: python
+
+    qb = QueryBuilder()
+    qb.append(
+        CalcJobNode,
+        filters=CalcJobNode.fields.process_state.in_(['finished', 'excepted']),
+    )
+
+.. note::
+
+    Logical operations are distributed by type. As such, ``Node.fields.<some_field>.`` will `only` provide the :ref:`supported operations<topics:database:advancedquery>` for the type of ``some_field``, in this case ``==``, ``in_``, ``like``, and ``ilike``, for type ``str``.
+
+Logical expressions may be strung together with ``&`` and ``|`` to construct complex queries.
+
+.. code-block:: python
+
+    filters=(
+        (Node.fields.ctime < datetime(2030, 1, 1))
+        & (
+            (Node.fields.pk.in_([4, 8, 15, 16, 23, 42]))
+            | (Node.fields.label.like("%some_label%"))
+        )
+        & (Node.fields.extras.has_key("some_key"))
+    )
+
+.. tip::
+
+    ``()`` may be used to override the natural precedence of ``|``.
+
 .. _how-to:query:filters:operator-negations:
 
 Operator negations
@@ -185,6 +237,14 @@ So, to query for all calculation jobs that are not a ``finished`` or ``excepted`
 
 A complete list of all available operators can be found in the :ref:`advanced querying section<topics:database:advancedquery:tables:operators>`.
 
+.. _how-to:query:filters:operator-negations:new:
+
+.. versionadded:: 2.6
+    Programamtic filter negation
+
+    In the new :ref:`logical expression syntax<how-to:query:filters:programmatic>`, negation can be achieved by prepending ``~`` to any expression.
+    For example ``~(Int.fields.value < 5)`` is equivalent to ``Int.fields.value >= 5``.
+
 .. _how-to:query:relationships:
 
 Relationships
@@ -273,10 +333,28 @@ This can be used to project the values of nested dictionaries as well.
     Be aware that for consistency, ``QueryBuilder.all()`` / ``iterall()`` always returns a list of lists, even if you only project one property of a single entity.
     Use ``QueryBuilder.all(flat=True)`` to return the query result as a flat list in this case.
 
-As mentioned in the beginning, this section provides only a brief introduction to the :class:`~aiida.orm.querybuilder.QueryBuilder`'s basic functionality.
-To learn about more advanced queries, please see :ref:`the corresponding topics section<topics:database:advancedquery>`.
+.. _how-to:query:projections:programmatic:
+
+.. versionadded:: 2.6
+    Programmatic syntax for projections
 
+    Similar to :ref:`filters<how-to:query:filters:programmatic>`, projections may also be provided programmatically, leveraging the autocompletion feature of modern IDEs.
 
+    .. code-block:: python
+
+        qb = QueryBuilder()
+        qb.append(
+            Int,
+            project=[
+                Int.fields.pk,
+                Int.fields.value,
+            ],
+        )
+
+----
+
+As mentioned in the beginning, this section provides only a brief introduction to the :class:`~aiida.orm.querybuilder.QueryBuilder`'s basic functionality.
+To learn about more advanced queries, please see :ref:`the corresponding topics section<topics:database:advancedquery>`.
 
 .. _how-to:query:shortcuts:
 

docs/source/topics/data_types.rst

@@ -1375,3 +1375,34 @@ Therefore, big data (think large files), whose content does not necessarily need
 A data type may safely use both the database and file repository in parallel for individual properties.
 Properties stored in the database are stored as *attributes* of the node.
 The node class has various methods to set these attributes, such as :py:meth:`~aiida.orm.nodes.attributes.NodeAttributes.set` and :py:meth:`~aiida.orm.nodes.attributes.NodeAttributes.set_many`.
+
+Fields
+------
+
+.. versionadded:: 2.6
+
+The attributes of new data types may be exposed to end users by explicitly defining each attribute field under the ``__qb_fields__`` class attribute of the new data class.
+
+.. code-block:: python
+
+    from aiida.orm.fields import add_field
+
+    class NewData(Data):
+        """A new data type."""
+
+        __qb_fields__ = [
+            add_field(
+              key='frontend_key',
+              alias='backend_key',  # optional mapping to a backend key, if different (only allowed for attribute fields)
+              dtype=str,
+              is_attribute=True,  # signalling if field is an attribute field (default is `True`)
+              is_subscriptable=False,  # signalling subscriptability for dictionary fields
+              doc='An example field',
+            )
+        ]
+
+The internal mechanics of ``aiida.orm.fields`` will dynamically add ``frontend_key`` to the ``fields`` attribute of the new data type. The construction of ``fields`` follows the rules of inheritance, such that other than its own fields, ``NewData.fields`` will also inherit the fields of its parents, following the inheritance tree up to the root ``Entity`` ancestor. This enhances the usability of the new data type, for example, allowing the end user to programmatically define  :ref:`filters<how-to:query:filters:programmatic>` and :ref:`projections<how-to:query:projections:programmatic>` when using AiiDA's :py:class:`~aiida.orm.querybuilder.QueryBuilder`.
+
+.. note::
+
+  :py:meth:`~aiida.orm.fields.add_field` will return the flavor of :py:class:`~aiida.orm.fields.QbField` associated with the type of the field defined by the ``dtype`` argument, which can be given as a primitive type or a ``typing``-module type hint, e.g. ``Dict[str, Any]``. When using the data class in queries, the logical operators available to the user will depend on the flavor of :py:class:`~aiida.orm.fields.QbField` assigned to the field.

docs/source/topics/database.rst

@@ -178,13 +178,26 @@ List of all operators:
 | ``contains`` |    lists    | ``'attributes.some_key': {'contains': ['a', 'b']}``   | Filter for lists that should contain certain values.                         |
 +--------------+-------------+-------------------------------------------------------+------------------------------------------------------------------------------+
 
-As mentioned in the :ref:`section about operatior negations<how-to:query:filters:operator-negations>` all operators can be turned into their associated negation (``NOT`` operator) by adding a ``!`` in front of the operator.
+.. versionadded:: 2.6
+    Logical expression syntax
+
+    In the new logical expression syntax for :ref:`filters<how-to:query:filters:programmatic>` and :ref:`projections<how-to:query:projections:programmatic>`, the above operations are distributed by type.
+    When using the autocompletion feature in constructing query expressions, only operations associated with the type of the queried field will be presented.
+
+    For example, ``Node.fields.uuid`` is a string type, and as such, ``Node.fields.uuid.`` will, in addition to ``==`` and ``in_``, only suggest ``like`` and ``ilike``.
+
+
+As mentioned in the :ref:`section about operator negations<how-to:query:filters:operator-negations>` all operators can be turned into their associated negation (``NOT`` operator) by adding a ``!`` in front of the operator.
+
+.. versionadded:: 2.6
+    Programmatic filter negation
+
+    In the new logical expression syntax, negation is defined with ``~``, such that ``~(Node.fields.some_field < 1)`` is equivalent to ``Node.fields.some_field >= 1``.
 
 .. note::
     The form of (negation) operators in the rendered SQL may differ from the ones specified in the ``QueryBuilder`` instance.
     For example, the ``!==`` operator of the ``QueryBuilder`` will be rendered to ``!=`` in SQL.
 
-
 .. _topics:database:advancedquery:tables:relationships:
 
 List of all relationships:

src/aiida/orm/__init__.py

@@ -83,7 +83,6 @@
     'PortableCode',
     'ProcessNode',
     'ProjectionData',
-    'QbAttrField',
     'QbField',
     'QbFieldFilters',
     'QbFields',

src/aiida/orm/authinfos.py

@@ -14,7 +14,7 @@
 from aiida.plugins import TransportFactory
 
 from . import entities, users
-from .fields import QbField
+from .fields import add_field
 
 if TYPE_CHECKING:
     from aiida.orm import Computer, User
@@ -45,13 +45,38 @@ class AuthInfo(entities.Entity['BackendAuthInfo', AuthInfoCollection]):
 
     _CLS_COLLECTION = AuthInfoCollection
 
-    __qb_fields__ = (
-        QbField('enabled', dtype=bool, doc='Whether the instance is enabled'),
-        QbField('auth_params', dtype=Dict[str, Any], doc='Dictionary of authentication parameters'),
-        QbField('metadata', dtype=Dict[str, Any], doc='Dictionary of metadata'),
-        QbField('computer_pk', 'dbcomputer_id', dtype=int, doc='The PK of the computer'),
-        QbField('user_pk', 'aiidauser_id', dtype=int, doc='The PK of the user'),
-    )
+    __qb_fields__ = [
+        add_field(
+            'enabled',
+            dtype=bool,
+            is_attribute=False,
+            doc='Whether the instance is enabled',
+        ),
+        add_field(
+            'auth_params',
+            dtype=Dict[str, Any],
+            is_attribute=False,
+            doc='Dictionary of authentication parameters',
+        ),
+        add_field(
+            'metadata',
+            dtype=Dict[str, Any],
+            is_attribute=False,
+            doc='Dictionary of metadata',
+        ),
+        add_field(
+            'computer_pk',
+            dtype=int,
+            is_attribute=False,
+            doc='The PK of the computer',
+        ),
+        add_field(
+            'user_pk',
+            dtype=int,
+            is_attribute=False,
+            doc='The PK of the user',
+        ),
+    ]
 
     PROPERTY_WORKDIR = 'workdir'
 

src/aiida/orm/comments.py

@@ -13,7 +13,7 @@
 from aiida.manage import get_manager
 
 from . import entities, users
-from .fields import QbField
+from .fields import add_field
 
 if TYPE_CHECKING:
     from aiida.orm import Node, User
@@ -64,14 +64,44 @@ class Comment(entities.Entity['BackendComment', CommentCollection]):
 
     _CLS_COLLECTION = CommentCollection
 
-    __qb_fields__ = (
-        QbField('uuid', dtype=str, doc='The UUID of the comment'),
-        QbField('ctime', dtype=datetime, doc='Creation time of the comment'),
-        QbField('mtime', dtype=datetime, doc='Modified time of the comment'),
-        QbField('content', dtype=str, doc='Content of the comment'),
-        QbField('user_pk', 'user_id', dtype=int, doc='User PK that created the comment'),
-        QbField('node_pk', 'dbnode_id', dtype=int, doc='Node PK that the comment is attached to'),
-    )
+    __qb_fields__ = [
+        add_field(
+            'uuid',
+            dtype=str,
+            is_attribute=False,
+            doc='The UUID of the comment',
+        ),
+        add_field(
+            'ctime',
+            dtype=datetime,
+            is_attribute=False,
+            doc='Creation time of the comment',
+        ),
+        add_field(
+            'mtime',
+            dtype=datetime,
+            is_attribute=False,
+            doc='Modified time of the comment',
+        ),
+        add_field(
+            'content',
+            dtype=str,
+            is_attribute=False,
+            doc='Content of the comment',
+        ),
+        add_field(
+            'user_pk',
+            dtype=int,
+            is_attribute=False,
+            doc='User PK that created the comment',
+        ),
+        add_field(
+            'node_pk',
+            dtype=int,
+            is_attribute=False,
+            doc='Node PK that the comment is attached to',
+        ),
+    ]
 
     def __init__(
         self, node: 'Node', user: 'User', content: Optional[str] = None, backend: Optional['StorageBackend'] = None

src/aiida/orm/computers.py

@@ -16,7 +16,7 @@
 from aiida.plugins import SchedulerFactory, TransportFactory
 
 from . import entities, users
-from .fields import QbField
+from .fields import add_field
 
 if TYPE_CHECKING:
     from aiida.orm import AuthInfo, User
@@ -72,15 +72,50 @@ class Computer(entities.Entity['BackendComputer', ComputerCollection]):
 
     _CLS_COLLECTION = ComputerCollection
 
-    __qb_fields__ = (
-        QbField('uuid', dtype=str, doc='The UUID of the computer'),
-        QbField('label', dtype=str, doc='Label for the computer'),
-        QbField('description', dtype=str, doc='Description of the computer'),
-        QbField('hostname', dtype=str, doc='Hostname of the computer'),
-        QbField('transport_type', dtype=str, doc='Transport type of the computer'),
-        QbField('scheduler_type', dtype=str, doc='Scheduler type of the computer'),
-        QbField('metadata', dtype=Dict[str, Any], doc='Metadata of the computer'),
-    )
+    __qb_fields__ = [
+        add_field(
+            'uuid',
+            dtype=str,
+            is_attribute=False,
+            doc='The UUID of the computer',
+        ),
+        add_field(
+            'label',
+            dtype=str,
+            is_attribute=False,
+            doc='Label for the computer',
+        ),
+        add_field(
+            'description',
+            dtype=str,
+            is_attribute=False,
+            doc='Description of the computer',
+        ),
+        add_field(
+            'hostname',
+            dtype=str,
+            is_attribute=False,
+            doc='Hostname of the computer',
+        ),
+        add_field(
+            'transport_type',
+            dtype=str,
+            is_attribute=False,
+            doc='Transport type of the computer',
+        ),
+        add_field(
+            'scheduler_type',
+            dtype=str,
+            is_attribute=False,
+            doc='Scheduler type of the computer',
+        ),
+        add_field(
+            'metadata',
+            dtype=Dict[str, Any],
+            is_attribute=False,
+            doc='Metadata of the computer',
+        ),
+    ]
 
     def __init__(
         self,

src/aiida/orm/entities.py

@@ -21,7 +21,7 @@
 from aiida.common.warnings import warn_deprecation
 from aiida.manage import get_manager
 
-from .fields import EntityFieldMeta, QbField, QbFields
+from .fields import EntityFieldMeta, QbField, QbFields, add_field
 
 if TYPE_CHECKING:
     from aiida.orm.implementation import BackendEntity, StorageBackend
@@ -176,7 +176,14 @@ class Entity(abc.ABC, Generic[BackendEntityType, CollectionType], metaclass=Enti
 
     fields: QbFields = QbFields()
 
-    __qb_fields__: Sequence[QbField] = (QbField('pk', 'id', dtype=int, doc='The primary key of the entity'),)
+    __qb_fields__: Sequence[QbField] = [
+        add_field(
+            'pk',
+            dtype=int,
+            is_attribute=False,
+            doc='The primary key of the entity',
+        ),
+    ]
 
     @classproperty
     def objects(cls: EntityType) -> CollectionType:  # noqa: N805