diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 2c65098c5ec..d5950fdbcb4 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -700,6 +700,8 @@ peps/pep-0819.rst @emmatyping peps/pep-0820.rst @encukou peps/pep-0821.rst @JelleZijlstra peps/pep-0822.rst @methane +peps/pep-0823.rst @gvanrossum +# ... peps/pep-0825.rst @warsaw @dstufft peps/pep-0826.rst @savannahostrowski peps/pep-0827.rst @1st1 diff --git a/peps/pep-0823.rst b/peps/pep-0823.rst new file mode 100644 index 00000000000..74571808bd8 --- /dev/null +++ b/peps/pep-0823.rst @@ -0,0 +1,1219 @@ +PEP: 823 +Title: None-aware access operators +Author: Marc Mueller +Sponsor: Guido van Rossum +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 16-Jun-2026 +Python-Version: 3.16 + + +Abstract +======== + +This PEP proposes adding two new operators. + +* The "``None``-aware attribute access" operator ``?.`` ("maybe dot") +* The "``None``-aware indexing" operator ``?[ ]`` ("maybe subscript") + +Both operators evaluate the left hand side, check if it is ``not None`` +and only then evaluate the full expression. They are roughly equivalent +to: + +.. code-block:: python + + # a.b?.c.d + _t.c.d if ((_t := a.b) is not None) else None + + # a.b?[c].d + _t[c].d if ((_t := a.b) is not None) else None + +See the `Specification`_ section for more details. + + +Terminology +=========== + +An attribute or value is ``optional`` + In the context of this PEP an attribute or value is considered + ``optional`` if it is always present but can be ``None``. + + .. code-block:: python-console + + >>> class A: + ... def __init__(self, val: int | None) -> None: + ... self.val = val + ... + >>> a = A(None) + >>> hasattr(a, "val") + True + +An attribute or value is ``missing`` + An attribute or value is considered ``missing`` if it is not present + at all. + + .. code-block:: python-console + + >>> class B: + ... ... + ... + >>> b = B() + >>> hasattr(b, "val") + False + + +Motivation +========== + +First officially proposed over ten years ago in (the now deferred) :pep:`505`, +the idea to add ``None``-aware access operators has been along for +some time now, discussed at length in numerous threads, most recently +in [#discuss_revisit_505]_ and [#discuss_safe_navigation_op]_. This PEP +aims to capture the current state of discussion and proposes a specification +for addition to the Python language. In contrast to :pep:`505`, it will +only focus on the two access operators. See the `Deferred Ideas`_ section +for more details. + +``None``-aware access operators are not a new invention. Several other +modern programming languages have so called "``null``-aware" or +"optional chaining" operators, including TypeScript [#ts]_, +ECMAScript (a.k.a. JavaScript) [#js]_, C# [#csharp]_, Dart [#dart]_, +Swift [#swift]_, Kotlin [#kotlin]_, Ruby [#ruby]_, PHP [#php]_ and more. + +The general idea is to provide access operators which can traverse +``None`` values without raising exceptions. + +Nested objects with ``optional`` attributes +------------------------------------------- + +When writing Python code, it is common to encounter objects with ``optional`` +attributes. Accessing attributes, subscripts or function calls can raise +``AttributeError`` or ``TypeError`` at runtime if the value is ``None``. +To ensure these operations will not raise, a common option is to add +conditional ``is not None`` checks. Even for fairly simple objects, +this often adds nesting and code duplication. Consider the following +simplified example: + +.. code-block:: python + + from dataclasses import dataclass + + @dataclass + class Data: + customer: Customer | None + total_customers: int + + @dataclass + class Customer: + user: User | None + shopping_card: list[Item] + + @dataclass + class User: + name: str + + def get_customer_name(data: Data) -> str | None: + """Get customer name in lower case if it exists.""" + customer = data.customer + if customer is not None: + user = customer.user + if user is not None: + return user.name.lower() + return None + +This could also be written using assignment expressions: + +.. code-block:: python + + def get_customer_name(data: Data) -> str | None: + if ( + (customer := data.customer) is not None + and (user := customer.user) is not None + ): + return user.name.lower() + return None + +While both options are correct, it requires more effort than necessary +to understand the simple checks being added here. Additionally, even +writing these statements and expressions can become quite cumbersome, +to a point that it is not uncommon to skip certain parts: be it the +explicit ``is not None`` and instead defaulting to implicit +boolean checks or the assignments to temporary variables and instead +repeating the previous expressions. + +The goal for ``?.`` and ``?[ ]`` is to make reading and writing these +kinds of expressions much simpler while being predictable and doing +the correct things intuitively. Using these operators, the function +could instead be written as: + +:: + + def get_customer_name(data: Data) -> str | None: + return data.customer?.user?.name.lower() + +Here ``?`` are inserted after each ``optional`` subexpression to change +the attribute access to "``None``-aware attribute access" operators. +In contrast to before, it is not necessary to add additional ``if`` +statements, assignments or nesting, so starting from "normal" +attribute access operators and changing these to ``?.`` where +necessary afterwards, is an easy way to write these expressions. + +Parsing structured data +----------------------- + +The ``?.`` and ``?[ ]`` operators can also aid in the traversal of +structured data, oftentimes coming from JSON and parsed as nested +dicts and lists. It is worth noting though that the operators do +**NOT** handle ``missing`` attributes / data. In these cases, at +least for dictionaries a useful helper method is ``dict.get(key)``. + +.. code-block:: python + + from typing import NotRequired, TypedDict + + class Data(TypedDict): + # Note the 'NotRequired' here, i.e. 'customer' can be **missing**. + # That is different from the previous example! + + customer: NotRequired[Customer | None] + total_customers: int + + class Customer(TypedDict): + user: User | None + shopping_card: list[Item] + + class User(TypedDict): + name: str + + def get_customer_name(data: Data) -> str | None: + customer = data.get("customer") + if customer is not None: + user = customer["user"] + if user is not None: + return user.name.lower() + return None + +Writing it using ``?.`` and ``?[ ]`` would look like this: + +:: + + def get_customer_name(data: Data) -> str | None: + return data.get("customer")?["user"]?.name.lower() + +Other common patterns +--------------------- + +A collection of additional patterns which could be improved with +``?.`` and ``?[ ]``. It is not the goal to list every foreseeable +use case but rather to help recognize these patterns which often +hide in plain sight. Attribute and function names have been shortened. + +.. note:: + + Most patterns below are **not** fully identical. As mentioned + `earlier `_, it is common + to use boolean expressions to filter out ``None`` values. Other + falsy values, e.g. ``False``, ``""``, ``0``, ``[]``, ``{}`` or custom + objects which overwrite ``__bool__``, are filtered out too though. + If code relied on this property, the expression cannot necessarily + be replaced with ``?.`` or ``.[ ]``. + +:: + + # In assignments + + x = a.b if (a is not None) else None + x = a?.b + +:: + + # In if statements often used as guard clause with early + # return or raising of an exception + + if not (a and a.b == val): ... + if a?.b != val: ... + + if not (a and a.lower()): ... + if not a?.lower(): ... + +:: + + # Misc expressions + + # Each of these results still needs to be checked for the + # negative case, though falsy values are usually enough. + # It often does not matter whether it is 'False' or 'None'. + + a and a.b and a.b.c + a?.b?.c + + a.b and a.b[0].c and a.b[0].c.d and a.b[0].c.d[0].e + a.b?[0].c?.d?[0].e + + d1: dict | None + d1 and key in d1 and d1[key] + d1?.get(key) + + d2: dict + key in d2 and d2[key][other] + d2.get(key)?[other] + + key in d2 and d2[key].do_something() + d2.get(key)?.do_something() + + (c := a.b) and c.startswith(key) + a.b?.startswith(key) + + (b := a.get(key)) and b.get(other) == 2 + a.get(key)?.get(other) == 2 + + (b := a.get(key)) and b.strip().lower() + a.get(key)?.strip().lower() + + + # Sometimes actions should only be performed + # if a variable is not None + + if timeout_handle is not None: + timeout_handle.cancel() + timeout_handle?.cancel() + + +Specification +============= + +The maybe-dot and maybe-subscript operators +------------------------------------------- + +Two new operators are added, ``?.`` ("maybe-dot") and ``?[ ]`` +("maybe subscript"). Both operators first evaluate the left hand side +(the ``base``). The result is cached, so that the expression is not +evaluated again. It is checked if the result is not ``None`` and only +then is the remaining expression (the ``tail``) evaluated as if normal +attribute or subscript access were used. + +.. code-block:: python + + # base?.tail + _t.tail if ((_t := base) is not None) else None + +The ``base`` can be replace with any number of expressions, including +`Parenthesized ones `_ +while the ``tail`` is limited to attribute access, subscript, +their ``None``-aware variants and call expressions. + +.. code-block:: python + + # a.b?.c + _t.c if ((_t := a.b) is not None) else None + + # a.b?[c] + _t[c] if ((_t := a.b) is not None) else None + + # a.b?.c() + _t.c() if ((_t := a.b) is not None) else None + + # a?.b?.c.d + _t2.c.d if ( + (_t2 := _t1.b if ((_t1 := a) is not None) else None) is not None + ) else None + +Short-circuiting +**************** + +If the left hand side (the ``base``) for ``?.`` or ``?[ ]`` evaluates to +``None``, the remaining expression (the ``tail``) is skipped and the +result will be set to ``None`` instead. The ``AttributeError`` for +accessing a member of ``None`` or ``TypeError`` for trying to subscribe +to ``None`` are omitted. It is therefore not necessary to change +subsequent ``.`` or ``[ ]`` on the right hand side just because a +``?.`` or ``?[ ]`` is used prior. + +:: + + >>> a = None + >>> print(a?.b.c[0].some_function()) + None + +The ``None``-aware access operators will only short-circuit expressions +containing primary expressions (name, attribute access, subscript, +their ``None``-aware counterparts, and call expressions). As a rule of +thumb, short-circuiting is broken once an operator other than +``.``, ``[ ]``, ``?.``, ``?[ ]`` is reached. + +:: + + >>> a = None + >>> print(a?.b.c) + None + >>> print(a?.b.c or "Hello") + 'Hello' + >>> 2 in a?.b.c + Traceback (most recent call last): + File "", line 1, in + 2 in a?.b.c + TypeError: argument of type 'NoneType' is not a container or iterable + >>> 2 in (a?.b.c or ()) + False + +Parenthesized expressions - groupings +************************************* + +Using ``?.`` and ``?[ ]`` inside groups is possible. In addition to the +rules laid out in the `previous `_ section, +short-circuiting will also be broken at the end of a group. For example +the expression ``(a?.b).c`` will raise an ``AttributeError`` on ``.c`` +if ``a is None``. This is conceptually identical to extracting the group +contents and storing the result in a temporary variable before +substituting it back into the original expression. + +:: + + # (a?.b).c + + _t = a?.b + _t.c + +Common use cases for ``None``-aware access operators in groups are +boolean or conditional expressions which can provide a fallback value +in case the first part evaluates to ``None``. + +:: + + (a.b?.c or d).e?.func() + + # a.b?.c + _t2 = _t1.c if ((t1 := a.b) is not None) else None + + # (... or d) + _t3 = _t2 if _t2 else d + + # (...).e?.func() + _t4.func() if ((_t4 := _t3.e) is not None) else None + +Assignments +*********** + +``None``-aware access operators may only be used in a ``Load`` context. +Assignments are not permitted and will raise a ``SyntaxError``. + +:: + + >>> a?.b = 1 + File "", line 1 + a?.b = 1 + ^^^^ + SyntaxError: cannot assign to none aware expression + + >>> (a?.b) = 1 + File "", line 1 + (a?.b) = 1 + ^^^^ + SyntaxError: cannot assign to none aware expression here. Maybe you meant '==' instead of '='? + +It is however possible to use them in **non-trivial** +`Parenthesized expressions `_ +since in these the "``None``-aware" expressions are in a ``Load`` +context. Though care must be taken so they can be evaluated properly, +e.g. with a fallback value. + +:: + + class D: + c = 0 + + a = None + d = D() + + (a?.b or d).c = 1 + + # This would be evaluated as + _t2 = _t1.b if ((_t1 := a) is not None) else None + if _t2: + _t2.c = 1 + else: + d.c = 1 + +Await expressions +***************** + +``None``-aware access operations are permitted in ``await`` expressions. +It is up to the developer to make sure they do not evaluate to ``None`` +at runtime, otherwise a ``TypeError`` is raised. This behavior is similar +to awaiting any other variable which can be ``None``. + +AST changes +----------- + +Two new AST nodes are added ``NoneAwareAttribute`` and ``NoneAwareSubscript``. +They are the counterparts to the existing ``Attribute`` and ``Subscript`` +nodes. Notably there is no ``expr_context`` attribute because the new nodes +do not support assignments themselves and thus the context will always be +``Load``. Furthermore, an optional ``group`` attribute is added for all +expression nodes. It is set to ``1`` if an expression is the topmost +node in a group, ``0`` otherwise. + +:: + + expr = ... + | Attribute(expr value, identifier attr, expr_context ctx) + | Subscript(expr value, expr slice, expr_context ctx) + ... + | NoneAwareAttribute(expr value, identifier attr) + | NoneAwareSubscript(expr value, expr slice) + + attributes (int? group, int lineno, int col_offset, + int? end_lineno, int? end_col_offset) + +Grammar changes +--------------- + +A new ``?`` token is added. In addition the ``primary`` grammar rule is +updated to include ``none_aware_attribute`` and ``none_aware_subscript``. + +.. code-block:: PEG + + primary: + | primary '.' NAME + | none_aware_attribute + | primary genexp + | primary '(' [arguments] ')' + | primary '[' slices ']' + | none_aware_subscript + | atom + + none_aware_attribute: + | primary '?' '.' NAME + + none_aware_subscript: + | primary '?' '[' slices ']' + +Multiline formatting +******************** + +Using two separate tokens to express ``?.`` and ``?[`` allows developers +to insert a space or line break as needed. For multiline expressions it +enables that ``?`` is appended to the ``optional`` subexpression whereas +``.`` or ``[`` could be moved to the next line. This is indented merely +as an option for developers. Everyone is free to choose a style that fits +their needs, especially code formatters might prefer a style which +conforms better to their existing preferences. An example of what +is possible: + +:: + + def get_customer_name(data: Data) -> str | None: + return ( + data.customer? + .user? + .name.lower() + ) + + +Backwards Compatibility +======================= + +``None``-aware access operators are **opt-in**. Existing programs will +continue to run as is. So far code which used either ``?.`` or ``?[ ]`` +raised a ``SyntaxError``. + + +Security Implications +===================== + +There are no new security implications from this proposal. + + +How to Teach This +================= + +After students know how the attribute access ``.`` and subscript ``[ ]`` +operators work, they may learn about the "``None``-aware" variants for +both. + +Students may find it helpful to think of ``?.`` and ``?[ ]`` as a +combination of two different actions. First the ``?`` postfix represents +an ``is not None`` check on the subexpression with short-circuiting +if the check fails. If it succeeds, the attribute and subscript access +are performed like normal. + +Experienced developers may find that, after learning about the PEP, they +start to notice the patterns described in the `Motivation`_ section in +their own code bases. + + +Reference Implementation +======================== + +A reference implementation is available at +https://github.com/cdce8p/cpython/tree/pep823-none-aware-access-operators. +A online demo can be tested at https://pep823-and-pep824-demo.pages.dev/. + + +Deferred Ideas +============== + +Coalesce ``??`` and coalesce assignment operator ``??=`` +-------------------------------------------------------- + +:pep:`505` also suggested the addition of a "``None`` coalescing" operator +``??`` and a "``None`` coalescing assignment" operator ``??=``. As the +``None``-aware access operators have their own use cases, the coalescing +operators were moved into a separate document, see PEP-824. +Both proposals can be adopted independently of one another. + +``None``-aware function calls +----------------------------- + +The ``None``-aware access operators work for attribute and index access. +It seems natural to ask if there should be a variant which works for +function invocations. It might be written as ``a.foo?()`` which would +be equivalent to: + +.. code-block:: python + + _t1() if ((_t1 := a.foo) is not None) else None + +This has been deferred on the basis that the proposed operators are +intended to help for nested objects with ``optional`` attributes and +the parsing of structured data, *not* the traversal of arbitrary class +hierarchies. + +A workaround would be to write ``a.foo?.__call__(arguments)``. + +Add ``list.get(key)`` +--------------------- + +It was suggested to add a ``.get(key)`` method to ``list`` +and ``tuple`` objects, similar to the existing ``dict.get`` method. This +could further make parsing of structured data easier since it would no +longer be necessary to check if a ``list`` or ``tuple`` is long enough +before trying to access the n-th element avoiding a possible +``IndexError``. While potentially useful, the idea is out of the scope +for this PEP. + + +Rejected Ideas +============== + +Exception-aware operators +------------------------- + +Arguably, the reason to short-circuit an expression when ``None`` is +encountered is to avoid the ``AttributeError`` or ``TypeError`` that +would be raised under normal circumstances. Instead of testing for +``None``, it was suggested that ``?.`` and ``?[ ]`` could instead handle +``AttributeError`` and ``TypeError`` and skip the remainder of the +expression. Similar to nested try-except blocks. + +While this would technically work, it's not at all clear what the result +should be if an error is caught. Furthermore, this approach would hide +genuine issues like a misspelled attribute which would have raised an +``AttributeError``. There are also already established patterns to +handle these kinds of errors in the form of ``getattr`` and +``dict.get(key)``. + +As catching exceptions would be unexpected and hide potential errors, +it is rejected. + +Add a ``maybe`` keyword +----------------------- + +The ``None``-aware access operators only check for ``None`` in the place +there are used. If multiple attributes in an expression can return ``None``, +it might be necessary to add them multiple times ``a?.b.c?[0].d?.e()``. +It was suggested to instead add a new soft-keyword ``maybe`` to prefix +the expression: ``maybe a.b.c[0].d.e()``. A ``None`` check would then be +added for each attribute and item access automatically. + +While this might be easier to write at first, it introduces new issues. +When using explicit ``?.`` and ``?[ ]`` operators, the input space is well +defined. Only ``a``, ``.c`` and ``.d`` are expected to possibly be ``None``. +If ``.b`` all of the sudden is also ``None``, it would still raise an +``AttributeError`` since it was unexpected. That would not happen for +``maybe``. This behavior is problematic since it can subtly hide real +issues. As the expression output can already be ``None``, the space of +potential outputs did not change and as such no error would appear. + +If it is the intent to catch all ``AttributeError`` and ``TypeError``, +a try-except block can be used instead. + +As the ``?.`` and ``?[ ]`` would allow developers to be more explicit in +their intent, this suggestion is rejected. + +Remove short-circuiting +----------------------- + +It was suggested to remove the `Short-circuiting`_ behavior completely +because it might be too difficult to understand. Developers should +instead change any subsequent attribute access or subscript to their +``None``-aware variants. + +:: + + # before + a.b.optional?.c.d.e + + # after + a.b.optional?.c?.d?.e + +The idea has some of the same challenges as `Add a maybe keyword`_. +By forcing the use of ``?.`` or ``?[ ]`` for attributes which are +``not-optional``, it will be difficult to know if the ``not-optional`` +attributes ``.c`` or ``.d`` suddenly started to return ``None`` as well. +The ``AttributeError`` would have been silenced. + +Another issue especially for longer expressions is that **all** +subsequent attribute access and subscript operators need to be changed +as soon as just one attribute in a long chain is ``optional``. Missing +just one can instantly cause a new ``AttributeError`` or ``TypeError``. + +``?`` Unary Postfix operator +---------------------------- + +To generalize the ``None``-aware behavior and limit the number of new +operators introduced, a unary, postfix operator ``?`` was considered. +``?.`` or ``?[ ]`` could then be considered to be two separate operators. + +While this might have made teaching the operators a bit easier, just one +instead of two new operators, it may also be **too general**, in a sense +that it can be combine with any other operator. For example it is not +clear what the following expressions would mean: + +:: + + >>> x? + 1 + >>> x? -= 1 + >>> x? == 1 + >>> ~x? + >>> [*x?] + +Even if a default meaning of ``is not None else None`` is assumed, the +expressions are likely to raise errors at some point. + +:: + + >>> x? + 1 + >>> (_t1 if ((_t1 := x) is not None) else None) + 1 + +This degree of generalization is not useful. The ``None``-aware access +operators where intentionally chosen to make it easier to access +values in nested objects with ``optional`` attributes. + +If future PEPs want to introduce new operators to access attributes or +call methods, e.g. a chaining operator, it would be advisable to consider +if a ``None``-aware variant for it could be useful, at that time. + +Builtin function for traversal +------------------------------ + +There are a number of libraries which provide some kind of object +traversal functions. The most popular likely being ``glom`` [#glom]_. +Others include ``jmespath`` [#jmespath]_ and ``nonesafe`` [#nonesafe]_. +The idea is usually to pass an object and the lookup attributes as +string to a function which handles the evaluation. It was suggested +to add a ``traverse`` or ``deepget`` function to the stdlib. + +.. code-block:: python-console + + # pip install glom + >>> from glom import glom + >>> data = {"a": {"b": {"c": "d"}}} + >>> glom(data, "a.b.c") + 'd' + >>> glom(data, "a.b.f.g", default=2) + 2 + +While these libraries do work and have its use cases, especially +``glom`` provides an excellent interface to extract and combine multiple +data points from deeply nested objects, they do also have some +disadvantages. Passing the lookup attributes as a string means that often +times there are no more IDE suggestions. Type checking these expressions +is also limited. Furthermore, normal function calls can not provide +short-circuiting, so they would still need to be combined with assignment +and conditional expressions. + +Maybe function +-------------- + +Another suggestion was to add a ``maybe`` function which would return +either an instance of ``Something`` or an instance of ``Nothing``. +``Nothing`` would override the dunder methods in order to allow chaining +on ``optional`` attributes. + +A Python package called ``pymaybe`` [#pymaybe]_ provides a rough +approximation. An example could look like this: + +.. code-block:: python-console + + # pip install pymaybe + >>> from pymaybe import maybe + >>> data = {"a": {"b": {"c": "d"}}} + >>> type(maybe(data)["a"]["b"]["c"]) + + >>> maybe(data)["a"]["b"]["c"] + 'd' + >>> type(maybe(data)["a"]["b"]["c"]["e"]) + + >>> maybe(data)["a"]["b"]["c"]["e"] + None + +While this could work, ``Something`` and ``Nothing`` are only wrapper +classes for the actual values which adds its own challenges. For example +to filter out ``None`` in a subsequent operation an ``is not None`` +check would always return ``True`` and instead ``.is_some()`` would need +to be used. This would make adopting it across a large codebase +difficult and limit its usefulness. Additionally any pure Python +implementation can not really short-circuit the expression. The best +it can do is to implement no-ops on the wrapper classes. + +As such a builtin ``maybe`` function to support accessing nested objects +with ``optional`` attributes is rejected. + +Result object +------------- + +It was suggested to introduce a ``Result`` object similar to how +``asyncio.Future`` works today. Expressions marked with a special +keyword or syntax would then return an instance of ``Result`` instead +of the evaluated expression. The actual value could then be retrieved +by calling ``.result()`` or ``.exception()`` on it. With that it could +be possible to gracefully handle ``None``-aware expression as well. + +While this is an interesting idea, it would be a disruptive change how +expressions need to be written and evaluated today. + +An advantages of the ``?.`` and ``?[ ]`` operators is that they do not +change the result much aside from adding ``None`` as a possible return +value of an expression. As such they are a better solution for the use +cases outlined in the `Motivation`_ section. + +No-Value Protocol +----------------- + +The ``None``-aware access operators could be generalized to user-defined +types by defining a protocol to indicate when a value represents +"no value". Such a protocol may be a dunder method ``__has_value__(self)`` +that returns ``True`` if the value should be treated as having a value +and ``False`` if the value should be treated as no value. + +In the specification section, all uses of ``x is not None`` would be +replaced with ``x.__has_value__()``. + +There are a few obvious candidates like ``math.nan`` and ``NotImplemented``. +However, while these could be interpreted as representing no value, the +interpretation is **domain specific**. For the language itself they *should* +still be treated as values. For example ``math.nan.imag`` is well defined +(it is ``0.0``) and so short-circuiting ``math.nan?.imag`` to return +``None`` would be incorrect. + +As ``None`` is already defined by the language as being the value that +represents "no value" the idea is rejected. + +Use existing syntax or keyword +------------------------------ + +Some comments suggested to use existing syntax like ``->`` for the +``None``-aware access operators, e.g. ``a->b.c``. + +Though possible, the ``->`` operator is already used in Python for +something completely different. Additionally, a majority of other +languages which support "``null``-aware" or "optional chaining" +operators use ``?.``. Some exceptions being Ruby [#ruby]_ with ``&.`` +or PHP [#php]_ with ``?->``. The ``?`` character does not have an +assigned meaning in Python just yet. As such it makes sense to adopt +the most common spelling for the ``None``-aware access operators. +Especially considering that it also works well with the "normal" +``.`` and ``[ ]`` operators. + +Defer ``None``-aware indexing operator +-------------------------------------- + +A point of discussion was the ``?[ ]`` operator. Some thought it might be +missed to easily in an expression ``a.b?[c]``. To move the discussion +forward, it was suggested to defer the operator for later. + +Though it is often helpful to reduce the scope to move forward at all, +the ``?[ ]`` operator is necessary to efficiently get items from +``optional`` objects. While for dictionaries a suitable alternative is to +use ``d?.get(key)``, for general objects developers would have needed +to defer to ``o?.__getitem__(key)``. + +Furthermore, any future PEP just for a ``?[ ]`` would likely have needed +to included a lot of the arguments and objections listed in this one +again. As such it makes sense to include both operators in the same PEP. + +While adding ``list.get(key)`` as suggested in +`Add list.get(key)`_ would reduce the need for ``?[ ]`` +for lists and tuples and as such would be a valuable addition to the +language itself, it does not remove the need for arbitrary objects which +implement a custom ``__getitem__`` method. + +Ignore groups for short-circuiting +---------------------------------- + +An earlier version of this PEP suggested the short-circuiting +behavior should be indifferent towards grouping. It was assumed that +short-circuiting would be broken already for more complex group +expressions like ``(a?.b or c).d`` by the behavior outline in the +`Short-circuiting`_ section, while for simpler ones like ``(a?.b).c`` +the grouping was considered trivial and the expression would be equal to +``a?.b.c``. The advantage being that developers would not have to +look for groupings when evaluating simpler expressions. As long as +any ``None``-aware access operator was used and the expression was not +broken by any other unrelated operator, it would return ``None`` +instead of raising an ``AttributeError`` or ``TypeError``. + +This suggestion was rejected in favor of the specification outline in +the `Parenthesized expressions - groupings`_ section since it violates +the substitution principle. An expression ``(a?.b).c`` should behave +the same whether or not ``a?.b`` is written inline inside a group or +defined as a separate variable. + +:: + + (a?.b).c + + _t = a?.b + _t.c + +Furthermore, defining the short-circuiting behavior that way would have +been a deviation from the already established behavior in +languages like JS [#js_short_circuiting]_ and C# [#csharp]_. + +Change ``primary`` rule to be right-recursive +--------------------------------------------- + +The ``primary`` grammar rule as it is defined [#py_grammar]_ is +left-recursive. This can make it difficult to reason about especially +with regards to the `Short-circuiting`_ and +`Parenthesized expressions `_ +behavior. + +It was therefore proposed to make the ``None``-aware access operators +part of the ``primary`` rule right-recursive instead. The expression +``a.b?.c[0].func()`` would then roughly be parsed as: + +.. code-block:: python + + NoneAwareAttribute( + base=Attribute( + expr=Name(identifier="a"), + identifier="b" + ) + tail=[ + AttributeTail(identifier="c"), + SubscriptTail(slice=Constant(value=0)), + AttributeTail(identifier="func") + CallTail(args=[], keyword=[]) + ] + ) + +While this approach would clearly define which parts of an expression +would be short-circuited, it has several drawbacks. To implement it, at +least three additional AST nodes have to be added ``AttributeTail``, +``SubscriptTail`` and ``CallTail``. As these are right-recursive now, +reusing code from the ``Attribute``, ``Subscript`` and ``Call`` nodes +might prove difficult. Not to mention that the ``NoneAware*`` nodes +would contain parts that are both left- and right-recursive which +would be confusing. + +In comparison the proposed `Grammar changes`_ are intentional kept to +a minimum. The ``None``-aware access operators should behave more or +less like a drop-in replacement for ``.`` and ``[ ]``, only with the +behavior outline in this PEP. + + +Common objections +================= + +Difficult to read +----------------- + +A common objection raised during the discussion was that ``None``-aware +operators are difficult to read in expressions and add line noise. They +might be too easy to miss besides "normal" attribute access and subscript +operators. + +This is a valid concern. Especially for long lines, it is not difficult +to imaging a ``?`` hiding somewhere. However, as with all proposals the +downsides have to be weight against the alternatives. As shown in the +`Motivation`_ section, accessing nested values from objects with +``optional`` attributes can be quite cumbersome. Getting all steps right, +often involves a combination of assignment expressions, temporary +variables and chained conditionals. Especially for beginners this can be +overwhelming and it is frequently just faster to repeat each subexpression +as well as only relying on the implicit ``bool(...) is True`` check instead +of ``is not None``, which can fail in unexpected ways. It is also +a bit slower. Even if the statements and expressions are written +correctly, reading them again is far from simple. + +In contrast, ``?.`` and ``?[ ]`` leave the core of the expression mostly +untouched. It is thus fairly strait forward to see what is happening. +Furthermore, it will be easier to write since one can start from the +normal attribute access and subscript operators and just insert ``?`` +as needed. The Python error messages for accessing a member or subscript +of ``None`` can help here, similarly type checkers and IDEs will be +able to assist. For conditional expressions it is necessary to first +split it up the expression, combine both parts with ``and``, add an +assignment expression (do not miss the brackets!) and add the +``is not None`` check. The whole process is a lot more involved. + +Easy to get ``?.`` wrong +------------------------ + +It was pointed out that it is too easy to switch up the characters in +``?.`` . + +During the PEP discussion numerous alternatives have been proposed. This +contributed to a sense of not knowing which is the "current" or "right" +one. The author believes that this is only temporary and will resolve +itself once a PEP has been accepted. Any spelling mistake will also +raise a ``SyntaxError``. + +As described in the `How to Teach This`_ section, it can be helpful to +think of ``?.`` and ``?[ ]`` as a combination of two different actions +which need to be performed in a certain order. The ``?`` postfix +represents an ``is not None`` check on the subexpression and as such +should always come first. + +Not obvious what ``?.`` and ``?[ ]`` do +--------------------------------------- + +A lot of the discussion centered around the interpretation of the +``None``-aware operators, should they only work for ``optional``, +i.e. perform ``is not None`` checks, or also work for ``missing`` +attributes, i.e. do ``getattr(obj, attr, None) is not None``. + +It was decided that the ``None``-aware operators should only handle +``optional`` attributes, see the `Exception-aware operators`_ section +section for more details why the latter interpretation was rejected. + +Similar to `Easy to get ?. wrong`_ the discussion created a lot of +confusion what the agreed upon interpretation should be. This will also +resolve itself once a PEP has been accepted. + +``?.`` and ``?[ ]`` should handle missing attributes +---------------------------------------------------- + +Some comments pointed out that the ``None``-aware operators should +handle ``missing`` attributes to be useful. + +As shown in the `Motivation`_ section, the operators are not designed +to handle arbitrary data, rather to make it easier to work with nested +objects with ``optional`` attributes. If arbitrary data handling is the +goal, other language concepts are likely better suited, like try-except, +a match statement or data traversal libraries from PyPI. + +See the `Exception-aware operators`_ section for more details why +this was rejected. + +Short circuiting is difficult to understand +------------------------------------------- + +Some have pointed the `Short-circuiting`_ behavior might be difficult +to understand and suggested to remove it in favor of a simplified +proposal. + +It is true that using short-circuiting that way is uncommon in Python. +That closest analogs would be boolean expressions like +``a or b``, ``a and b`` which do also short-circuit the expression if +the first value is truthy or falsy respectively. However, while +the details are complex, the behavior itself is rather intuitive. +To understand it, it is often enough to know that once a subexpression +for an ``optional`` value evaluates to ``None``, the result will be +``None`` as well. Any subsequent attribute access, subscript or call +will be skipped. In the example below, if ``a.b`` is ``None``, so will +be ``a.b?.c``: + +:: + + a.b?.c + ^^^ + +On a technical level, removing short-circuiting would make it difficult +to detect if ``not-optional`` attributes suddenly started to return +``None`` as well. See `Remove short-circuiting`_ in the rejected ideas +section for more details on that. + +Lastly, the short-circuiting behavior as proposed matches that +of other languages like TS [#ts]_, JS [#js_short_circuiting]_ and +C# [#csharp]_. + +Just use ... +------------ + +A common reply towards the proposal was to use an existing language +concept instead of introducing a new syntax. A few alternatives have +been proposed. + +... a conditional expression +**************************** + +Conditional expressions provide the basis for this PEP. As shown in the +`Motivation`_ section, each operator can effectively be written as such. +The ``None``-aware access operators could therefore simply be considered +syntactic sugar. While this is true, this PEP has highlighted repeatedly +that conditional expressions can often times get fairly complex and are +difficult to get right to the point that it is not uncommon for +developers to prefer a less safe and slower alternative with repetitions +just because it is easier to write. The ``?.`` and ``?[ ]`` operators +will provide a better alternative for these exact situations. + +... a match statement +********************* + +Match statements can be a great option to parse any kind of data and +could also be used to deal with ``optional`` attributes. + +.. code-block:: python + + def get_customer_name(data: Data) -> str | None: + match data.customer: + case Customer(user=User(name=name)): + return name.lower() + case _: + return None + +However, getting them right can be equally +tricky as the conditional expression. There are a few pitfalls to watch +out for, only using keyword attributes for class patterns, making sure +the attribute names are correct since the class pattern can also match +``missing`` once and as such will not emit an error if it is misspelled, +and the performance impact from an often times unnecessary ``isinstance`` +check. Furthermore, great care must be taking during refactorings as +patterns often can not be updated automatically. + +While the match statement has been available in Python since 3.10, +anecdotal evidence suggests that developers still prefer other +alternatives for ``optional`` attributes, at least for simple, +strait-forward expression. Pattern matching starts to become much more +useful once multiple attributes or values need to be checked at the +same time. + +... try ... except ... +********************** + +Especially for deeply nested data, a common suggestion was to use +try-except: + +.. code-block:: python + + def get_customer_name(data: Data) -> str | None: + try: + return data.customer.user.name.lower() + except AttributeError, TypeError: + return None + +While this will likely work, it does not provide nearly the same level +as granularity as a conditional expression would. It is often desired +to raise an exception if something unexpected changes. The try-except +block would simply swallow it. + +... a traversal library +*********************** + +Data traversal libraries like ``glom`` [#glom]_ can be used to access +data from nested objects with ``optional`` attributes. However, as +highlighted in the `Builtin function for traversal`_ section passing +the attribute lookup as a string usually means developers will not +get any more IDE suggestions and type checking these expressions is +also fairly limited. Furthermore, normal functions do not support +short-circuiting for the remaining expression. + +Proliferation of ``None`` in code bases +--------------------------------------- + +One of the reasons why :pep:`505` stalled was that some expressed their +concern how ``None``-aware access operators will affect the code written +by developers. If it is easier to work with ``optional`` attributes, +this will encourage developers to use them more. They believe that e.g. +returning an ``optional`` (can be ``None``) value from a function is +usually an anti-pattern. In an ideal world the use of ``None`` would +be limited as much as possible, for example with early data validation. + +It is certainly true that new language features affect how the language +as a whole develops. Therefore any changes should be considered carefully. +However, just because ``None`` represents an anti-pattern for some, has +not prevented the community as a whole from using it extensively. Rather +the lack of ``None``-aware access operators has stopped developers from +writing concise expressions to access ``optional`` attributes and instead +often leads to code which is difficult to understand and can contain +subtle errors. + +``None`` is not special enough +------------------------------ + +Some mentioned that ``None`` is not special enough to warrant dedicated +operators. + +"``null``-aware" or "optional chaining" operators have been added to a +number of other modern programming languages. Furthermore, adding +``None``-aware access operators is something which was suggested numerous +times since :pep:`505` was first proposed over ten years ago. + +In Python ``None`` is frequently used to indicate the absence of +something better or a missing value, e.g. ``dict.get(key)``. Other +languages do often have two separate sentinels for that, e.g. JavaScript +with ``null`` and ``undefined``. This only contributes further to the +prevalence of ``None`` in Python. + +``?`` last available ASCII character +------------------------------------ + +Another objections was that ``?`` is one of the last available ASCII +characters for new syntax. It was suggested to use something else. + +While this is true, the use of ``?.`` and ``?[ ]`` for "null"- / +``None``-aware operators in other languages means that it would be +difficult to use ``?`` for anything else. + +Furthermore, it is common for developers to use / be fluent in multiple +programming languages. It is up the Python language specification to +provide a meaning for these operators which roughly matches those in +other languages while still respecting the norms in Python itself. + + +Footnotes +========= + +.. [#discuss_revisit_505] discuss.python.org: Revisiting PEP 505 - None-aware operators + (https://discuss.python.org/t/revisiting-pep-505-none-aware-operators/74568) +.. [#discuss_safe_navigation_op] discuss.python.org: Introducing a Safe Navigation Operator in Python + (https://discuss.python.org/t/introducing-a-safe-navigation-operator-in-python/35480) +.. [#ts] TypeScript: Optional Chaining + (https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html#optional-chaining) +.. [#js] JavaScript: Optional chaining (?.) + (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining) +.. [#js_short_circuiting] JavaScript: Optional chaining (?.) - Short-circuiting + (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining#short-circuiting) +.. [#csharp] C# Reference: Member access operators + (https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/operators/member-access-operators#null-conditional-operators--and-) +.. [#dart] Dart: Other operators + (https://dart.dev/language/operators#other-operators) +.. [#swift] Swift: Optional Chaining + (https://docs.swift.org/swift-book/documentation/the-swift-programming-language/optionalchaining/) +.. [#kotlin] Kotlin: Safe call operator + (https://kotlinlang.org/docs/null-safety.html#safe-call-operator) +.. [#ruby] Ruby: Safe navigation operator + (https://ruby-doc.org/core-2.6/doc/syntax/calling_methods_rdoc.html#label-Safe+navigation+operator) +.. [#php] PHP: Nullsafe operator + (https://wiki.php.net/rfc/nullsafe_operator) +.. [#glom] PyPI: glom + (https://pypi.org/project/glom/) +.. [#jmespath] PyPI: jmespath + (https://pypi.org/project/jmespath/) +.. [#nonesafe] PyPI: nonesafe + (https://pypi.org/project/nonesafe/) +.. [#pymaybe] PyPI: pymaybe + (https://pypi.org/project/pymaybe/) +.. [#py_grammar] Python documentation: Full Grammar specification + (https://docs.python.org/3/reference/grammar.html) + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.