C API: Remove private C API functions (move them to the internal C API) #106320

vstinner · 2023-07-01T21:35:51Z

Over the years, we accumulated many private functions as part of the public C API header files. I propose to remove them: move them to the internal C API.

If many C extensions are affected by these removals, it's a sign that we should consider promoted used private functions as public functions: provide a better API, add error handling, write documentation, write tests.

Summary: My plan to clarify private vs public functions in Python 3.13

Private functions converted to public functions:
- PyLong_AsInt(): issue [C API] Add PyLong_AsInt() public function #108444
- Py_IsFinalizing(): PR gh-108014: Add Py_IsFinalizing() function #108032
Discussions:
- (pssst) Let’s treat all API in public headers as public
- list of PyPI affected projects
- C API: How much private is the private _Py_IDENTIFIER() API? -- _Py_Identifier
- _Py_c_xxx complex functions (ex: _Py_c_abs())
- PR gh-107137: Add _PyTupleBuilder API to the internal C API #107139: Add _PyTupleBuilder API to the internal C API
Notes:
- While being private, _PyBytes_Resize() and _PyTuple_Resize() are documented
- The private _PyCrossInterpreterData API is used by 3rd party project on PyPI: gh-106320: Remove private _PyCrossInterpreterData API #107068 (comment). I closed my PR gh-106320: Remove private _PyCrossInterpreterData API #107068.

Linked PRs

The text was updated successfully, but these errors were encountered:

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

vstinner · 2023-07-01T22:04:07Z

See also issue #106316: remove pytime.h header file (_PyTime functions).

vstinner · 2023-07-01T22:04:33Z

See also issue #106084: C API: Remove private functions from abstract.h.

vstinner · 2023-07-01T22:16:19Z

See also commit 0b51463:

Remove the following private functions of the C API:

* _PyCodecInfo_GetIncrementalDecoder()
* _PyCodecInfo_GetIncrementalEncoder()
* _PyCodec_DecodeText()
* _PyCodec_EncodeText()
* _PyCodec_Forget()
* _PyCodec_Lookup()
* _PyCodec_LookupTextEncoding()

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

* Remove private _PyTracemalloc C API functions: move them to the internal C API. * Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h

* Remove private _PyTracemalloc C API functions: move them to the internal C API. * Don't export private _PyTracemalloc functions anymore. * Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h

* Remove private _PyTraceMalloc C API functions: move them to the internal C API. * Don't export most of these functions anymore, but still export _PyTraceMalloc_GetTraceback() used by tests. * Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h

Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

* Remove private _PyTraceMalloc C API functions: move them to the internal C API. * Don't export most of these functions anymore, but still export _PyTraceMalloc_GetTraceback() used by tests. * Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h

Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

Add internal pycore_complexobject.h header file. Move _Py_c_xxx() functions and _PyComplex_FormatAdvancedWriter() function to this new header file.

…ng() Replace private _PyUnicode_EqualToASCIIString() with public PyUnicode_CompareWithASCIIString().

…#111162) * pythongh-106320: Re-add _PyLong_FromByteArray(), _PyLong_AsByteArray() and _PyLong_GCD() to the public header files since they are used by third-party packages and there is no efficient replacement. See python#111140 See python#111139 * pythongh-111262: Re-add _PyDict_Pop() to have a C-API until a new public one is designed.

See python/cpython#106320 and python/cpython#107032 Closes #1526

…#106335) Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

The `_PyErr_FormatFromCause` function is a private function, which was never documented. While it existed for a long time, it was removed in Python 3.13 [1]. Use instead the public function `PyErr_Format`, which is also part of the stable API. [1] python/cpython#106320 Signed-off-by: Arthur Zamarin <[email protected]>

…#111162) * pythongh-106320: Re-add _PyLong_FromByteArray(), _PyLong_AsByteArray() and _PyLong_GCD() to the public header files since they are used by third-party packages and there is no efficient replacement. See python#111140 See python#111139 * pythongh-111262: Re-add _PyDict_Pop() to have a C-API until a new public one is designed.

Python 3.13 moved some private C API functions to internal header files. See python/cpython#106320 for a general discussion on this (glad to see that gnureadline was at least listed as an affected PyPI package :-) ). These private functions affect us: - _Py_SetLocaleFromEnv -> pycore_pylifecycle.h (see python/cpython#106400) - _PyArg_CheckPositional, _PyArg_BadArgument -> pycore_modsupport.h (see python/cpython#110964) Since we can't include these anymore, patch the relevant declarations into the module code. The alternative (add the internal headers to this package) is much messier, as we would have to pull in most of those headers.

clin1234 · 2024-12-17T14:52:03Z

@vstinner Given the number of commits in various projects affected by this, perhaps there should a comprehensive list of replacement functions for each corresponding private function included in the What's New section within 3.13's documentation.

vstinner · 2025-01-13T15:08:46Z

@clin1234:

@vstinner Given the number of commits in various projects affected by this, perhaps there should a comprehensive list of replacement functions for each corresponding private function included in the What's New section within 3.13's documentation.

Good idea: I created #128787 for that.

) (cherry picked from commit 43ef958) Co-authored-by: Victor Stinner <[email protected]>

…128837) gh-106320: Document replacement for removed C API (GH-128787) (cherry picked from commit 43ef958) Co-authored-by: Victor Stinner <[email protected]>

vstinner added the topic-C-API label Jul 1, 2023

vstinner added a commit to vstinner/cpython that referenced this issue Jul 1, 2023

pythongh-106320: Remove _PyInterpreterState_Get() alias

8cf018e

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

vstinner added a commit to vstinner/cpython that referenced this issue Jul 1, 2023

pythongh-106320: Remove _PyInterpreterState_Get() alias

c053d2b

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

vstinner added a commit to vstinner/cpython that referenced this issue Jul 1, 2023

pythongh-106320: Remove _PyInterpreterState_Get() alias

1b38110

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

bedevere-bot mentioned this issue Jul 1, 2023

gh-106320: Remove _PyInterpreterState_Get() alias #106321

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 1, 2023

pythongh-106320: Remove _PyInterpreterState_Get() alias

096b84a

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

vstinner added a commit that referenced this issue Jul 1, 2023

gh-106320: Remove _PyInterpreterState_Get() alias (#106321)

18b1fde

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Remove private _PyTraceMalloc C API functions #106324

Merged

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Remove private _PyInterpreterState functions #106325

Merged

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Remove private _PyInterpreterState functions #106335

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 2, 2023

pythongh-106320: Use _PyInterpreterState_GET()

72f9063

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

vstinner added a commit to vstinner/cpython that referenced this issue Jul 2, 2023

pythongh-106320: Use _PyInterpreterState_GET()

85ea281

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Use _PyInterpreterState_GET() #106336

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 2, 2023

pythongh-106320: Use _PyInterpreterState_GET()

606da6d

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

vstinner added a commit that referenced this issue Jul 2, 2023

gh-106320: Use _PyInterpreterState_GET() (#106336)

bc7eb17

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Add pycore_complexobject.h header file #106339

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 2, 2023

pythongh-106320: _testcapi: avoid private _PyUnicode_EqualToASCIIStri…

3ea652f

…ng() Replace private _PyUnicode_EqualToASCIIString() with public PyUnicode_CompareWithASCIIString().

musicinmybrain mentioned this issue Jan 23, 2024

Internal API _PyBytes_Join() is removed in Python 3.13 pycompression/python-zlib-ng#30

Closed

lachbr pushed a commit to toontownretro/panda that referenced this issue Feb 17, 2024

event: Replace _PyGen_FetchStopIterationValue in Python 3.13+

043417b

See python/cpython#106320 and python/cpython#107032 Closes #1526

rogerbinns mentioned this issue Mar 14, 2024

_PyErr_ChainExceptions1 no longer avallable to extension modules in Python 3.13 alpha headers #116809

Open

cdce8p mentioned this issue Jul 8, 2024

[C API] Private functions used by mypyc #121489

Closed

arthurzam mentioned this issue Aug 28, 2024

binding/query.c: use PyErr_Format instead of _PyErr_FormatFromCause tree-sitter/py-tree-sitter#284

Closed

stefanor mentioned this issue Sep 13, 2024

Fails to build on Python 3.13: _PyLong_AsByteArray is a private API googlefonts/compreffor#156

Open

eirnym mentioned this issue Oct 17, 2024

fails to build with Python 3.13 MagicStack/asyncpg#1181

Closed

clin1234 mentioned this issue Dec 16, 2024

Moving _PyMem_RawStrdup to private API causes build failure #127991

Closed

Hoikas mentioned this issue Jan 5, 2025

_PyModule_Clear missing in Python 3.13 H-uru/Plasma#1649

Open

vstinner added a commit to vstinner/cpython that referenced this issue Jan 13, 2025

pythongh-106320: Document replacement for removed C API

7f25da3

vstinner added a commit to vstinner/cpython that referenced this issue Jan 13, 2025

pythongh-106320: Document replacement for removed C API

299d662

bedevere-app bot mentioned this issue Jan 13, 2025

gh-106320: Document replacement for removed C API #128787

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jan 13, 2025

pythongh-106320: Document private C APIs promoted to public C API

eddf0ed

bedevere-app bot mentioned this issue Jan 13, 2025

gh-106320: Document private C APIs promoted to public C API #128788

Merged

vstinner mentioned this issue Jan 14, 2025

gh-69639: add mixed-mode rules for complex arithmetic (C-like) #124829

Merged

vstinner added a commit that referenced this issue Jan 14, 2025

gh-106320: Document private C APIs promoted to public C API (#128788)

41f7350

skirpichev mentioned this issue Jan 14, 2025

Make _Py_c_sum(), _Py_c_diff(), etc (elementary operations on Py_complex) - part of the public API #128813

Open

vstinner added a commit that referenced this issue Jan 14, 2025

gh-106320: Document replacement for removed C API (#128787)

43ef958

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 14, 2025

pythongh-106320: Document replacement for removed C API (pythonGH-128787

6baa1d4

) (cherry picked from commit 43ef958) Co-authored-by: Victor Stinner <[email protected]>

bedevere-app bot mentioned this issue Jan 14, 2025

[3.13] gh-106320: Document replacement for removed C API (GH-128787) #128837

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

C API: Remove private C API functions (move them to the internal C API) #106320

C API: Remove private C API functions (move them to the internal C API) #106320

vstinner commented Jul 1, 2023 •

edited by bedevere-app bot

Loading

vstinner commented Jul 1, 2023

vstinner commented Jul 1, 2023

vstinner commented Jul 1, 2023

clin1234 commented Dec 17, 2024

vstinner commented Jan 13, 2025

C API: Remove private C API functions (move them to the internal C API) #106320

C API: Remove private C API functions (move them to the internal C API) #106320

Comments

vstinner commented Jul 1, 2023 • edited by bedevere-app bot Loading

Linked PRs

vstinner commented Jul 1, 2023

vstinner commented Jul 1, 2023

vstinner commented Jul 1, 2023

clin1234 commented Dec 17, 2024

vstinner commented Jan 13, 2025

vstinner commented Jul 1, 2023 •

edited by bedevere-app bot

Loading