diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 6ac66492831..fbacbd2388a 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -648,6 +648,7 @@ peps/pep-0767.rst @carljm peps/pep-0768.rst @pablogsal peps/pep-0769.rst @facundobatista peps/pep-0770.rst @sethmlarson @brettcannon +peps/pep-0773.rst @zooba # ... peps/pep-0777.rst @warsaw # ... diff --git a/peps/pep-0773.rst b/peps/pep-0773.rst new file mode 100644 index 00000000000..c26ce1f695c --- /dev/null +++ b/peps/pep-0773.rst @@ -0,0 +1,1622 @@ +PEP: 773 +Title: A Python Installation Manager for Windows +Author: Steve Dower +Discussions-To: https://discuss.python.org/t/77900/ +Status: Draft +Type: Standards Track +Topic: Release +Created: 21-Jan-2025 +Post-History: + `18-Dec-2024 `__, + `21-Jan-2025 `__, +Replaces: 397, 486 + + +Abstract +======== + +Installation of the python.org Python distribution on Windows is complex. +There are three main approaches with roughly equivalent levels of user +experience, and yet all of these suffer from different limitations, including +failing to satisfy modern usage scenarios. This PEP proposes a design for +a single Windows install workflow tool that satisfies all the needs of the +existing installers for the platform, while avoiding most of their limitations, +and provides the core team with the ability to manage releases for many years +to come. + +Reader's Note +------------- + +This document is a detailed *proposal*, intended to assist in an "approve" or +"reject" decision on the replacement of the existing install formats for Python +on Windows with a new format. It is not intended to be a binding specification, +nor is it user documentation or an interoperability specification. The actual +implementation may vary over time as needs change, and that is not to be +considered a "violation" of this PEP. Any interfaces and protocols meant for +public use will be independently documented and maintained in accordance with +standard deprecation timelines. + +Documentation on installing Python on Windows can be found at +`docs.python.org/using/windows.html `__. + +Background +========== + +There are a large range of needs users may have that lead to them wanting +to install a Python runtime. Many, likely most, are interested in running +(perhaps writing) short scripts, such as those that perform a simple task, +or help teach someone a concept. Some users are looking for a specific version +to integrate with existing code or another application. Some are after a full +set of different interpreter versions to perform testing. + +In this section, we discuss the expectations that users have of "installing +Python", provide an overview of the existing installers for Windows, and +identify some of the gaps and challenges inherent in these offerings. + +Expectations +------------ + +Based on significant anecdotal experience and analysis of quantitative data +available (though not necessarily public), we make the following assertions +about the majority of Python users on Windows: + +* most users just want the latest stable version +* most users want a "one-click" (or fewer) install +* most users do not want to use administrator privileges +* most users will benefit from installing maintenance updates +* most users expect the ``python`` command to work after installation + +The primary support for these assertions is that the most popular installers +actively chosen by users are the latest stable release on python.org, and +the latest stable release on the Windows Store, both of which meet these +requirements. + +We make the following assumptions about other significant sets of users. +These may have some overlap between groups, and at least some users expect +all of them. + +* some users want to install Python programmatically +* some users want to install a particular version +* some users want to install many versions +* some users want to install for all users of their machine +* some users do not want Start Menu shortcuts +* some users want to install as part of their project's build process +* some users want to install as part of their project's install process +* some users intend to never update their install +* some users expect the ``py`` command to work after installation + +These assumptions have all been demonstrated over time to exist, though the +relative importance has not been quantified. The :ref:`NuGet packages +` and the :ref:`embeddable distro +` can meet most of these needs. + +Traditional Installer +--------------------- + +The traditional installer is an executable downloadable directly from +python.org that installs the entire development kit for Python. This includes +the CPython interpreter, the standard library, Python headers and import +libraries, builds of Tcl and Tk, the documentation as HTML files, the runtime +and standard library test suite, Start Menu shortcuts for Python and IDLE, +debugging symbols and debug builds of the binaries, the ``py.exe`` launcher +and its file associations, and functionality to modify the user's ``PATH`` +environment variable, enable long-path support on their system, pre-generate +``.pyc`` files for the standard library, and install pip. +As of 3.13, it also includes a set of experimental free-threaded binaries. +Many of these components are optional. + +After downloading the executable, users are presented with a "quick install" +option, which installs into their user directory with most options enabled. +We believe that most users select this option. + +A second option alongside the quick install takes the user to two pages worth +of options, listing the components that they need not install, as well as other +options such as the install directory and whether to install for all users. + +All of these options may be specified on the command line, and there is also an +option to proceed with the install without displaying any UI. +Based on feedback and bug reports, all of these options are used by at least +some users. However, as we do not track install telemetry, we have no way to +know which options are more important than others. + +Behind the scenes, the traditional installer is a Burn bundle, generated using +the Wix Toolset installer framework, containing one or more MSI files for each +feature. This framework is used extensively by Microsoft themselves, and +provides the most direct method of using Windows Installer. The bundle is a +custom C++ application, based on their template, which allows us to customise +the overall behaviour of the installer to determine precisely which MSI files +should actually be installed. The process of copying files, updating the +registry, and generating shortcuts is handled entirely by Windows Installer. + +As well as the intended uses, it is understood that many users will (attempt to) +use the traditional installer for other scenarios, such as unregistered installs +and automated CI system installs. While better alternatives are available, they +are not as obvious, and the hope is that a future design would make these +scenarios easier. + +Windows Store +------------- + +The Windows Store packages for CPython are produced as part of our normal +release process using almost all identical binaries to the other packages. +Due to being in an app store package, the primary ``python.exe`` is enhanced +to be able to determine its location properly, and alternative ``pip.exe`` and +other shortcuts are included to make up for the lack of ``PATH`` environment +variable settings. These are implemented in :file:`PC\\python_uwp.cpp` in our repo. + +These packages are installed by searching for Python in the Microsoft Store +app, which will find results for each major version since 3.8. Users then have +to select a version and install it. These packages include the CPython +interpreter, standard library, Tcl/Tk, IDLE, and pip, and create file +associations, Start Menu shortcuts, and global commands for ``python.exe``, +``python3.exe``, ``python3.X.exe``, ``pip[3[.X]].exe`` and ``idle[3[.X]].exe``. +No ``PATH`` modification is possible or required, though users may need to +manage their global shortcuts through the "Manage App Execution Alias" settings +page. + +In addition, Microsoft has added to a clean Windows install a default +``python.exe`` command. This captures attempts by users to launch Python on +a machine that has not yet installed it. When launched directly, the command +will open the Microsoft Store app to the page containing the recommended +Python app, typically the latest version. This app is entirely controlled by +Microsoft. Based on telemetry associated with the Python app (which *is* +controlled by the upstream Python project), approximately 300,000 installs +per month come through this redirector, making up about 90% of the total +installs of that version. + +Behind the scenes, the Store package is based on Microsoft's new installer +technology for apps known as APPX or MSIX. These are essentially plain ZIP +files with a small amount of metadata, except that installation is handled +by the operating system. They are always extracted to a fixed location, +accessible to all users but not modifiable by any, and automatically updated +to the latest release. The user's own data is stored in an OS-managed location +in their user profile, and is able to be reset, backed-up and restored using +regular OS functionality. + +Nuget Package +------------- + +The Nuget packages for CPython are produced and published as part of our +normal release process. The contents are identical to the traditional +installer. A Nuget package is published to nuget.org, which is a package +manager typically associated with .NET languages, but highly integrated with +any project supported by Visual Studio. This makes it a nice format for users +who want a lightweight install of Python as part of their regular build process, +and can simplify embedding scenarios. + +The packages are installed using any tool capable of using the Nuget API, or +may be downloaded directly once the URL of the package is known. The package is +a plain ZIP file with some metadata. It contains the CPython interpreter, the +standard library, development headers and import libraries, and pip. It does +not execute any code at install time, and users must locate the package +themselves in order to launch the ``python.exe`` contained within. + +Embeddable Package +------------------ + +The embeddable package for CPython is produced and published as part of our +normal release process. It is published to python.org alongside the +traditional installer. The contents are identical, however, the layout is +changed to store all binaries at the top level, with the standard library +packed into a ZIP file. A ``._pth`` file is included to override ``sys.path`` +so that only the files that are part of the distro are used, and environment +variables or registry entries are ignored. + +This package does not include pip, as the intention is for it to be embedded +into a broader application. Other libraries should be installed at build time, +since after distribution the runtime is meant to be an internal implementation +detail of the app it is a part of. + +As well as its intended use, some users attempt to use this package as a +development kit rather than a runtime package. This is believed to be due to +those users preferring to avoid "heavyweight" installers, and believing that +this package is intended to be a "portable" install (extract and run), likely +because it is the only ZIP file option listed on the python.org download pages +(speaking to the importance of clarity and limiting options on those pages). +It is hoped that a future installer design will avoid or limit this confusion. + +Alternate Distributions +----------------------- + +While outside of our purview as the core team, alternate distributions of Python +for Windows often use a project, workflow or environment-centric model for +installation of the runtime. By this, we mean that the tool is installed first, +and is used to create a working space that includes a runtime, as well as other +dependencies. Examples of these tools include conda and uv. + +Two observations are worth making about these tools. Firstly, they are often +praised for being low impact, in that they usually don't install additional +entry points or files for the runtime, making the install fast and also isolated +to a single project. Secondly, their users often appreciate the ease of +selecting a particular version of a runtime, or alternatively, not having to +select at all because existing specifications (or constraints) can choose for +them. + +These tools tend to meet many of the second set of expectations described above, +usually combining multiple tasks in a single command to reduce the cognitive +overhead of learning how to use and combine multiple commands. + +It's also worth pointing out that the core team does not view these alternate +distributions as competitors to any upstream distribution. They are a +fundamental part of how the open source ecosystem is intended to work. Our own +distributions are a convenience for those who choose to use them, as not all +scenarios are well served by a workflow tool or even a pre-built package. + + +Challenges +---------- + +There are numerous challenges we face with the current set of installers, +which largely break down into two categories: mismatched or unachievable +user expectations, and general unreliability. + +The traditional installer has the highest level of unreliability. The Windows +Installer technology is very old, and effectively no longer under development. +While its basic functionality is okay, interference may come from many sources, +such as virus scanners, other installers, system configuration, admin policies, +and even other files in the same directory as the installer. On top of this, +most of its advanced and beneficial functionality such as update patches, +incremental updates, and automatic rollback are unimportant for Python users. + +Most user expectations are *defined* by the traditional installer, and so by +definition, it meets them. One primary gap is that it is not able to create an +"unmanaged" install - that is, the equivalent of only copying files onto the +user's system without registration. If you have installed it once, and you +try to install it again, you will only even be able to manage (or upgrade) the +existing install. This can lead to installs moving on update, which will +break users. + +Additionally, the ``PATH`` environment variable cannot be intelligently +modified - at best, we can prepend or append the install path. This usually +results in the most recent install of Python being the highest priority. For +example, if the user has Python 3.14 installed and then installs (or updates) +3.13, the ``python`` command will switch from the later version to the earlier +version. + +The ``py.exe`` launcher, defined in :pep:`397` and implicitly updated by :pep:`514`, +is an attempt to avoid this particular issue. It uses its own logic for finding +installed versions without relying on ``PATH``. However, the PEP 514 logic does +not allow for prerelease or experimental builds to be treated specially, and so +``py.exe`` often prefers these builds by default over the non-experimental +version expected by the user. + +The Windows Store package is very reliable, with the exception of the global +shortcuts. Rather than modifying ``PATH`` to add its own directory, these +shortcuts are created in a single OS managed directory that has all the +shortcuts defined by any app. Users are able to modify their ``PATH`` to exclude +or de-prioritise this directory, leading to unreliable or inconsistent +behaviour, and historically we have also seen this caused by installers. +For example, installing Python from the Store followed by Python from the +traditional installer with its ``PATH`` modification enabled will almost always +shadow the Store package's Python with the later install. + +User expectations that are un-met by the Store package tend to be performance +and technical. Due to the overhead of launching an app, Python starts up slower. +Because apps are designed to be isolated from each other, it is more difficult +to use hidden directories (such as ``AppData`` or ``TEMP``) to communicate +between different versions of Python, as each version has its own space. Apps +are subject to stricter security requirements that legacy applications usually +have disabled, such as DLL hijacking protection, which causes some libraries to fail. +The ``python3`` and ``python`` shortcuts are managed through system settings, +and the user interface is not very good (and not going to be improved, according +to Microsoft). Without managing these, it is relatively easy for an undesired +version to be launched, though in general the targets can only be changed +manually by the user, and not by merely installing another app. + +Both the Nuget package and the embeddable distro are as simple and reliable to +install as extracting an archive file, though it's worth noting that for many +Python users this is not a common task. They provide no install management at +all, and cannot be reliably updated other than by deleting and re-extracting. +User expectations that are un-met are almost always due to users selecting the +wrong installer. Both these packages are for specialised cases, and while they +are documented as such, the attraction of a plain ZIP file leads some users into +failure. + +Overview of PyManager +===================== + +("PyManager" name open for bikeshedding) + +PyManager is the internal name of our proposed replacement installer tool. It +will be distributed both in the Windows Store and on python.org as an MSIX +package. Downloading from either source will get an identical package, and +both will support automatic updates (through the Store) for new releases. + +The user visible name will be "Python Install Manager", published by the +Python Software Foundation. After publishing, we will request that Microsoft +adjust their ``python.exe`` stub to open to this new app. + +This app does not directly provide a version of Python, but it does provide the +global commands that users expect to work, as well as file associations and +Start menu shortcuts. The OS will prompt users to launch the app after install, +which will trigger an automatic install of the current release of CPython and +then launch it. From the user's perspective, they have the same initial +experience as today, with one added progress bar on first launch. + +The global commands provided by the app must be static and bundled into the app +itself. They can only change their behaviour at runtime, and cannot be +redirected to different executables except by the user (and then only to another +installed app). So the commands to be provided by PyManager are ``python.exe``, +``python3.exe``, ``py.exe``, ``pymanager.exe``. Each of these must have the +ability to inspect the user's system and choose the correct runtime to launch. +Additionally, ``py`` and ``pymanager`` will have management subcommands to allow +adding and removing runtimes. + +In line with :pep:`394` and the default behaviour of Windows, the recommended +command for launching Python is ``python.exe``. As provided by PyManager, this +will locate an existing install, either among those that PyManager manages or +using :pep:`514`, or it will install the latest available version of CPython and +select that. The ``python3.exe`` command behaves similarly, but is only allowed +to find 3.x installs from python.org. + +The ``py.exe`` command provided by PyManager will be recommended for most +management use, due to its brevity. ``py install ...``, ``py list ...`` and so +on. The proposed commands are detailed later. The existing behaviour of the +:pep:`397` launcher is preserved, however, launching through ``py`` will not +automatically install runtimes (by default). If one is requested but is not +installed, users will just get an error. The ``py run ...`` subcommand, however, +will install automatically, and supports the same options as bare ``py``. + +These commands are added at very low priority in the user's ``PATH`` by the OS. +Every existing configuration we may have created on a user's machine will take +precedence over these commands, and so these are a last resort in place of an +error message. As a result, we can generally assume that a user is launching +these commands because they haven't configured a stronger preference (for +example, a user who has activated an environment will never launch our +``python.exe``, because activation will put a different one ahead of it, and a +user who wants precisely the behaviour of the existing ``py.exe`` can just +install it and will never launch our new one). + +The ``pymanager.exe`` command is to allow for handling ambiguous situations. +Existing installs of Python and the launcher may shadow ``python.exe`` and +``py.exe``, but in an automated environment, this can make administrative +scripts unreliable, and so the ``pymanager`` command is unlikely to refer to +something other than PyManager. It has all the subcommands, and launching it +with no command specified will print help for the user. + + +Replacing other installers +-------------------------- + +Our intent is to immediately stop publishing individual versions to the Windows +Store, and to deprecate and phase out the traditional installer and Nuget +packages by Python 3.16. The embeddable distro would remain, but its listing on +python.org download pages would be phased out and it will be available only +through PyManager. + +PyManager will be made available as an app package downloadable manually from +python.org, and the double-click install experience is generally smooth. This +provides an equivalent to the current approach of downloading from our site. +It will bundle a recent (unspecified) version of CPython so that the download +can be moved to a non-internet connected machine and still provide a Python +runtime after install. + +Some automated deployment scenarios do not work with the newer MSIX format, and +so a simple MSI will also be provided on python.org. This will have no options +or user interface, and require administrative privileges, which are typically +available for these kinds of scenarios. This MSI would be discouraged for most +users, and the MSIX should be the default. + +It's worth noting that there is no way to make the MSI install fully compatible +with the MSIX, and users with both will likely encounter confusion or problems. +It is anticipated that only users without Store app support will use the MSI. + +Our release processes will start publishing plain ZIP packages to python.org. +These will be available from the FTP pages, but will not be listed directly on +regular download pages. + +Third-party tools that currently distribute their own builds of CPython will be +welcome to use ours, though will be expected to be the initial point of contact +for their users requiring support. + + +Project ownership and development +--------------------------------- + +PyManager will be developed and maintained in its own repository adjacent to +the CPython repository, and under the same terms. The CPython CLA will apply, +and all (and only) core developers will have commit rights. + +PyManager releases are independent from CPython releases. There is no need for +versions to match, or releases to be simultaneous. Unless otherwise arranged, +the PyManager release manager is whoever is the build manager for Windows. + + +Specification +============= + +.. note:: + In this document, all command line options will be shown with one or two + hyphens. In implementation, all options will support one or two hyphens or a + forward slash, to be permissive of both Windows and UNIX conventions. + +Exec subcommand +--------------- + +.. code:: text + + pymanager exec -V:tag [interpreter opts] [script.py|-m module|-c code] [script args] + pymanager exec -3.* ... + pymanager exec [--only-managed] [--[no-]install] [-V:|-3.*] ... + py [-V:|-3.*] ... + python ... + python3 ... + +This subcommand is used to select and launch a runtime. It is the default action +for the ``py`` command, and the only action supported by the ``python`` and +``python3`` commands. The default options are subtly different in each case for +consistency with existing use of these commands. + +This subcommand is available on both ``py`` and ``pymanager``. However, since +``py`` offers it by default, we would not expect users to use it there. The +intent is that the ``py``, ``python`` and ``python3`` commands are the default +ways to launch a runtime, and ``pymanager exec`` is for advanced scenarios. + +The ``-V:tag`` command is used to request a specific runtime from the command +line. The tag is a ``Company\\Tag`` pair, or just ``Tag`` if no slash is +included, and is used as defined by PEP 514. The ``-3.*`` option is interpreted +as ``-V:PythonCore\\3.*``. This option is only available for ``py`` and +``py exec`` variants. + +If no tag is specified on the command line, and a script file is specified, +the script will be inspected for a shebang. If one is found that matches a +recognised pattern, it will either provide the tag to be used for search, or it +will override all other processing and its specified executable will be launched +without further effort being made. This is to handle the (unfortunate) legacy +support of arbitrary Windows-specific paths being allowed in what was meant to +be a portability feature. In general, shebangs including simple patterns like +``/usr/bin/python3.13`` are intended, while those that use ``/usr/bin/env +python`` are unlikely to be of benefit since the environment tends to be less +reliable than our search. + +If no tag is yet requested, the ``VIRTUAL_ENV`` environment variable will be +consulted to see if an environment has been activated. If so, that will become +the request. + +If a tag has been requested at this stage, the ``python3`` command will verify +that it matches ``PythonCore\\3.*`` and exit with an error if not. This allows +allows the ``python3`` command to be used in an active environment consistent +with other platforms, but not if the environment would not have included the +command. This applies to most existing versions of Python on Windows. (The +alternative to this behaviour is to make ``python3`` always error when an +environment is active, as anything else would behave inconsistently for the +user.) + +If no tag is requested, the default will be consulted. For ``python3``, this is +``PythonCore\\3``, but for all other commands it is read from configuration +(which might involve an environment variable). If it's still empty, any tag will +be allowed. + +The best installed runtime matching the tag is then selected and launched with +the remaining command line. + +If no matching runtime is found, and the ``--install`` option is set, it will be +installed and then launched. This is the default for ``py exec``, and also for +``python`` but only when no runtime was requested (which implies that no +runtimes were installed). In all other cases, if no install is found, an error +is printed and the process exits. + + +Install subcommand +------------------ + +.. code:: text + + py install [-s|--source ] [-f|--force] [-u|--upgrade] [tag ...] + py install [-s|--source ] [-t|--target ] [tag ...] + py install [-s|--source ] [-d|--download ] [tag ...] + py install --refresh + +.. note:: + This and all later subcommands are also available under ``pymanager``. + However, as we intend for ``py`` to be the usual command, we only show that + one. + +This subcommand will install one or more runtimes onto the current machine. +The tags are ``Company\\Tag`` pairs (or just ``Tag`` if no slash is included), +and are used to search the index file. Company names match as case-insensitive +prefixes, preferring a full match over a prefix, and tags use case-insensitive, +number-aware matches, with dotted numbers treated as versions. Tags must match +one of the listed "install for" tags, and entries list multiple such tags to +handle abbreviated requests. + +For example, the ``3.10.5`` entry would list all of ``3``, ``3.10`` and +``3.10.5`` as tags to be installed for. A request for ``3.10`` would match one +of these and so the entry is selected. Due to the number-aware matches, a +request for ``03.0010`` would also match, and ``3.10.50`` would not. + +Tags may also be specified as a constraint, using ``>``, ``>=``, ``<``, ``<=`` +or ``!=`` followed by the ``Company\\Tag`` or ``Tag`` value. When matching a +constraint only the primary tag metadata is used for comparisons. Since the +comparisons are version-aware, constraints such as ``>3.10`` will select +3.11 as a minimum, while ``>3.10.0`` may select 3.10.1. + +The behaviour of constraints against arbitrary tags is likely to be unintuitive +in some circumstances. It is anticipated that constraints will mainly be used +with upstream releases, which typically use version-shaped tags, and primarily +for cases where other metadata such as ``Requires-Python`` are being handled. +Users are expected to use shorter tags for convenience, rather than ranges. + +The default index file is hosted on python.org, and contains install information +including package URLs and hashes for all installable versions. An alternate +index may be specified by the user or their administrator (see Configuration +below). Entries in the index file list the full tags they should be installed +for, and if an exact match is found the package will be selected. In the case +of no exact match, a prefix match will be used. In both cases, numbers in the +tag are treated logically - that is, ``3.1`` is a prefix of ``3.1.2`` but not of +``3.10``. + +If a tag is already satisfied by an existing install, nothing will be installed. +The user must pass an ``--upgrade`` or ``--force`` option to replace the +existing install; the former will only replace it with a newer version, while +the latter will remove and replace even with the same version. + +Calling the command without providing any tags will install the latest default +version (in effect, the first non-prerelease entry in the index). +Passing ``--upgrade`` with no tags is an error. + +Passing ``--refresh`` will regenerate all metadata and shortcuts for all +installs. This is intentionally applied to all installs at once, as shortcut +prioritisation relies on all installs being consistent (for example, the latest +3.x version should get the ``python3.exe`` shortcut, which gets complicated if +users can choose to only refresh an older install). + +If a ``--target `` option is passed with only a single tag, that runtime +will be extracted to the specified directory without being registered as an +install (or generating aliases or shortcuts). This is intended to cover +embedding cases, or downloading the files for incompatible platforms. Passing +multiple tags with ``--target`` is an error. + +If the ``--download `` option is passed, runtimes will be downloaded to the +specified directory as their source packages, and an ``index.json`` will be +created referencing these files. This index can be referenced later to perform +offline installs with ``python install --source [tag ...]``. + + +Uninstall subcommand +-------------------- + +.. code:: text + + py uninstall [-y|--yes] [--purge] [tag ...] + +This subcommand will uninstall one or more runtimes on the current machine. Tags +are exactly as for the install command, including prefix matching, but only +inspect existing installs. Unless the ``--yes`` option is passed, the user will +be prompted before uninstalling each runtime. + +If the ``--purge`` option is passed with no tags, then (after confirmation) all +runtimes will be removed, along with shortcuts and any cached files. + + +List subcommand +--------------- + +.. code:: text + + py list [-f|--format ] [-1|--one] [--only-managed] [tag ...] + py list [-f|--format ] [-1|--one] [--online] [--source ] [tag ...] + py [--list|-0] + py [--list-paths|-0p] + +This subcommand will list any or all installs matching the specified tags or +ranges. If no tags are provided, lists all installs. Runtimes not managed by +PyManager (including an active virtual environment) may be listed separately. + +The default format is user-friendly. Other formats will include machine-readable +and single string formats (e.g. ``--format=prefix`` simply prints ``sys.prefix`` +on a line by itself). The exact list of formats is left to implementation. + +If ``--one`` is provided, only the best result is listed. This is to assist +shell scripts that want to locate the default (or a suitable) runtime without +launching it. (Note that "best" is loosely defined, but will always be the +user's preferred default environment if it is included in the results.) + +The ``--only-managed`` option omits runtimes that were discovered but are not +managed by PyManager, for example, those found using a regular PEP 514 lookup. + +Passing ``--source`` (or ``--online`` to implicitly pass the default source) +will search an online index rather than currently installed runtimes. The option +is here rather than on the ``install`` subcommand because the filtering and +formatting options are already available on ``list``. + +The legacy ``--list``, ``--list-paths``, ``-0`` and ``-0p`` arguments from the +``py.exe`` launcher will also be provided. However, they will not support the +new options listed here, and are limited to reproducing the output from the +existing launcher. Unmanaged installs are not distinguishable in this listing. + + +Help subcommand +--------------- + +.. code:: text + + py help + +This subcommand will display the help text for each specified command, or if +none specified, will show the list of commands. Specifying one command is the +equivalent of ``py --help``. Showing the list of subcommands is the +default action for the ``pymanager`` command. + +The command is added primarily to offer a simple way to tell users how to find +more information: they can be told to run ``py help``. This avoids having to +override or extend the ``python -?`` output, which otherwise forwards to the +selected runtime and already prints at least one screen's worth of text. + +After an automatic install (e.g. running ``python`` with nothing installed), a +message will be displayed telling users that they can run ``py help`` for more +information on how to manage their installs. + + +Environment Variables +--------------------- + +No environment variables can be updated automatically when installing a Store +app, and so no updates will be done automatically. The core commands should +already be available on a correctly functioning machine. + +One directory within the user's PyManager data directory is set aside for +generated aliases. If desired, the user can add this directory to their ``PATH`` +themselves. The contents of this directory will be managed by PyManager, and +will contain executables to directly launch installed runtimes (for example, +``python3.exe`` and ``python3.13.exe`` for an install of Python 3.13). Whenever +aliases are added to this directory, ``PATH`` will be checked and if it is +missing, the user will be presented a message containing the path to add. + +Scripts installed by packages installed into a runtime will be in yet another +directory. Due to the current design, we do not believe it is safe to have them +all install into a single directory, or a directory shared by multiple runtimes. +However, a future development may include a command for PyManager to generate +its own entry points based on metadata in installed packages. + + +Start Menu Shortcuts +-------------------- + +A Start Menu shortcut will be added to launch PyManager documentation in the +user's default web browser. No applications are added to the Start Menu. + +When installing Python runtimes, the install definition may specify Start Menu +shortcuts to create for the install. + + +File Associations +----------------- + +Standard file associations will be created when installing PyManager, and will +launch scripts and packaged apps with PyManager's global ``python.exe`` alias. +This provides sensible behaviour for users who are double-clicking on scripts or +``.pyz`` files. + + +Windowed Executables +-------------------- + +For each of the global aliases described earlier, a ``*w.exe`` also exists. +These launch without creating or attaching a console window, which typically +means they will only display UI created by the script. For example, IDLE always +launches using ``pythonw.exe``, as this avoids an unnecessary native console. + +These commands otherwise behave identically to their console counterparts. + + +Configuration +------------- + +PyManager is configured using a hierarchy of JSON-based configuration files. +Command-line options always override configuration file options. Configuration +files in user editable locations may be disabled by a configuration or +command-line option. + +In ascending order of priority, these will be located: + +* within the app package +* specified by admin-only configuration (see below) +* in the ``base_config`` setting (default: none) +* in the ``user_config`` setting (default: ``%AppData%\\Python\\PyManager.json``) +* in the ``additional_config`` setting (default: ``%PYTHON_MANAGER_CONFIG%``) +* specified with the ``-c`` command line option + +The specific behaviour of each configuration option is left to implementation. +However, a number of intended options are discussed in other sections. + +App package configuration is provided to allow PyManager to be embedded in other +applications or packages. For example, an alternative distribution may want to +include PyManager but have it locate installs from their own index. The app +package configuration allows reusing our build and overriding the default +settings. + +The ``user_config`` and ``additional_config`` settings are pre-configured in +earlier configuration files, allowing them to be overridden by admin-only +configuration or an alternate root configuration. If a configuration file +overwrites the setting that caused the file to be loaded, it is ignored. +The ``base_config`` setting is similar, but starts empty and is intended for +easy overriding through admin configuration. + +Admin-only configuration is provided to allow administrators to manage systems +under their control using existing tools, such as group policy or registry +updates. By design, these controls cannot be overridden, such that it is +possible for administrators to deploy policy that prevents or limits the use of +PyManager. These controls are essential to allow PyManager to be deployed safely +into certain environments, and without them, it would simply be disallowed and +those users would have no access to Python. + +The intent is for the main admin-only configuration to be a path to a new +``base_config`` configuration file that an administrator can deploy to any +controlled location. This allows a network administrator to control the source +of their users' default Python runtimes, without forcibly restricting them, or +to override the other sources for configuration files (apart from the command +line option). + + +Index Schema +------------ + +The index file is made available either online or locally, and provides +PyManager with all the information needed to find, select, install, and manage +any Python runtime. + +The index is stored as JSON. The main top level key is ``versions``, which +contains a list of objects. Each version object has its own schema version, and +there is no overall file schema version. Future changes may add additional +top-level keys to provide functionality that cannot be safely integrated into +an existing one. + +Version objects may be split between the index file and a ``__install__.json`` +stored in the root of each package archive. The entries in the bundled file will +fill in any gaps from the index file. This is intended to allow the typically +large ``shortcuts`` key to be removed from the index file, but may also extend +to ``alias``, ``executable`` and ``executable_args``. Omitting other keys from +the index may result in problems installing the package. + +A second top-level key ``next`` contains an optional URL to another index. This +may be used if PyManager cannot find a suitable package in the included +versions. The intent is to allow for older indexes to be archived and only +accessed when required, reducing the size of the initial download without +cutting off users from older versions. + +The initial schema is shown below: + +.. code:: python + + SCHEMA = { + "versions": [ + { + # Should be 1. + "schema": int, + + # Unique ID used for install detection/side-by-side. + # Must be valid as a filename. + "id": str, + + # Name to display in the UI + "displayName": str, + + # Version used to sort packages. Also determines prerelease status. + # Should follow Python's format, but is only compared among releases + # with the same Company. + "sort-version": Version, + + # Specifies platforms to consider this package for. + # Initially, 'win32' is the only supported value. Others may be + # defined in the future. This condition is evaluated silently, and + # is not intended to replace platform requests in "install-for". + "platform": [str], + + # Company field, used for filtering and displayed as the publisher. + "company": str, + + # Default tag, mainly for UI purposes. + # It should also be specified in 'install-for' and 'run-for'. + "tag": str, + + # List of tags to install this package for. This does not have to be + # unique across all installs; the first match will be selected. + "install-for": [str], + + # List of tags to run this package for. Does not have to be unique + # across all installs; the first match will be selected. The target + # is the executable path relative to the root of the archive. + # Explicit args (optional) are inserted before user args. + "run-for": [{"tag": str, "target": str, "args": [str], "windowed": int}, ...], + + # List of global CLI aliases to create for this package. Does not + # have to be unique across all installs; the first match will be + # created. + "alias": [{"name": str, "target": str, "windowed": int}, ...], + + # List of shortcuts to create for this package. Additional keys on + # each instance are allowed based on the value of 'kind'. + # Initially, 'kind' supports the following values: + # * 'pep514' - other keys define registry values to set + # * 'start' - generate shortcuts in the user's Start Menu + # * 'uninstall' - generate an Add/Remove Programs entry + "shortcuts": [{"kind": str, ...}, ...] + + # Default executable path, relative to the root of the archive. + # Usually the values from 'run-for' will be used instead, and this + # is mainly for display purposes. + "executable": str, + # Default executable args + "executable_args": [str], + + # URL to download the package archive from + "url": str, + + # Optional set of hashes to validate the download. Hashes are stored + # as hex digests. Any hash supported by hashlib without OpenSSL is + # permitted. + "hash": { + "": str, + } + } + ], + + # Full or partial URL to the next index file + "next": str, + } + + +Shebang Processing +------------------ + +For limited compatibility with scripts designed for sh-like shells, PyManager +will check scripts for a shebang line. A shebang line specifying a Python +command will be used (when not overridden on the command line) to select a +suitable runtime for the script. + +Unlike the support currently in the ``py.exe`` launcher, we propose to reduce +this functionality to only support Python commands where the command matches +a global alias listed for an install. (The existing launcher is able to run any +executable, and attempts to extract version information from the command, rather +than simple matching.) + +The specific patterns to be detected are left to the implementation. + + +Rationale +========= + +"Changing" the python.exe command +--------------------------------- + +It may be argued that the global ``python.exe`` alias provided by PyManager is +"not real Python" and so should use a different name. While this is strictly +true, there are three reasons we argue that it should be used. + +Firstly, thousands of users *daily* install through the Store page after being +led there by having typed ``python`` at the terminal of a clean machine. Due to +how this redirection is implemented, if the app they install does not provide a +``python`` command, then the redirection will remain in place. In order to +ensure that users do not get stuck always going back to the Store, we need to +provide this command. (The same applies to ``python3``.) + +Second, the alternative to the "not real" alias is not "the real" one. It's +nothing. We don't have the ability to replace the global static alias with one +that follows the user's preference or installs, and so the alternative would be +to provide nothing and have ``python`` be an error in all cases. This is worse, +and in our opinion, actively harmful to Python's reputation. + +Third, although the underlying implementation of the ``python`` alias is more +complex than the default ``Programs/python.c``, the experience of using it is +identical. The alias is only launched in the absence of another expressed +preference (that is, there's nothing else on ``PATH``), it respects any +indirect preferences (such as through configuration or shebangs), and it +launches the most appropriate version of Python available on the user's machine. +This is much closer to the desired behaviour of the global ``python`` command +than any alternative. + +TODO: Note that this follows Gentoo's design as well + + +Replacing py.exe +---------------- + +The ``py.exe`` launcher exists to provide some of the functionality that will be +replicated by PyManager - specifically, the ability to launch an already +installed runtime. Despite its long history, the launcher does not seem to have +become the preferred method for most users, with many preferring the global +modifications to the ``PATH`` environment variable. However, the command itself +has come to be relied upon, and should be preserved as long as possible. This is +achieved in two ways. + +Firstly, we install our own ``py.exe`` alias with PyManager that provides the +same functionality, along with PyManager specific functionality. This is +intended to become the default/preferred install of ``py.exe`` over time. + +Second, we generate PEP 514 metadata (when requested) for each install, which +allows a legacy ``py.exe`` to continue to work normally with installs managed by +PyManager. + +Due to how the existing ``py.exe`` launcher configures itself, and how the MSIX +package for PyManager is constrained, it is not possible for PyManager's ``py`` +alias to override the launcher. As a result, users who install the launcher will +always find ``py`` resolving to the launcher. Ultimately, the only way to +resolve this in favour of PyManager is to uninstall the launcher, which can be +done through the standard Installed Apps control panel. + + +Interaction with venv +--------------------- + +An activated virtual environment, as implemented by the standard library +:mod:`venv` module, will modify the user's ``PATH`` environment variable to ensure +that the venv launcher will take precedence over other executables. As a result, +when a venv has been activated, PyManager can only be launched by its aliases +other than ``python``. + +This means that working virtual environments will behave as they do today with +no additional support from PyManager. + + + +Backwards Compatibility +======================= + +In general, there are no compatibility guarantees to the install process between +minor versions (``3.x`` to ``3.y``), and so "having to use a different +installer" is not considered compatibility breakage. The versions of Python +installed are only impacted by this change to the extent that the install method +modified their behaviour. In general, most installs will be closer to the +behaviour of having been built from source by the user themselves. + +That said, there are a number of changes that will impact certain users when +they do move to a new install process. This section outlines as many of these +changes as we are aware of, in no particular order, and will likely form the +basis of a migration guide. + + +Scripted downloads +------------------ + +Users who wrote scripts to generate the download filename of our old installer +will find those scripts are broken. These URLs were never guaranteed stable or +predictable, and so we have no recourse to do anything other than apologise and +suggest users use our own tooling for downloads. + +The deprecation period for the traditional installer allows time for these users +to learn about the upcoming change. + + +Scripted installs +----------------- + +Users who wrote scripts to execute our installer with particular options will +have to change their script. Most options have been removed, to begin with, and +those that remain have new spelling. Since it is not possible to reach a state +where options for the old installer are being passed to the new without manual +intervention (that is, someone has to change the command already), this is +considered an acceptable change. + +The deprecation period for the traditional installer allows time for these users +to learn about the upcoming change. + + +Old runtime installed +--------------------- + +Users with existing runtimes installed will find them selected by PyManager and +its aliases, provided the registration is not corrupt. + +The priority order among installed runtimes has changed to only include +prerelease versions when specifically requested (for example, ``-V:3`` will +match 3.14.0 rather than 3.15.0a1, but ``-V:3.15`` will match 3.15.0a1), and to +correctly sort text suffixes on tags (for example, 3.14t is now *lower* +priority than 3.14). + +While it is possible to provide warnings in cases where this may be impacting a +user, such warnings would be considered very noisy (e.g. a message every time +you launch ``python`` because you have a prerelease installed that wasn't +selected) and require complicating the selection logic unnecessarily. This +change will be documented only. + + +Old ``py.exe`` launcher installed +--------------------------------- + +Users who do not manually uninstall an old ``py.exe`` launcher will find that +both their existing and new installs of Python are found, though where versions +match the existing install will take priority over the new install (whereas the +new ``py`` would select the new install). + +They will also find that commands such as ``py list`` do not work. The solution +here is to use Windows Settings to uninstall the launcher. + +There is no way to detect that a user has accidentally left an old ``py`` +installed, or to remove it for them. This change will be documented only. + + +Misconfigured venv activated +---------------------------- + +Users who activate a corrupt or misconfigured virtual environment that is either +missing its ``python.exe`` or has it not on ``PATH`` may receive a different +error from before. + +PyManager's global ``python`` alias will be found and executed instead, +suppressing any system "not found" error. As it fails to find the environment's +actual runtime, it will then fail, though the code and message may be different. + +As this scenario requires an already corrupt system, this change will be +documented only. + + +Old version availability +------------------------ + +Python versions prior to the first release of PyManager can be backfilled into +the python.org index, either based on newly repackaged archives or using the +almost equivalent packages from Nuget (the latter does not include Tcl/Tk, +making them significantly incompatible for some users, but this is likely okay +for especially old versions). + + +Administrator installs +---------------------- + +Installing a copy of Python for all users is no longer possible, as PyManager +will only install into the user's own directory. No scenario has been presented +to show that per-machine installs are in line with our intent for the upstream +distribution, and so we will simply not provide an option for them. Third +parties who desire this functionality are encouraged to provide their own +distributions. + +PyManager can only be installed for all users, and can be extensively configured +by an administrator, including to constrain the actual runtimes which users may +install. Additionally, PyManager supports local extraction for bundling, and so +embedding apps can easily generate their own layout, which can be installed for +all users if they so desire. + +As this scenario requires administrator intervention with or without any +changes, this will be documented only. + + +Build-time installs +------------------- + +Users currently using Nuget packages will also have to change to a new workflow. +Further investigation is required to determine how best to support this, as it +is possible that the PyManager MSIX package may not be installable on all +continuous integration systems. No differences are anticipated between a package +installed by PyManager compared to one installed by Nuget. + +Users using the embeddable distro may have to change to a new method for +discovering the URL to the packages, though the recommendation would be to use +PyManager to discover and install. No differences are anticipated due to the +change of installer, and the embeddable distro package would be identical to +today. + + +Security Implications +===================== + +In this section we compare the security implications of the installer itself to +the existing installers. The implications of Python being installed on a machine +are out of scope, and the ability of a malicious user to execute the installer +is also out of scope. + +The typical risk introduced by an installer is that an elevated install may make +changes to a system that allow a low-privileged user to later affect a +high-privilege user, for example, by inappropriately setting access control on +shared folders. PyManager only operates as the same privilege level as the user, +and therefore cannot introduce any escalation path. + +An install using the MSI described earlier may introduce additional risks, due +to using older installer technology. Typical users are directed towards the MSIX +or Windows Store install paths, which are safe, and it is assumed that users +of the MSI are capable of ensuring the security of their install process (for +example, by correctly quoting their commands to launch the installer and +ensuring the initial system configuration is suitable). + +Once PyManager is installed on a machine, it is likely that malicious users will +use it to install Python. The admin-only configuration described earlier in +"Configuration" is intended to control these scenarios. Ultimately though, an +attacker who can run PyManager is able to do whatever the user can do, and only +a complete application whitelisting approach can prevent the use of Python. + +TODO: Notes about security of package acquisition (tl;dr: TLS) + +Runtime installs by PyManager are fully accessible by, and modifiable by, the +current user. This is equivalent to typical installs using the traditional +installer or a Nuget package, but is more vulnerable to tampering than a Store +install or a per-machine install with the traditional installer. It is not +possible to fully protect an install from the user who installed it. + +The aliases generated by PyManager when installing a runtime are designed to +use a signed, unmodified executable that uses an adjacent data file to launch +the correct target. This can be easily abused to direct the launcher to launch +an alternative, however, the only way to resolve this would sacrifice the trust +in the executable itself, making it trivial to replace it instead of the data. +Such risk already exists, and is equivalent to replacing the script that a user +may launch, or any part of the standard library. Importantly, since aliases are +not shared between users, there is no escalation of privilege along this route. + +PyManager has no mechanism to perform a per-machine install. This may be useful +functionality to some users, as it would allow an install to be completely +unmodifiable by the regular user (excluding virtual environments and the user +site folders). Such functionality may be manually imitated by an administrator +using PyManager and other OS commands, but it is not considered a critical +workflow. The recommended alternative is for an administrator to provide +PyManager and override its configuration. + + +Impact on Existing PEPs +======================= + +This proposal would effectively replace :pep:`397` ("Python launcher for Windows") +and :pep:`486` ("Make the Python Launcher aware of virtual environments") by +defining the same functionality as part of a new tool with the same name. Both +are already considered final, and the launcher is defined by its documentation +and normal compatibility processes. New functionality is based on the current +implementation, and not the original PEP text. + +This proposal has no impact on :pep:`394` ("The “python” Command on Unix-Like +Systems"), and is believed to be consistent with it in devising an approach for +Windows that allows similar guidance to be given to users on all platforms. + +This proposal has no impact on :pep:`514` ("Python registration in the Windows +registry"), and in fact improves our ability to follow it with a more flexible +system for registering our own runtimes. Tools that follow PEP 514 will find any +runtimes that choose to use the registration, regardless of how they were +installed. + + +How to Teach This +================= + +Basic Use +--------- + +A central goal of this proposal is that "type 'python' in your terminal" will be +sufficient instruction for the most basic cases. Thanks to the redirector added +by Microsoft, following this instruction will at least result in something +useful happening, and with PyManager we can ensure that "something useful" means +that the user is running the latest version. + +To explain what is actually happening, we propose the following as introductory +text: + +.. code:: text + + Python installs on Windows are managed using an installer tool. After it has + been installed, you can run ``python`` to launch the interpreter, and it will + choose the best version already installed, available online, or referenced by + the script you are launching (if any). If you have a preference for a + particular version, you can specify it with ``py -V:`` followed + by the rest of your command. + + To install a version of Python without running any command, use ``py install + ``. You can see all of your installs with ``py list`` and remove them + with ``py uninstall ``. Run ``py help`` to see all the options that + are available. + + Because each version of Python will be shared by all your projects, we + recommend using virtual environments. This will usually be created for a + particular Python version by running ``py -V: -m venv .venv``, and + activated with ``.venv\Scripts\Activate``. Now, rather than the install + manager, ``python`` or ``py`` will always launch your virtual environment, and + any packages you install are only available while this environment is active. + To get access to the manager again, you can ``deactivate`` the environment, or + use ``py ``. + +Many Python projects include information about how to launch their projects as +part of their own README files. Historically, such information has been +complicated due to the range of options available to users. We propose that, +after the install manager is published, such guidance could be written along +these lines: + +.. code:: text + + To install and use our application, first install Python following the + guidance for your operating system at https://docs.python.org/using/. Then, + create a virtual environment and use 'pip' to install. + + ``python3 -m venv .venv`` + ``source .venv/bin/activate`` or ``.venv\Scripts\Activate`` (on Windows) + ``python -m pip install OurAwesomePackage`` + ... + +If instructions will not include information about virtual environments, then +the ``python`` or ``python3`` command can be shown, and on Windows both will +operate as intended for users with the install manager. + +Instructions currently referring to ``py`` for Windows can continue to do so, as +the install manager provides a practically equivalent command. Projects that +wish to provide Windows-specific instructions, such as by using the ``-V:`` +or ``--install`` options to install the correct version, should also link to the +:ref:`documentation ` as guidance for +ensuring that the install manager is installed. + + +Uninstallation +-------------- + +Complete uninstallation is an important topic to cover before a user is likely +to consider removing the install manager. Since not all parts of installs can be +automatically cleaned up when removing the manager, we choose not to remove most +of them. So while the default ``python`` and ``py`` commands will go away, all +the runtimes that were installed are still present and usable. + +We suggest an explanation like this: + +.. code:: text + + Before you uninstall the Python install manager, you'll want to uninstall any + runtimes that you added. This can be done easily with the "purge" option: + + ``py uninstall --purge`` + + This will remove all installs and any shortcuts that would otherwise be left + behind. If you already removed the manager, you can reinstall it and run the + above uninstall command again to clean up. Individual runtimes can be + uninstalled by replacing the ``--purge`` option with the tag, found by looking + at ``python list``. + + +Configuration +------------- + +Configuration files are a common feature that will be documented, but do not +need to be taught to regular users. Similarly, advanced deployment options, such +as those that might be used by system administrators or organisations wanting +their users to use a preferred index, are best covered in reference material. + + +Custom Index +------------ + +We suggest that indexes only need to be introduced when instructing users to +install a specialised runtime or distribution. Administrators seeking to provide +an index are anticipated to actively seek out the relevant information in the +documentation. + +To explain how and when to use an alternate index, we propose text along these +lines: + +.. code:: text + + Our distribution can be installed on Windows using the Python Install Manager + (include link) by referencing our index: + + ``py install --source latest`` + + This index contains all our versions. Use ``py list --source `` to + see everything that is available. + + +Reference Implementation +======================== + +The reference implementation is available at https://github.com/zooba/pymanager/ +with a precompiled MSIX package under the Releases at +https://github.com/zooba/pymanager/releases. This sample includes a bundled +index, rather than a hosted one, and references a range of existing Nuget +packages to allow install testing. + + +Rejected Ideas +============== + +Make PyManager available on all platforms +----------------------------------------- + +While we are not inherently opposed to this idea, it relies on many more +components being aligned before it can become possible. + +Firstly, as it stands, the reference implementation has a lot of +Windows-specific knowledge. Equivalent knowledge for other platforms would need +to be collated and implemented, as well as any additional behaviours specific to +non-Windows platforms. + +Second, we need a source of pre-built, relocatable binaries that can be +extracted onto the system. While such sources do exist, due to our position in +the supply chain, we cannot justifiably use them (they should be using us). For +Windows, our own binaries already meet these criteria, so we can repackage them +without modification. + +Third, the current implementation relies on a bundled Python runtime, which must +be isolated from any user interference for obvious reasons. This would also +require the relocatable binaries mentioned above, which we currently only have +for Windows. + +Due to the additional steps needed to make this functional on other platforms, +and the fact that there isn't a need to replace existing installers for those +platforms, we consider this idea out of scope for this PEP. It may be pursued in +the future. + + +Include a runtime pre-installed with the manager +------------------------------------------------ + +The proposal is to have a full Python runtime included with PyManager, so that +its ``python.exe`` alias can refer directly to it rather than resolving to the +best available version dynamically. + +It is very important for stability and updates that runtime releases are +fully independent of the manager. Updating the manager should be possible +without affecting any existing runtime installs, and likewise there should be +no requirement to update the manager to get a newer runtime. + +Hypothetically, if we were to include Python 3.14.0 with the manager such that +it did not need to be installed, it would be a breaking change to later replace +that with 3.15.0. As we only have a single install for the manager, this would +result in the newest installs getting the oldest runtime. + +This would also ignore the status of PyManager's ``python`` alias as being of an +unspecified version - when the user is launching this alias, it's because they +didn't care what version they get enough to specify one. In that situation, we +ought to select the best available, and allow them the options to stabilise it +as is appropriate (whether through a shebang or an active environment). + + +Include a runtime INSTEAD OF the manager +---------------------------------------- + +This is the current situation, which we are trying to change. If you made this +argument, and somehow still read this far, please go back and start again. + + +Use a built-in module rather than subcommands +--------------------------------------------- + +Two alternatives to using commands like ``py list`` or ``py install`` that have +been proposed are to use either dedicated modules, invoked like ``py -m list`` +and ``py -m install``, or a single dedicated module invoked like ``py -m manage +list``. This idea is rejected on the basis that it attempts to reuse existing +semantics for a scenario that cannot be reliably implemented by those semantics, +and so will require a special case that is harder to explain, understand, and +maintain. + +The main reason this idea is rejected is due to the interaction of two otherwise +desirable semantics: first, that the default ``py`` command should launch the +latest available runtime as if it were launched directly; and second, that the +behaviour of ``-m`` should not be treated as a special case in some +circumstances. If the first part were dropped, we would freely modify the +command to behave as users expect - nobody would be raising compatibility +concerns at all if we were agreed to completely break compatibility. However, if +the second constraint were dropped, users would bear the burden of the ensuring +confusion. (We aren't proposing dropping either - this is a rejected idea, after +all - but it helps to illustrate what the options are.) + +First, since one of the subcommands is intended to install your first runtime, +we cannot treat ``python -m [manage] install`` as if it is running through the +default runtime - there isn't one! It inherently requires special case handling +in order to read the command and execute it through a different program. + +Additionally, Python allows other options to precede or mingle with the ``-m``, +which would have to be supported by this special case. + +Finally, the semantics of the ``-m`` option include searching the initial +``sys.path`` for matching module names. This is a considerably more broad search +than a bare name. ``py -m install`` would gladly execute ``install.py``, +``install.pyc``, ``install.pyd``, ``install\\__init__.py``, and more after +searching a number of directories found by inspecting the file system, the +environment, the registry, as well as any transitively included paths found in +those. Compared to ``py install``, which would *only* look for a file called +precisely ``install`` in the current working directory, the ``-m`` behaviour is +far more likely to be already relied upon by real scenarios. (For example, +Django projects typically have a ``manage.py`` script, meaning that ``py -m +manage`` would always behave incorrectly.) + +Changing ``py -m install`` to *not* behave like ``-m``, but instead to execute +an internal command, is vastly more likely to break users than changing +``py install``. As such, this idea is rejected. + + +Use a new command-line option rather than subcommands +----------------------------------------------------- + +A reasonable alternative to subcommands is to specify their names with leading +punctuation, like an option rather than a subcommand. For example, this may look +like ``py /install ...`` rather than ``py install``, or ``py --list``. Because +some of these are currently errors for a normal CPython interpreter, they +could be added without any backwards compatibility concern. + +Notably, however, the typical Windows format of a leading slash is not an error +in CPython. Windows users therefore cannot directly transfer existing knowledge +and must learn a new way to specify options. As we are proposing a Windows +specific tool, this is a terrible start. Additionally, those users familiar with +Unix-style command lines will recognise the misuse of options as commands. + +We desire to create a clean interface, and starting with a design that includes +obvious warts or learning challenges is counter to that goal. Modern tools +universally use subcommands for these purposes, and so the idea to use something +different is rejected. + + +Improving the current traditional installer instead +--------------------------------------------------- + +Rather than creating a new install mechanism, we could invest in maintaining the +current installer. At this stage, however, our current installer is based +entirely on retired technology. Windows is no longer developing the Windows +Installer service, and Wix are no longer improving the version of their toolset +that we use. Migrating to a newer Wix Toolset is a significant amount of work, +and ultimately still leaves us tied to old technologies. + +As mentioned earlier, the most beneficial functionality provided by Windows +Installer is not used for CPython, and generally has caused more issues than it +has ever solved (for example, accidental downgrades due to automatically +collected file version information). + +The implementation of the Burn bundle, which is our primary source of installer +logic, is in C++ and integrated into a framework that few core developers are +familiar with. This makes maintenance challenging, and is not a good long term +position to take. Migrating desired features such as registration-free installs +into the Burn bundle is not possible (without writing the end-to-end +reimplementation and integrating it as an afterthought). + +Our view is that maintaining the current traditional installer is at least as +much effort as implementing a new installer, and would not provide meaningful +benefits for the core team or for our users. As such, this idea is rejected. + + +Delete the Store package completely +----------------------------------- + +Removing the Store packages would reduce the number of options users face when +choosing a Python runtime. By all measures apart from reliability and security, +the traditional installer is entirely sufficient as a substitute. The effort to +migrate parts of the ecosystem to more secure settings (such as not relying on +DLL hijacking) has largely occurred, but some packages remain that still only +work with less secure configurations, and moving all users back to these +configurations would ensure that users of these packages would not face the +issues they face today. + +However, the majority of users of the Store packages appear to have no +complaints. Anecdotally, they are often fully satisfied by the Store install, +and particularly appreciate the ease and reliability of installation. (And on a +personal note, this author has been using Store packages exclusively since +Python 3.8 with no blocking issues.) + +The greatest number of issues have been caused by misconfigured ``PATH`` +variables and the default ``python.exe`` redirector installed by Microsoft. In +other words, entirely unrelated to our own package (though sometimes related to +unresolvable issues in our other installer). For the sake of the high number of +successful installs through this path, we consider the burden of diagnosing and +assisting impacted users to be worthwhile, and consider the idea to simply drop +the Store package rejected. + +That said, when PyManager is published to the Store, we would plan to delist all +existing runtimes on the Store to ensure users find the manager. This only +impacts new installs, and anyone who has previously installed a particular +version (even on another machine, if they were logged in) will be able to +continue to use and install those versions. + + +Rely on WinGet or equivalents +----------------------------- + +WinGet, Chocolatey, and other similar tools are not installers in the sense that +we require. They use their own repository of metadata to download, validate, and +run installers. Without our own installer, they have nothing to run, and so +cannot be used. + +It is possible that their metadata will not support installing PyManager and +then running it to install a particular runtime. If this is the case, they may +need to investigate using our binary packages directly. + +Currently, none of these install tools are officially supported by CPython, and +so we have no obligation to make them work. + + +Make every version a Windows Store package +------------------------------------------ + +It is possible to release each version to the Windows Store as we currently do, +but make them unlisted and rely on an installer (potentially PyManager, WinGet, +or another tool that can install Store packages). This would avoid the risk of +overwhelming the user, while greatly simplifying our own reponsibilities for +package management. + +This approach would leave a significant burden on whichever contributor has +access to the Store publishing interface, as updating packages is a manual +operation. Additionally, it would leave every Python runtime with the technical +limitations outlined earlier. As such, this idea is rejected. + +Making every version a MSIX package rather than a ZIP, even though this avoids +the Store publishing interface, would still impose technical limitations on +users. It is also rejected. + + +Just publish the plain ZIP file +------------------------------- + +Publishing the plain ZIP file is part of the plan, however, it will not be +visibly listed (for example, on the python.org downloads pages, though they will +be visible in the FTP view). An alternative would be to publish and list these +packages, and expect users to download and manually extract and configure them. + +Given the workflows we see, we believe that most users do not want to configure +a Python install at all. Not only do they not want to choose the install +location, they do not want to choose a version, or even have to search for a +download provider or instructions. However, they do want to be able to find an +install later, launch, update or remove it, or list all known installs. + +It is also worth recognising that there will be more ZIP files than are +currently listed on the Download pages, and so the list of files will become +longer. Choosing the correct download is already challenging for users (those +who bypass the primary "Download" button and view the list of all available +versions and then files), and we have no desire to make it more challenging. + +The index protocol and download list will be available for tools that wish to +use it, or for users who are willing to navigate JSON in order to find the URL. +The ``--target`` option on the install command also provides a trivial download +and extract operation, allowing users to have the same experience as a ZIP file. + + +Only publish PyManager to one place +----------------------------------- + +Whether the Windows Store or python.org, it would be viable to publish to only +one location. + +However, users strongly expect to be able to download *something* from +python.org. If we were to remove any option at all, we would inevitably hurt our +users. Without an MSIX available on python.org, users have no way to transfer +the package to another machine, or to fully script the initial install of the +manager. + +Many users rely on the Windows Store app to install packages, and the built-in +redirector in Windows can only open to a Store page. As such, removing the Store +app is equivalent to denying hundreds of thousands of installs each month. + +Additionally, automatic updates are only supported through the Store. We would +have to implement automatic updates manually if we did not publish there. It is +possible to have the MSIX from python.org find its own updates on the Store, and +we can assume that machines without access are responsible for their own +updates. + +The two builds are practically identical. The only difference between the MSIX +we provide to the Store and the one that goes to python.org is package signing: +we sign the python.org package ourselves, while the Store package is signed as +part of the publish process. Otherwise, there is no additional cost to producing +and publishing both packages. + + +Inline Script Metadata +---------------------- + +PEP 723 introduced inline script metadata, a structured comment intended for +third-party tools to interpret and then launch a Python script in the correct +environment. An example taken from that PEP: + +.. code:: python + + # /// script + # requires-python = ">=3.11" + # dependencies = [ + # "requests<3", + # "rich", + # ] + # /// + +PyManager has no integrated support for installing dependencies, and does not +propose adding any. As a result, we could not fully implement handling of this +metadata, and as we consider partial handling to be worse than nothing, we +choose not to implement any. + +It is possible for a user to specify the constraint directly as an option, for +example, ``py -V:>=3.11 my-script.py`` to get the selection behaviour. + +We could also detect the metadata and warn if the selected runtime does not +match its requirement, but this is not part of the initial proposal. + + +Open Issues +=========== + +TODO: Raise open issues + + +Footnotes +========= + +TODO: Collate references + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.