MSVC: detection fixes

CHANGES.txt

@@ -12,6 +12,64 @@ NOTE: Python 3.6 support is deprecated and will be dropped in a future reease.
 
 RELEASE  VERSION/DATE TO BE FILLED IN LATER
 
+  From Joseph Brill:
+    - For msvc version specifications without an 'Exp' suffix, an express installation
+      is used when no other edition is detected for the msvc version.  Similarly, an
+      express installation of the IDE binary is used when no other IDE edition is
+      detected.
+    - VS2015 Express (14.0Exp) does not support the sdk version argument.  VS2015 Express
+      does not support the store argument for target architectures other than x86.
+      Script argument validation now takes into account these restrictions.
+    - VS2015 BuildTools (14.0) does not support the sdk version argument and does not
+      support the store argument.  Script argument validation now takes into account
+      these restrictions.
+    - The Windows SDK for Windows 7 and .NET Framework 4" (SDK 7.1) populates the
+      registry keys in a manner in which the msvc detection would report that VS2010
+      (10.0) is installed when only the SDK was installed.  The installed files are
+      intended to be used via the sdk batch file setenv.cmd. The installed msvc
+      batch files will fail. The msvc detection logic now ignores SDK-only VS2010
+      installations.  Similar protection is implemented for the sdk-only installs that
+      populate the installation folder and registry keys for VS2008 (9.0), if necessary.
+    - For VS2005 (8.0) to VS2015 (14.0), vsvarsall.bat is employed to dispatch to a
+      dependent batch file when configuring the msvc environment.  Previously, only the
+      existence of the compiler executable was verified. In certain installations, the
+      dependent batch file (e.g., vcvars64.bat) may not exist while the compiler
+      executable does exist resulting in build failures.  The existence of vcvarsall.bat,
+      the dependent batch file, and the compiler executable are now validated.
+    - MSVC configuration data specific to versions VS2005 (8.0) to VS2008 (9.0) was added
+      as the dependent batch files have different names than the batch file names used
+      for VS2010 (10.0) and later.  The VS2008 (9.0) Visual C++ For Python installation
+      is handled as a special case as the dependent batch files are: (a) not used and (b)
+      in different locations.
+    - When VS2008 (9.0) Visual C++ For Python is installed using the ALLUSERS=1 option
+      (i.e., msiexec /i VCForPython27.msi ALLUSERS=1), the registry keys are written to
+      HKEY_LOCAL_MACHINE rather than HKEY_CURRENT_USER.  An entry was added to query the
+      Visual C++ For Python keys in HKLM following the HKCU query, if necessary.
+    - For VS2008, a full development edition (e.g., Professional) is now selected before
+      a Visual C++ For Python edition.  Prior to this change, Visual C++ For Python was
+      selected before a full development edition when both editions are installed.
+    - The registry detection of VS2015 (14.0), and earlier, is now cached at runtime and
+      is only evaluated once for each msvc version.
+    - The vswhere executable is frozen upon initial detection.  Specifying a different
+      vswhere executable via the construction variable VSWHERE after the initial
+      detection now results in an exception.  Multiple bugs in the implementation of
+      specifying a vswhere executable via the construction variable VSWHERE have been
+      fixed.  Previously, when a user specified vswhere executable detects new msvc
+      installations after the initial detection, the internal msvc installation cache
+      and the default msvc version based on the initial detection are no longer valid.
+      For example, when no vswhere executable is found for the initial detection
+      and then later an environment is constructed with a user specified vswhere
+      executable that detects new msvc installations.
+    - The vswhere detection of VS2017 (14.1), and later, is now cached at runtime and is
+      only evaluated once using a single vswhere invocation for all msvc versions.
+      Previously, the vswhere executable was invoked for each supported msvc version.
+    - The vswhere executable locations for the WinGet and Scoop package managers were
+      added to the default vswhere executable search list after the Chocolatey
+      installation location.
+    - Fix issue #4543: add support for msvc toolset versions 14.4X installed as the
+      latest msvc toolset versions for msvc buildtools v143.  The v143 msvc buildtools
+      may contain msvc toolset versions from 14.30 to 14.4X.
+
   From Thaddeus Crews:
     - GetSConsVersion() to grab the latest SCons version without needing to
       access SCons internals.
@@ -1490,7 +1548,7 @@ RELEASE 3.1.2 - Mon, 17 Dec 2019 02:06:27 +0000
       in the case where a child process is spawned while a Python action has a
       file open. Original author: Ryan Beasley.
     - Added memoization support for calls to Environment.Value() in order to
-	  improve performance of repeated calls.
+      improve performance of repeated calls.
 
 
   From Jason Kenny

RELEASE.txt

@@ -44,6 +44,15 @@ CHANGED/ENHANCED EXISTING FUNCTIONALITY
   of CCFLAGS; the latter variable could cause a compiler warning.
 - The implementation of Variables was slightly refactored, there should
   not be user-visible changes.
+- MSVC: For msvc version specifications without an 'Exp' suffix, an express
+  installation is used when no other edition is detected for the msvc version.
+  This was the behavior for Visual Studio 2008 (9.0) through Visual Studio 2015
+  (14.0).  This behavior was extended to Visual Studio 2017 (14.1) and Visual
+  Studio 2008 (8.0).  An express installation of the IDE binary is used when no
+  other IDE edition is detected.
+- The vswhere executable locations for the WinGet and Scoop package managers were
+  added to the default vswhere executable search list after the Chocolatey
+  installation location.
 - SCons.Environment.is_valid_construction_var() now returns a boolean to
   match the convention that functions beginning with "is" have yes/no
   answers (previously returned either None or an re.match object).
@@ -73,6 +82,37 @@ FIXES
 - Improved the conversion of a "foreign" exception from an action
   into BuildError by making sure our defaults get applied even in
   corner cases. Fixes Issue #4530
+- MSVC: Visual Studio 2010 (10.0) could be inadvertently detected due to an
+  sdk-only install of Windows SDK 7.1.  An sdk-only install of Visual Studio
+  2010 is ignored as the msvc batch files will fail. The installed files are
+  intended to be used in conjunction with the SDK batch file.  Similar protection
+  was added for Visual Studio 2008 (9.0).
+- MSVC: For Visual Studio 2005 (8.0) to Visual Studio 2015 (14.0), detection of
+  installed files was expanded to include the primary msvc batch file, dependent
+  msvc batch file, and compiler executable. In certain installations, the
+  dependent msvc batch file may not exist while the compiler executable does exist
+  resulting in a build failure.
+- MSVC: Visual Studio 2008 (9.0) Visual C++ For Python was not detected when
+  installed using the ALLUSERS command-line option:
+      msiexec /i VCForPython27.msi ALLUSERS=1
+  When installed for all users, Visual Studio 2008 (9.0) Visual C++ For Python is
+  now correctly detected.
+- MSVC: For Visual Studio 2008 (9.0), a full development edition (e.g., Professional)
+  is now selected before a Visual C++ For Python edition. Prior to this change,
+  Visual C++ For Python was selected before a full development edition when both
+  editions are installed.
+- The vswhere executable is frozen upon initial detection.  Specifying a different
+  vswhere executable via the construction variable VSWHERE after the initial
+  detection now results in an exception.  Multiple bugs in the implementation of
+  specifying a vswhere executable via the construction variable VSWHERE have been
+  fixed.  Previously, when a user specified vswhere executable detects new msvc
+  installations after the initial detection, the internal msvc installation cache
+  and the default msvc version based on the initial detection are no longer valid.
+  For example, when no vswhere executable is found for the initial detection
+  and then later an environment is constructed with a user specified vswhere
+  executable that detects new msvc installations.
+- MSVC: Visual Studio 2022 v143 BuildTools now supports msvc toolset versions from
+  14.30 to 14.4X.  Fixes Issue #4543.
 - The Clone() method now respects the variables argument (fixes #3590)
 
 IMPROVEMENTS
@@ -84,6 +124,17 @@ IMPROVEMENTS
 - Performance tweak: the __setitem__ method of an Environment, used for
   setting construction variables, now uses the string method isidentifier
   to validate the name (updated from microbenchmark results).
+- MSVC: Visual Studio 2015 Express (14.0Exp) does not support the sdk version
+  argument. Visual Studio 2015 Express does not support the store argument for
+  target architectures other than x86.  Script argument validation now takes into
+  account these restrictions.
+- MSVC: Visual Studio 2015 BuildTools (14.0) does not support the sdk version
+  argument and does not support the store argument.  Script argument validation now
+  takes into account these restrictions.
+- MSVC: The registry detection of VS2015 (14.0), and earlier, is now cached at runtime
+  and is only evaluated once for each msvc version.
+- MSVC: The vswhere detection of VS2017 (14.1), and later, is now cached at runtime and
+  is only evaluated once using a single vswhere invocation for all msvc versions.
 
 PACKAGING
 ---------

SCons/Tool/MSCommon/MSVC/Config.py

@@ -118,59 +118,126 @@
     for vc_runtime_alias in vc_runtime_alias_list:
         MSVC_RUNTIME_EXTERNAL[vc_runtime_alias] = vc_runtime_def
 
-MSVC_BUILDTOOLS_DEFINITION = namedtuple('MSVCBuildtools', [
-    'vc_buildtools',
-    'vc_buildtools_numeric',
+MSVC_BUILDSERIES_DEFINITION = namedtuple('MSVCBuildSeries', [
+    'vc_buildseries',
+    'vc_buildseries_numeric',
     'vc_version',
     'vc_version_numeric',
     'cl_version',
     'cl_version_numeric',
+])
+
+MSVC_BUILDSERIES_DEFINITION_LIST = []
+
+MSVC_BUILDSERIES_INTERNAL = {}
+MSVC_BUILDSERIES_EXTERNAL = {}
+
+VC_BUILDTOOLS_MAP = {}
+
+VC_VERSION_MAP = {}
+CL_VERSION_MAP = {}
+
+for (vc_buildseries, vc_version, cl_version) in [
+    ('144', '14.4', '19.4'),
+    ('143', '14.3', '19.3'),
+    ('142', '14.2', '19.2'),
+    ('141', '14.1', '19.1'),
+    ('140', '14.0', '19.0'),
+    ('120', '12.0', '18.0'),
+    ('110', '11.0', '17.0'),
+    ('100', '10.0', '16.0'),
+    ('90', '9.0', '15.0'),
+    ('80', '8.0', '14.0'),
+    ('71', '7.1', '13.1'),
+    ('70', '7.0', '13.0'),
+    ('60', '6.0', '12.0'),
+]:
+
+    vc_buildseries_def = MSVC_BUILDSERIES_DEFINITION(
+        vc_buildseries=vc_buildseries,
+        vc_buildseries_numeric=int(vc_buildseries),
+        vc_version=vc_version,
+        vc_version_numeric=float(vc_version),
+        cl_version=cl_version,
+        cl_version_numeric=float(cl_version),
+    )
+
+    MSVC_BUILDSERIES_DEFINITION_LIST.append(vc_buildseries_def)
+
+    MSVC_BUILDSERIES_INTERNAL[vc_buildseries] = vc_buildseries_def
+    MSVC_BUILDSERIES_EXTERNAL[vc_buildseries] = vc_buildseries_def
+    MSVC_BUILDSERIES_EXTERNAL[vc_version] = vc_buildseries_def
+
+    VC_VERSION_MAP[vc_version] = vc_buildseries_def
+    CL_VERSION_MAP[cl_version] = vc_buildseries_def
+
+MSVC_BUILDTOOLS_DEFINITION = namedtuple('MSVCBuildtools', [
+    'vc_buildtools',
+    'vc_buildtools_numeric',
+    'vc_buildseries_list',
     'vc_runtime_def',
     'vc_istoolset',
+    'msvc_version',
+    'msvc_version_numeric',
 ])
 
 MSVC_BUILDTOOLS_DEFINITION_LIST = []
 
 MSVC_BUILDTOOLS_INTERNAL = {}
 MSVC_BUILDTOOLS_EXTERNAL = {}
 
-VC_VERSION_MAP = {}
-
-for vc_buildtools, vc_version, cl_version, vc_runtime, vc_istoolset in [
-    ('v143', '14.3', '19.3', '140', True),
-    ('v142', '14.2', '19.2', '140', True),
-    ('v141', '14.1', '19.1', '140', True),
-    ('v140', '14.0', '19.0', '140', True),
-    ('v120', '12.0', '18.0', '120', False),
-    ('v110', '11.0', '17.0', '110', False),
-    ('v100', '10.0', '16.0', '100', False),
-    ('v90',   '9.0', '15.0',  '90', False),
-    ('v80',   '8.0', '14.0',  '80', False),
-    ('v71',   '7.1', '13.1',  '71', False),
-    ('v70',   '7.0', '13.0',  '70', False),
-    ('v60',   '6.0', '12.0',  '60', False),
+MSVC_VERSION_NEWEST = None
+MSVC_VERSION_NEWEST_NUMERIC = 0.0
+
+for vc_buildtools, vc_buildseries_list, vc_runtime, vc_istoolset in [
+    ('v143', ['144', '143'], '140', True),
+    ('v142', ['142'], '140', True),
+    ('v141', ['141'], '140', True),
+    ('v140', ['140'], '140', True),
+    ('v120', ['120'], '120', False),
+    ('v110', ['110'], '110', False),
+    ('v100', ['100'], '100', False),
+    ('v90', ['90'], '90', False),
+    ('v80', ['80'], '80', False),
+    ('v71', ['71'], '71', False),
+    ('v70', ['70'], '70', False),
+    ('v60', ['60'], '60', False),
 ]:
 
     vc_runtime_def = MSVC_RUNTIME_INTERNAL[vc_runtime]
 
+    vc_buildseries_list = tuple(
+        MSVC_BUILDSERIES_INTERNAL[vc_buildseries]
+        for vc_buildseries in vc_buildseries_list
+    )
+
+    vc_buildtools_numstr = vc_buildtools[1:]
+
+    msvc_version = vc_buildtools_numstr[:-1] + '.' + vc_buildtools_numstr[-1]
+    msvc_version_numeric = float(msvc_version)
+
     vc_buildtools_def = MSVC_BUILDTOOLS_DEFINITION(
         vc_buildtools = vc_buildtools,
         vc_buildtools_numeric = int(vc_buildtools[1:]),
-        vc_version = vc_version,
-        vc_version_numeric = float(vc_version),
-        cl_version = cl_version,
-        cl_version_numeric = float(cl_version),
+        vc_buildseries_list = vc_buildseries_list,
         vc_runtime_def = vc_runtime_def,
         vc_istoolset = vc_istoolset,
+        msvc_version = msvc_version,
+        msvc_version_numeric = msvc_version_numeric,
     )
 
     MSVC_BUILDTOOLS_DEFINITION_LIST.append(vc_buildtools_def)
 
     MSVC_BUILDTOOLS_INTERNAL[vc_buildtools] = vc_buildtools_def
     MSVC_BUILDTOOLS_EXTERNAL[vc_buildtools] = vc_buildtools_def
-    MSVC_BUILDTOOLS_EXTERNAL[vc_version] = vc_buildtools_def
+    MSVC_BUILDTOOLS_EXTERNAL[msvc_version] = vc_buildtools_def
+
+    for vc_buildseries_def in vc_buildseries_list:
+        VC_BUILDTOOLS_MAP[vc_buildseries_def.vc_buildseries] = vc_buildtools_def
 
-    VC_VERSION_MAP[vc_version] = vc_buildtools_def
+    if vc_buildtools_def.msvc_version_numeric > MSVC_VERSION_NEWEST_NUMERIC:
+        MSVC_VERSION_NEWEST_NUMERIC = vc_buildtools_def.msvc_version_numeric
+        MSVC_VERSION_NEWEST = vc_buildtools_def.msvc_version
 
 MSVS_VERSION_INTERNAL = {}
 MSVS_VERSION_EXTERNAL = {}
@@ -181,8 +248,6 @@
 
 MSVS_VERSION_MAJOR_MAP = {}
 
-CL_VERSION_MAP = {}
-
 MSVC_SDK_VERSIONS = set()
 
 VISUALSTUDIO_DEFINITION = namedtuple('VisualStudioDefinition', [
@@ -247,15 +312,15 @@
 
     vc_buildtools_def.vc_runtime_def.vc_runtime_vsdef_list.append(vs_def)
 
-    vc_version = vc_buildtools_def.vc_version
+    msvc_version = vc_buildtools_def.msvc_version
 
     MSVS_VERSION_INTERNAL[vs_product] = vs_def
     MSVS_VERSION_EXTERNAL[vs_product] = vs_def
     MSVS_VERSION_EXTERNAL[vs_version] = vs_def
 
-    MSVC_VERSION_INTERNAL[vc_version] = vs_def
+    MSVC_VERSION_INTERNAL[msvc_version] = vs_def
     MSVC_VERSION_EXTERNAL[vs_product] = vs_def
-    MSVC_VERSION_EXTERNAL[vc_version] = vs_def
+    MSVC_VERSION_EXTERNAL[msvc_version] = vs_def
     MSVC_VERSION_EXTERNAL[vc_buildtools_def.vc_buildtools] = vs_def
 
     if vs_product in VS_PRODUCT_ALIAS:
@@ -264,14 +329,12 @@
             MSVS_VERSION_EXTERNAL[vs_product_alias] = vs_def
             MSVC_VERSION_EXTERNAL[vs_product_alias] = vs_def
 
-    MSVC_VERSION_SUFFIX[vc_version] = vs_def
+    MSVC_VERSION_SUFFIX[msvc_version] = vs_def
     if vs_express:
-        MSVC_VERSION_SUFFIX[vc_version + 'Exp'] = vs_def
+        MSVC_VERSION_SUFFIX[msvc_version + 'Exp'] = vs_def
 
     MSVS_VERSION_MAJOR_MAP[vs_version_major] = vs_def
 
-    CL_VERSION_MAP[vc_buildtools_def.cl_version] = vs_def
-
     if vc_sdk:
         MSVC_SDK_VERSIONS.update(vc_sdk)
 
@@ -292,7 +355,7 @@
 for vs_def in VISUALSTUDIO_DEFINITION_LIST:
     if not vs_def.vc_buildtools_def.vc_istoolset:
         continue
-    version_key = vs_def.vc_buildtools_def.vc_version
+    version_key = vs_def.vc_buildtools_def.msvc_version
     MSVC_VERSION_TOOLSET_DEFAULTS_MAP[version_key] = [version_key]
     MSVC_VERSION_TOOLSET_SEARCH_MAP[version_key] = []
     if vs_def.vs_express:
@@ -305,11 +368,11 @@
 for vs_def in VISUALSTUDIO_DEFINITION_LIST:
     if not vs_def.vc_buildtools_def.vc_istoolset:
         continue
-    version_key = vs_def.vc_buildtools_def.vc_version
+    version_key = vs_def.vc_buildtools_def.msvc_version
     for vc_buildtools in vs_def.vc_buildtools_all:
         toolset_buildtools_def = MSVC_BUILDTOOLS_INTERNAL[vc_buildtools]
-        toolset_vs_def = MSVC_VERSION_INTERNAL[toolset_buildtools_def.vc_version]
-        buildtools_key = toolset_buildtools_def.vc_version
+        toolset_vs_def = MSVC_VERSION_INTERNAL[toolset_buildtools_def.msvc_version]
+        buildtools_key = toolset_buildtools_def.msvc_version
         MSVC_VERSION_TOOLSET_SEARCH_MAP[buildtools_key].extend(MSVC_VERSION_TOOLSET_DEFAULTS_MAP[version_key])
 
 # convert string version set to string version list ranked in descending order