Skip to content

Commit

Permalink
[tool/ci] Add iOS/macOS and Dart support to fetch-deps (flutter#4562)
Browse files Browse the repository at this point in the history
Adds `fetch-deps` support for:
- iOS/macOS dependencies, using `pod install`
- Dart package dependencies, using `pub get`

To make avoid doing extra work in the Dart dependencies step when using this with `*_platform_tests` CI, also adds flags for all of the other platforms, and adds a flag that allows skipping Dart dependencies for any package that doesn't have an example supporting any requested platform. This means that we can pass, e.g., `--windows --supporting-target-platforms-only` to only fetch Dart packages for packages with examples that will be build during the build-and-drive Windows tests.

Adds this as a new step in every platform tests CI task, and in the standard analyze step, so that we will pre-fetch Dart packages (and for iOS/macOS, pods). This won't yet fully eliminate later network access (see flutter/flutter#131204), but will give us early warning on any major failures, such as pub being entirely unreachable from the bots.
- These are marked as an infrastructure step; we'll have to see if this ends up being confusing in practice. If `pub` resolution fails for legitimate reasons, such as a PR that tries to require a version of a package that doesn't exist or that has conflicts, this will cause a failure that is marked as infra. My assumption is that the much more common case is going to be that it is actually an infra failure.

Fixes flutter/flutter#130280
  • Loading branch information
stuartmorgan authored Jul 26, 2023
1 parent 406eac1 commit a99fc87
Show file tree
Hide file tree
Showing 15 changed files with 903 additions and 65 deletions.
4 changes: 4 additions & 0 deletions .ci/targets/analyze.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ tasks:
infra_step: true # Note infra steps failing prevents "always" from running.
- name: analyze repo tools
script: .ci/scripts/analyze_repo_tools.sh
- name: download Dart deps
script: script/tool_runner.sh
args: ["fetch-deps"]
infra_step: true
- name: analyze
script: script/tool_runner.sh
# DO NOT change the custom-analysis argument here without changing the Dart repo.
Expand Down
4 changes: 2 additions & 2 deletions .ci/targets/android_platform_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@ tasks:
- name: prepare tool
script: .ci/scripts/prepare_tool.sh
infra_step: true # Note infra steps failing prevents "always" from running.
- name: download android deps
- name: download Dart and Android deps
script: script/tool_runner.sh
infra_step: true
args: ["fetch-deps"]
args: ["fetch-deps", "--android", "--supporting-target-platforms-only"]
- name: build examples
script: script/tool_runner.sh
args: ["build-examples", "--apk"]
Expand Down
4 changes: 4 additions & 0 deletions .ci/targets/ios_platform_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ tasks:
- name: create simulator
script: .ci/scripts/create_simulator.sh
infra_step: true # Note infra steps failing prevents "always" from running.
- name: download Dart and iOS deps
script: script/tool_runner.sh
args: ["fetch-deps", "--ios", "--supporting-target-platforms-only"]
infra_step: true
- name: build examples
script: script/tool_runner.sh
args: ["build-examples", "--ios"]
Expand Down
5 changes: 5 additions & 0 deletions .ci/targets/linux_platform_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@ tasks:
script: .ci/scripts/prepare_tool.sh
- name: set default apps
script: .ci/scripts/set_default_linux_apps.sh
infra_step: true
- name: download Dart deps
script: script/tool_runner.sh
args: ["fetch-deps", "--linux", "--supporting-target-platforms-only"]
infra_step: true
- name: build examples
script: script/tool_runner.sh
args: ["build-examples", "--linux"]
Expand Down
4 changes: 4 additions & 0 deletions .ci/targets/macos_platform_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@ tasks:
- name: prepare tool
script: .ci/scripts/prepare_tool.sh
infra_step: true # Note infra steps failing prevents "always" from running.
- name: download Dart and macOS deps
script: script/tool_runner.sh
args: ["fetch-deps", "--macos", "--supporting-target-platforms-only"]
infra_step: true
- name: build examples
script: script/tool_runner.sh
args: ["build-examples", "--macos"]
Expand Down
4 changes: 4 additions & 0 deletions .ci/targets/web_platform_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@ tasks:
- name: prepare tool
script: .ci/scripts/prepare_tool.sh
infra_step: true # Note infra steps failing prevents "always" from running.
- name: download Dart deps
script: script/tool_runner.sh
args: ["fetch-deps", "--web", "--supporting-target-platforms-only"]
infra_step: true
- name: build examples
script: script/tool_runner.sh
args: ["build-examples", "--web"]
Expand Down
4 changes: 4 additions & 0 deletions .ci/targets/windows_build_and_platform_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@ tasks:
- name: prepare tool
script: .ci/scripts/prepare_tool.sh
infra_step: true # Note infra steps failing prevents "always" from running.
- name: download Dart deps
script: script/tool_runner.sh
args: ["fetch-deps", "--windows", "--supporting-target-platforms-only"]
infra_step: true
- name: build examples (Win32)
script: .ci/scripts/build_examples_win32.sh
- name: native unit tests (Win32)
Expand Down
10 changes: 2 additions & 8 deletions script/tool/lib/src/build_examples_command.dart
Original file line number Diff line number Diff line change
Expand Up @@ -176,9 +176,8 @@ class BuildExamplesCommand extends PackageLoopingCommand {
// supported platforms. For packages, just log and skip any requested
// platform that a package doesn't have set up.
if (!isPlugin &&
!example.directory
.childDirectory(platform.flutterPlatformDirectory)
.existsSync()) {
!example.appSupportsPlatform(
getPlatformByName(platform.pluginPlatform))) {
print('Skipping ${platform.label} for $packageName; not supported.');
continue;
}
Expand Down Expand Up @@ -304,11 +303,6 @@ class _PlatformDetails {
/// The `flutter build` build type.
final String flutterBuildType;

/// The Flutter platform directory name.
// In practice, this is the same as the plugin platform key for all platforms.
// If that changes, this can be adjusted.
String get flutterPlatformDirectory => pluginPlatform;

/// Any extra flags to pass to `flutter build`.
final List<String> extraBuildFlags;
}
19 changes: 19 additions & 0 deletions script/tool/lib/src/common/core.dart
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,25 @@ const String kEnableExperiment = 'enable-experiment';
// ignore: public_member_api_docs
enum FlutterPlatform { android, ios, linux, macos, web, windows }

const Map<String, FlutterPlatform> _platformByName = <String, FlutterPlatform>{
platformAndroid: FlutterPlatform.android,
platformIOS: FlutterPlatform.ios,
platformLinux: FlutterPlatform.linux,
platformMacOS: FlutterPlatform.macos,
platformWeb: FlutterPlatform.web,
platformWindows: FlutterPlatform.windows,
};

/// Maps from a platform name (e.g., flag or platform directory) to the
/// corresponding platform enum.
FlutterPlatform getPlatformByName(String name) {
final FlutterPlatform? platform = _platformByName[name];
if (platform == null) {
throw ArgumentError('Invalid platform: $name');
}
return platform;
}

// Flutter->Dart SDK version mapping. Any time a command fails to look up a
// corresponding version, this map should be updated.
final Map<Version, Version> _dartSdkForFlutterSdk = <Version, Version>{
Expand Down
20 changes: 13 additions & 7 deletions script/tool/lib/src/common/process_runner.dart
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,13 @@ class ProcessRunner {
String executable,
List<String> args, {
Directory? workingDir,
Map<String, String>? environment,
bool exitOnError = false,
}) async {
print(
'Running command: "$executable ${args.join(' ')}" in ${workingDir?.path ?? io.Directory.current.path}');
final io.Process process = await io.Process.start(executable, args,
workingDirectory: workingDir?.path);
workingDirectory: workingDir?.path, environment: environment);
await Future.wait(<Future<dynamic>>[
io.stdout.addStream(process.stdout),
io.stderr.addStream(process.stderr),
Expand All @@ -62,14 +63,19 @@ class ProcessRunner {
/// Defaults to `false`
///
/// Returns the [io.ProcessResult] of the [executable].
Future<io.ProcessResult> run(String executable, List<String> args,
{Directory? workingDir,
bool exitOnError = false,
bool logOnError = false,
Encoding stdoutEncoding = io.systemEncoding,
Encoding stderrEncoding = io.systemEncoding}) async {
Future<io.ProcessResult> run(
String executable,
List<String> args, {
Directory? workingDir,
Map<String, String>? environment,
bool exitOnError = false,
bool logOnError = false,
Encoding stdoutEncoding = io.systemEncoding,
Encoding stderrEncoding = io.systemEncoding,
}) async {
final io.ProcessResult result = await io.Process.run(executable, args,
workingDirectory: workingDir?.path,
environment: environment,
stdoutEncoding: stdoutEncoding,
stderrEncoding: stderrEncoding);
if (result.exitCode != 0) {
Expand Down
10 changes: 10 additions & 0 deletions script/tool/lib/src/common/repository_package.dart
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,16 @@ class RepositoryPackage {
return directory.childDirectory(directoryName);
}

/// Returns true if the package is an app that supports [platform].
///
/// The "app" prefix on this method is because this currently only works
/// for app packages (e.g., examples).
// TODO(stuartmorgan): Add support for non-app packages, by parsing the
// pubspec for `flutter:platform:` or `platform:` sections.
bool appSupportsPlatform(FlutterPlatform platform) {
return platformDirectory(platform).existsSync();
}

late final Pubspec _parsedPubspec =
Pubspec.parse(pubspecFile.readAsStringSync());

Expand Down
2 changes: 1 addition & 1 deletion script/tool/lib/src/drive_examples_command.dart
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ class DriveExamplesCommand extends PackageLoopingCommand {
for (final MapEntry<String, List<String>> entry
in _targetDeviceFlags.entries) {
final String platform = entry.key;
if (example.directory.childDirectory(platform).existsSync()) {
if (example.appSupportsPlatform(getPlatformByName(platform))) {
deviceFlags.addAll(entry.value);
} else {
final String exampleName =
Expand Down
Loading

0 comments on commit a99fc87

Please sign in to comment.