Commit 5dba9c97 authored by Simeon Anfinrud's avatar Simeon Anfinrud Committed by LUCI CQ

Pass large -j value to ninja when use_rbe = true

The goma-on-borg backend is turning down. While chromium can
continue using the goma client for build acceleration, the
backend is swapped out to instead use Remote Build Execution,
a gcloud service.

Other downstream projects (e.g. Nest) switched out the goma
frontend as well as the backend to RBE. These projects use a
|use_rbe| gn flag to enable build acceleration through RBE.

This change allows autoninja to recognize this flag and treat it
as if it was using goma for the purposes of passing an
appropriate -j flag.

Bug: None
Test: run on an out/ directory with use_rbe enabled

Change-Id: I52d5771e92e9163234811b154fd36816639bf1b8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2391009Reviewed-by: 's avatarBruce Dawson <brucedawson@chromium.org>
Commit-Queue: Simeon Anfinrud <sanfin@chromium.org>
parent 7592f932
...@@ -5,10 +5,10 @@ ...@@ -5,10 +5,10 @@
""" """
This script (intended to be invoked by autoninja or autoninja.bat) detects This script (intended to be invoked by autoninja or autoninja.bat) detects
whether a build is using goma. If so it runs with a large -j value, and whether a build is accelerated using a service like goma. If so, it runs with a
otherwise it chooses a small one. This auto-adjustment makes using goma simpler large -j value, and otherwise it chooses a small one. This auto-adjustment
and safer, and avoids errors that can cause slow goma builds or swap-storms makes using remote build acceleration simpler and safer, and avoids errors that
on non-goma builds. can cause slow goma builds or swap-storms on unaccelerated builds.
""" """
# [VPYTHON:BEGIN] # [VPYTHON:BEGIN]
...@@ -57,23 +57,23 @@ for index, arg in enumerate(input_args[1:]): ...@@ -57,23 +57,23 @@ for index, arg in enumerate(input_args[1:]):
# Support -Cout/Default # Support -Cout/Default
output_dir = arg[2:] output_dir = arg[2:]
use_goma = False use_remote_build = False
# Attempt to auto-detect goma usage. We support gn-based builds, where we # Attempt to auto-detect remote build acceleration. We support gn-based
# look for args.gn in the build tree, and cmake-based builds where we look for # builds, where we look for args.gn in the build tree, and cmake-based builds
# rules.ninja. # where we look for rules.ninja.
if os.path.exists(os.path.join(output_dir, 'args.gn')): if os.path.exists(os.path.join(output_dir, 'args.gn')):
with open(os.path.join(output_dir, 'args.gn')) as file_handle: with open(os.path.join(output_dir, 'args.gn')) as file_handle:
for line in file_handle: for line in file_handle:
# This regex pattern copied from create_installer_archive.py # Either use_goma or use_rbe activate build acceleration.
if re.match(r'^\s*use_goma\s*=\s*true(\s*$|\s*#.*$)', line): if re.match(r'^\s*(use_goma|use_rbe)\s*=\s*true(\s*$|\s*#.*$)', line):
use_goma = True use_remote_build = True
continue continue
elif os.path.exists(os.path.join(output_dir, 'rules.ninja')): elif os.path.exists(os.path.join(output_dir, 'rules.ninja')):
with open(os.path.join(output_dir, 'rules.ninja')) as file_handle: with open(os.path.join(output_dir, 'rules.ninja')) as file_handle:
for line in file_handle: for line in file_handle:
if re.match(r'^\s*command\s*=\s*\S+gomacc', line): if re.match(r'^\s*command\s*=\s*\S+gomacc', line):
use_goma = True use_remote_build = True
break break
# If GOMA_DISABLED is set to "true", "t", "yes", "y", or "1" (case-insensitive) # If GOMA_DISABLED is set to "true", "t", "yes", "y", or "1" (case-insensitive)
...@@ -85,7 +85,7 @@ elif os.path.exists(os.path.join(output_dir, 'rules.ninja')): ...@@ -85,7 +85,7 @@ elif os.path.exists(os.path.join(output_dir, 'rules.ninja')):
# autoninja uses an appropriate -j value in this situation. # autoninja uses an appropriate -j value in this situation.
goma_disabled_env = os.environ.get('GOMA_DISABLED', '0').lower() goma_disabled_env = os.environ.get('GOMA_DISABLED', '0').lower()
if goma_disabled_env in ['true', 't', 'yes', 'y', '1']: if goma_disabled_env in ['true', 't', 'yes', 'y', '1']:
use_goma = False use_remote_build = False
# Specify ninja.exe on Windows so that ninja.bat can call autoninja and not # Specify ninja.exe on Windows so that ninja.bat can call autoninja and not
# be called back. # be called back.
...@@ -98,7 +98,7 @@ args = [ninja_exe_path] + input_args[1:] ...@@ -98,7 +98,7 @@ args = [ninja_exe_path] + input_args[1:]
num_cores = psutil.cpu_count() num_cores = psutil.cpu_count()
if not j_specified and not t_specified: if not j_specified and not t_specified:
if use_goma: if use_remote_build:
args.append('-j') args.append('-j')
core_multiplier = int(os.environ.get('NINJA_CORE_MULTIPLIER', '40')) core_multiplier = int(os.environ.get('NINJA_CORE_MULTIPLIER', '40'))
j_value = num_cores * core_multiplier j_value = num_cores * core_multiplier
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment