autoninja - ninja wrapper to make goma easier

Using goma requires the developer to remember which build directories
use goma and which don't so that they can pass an appropriate -j number.
Getting this wrong makes builds slower, either by under utilizing
resources or by causing a self-inflicted DOS attack. Usage:

    autoninja -C out/debug

autoninja looks at the settings for the specified build directory and
then selects either -j num_cores*20 or no -j flag based on the
use_goma setting.

You can set the NINJA_CORE_MULTIPLIER variable to change from the
default 20* multiplier. You can also use NINJA_CORE_ADDITION if you
want non-goma builds to specify -j with an offset to the number of
cores, such as this Linux command:

    NINJA_CORE_ADDITION=-2 autoninja -C out/release base

This will tell autoninja to pass -j to ninja with num_cores-2 as the
parameter.

On Windows you can have a ninja.bat file (ahead of ninja on the path)
such that autoninja will automatically be used. It should contain this:
    @call autoninja.bat %*

Change-Id: I4003e3fc323d1cbab612999c945b5a8dc5bc6655
Reviewed-on: https://chromium-review.googlesource.com/517662Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

autoninja

+#!/usr/bin/env bash
+
+# Copyright (c) 2017 Google Inc. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+
+# Execute whatever is printed by autoninja.py.
+# Also print it to reassure that the right settings are being used.
+command=$(python $(dirname -- "$0")/autoninja.py "$@")
+echo $command
+exec $command

autoninja.bat

+@echo off
+:: Copyright 2017 The Chromium Authors. All rights reserved.
+:: Use of this source code is governed by a BSD-style license that can be
+:: found in the LICENSE file.
+
+REM Execute whatever is printed by autoninja.py.
+REM Also print it to reassure that the right settings are being used.
+FOR /f "usebackq tokens=*" %%a in (`python %~dp0autoninja.py %*`) do echo %%a & %%a
+

autoninja.py

+# Copyright (c) 2017 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""
+This script (intended to be invoked by autoninja or autoninja.bat) detects
+whether a build is using goma. If so it runs with a large -j value, and
+otherwise it chooses a small one. This auto-adjustment makes using goma simpler
+and safer, and avoids errors that can cause slow goma builds or swap-storms
+on non-goma builds.
+"""
+
+import multiprocessing
+import os
+import re
+import sys
+
+j_specified = False
+output_dir = '.'
+for index, arg in enumerate(sys.argv[1:]):
+  if arg == '-j':
+    j_specified = True
+  if arg == '-C':
+    # + 1 to get the next argument and +1 because we trimmed off sys.argv[0]
+    output_dir = sys.argv[index + 2]
+
+use_goma = False
+try:
+  with open(os.path.join(output_dir, 'args.gn')) as file_handle:
+    for line in file_handle:
+      # This regex pattern copied from create_installer_archive.py
+      m = re.match('^\s*use_goma\s*=\s*true(\s*$|\s*#.*$)', line)
+      if m:
+        use_goma = True
+except IOError:
+  pass
+
+if sys.platform.startswith('win'):
+  # Specify ninja.exe on Windows so that ninja.bat can call autoninja and not
+  # be called back.
+  args = ['ninja.exe'] + sys.argv[1:]
+else:
+  args = ['ninja'] + sys.argv[1:]
+
+num_cores = multiprocessing.cpu_count()
+if not j_specified:
+  if use_goma:
+    args.append('-j')
+    core_multiplier = int(os.environ.get("NINJA_CORE_MULTIPLIER", "20"))
+    args.append('%d' % (num_cores * core_multiplier))
+  else:
+    core_addition = os.environ.get("NINJA_CORE_ADDITION")
+    if core_addition:
+      core_addition = int(core_addition)
+      args.append('-j')
+      args.append('%d' % (num_cores + core_addition))
+
+# Specify a maximum CPU load so that running builds in two different command
+# prompts won't overload the system too much. This is not reliable enough to
+# be used to auto-adjust between goma/non-goma loads, but it is a nice
+# fallback load balancer.
+args.append('-l')
+args.append('%d' % num_cores)
+
+print ' '.join(args)