Skip to content

C
Chromium_Depot_tools
Switch branch/tag
  1. 24 Sep, 2020 1 commit
    • Mark Mentovai's avatar
      mac-arm64: Provide an arm64-native ninja-mac for builds in ¼ of the time · 4005bb74
      Mark Mentovai authored 
      ninja-mac is expanded to become a universal (“fat”) executable. The
existing x86_64 slice was taken from the x86_64 ninja-mac as it
previously existed in this location, and a new arm64 slice was added.

This is an aid to native development on arm64. In macOS, subprocesses
inherit their parent’s “binpref,” meaning that children of a translated
process will, by default, also be translated, if there’s a choice
between native and translated. Where ninja-mac is x86_64-only and runs
translated on arm64, all tools it invokes will also run translated, even
if native versions are available. This has an impact on any part of the
toolchain for which universal versions are available: in Chrome’s case,
tools such as ld64 are affected. In Crashpad’s build, which uses the
compiler provided by Xcode, it even affects the compiler, leading to
significant differences in build time on arm64.

mark@arm-and-hammer zsh% file $(which ninja-mac)
…/ninja-mac: Mach-O 64-bit executable x86_64
mark@arm-and-hammer zsh% time ninja -C out/release
ninja: Entering directory `out/release'
[1/1] Regenerating ninja files
[511/511] LINK ./crashpad_minidump_test
ninja -C out/release  1384.03s user 111.05s system 393% cpu 6:19.53 total

mark@arm-and-hammer zsh% file $(which ninja-mac)
…/ninja-mac: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
…/ninja-mac (for architecture x86_64):	Mach-O 64-bit executable x86_64
…/ninja-mac (for architecture arm64):	Mach-O 64-bit executable arm64
mark@arm-and-hammer zsh% time ninja -C out/release
ninja: Entering directory `out/release'
[1/1] Regenerating ninja files
[511/511] LINK ./crashpad_minidump_test
ninja -C out/release  606.80s user 34.71s system 701% cpu 1:31.41 total

This differs from the previous attempt, e64cd3ae
(https://chromium-review.googlesource.com/c/2288417) in that the arm64
version was built with Xcode 12.2b1 (12B5018i). The linker in this
version of Xcode produces linker-signed output for arm64 by default. As
of macOS 11.0db6 (20A5364e), all arm64 code must be signed. More
information at
https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11-universal-apps-beta-release-notes#Code-Signing.
In the previous attempt, an older Xcode was used and the output was not
signed at all. Because it did not run on systems upgraded to 11.0db6,
this version of ninja-mac was backed out in a070dd18
(https://chromium-review.googlesource.com/c/2393775).

To create this ninja-mac:

% git clone https://github.com/ninja-build/ninja.git
% cd ninja
% git checkout v1.8.2
% mkdir ../build
% cd ../build
% LDFLAGS=-Wl,-s ../ninja/configure.py --bootstrap
% lipo -create -output ninja-mac $(which ninja-mac) ninja

An explanation of the LDFLAGS: e64cd3ae used “strip” on the output,
matching the stripped x86_64 slice, but “strip” is not currently
compatible with linker-signed executables. This is documented at
https://developer.apple.com/documentation/xcode-release-notes/xcode-12_2-beta-release-notes#Apple-Clang-Compiler.
I was unable to use the recommended workaround in this case. Rather than
leave the output unstripped, I asked ld64 to strip symbols using its -s
option, which it believes is obsolete, but in fact provides the intended
behavior.

The x86_64 slice is unsigned, as it was previously. The new arm64 slice
is linker-signed with an ad-hoc signature. The x86_64 slice was built
with the macOS 10.12 SDK and runs on Mac OS X 10.6 or later. The arm64
slice was built with and has a minimum runtime requirement of macOS
11.0. In practice, the difference doesn’t matter, as the 11.0 SDK is
required to build for arm64, and no macOS version earlier than 11.0 will
run on arm64.

Bug: chromium:1103315
Change-Id: I9f201d0d6f476e187acc0008681c3fd2c1a72aa0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2427051Reviewed-by: 's avatarNico Weber <thakis@chromium.org>
Commit-Queue: Mark Mentovai <mark@chromium.org>
      4005bb74
  3. 04 Sep, 2020 1 commit
    • Nico Weber's avatar
      Revert "mac: Add an arm64 slice to ninja-mac." · a070dd18
      Nico Weber authored 
      This reverts commit e64cd3ae.

Reason for revert: A never macOS 11 beta made it so that
native arm binaries need to be signed, and the Xcode beta used
to build this arm64 slice here didn't do ad-hoc signing by default
yet, see https://crbug.com/1125021

Original change's description:
> mac: Add an arm64 slice to ninja-mac.
> 
> To build the arm ninja binary, I downloaded a zip with the ninja 1.8.2
> code, and ran `./configure --bootstrap` on an arm mac, with Xcode 12 beta
> installed, followed by `strip ninja`. That gave me a working ninja 1.8.2
> arm binary. I then copied it to my intel mac and ran
> 
>     lipo -create -output ninja-mac ninja-mac ~/Downloads/ninja
> 
> to make a fat binary.
> 
> No intended behavior change on non-arm macs.
> 
> (This means the x86_64 slice, which is the previously existing binary,
> has a LC_VERSION_MIN_MACOSX version 10.6, while the arm64 slice has
> LC_BUILD_VERSION minos 11.0, according to `otool -arch ax86_64 -l ninja-mac`
> and `otool -arch arm64 -l ninja-mac`.)
> 
> Bug: 1103315,762354
> Change-Id: I2c4bea53a01d7a6135f9cd5554e3123dba960664
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2288417
> Auto-Submit: Nico Weber <thakis@chromium.org>
> Reviewed-by: Mark Mentovai <mark@chromium.org>
> Commit-Queue: Mark Mentovai <mark@chromium.org>
> Commit-Queue: Nico Weber <thakis@chromium.org>

TBR=thakis@chromium.org,dpranke@google.com,mark@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 1103315
Bug: 762354
Change-Id: Icd968a2fecb9509722c43d9cb03db92e6e3829e4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2393775Reviewed-by: 's avatarNico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
      a070dd18
  5. 08 Jul, 2020 1 commit
    • Nico Weber's avatar
      mac: Add an arm64 slice to ninja-mac. · e64cd3ae
      Nico Weber authored 
      To build the arm ninja binary, I downloaded a zip with the ninja 1.8.2
code, and ran `./configure --bootstrap` on an arm mac, with Xcode 12 beta
installed, followed by `strip ninja`. That gave me a working ninja 1.8.2
arm binary. I then copied it to my intel mac and ran

    lipo -create -output ninja-mac ninja-mac ~/Downloads/ninja

to make a fat binary.

No intended behavior change on non-arm macs.

(This means the x86_64 slice, which is the previously existing binary,
has a LC_VERSION_MIN_MACOSX version 10.6, while the arm64 slice has
LC_BUILD_VERSION minos 11.0, according to `otool -arch ax86_64 -l ninja-mac`
and `otool -arch arm64 -l ninja-mac`.)

Bug: 1103315,762354
Change-Id: I2c4bea53a01d7a6135f9cd5554e3123dba960664
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2288417
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: 's avatarMark Mentovai <mark@chromium.org>
Commit-Queue: Mark Mentovai <mark@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
      e64cd3ae
  7. 11 Sep, 2017 1 commit
  9. 05 Sep, 2017 1 commit
    • Takuto Ikuta's avatar
      Update ninja to v1.8.0. · 372e0fdb
      Takuto Ikuta authored 
      New binaries for Mac, Linux32, Linux64, Windows.  Also update shell script.
See the bug for how these were built.
The main "new" thing is performance improvement when building chrome with goma.

After confirming the the behavior of ninja 1.8.0 on buildbot, I will roll ninja 1.8.1 includes fix for non-deterministic test.

TBR=dpranke@chromium.org
Bug: 761691, 738186
Change-Id: I108996dccfdf5d95d8815d6cece5eba46f3e8178
Reviewed-on: https://chromium-review.googlesource.com/648372
Commit-Queue: Takuto Ikuta <tikuta@google.com>
Reviewed-by: 's avatarNico Weber <thakis@chromium.org>
      372e0fdb
  11. 09 Nov, 2016 1 commit
  13. 28 Apr, 2016 1 commit
  15. 27 Apr, 2016 1 commit
  17. 29 Jun, 2015 1 commit
  19. 24 Nov, 2014 1 commit
  21. 28 Jun, 2014 1 commit
  23. 27 Jun, 2014 2 commits
  25. 13 Sep, 2013 1 commit
  27. 04 Jun, 2013 1 commit
  29. 20 May, 2013 2 commits
  31. 02 Jan, 2013 1 commit
  33. 14 Sep, 2012 1 commit
    • thakis@chromium.org's avatar
      Update ninja to v1.0.0 on Mac and Linux. · 79b09c47
      thakis@chromium.org authored 
      On both mac and linux, I ran:
git checkout v1.0.0
ninja
./ninja ninja_test
./ninja_test  # all passed
strip ninja

Changes since the last push:
* NINJA_STATUS can now print rates via the o and c format chars
* The version switch is now --version, -V no longer works
* -t list now works without a build.ninja
* Old .ninja_log files now trigger a rebuild instead of an error
* Missing .d files make the corresponding .o file dirty
* Allow @ and = in paths in depfiles
* Ninja's output is now lower-case ("error:" instead of "ERROR:" etc)
* Lots of Windows features (which doesn't affect Mac and Linux)

On Linux, I switched from gcc4.4.3 to clang r163126. The clang-built
binary is 125856 bytes (compared to 129944 bytes previously), and it does
an empty 'chrome' build about 70ms faster (0.94s instead of 1.01s).

The speed difference is almost completely in faster depfile parsing:

clang-built (0.943s):
$ time ../../ninja/ninja -C out/Release chrome -d stats
ninja: Entering directory `out/Release'
ninja: no work to do.
metric                  count   avg (us)        total (ms)
.ninja parse            716     507.8           363.6
canonicalize str        113486  0.2             27.7
canonicalize path       1305641 0.1             184.1
lookup node             1305641 0.2             200.9
.ninja_log load         1       11144.0         11.1
node stat               37828   2.6             98.0
depfile load            9266    80.9            749.3

path->node hash load 0.98 (48241 entries / 49157 buckets)

real    0m1.162s
user    0m0.980s
sys     0m0.170s


gcc-built (1.013s):
$ time ninja -C out/Release chrome -d stats
ninja: Entering directory `out/Release'
ninja: no work to do.
metric                  count   avg (us)        total (ms)
.ninja parse            716     516.4           369.7
canonicalize str        113486  0.2             27.6
canonicalize path       1305641 0.1             181.0
lookup node             1305641 0.2             260.0
.ninja_log load         1       11012.0         11.0
node stat               37828   2.6             98.2
depfile load            9266    86.3            799.3

path->node hash load 0.98 (48241 entries / 49157 buckets)

real    0m1.220s
user    0m0.970s
sys     0m0.240s



On Mac, I alsow used clang r163126. The binary size stayed at 125104 bytes,
performance didn't change. An empty build of chrome takes 1.26s. While
node stat is a bit slower, and there are more defiles to parse compared
to linux, most of the time difference is in slower .ninja parsing:

mac (1.263s):
$ time ../../ninja/ninja -C out/Release/ chrome -d stats
ninja: Entering directory `out/Release/'
ninja: no work to do.
metric                  count   avg (us)        total (ms)
.ninja parse            687     823.4           565.7
canonicalize str        115002  0.3             31.1
canonicalize path       1120853 0.2             175.6
lookup node             1120853 0.2             202.9
.ninja_log load         1       54126.0         54.1
node stat               38860   4.1             160.4
depfile load            9649    86.1            830.8

path->node hash load 0.99 (48522 entries / 49157 buckets)

real    0m1.478s
user    0m1.121s
sys     0m0.351s

TBR=scottmg
Review URL: https://codereview.chromium.org/10918242

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@156751 0039d316-1c4b-4281-b951-d872f2087c98
      79b09c47
  35. 19 Jul, 2012 1 commit
    • thakis@chromium.org's avatar
      Update ninja to v120715 · 45298e08
      thakis@chromium.org authored 
      On both mac and linux, I ran:

git checkout release-120715
ninja
./ninja ninja_test
./ninja_test  # all passed
strip ninja

Changes since the last push:
* ninja now has a functional -V flag that prints version information
* ninja is a lot faster. On my MBP, empty builds go from 1.8s to 1.1s,
  on my linux box from 1.4s to 0.9s
* ninja now has a -l N flag to do parallelsim based on load average

On linux, the binary grew 24 bytes, from 129920 to 129944. I used
gcc 4.4.3 to build ninja.

On mac, it grew from 4200 byte from 120904 to 125104 (likely caused by me
using a newer clang). I used clang r159409 to build ninja.

TBR=maruel@chromium.org

Review URL: https://chromiumcodereview.appspot.com/10810025

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@147557 0039d316-1c4b-4281-b951-d872f2087c98
      45298e08
  37. 27 Apr, 2012 1 commit
    • thakis@chromium.org's avatar
      Update ninja to 9b8e55d · f01196a6
      thakis@chromium.org authored 
      On Linux, I did a full ninja checkout and then ran:
python bootstrap.py
./configure.py --with-gtest=../chrome/src/testing/gtest
./ninja ninja_test
./ninja_test # (all tests passed)
strip ninja

I timed empty builds of 'chrome' with the old and the new version,
both took 1.1s for that. Binary size comparison:
121632 bytes old binary
129920 bytes new binary
I used gcc 4.4.3 to build ninja.


On Mac, I updated an existing checkout to the latest version and then ran:
CXX=../chrome-git/src/third_party/llvm-build/Release+Asserts/bin/clang++ \
./configure.py --with-gtest=../chrome-git/src/testing/gtest
./ninja
./ninja ninja_test # (all tests passed)
strip ninja

I timed empty builds of 'chrome' with the old and the new version,
both took 1.1s for that. Binary size comparison:
128976 bytes old binary
120904 bytes new binary
I used clang r155352 to build ninja. Last time I used gcc 4.2, which
explains the size drop.


The major new features in this update are:
* ninja cleans up files when interrupted mid-build
* NINJA_STATUS environment variable to tweak the status line
(see ninja manual)


Also set svn:mimetype to application/octet-stream on both files.
Review URL: https://chromiumcodereview.appspot.com/10221020/



git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@134307 0039d316-1c4b-4281-b951-d872f2087c98
      f01196a6
  39. 19 Jan, 2012 1 commit
  41. 13 Jan, 2012 1 commit