R=machenbach@chromium.org

Review-Url: https://codereview.chromium.org/2141013002
Cr-Commit-Position: refs/heads/master@{#37678}

Having presubmit called from within the test runner often
requires logic to remove the call again.

After the GN transition it would be better if presubmit is
called by a wrapper script if needed at all. It is run
on upload and on the tryservers anyways.

BUG=chromium:474921

Review-Url: https://codereview.chromium.org/2114653002
Cr-Commit-Position: refs/heads/master@{#37447}

This executes an action as part of the build, writing a json
configuration that includes all build flags relevant to v8
testing.

The test runner will derive all build-dependent flags from
the file if it detects it.

BUG=chromium:474921

Review-Url: https://codereview.chromium.org/2106423002
Cr-Commit-Position: refs/heads/master@{#37446}

This adds ignition to the testing variants that are run on all
bots.

Failing tests can only be skipped with the NO_IGNITION
keyword in status files. Existing expectations for the
ignition_turbofan variant are all duplicated and use the
NO_IGNITION keyword as well now.

BUG=v8:4280
LOG=N

Committed: https://crrev.com/132c09ed619f23fb7c6d26a4e3552c703389eabd
Cr-Commit-Position: refs/heads/master@{#35865}

Review-Url: https://codereview.chromium.org/1804003002
Cr-Commit-Position: refs/heads/master@{#35926}

Revert of [Ignition] Test ignition on all bots. (patchset #4 id:60001 of https://codereview.chromium.org/1804003002/ )

Reason for revert:
Will resubmit with proper fix and more test skips.

Original issue's description:
> [Ignition] Test ignition on all bots.
>
> This adds ignition to the testing variants that are run on all
> bots.
>
> Failing tests can only be skipped with the NO_IGNITION
> keyword in status files. Existing expectations for the
> ignition_turbofan variant are all duplicated and use the
> NO_IGNITION keyword as well now.
>
> BUG=v8:4280
> LOG=N
> NOTRY=true

TBR=rmcilroy@chromium.org,mstarzinger@chromium.org,yangguo@chromium.org,mythria@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=v8:4280

Review-Url: https://codereview.chromium.org/1930943002
Cr-Commit-Position: refs/heads/master@{#35871}

This adds ignition to the testing variants that are run on all
bots.

Failing tests can only be skipped with the NO_IGNITION
keyword in status files. Existing expectations for the
ignition_turbofan variant are all duplicated and use the
NO_IGNITION keyword as well now.

BUG=v8:4280
LOG=N
NOTRY=true

Review-Url: https://codereview.chromium.org/1804003002
Cr-Commit-Position: refs/heads/master@{#35865}

This adds a new ignition_turbofan testing variant that'll
allow switching turbofan off in the other ignition variant.

The switch will happen in a follow up after reconfiguring
the bots.

BUG=v8:4280
LOG=N

Review URL: https://codereview.chromium.org/1922623003

Cr-Commit-Position: refs/heads/master@{#35790}

BUG=chromium:605516
LOG=n

Review URL: https://codereview.chromium.org/1913013003

Cr-Commit-Position: refs/heads/master@{#35718}

BUG=v8:4437,v8:2899,chromium:604310
LOG=n

Review URL: https://codereview.chromium.org/1402373002

Cr-Commit-Position: refs/heads/master@{#35614}

The slow_archs list had only an effect when no timeout was
passed. Now it will also increase the timeout on the bots.

Also simplified the code a bit.

Review URL: https://codereview.chromium.org/1866953002

Cr-Commit-Position: refs/heads/master@{#35355}

TBR=jyan@ca.ibm.com
NOTRY=true

Review URL: https://codereview.chromium.org/1869683002

Cr-Commit-Position: refs/heads/master@{#35324}

The feature was removed from the bots a while ago. It was
superseeded by the flaky-test detection which reruns tests.

Remaining tests still marked as flaky most certainly pass
since a while.

Referencing all the bugs whose expectations lines get
removed by this.

BUG=v8:3838,v8:3525,v8:3125
LOG=n

Review URL: https://codereview.chromium.org/1802983002

Cr-Commit-Position: refs/heads/master@{#34804}

R=danno@chromium.org,jkummerow@chromium.org,jochen@chromium.org,joransiu@ca.ibm.com,michael_dawson@ca.ibm.com,mbrandy@us.ibm.com
BUG=

Review URL: https://codereview.chromium.org/1805533002

Cr-Commit-Position: refs/heads/master@{#34795}

This also runs the webkit test suite against Ignition. By now most of
the tests pass, the failing ones have been blacklisted.

R=machenbach@chromium.org
TEST=webkit

Review URL: https://codereview.chromium.org/1799603003

Cr-Commit-Position: refs/heads/master@{#34749}

This also runs the message test suite against Ignition. By now most of
the source positions (and exception messages) are accurate, the failing
ones have been blacklisted.

R=machenbach@chromium.org

Review URL: https://codereview.chromium.org/1783773003

Cr-Commit-Position: refs/heads/master@{#34708}

Changes include:
 - better test coverage for builds with snapshot
 - write snapshot blobs to buffer instead of test serialization files
 - renamed tests

R=machenbach@chromium.org

Review URL: https://codereview.chromium.org/1777213002

Cr-Commit-Position: refs/heads/master@{#34657}

With this, the test runner automatically merges sancov
files after testing. There's no need to do this by some
external infrastructure.

In a future CL, we could even merge during testing to lift
harddisk pressure.

BUG=chromium:568949
LOG=n
NOTRY=true

Review URL: https://codereview.chromium.org/1776123002

Cr-Commit-Position: refs/heads/master@{#34606}

This adds sanitizer-coverage compilation, test-runner
features and post processing.

Sanitizer coverage is expected to be used together with
asan.

During test runner execution, the produced sancov files
are disambiguated and match the pattern:
<executable name>.test.<test id>.sancov.

Two additional scripts are added for merging raw sancov
files and for generating json data containing all
instrumented lines + all covered lines from merged sancov
files. Both scripts use multiprocessing for speed.

The json data will later be uploaded to google storage
for further use, e.g. to show coverage data in rietveld.

Sancov documentation:
http://clang.llvm.org/docs/SanitizerCoverage.html

BUG=chromium:568949
LOG=n
NOTRY=true
TEST=python -m unittest sancov_formatter_test
TEST=python -m unittest sancov_merger_test

Review URL: https://codereview.chromium.org/1737263003

Cr-Commit-Position: refs/heads/master@{#34578}

BUG=chromium:577261
R=machenbach@chromium.org,jarin@chromium.org
LOG=n

Review URL: https://codereview.chromium.org/1604203002

Cr-Commit-Position: refs/heads/master@{#33508}

NOTRY=true

Review URL: https://codereview.chromium.org/1588073002

Cr-Commit-Position: refs/heads/master@{#33303}

BUG=v8:4280
LOG=N

Review URL: https://codereview.chromium.org/1559013004

Cr-Commit-Position: refs/heads/master@{#33114}

Add control for the ignition test suites. This will be
followed by a change on the infra side that targets the
ignition set instead of the hardcoded mjsunit.

After that, cctest can be enabled in a separate CL. This CL
already includes the logic for running cctest for ignition
on swarming.

BUG=v8:4280
LOG=N

Review URL: https://codereview.chromium.org/1559943002

Cr-Commit-Position: refs/heads/master@{#33089}

BUG=chromium:568949
LOG=n
NOTRY=true

Review URL: https://codereview.chromium.org/1520163003

Cr-Commit-Position: refs/heads/master@{#32840}

This will allow moving the test outcome check beyond the
multi-process boundary in a follow up. It'll allow wrapping
more complex test jobs like predicable mode on the multi-
process side, which will make the code easier to maintain.

BUG=

Review URL: https://codereview.chromium.org/1469833002

Cr-Commit-Position: refs/heads/master@{#32373}

The feature sometimes prevents subsequent swarming bots to
delete the work directory on windows.

The data file is not kept on swarming bots anyways, therefore
this switches off the feature completely.

BUG=chromium:535160
LOG=n

Review URL: https://codereview.chromium.org/1468933002

Cr-Commit-Position: refs/heads/master@{#32168}

Adds a blacklist of tests which are currently unsupported or broken in Ignition to
the mjsunit and test262 test status.

Also removes --ignition-script-filter flag, and adds a
--ignition_fallback_on_eval_and_catch flag which fallsback to fullcodegen for
functions which call eval or contain a catch block.

BUG=v8:4280
LOG=N

Review URL: https://codereview.chromium.org/1420963009

Cr-Commit-Position: refs/heads/master@{#31864}

The flake detection is done on the infra-side according to
the contents of the json test results. We don't want the
runner to fail after flakes.

This was controlled on the infra side by accepting any exit
codes so far. After the swarming switch, this is more
difficult, because the runner is wrapped by the swarming
collect script. There, failing exit codes can mean many
things, including network failures. Therefore, we now
force exit code 0 with test failures if those failures
are reported in the formal test results json.

The infrastructure will take care of reporting the flakes
and failures accordingly.

BUG=chromium:535160
LOG=n

Review URL: https://codereview.chromium.org/1416373005

Cr-Commit-Position: refs/heads/master@{#31740}

This is currently hardcoded in the infra-side test runner
setup, but will be removed there, so that swarming triggers
don't need a custom environment.

BUG=chromium:535160
LOG=n

Review URL: https://codereview.chromium.org/1411733008

Cr-Commit-Position: refs/heads/master@{#31682}

Review URL: https://codereview.chromium.org/1402353006

Cr-Commit-Position: refs/heads/master@{#31409}

Fix after https://codereview.chromium.org/1413023002/

BUG=chromium:535160
LOG=n
NOTRY=true
TBR=jkummerow@chromium.org

Review URL: https://codereview.chromium.org/1411973003

Cr-Commit-Position: refs/heads/master@{#31376}

The test suite is ran in 60% of the bots anyway and the
step is very short. For swarming, it's better to run this
together in one step as each step triggers a different bot.

BUG=chromium:535160
LOG=n
NOTRY=true

Review URL: https://codereview.chromium.org/1413023002

Cr-Commit-Position: refs/heads/master@{#31360}

NOTRY=true

Review URL: https://codereview.chromium.org/1413563002

Cr-Commit-Position: refs/heads/master@{#31328}

BUG=chromium:535160
LOG=n

Review URL: https://codereview.chromium.org/1397593004

Cr-Commit-Position: refs/heads/master@{#31308}

BUG=chromium:535160
LOG=n
NOTRY=true

Review URL: https://codereview.chromium.org/1395343002

Cr-Commit-Position: refs/heads/master@{#31207}

This adds the unittests to the "default" test set. Now that
the "default" and the DEFAULT_TESTS (i.e. runner with
no arguments) are the same, removed DEFAULT_TESTS and use
TEST_MAP["default"] instead.

On the bots, where unittests and default were run in
separation before, the explicit unittests step should now
be skipped.

This is necessary for swarming, as the unittests step is
too small to justify its own swarming job.

BUG=chromium:535160
LOG=n

Review URL: https://codereview.chromium.org/1374733006

Cr-Commit-Position: refs/heads/master@{#31122}

BUG=chromium:535160,v8:4459
LOG=n
NOTRY=true

Review URL: https://codereview.chromium.org/1384683002

Cr-Commit-Position: refs/heads/master@{#31073}

TEST=
BUG=

Review URL: https://codereview.chromium.org/1334793004

Cr-Commit-Position: refs/heads/master@{#31011}

[run-tests.py]
- adding more detailed information about the flags
- show more detailed error message on failing Popen commands

BUG=

Review URL: https://codereview.chromium.org/1369343002

Cr-Commit-Position: refs/heads/master@{#30997}

Review URL: https://codereview.chromium.org/1352803002

Cr-Commit-Position: refs/heads/master@{#30799}

The previous approach to this (in 1b1de2d2) did not work well with
bots that intentionally run no tests. But I think this warning message
is still useful for developers.

Review URL: https://codereview.chromium.org/1281313004

Cr-Commit-Position: refs/heads/master@{#30116}