1. 05 Apr, 2019 1 commit
    • Michael Achenbach's avatar
      [test] Expose variant flags in json output · ff0b9dd3
      Michael Achenbach authored
      For standard test failures, the variant name is enough to deduce the
      full command line of the test. This is used to create the flako
      command line on the infra side.
      
      Test failures from numfuzz have additional variant_flags, calculated
      by the fuzzer, which don't match a variant name. Exposing those in
      the test results will enable printing a proper flako command line
      on infra side for numfuzz cases.
      
      Bug: v8:8971
      Change-Id: Ie47d42a0b34037da458b474f2a9ab38f1a5d238a
      Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1554689Reviewed-by: 's avatarSergiy Belozorov <sergiyb@chromium.org>
      Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#60656}
      ff0b9dd3
  2. 03 Apr, 2019 1 commit
  3. 27 Feb, 2019 1 commit
  4. 19 Feb, 2019 1 commit
  5. 12 Feb, 2019 3 commits
    • Tamer Tas's avatar
      [testrunner] enable the progress indicator · baeb4e32
      Tamer Tas authored
      Using test generators meant that we had to remove the progress indicator since
      the total number of tests weren't known before-hand.
      
      This CL implements a progress indicator using test number estimations.
      
      cctest and unittests progress indicator is accurate, however estimating
      means the progress will terminate over 100% in big test suites and sometimes
      under 100%.
      
      R=machenbach@chromium.org
      CC=​sergiyb@chromium.org,yangguo@chromium.org
      
      Bug: v8:8769
      Change-Id: I40ca5b40f9b1223376d33707f0945900ea98cea3
      Reviewed-on: https://chromium-review.googlesource.com/c/1460471
      Commit-Queue: Tamer Tas <tmrts@chromium.org>
      Reviewed-by: 's avatarMichael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#59538}
      baeb4e32
    • Tamer Tas's avatar
      [fuzzer] create a test-suite for the fuzzer · e2207c67
      Tamer Tas authored
      Testrunner test-suite doesn't have automated tests for the Fuzzer. A significant
      change for the testrunner may break the fuzzers. We don't have a way to catch
      even the syntax errors for the fuzzer before landing CLs.
      
      This CL creates a rudimentary test-suite that runs the fuzzer without any tests
      to see if APIs still work as intended.
      
      R=machenbach@chromium.org
      CC=yangguo@chromium.org,sergiyb@chromium.org
      
      No-Try: true
      Bug: v8:8763
      Change-Id: Ib4519ca093de2af5279964edd164cee04aae77a8
      Reviewed-on: https://chromium-review.googlesource.com/c/1466301Reviewed-by: 's avatarMichael Achenbach <machenbach@chromium.org>
      Commit-Queue: Tamer Tas <tmrts@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#59535}
      e2207c67
    • Tamer Tas's avatar
      [testrunner] enable variant sharding · ad5b6d7d
      Tamer Tas authored
      Testrunner runs variants of a test sequentially without taking sharding into
      account. A slow test with slow variants slows down the whole test run no matter
      the sharding configuration.
      
      This CL implements a test hashing algorithm and variant sharding for test
      variants.
      
      R=machenbach@chromium.org
      CC=yangguo@chromium.org,sergiyb@chromium.org
      
      Bug: v8:8174
      Change-Id: I15f8c547fa2f361fb6c53bf8d5df055d3df38d3e
      Reviewed-on: https://chromium-review.googlesource.com/c/1458016
      Commit-Queue: Tamer Tas <tmrts@chromium.org>
      Reviewed-by: 's avatarMichael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#59530}
      ad5b6d7d
  6. 07 Feb, 2019 1 commit
  7. 06 Feb, 2019 1 commit
    • Tamer Tas's avatar
      Reland "Reland "[test] refactor testsuite configuration"" · df630e67
      Tamer Tas authored
      This is a reland of 81eec150
      
      Original change's description:
      > Reland "[test] refactor testsuite configuration"
      >
      > This is a reland of 7f92ad0a
      >
      > Original change's description:
      > > [test] refactor testsuite configuration
      > >
      > > Every testsuite configuration consist of at least 30% code duplication.
      > >
      > > The code age ranges from 10 years old to 5 years old. Implementing anything that
      > > touches the testsuite code becomes a technical fight to the death.
      > >
      > > This CL removes all the duplication by refactoring the common functionality.
      > >
      > > This CL contains structural changes without any logical changes % small bug
      > > fixes.
      > >
      > > R=machenbach@chromium.org
      > > CC=yangguo@chromium.org,sergiyb@chromium.org
      > >
      > > Bug: v8:8174, v8:8769
      > > Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
      > > Reviewed-on: https://chromium-review.googlesource.com/c/1445881
      > > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
      > > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
      > > Cr-Commit-Position: refs/heads/master@{#59361}
      >
      > Bug: v8:8174, v8:8769
      > Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
      > Reviewed-on: https://chromium-review.googlesource.com/c/1454485
      > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
      > Commit-Queue: Tamer Tas <tmrts@chromium.org>
      > Cr-Commit-Position: refs/heads/master@{#59369}
      
      Bug: v8:8174, v8:8790
      Change-Id: I38ab9d37bca76057441a970f26e2102e4387a857
      Reviewed-on: https://chromium-review.googlesource.com/c/1454724
      Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      Reviewed-by: 's avatarMichael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#59387}
      df630e67
  8. 05 Feb, 2019 4 commits
    • Tamer Tas's avatar
      Revert "Reland "[test] refactor testsuite configuration"" · ea4412ad
      Tamer Tas authored
      This reverts commit 81eec150.
      
      Reason for revert: windows mozilla test failures
      
      Original change's description:
      > Reland "[test] refactor testsuite configuration"
      > 
      > This is a reland of 7f92ad0a
      > 
      > Original change's description:
      > > [test] refactor testsuite configuration
      > > 
      > > Every testsuite configuration consist of at least 30% code duplication.
      > > 
      > > The code age ranges from 10 years old to 5 years old. Implementing anything that
      > > touches the testsuite code becomes a technical fight to the death.
      > > 
      > > This CL removes all the duplication by refactoring the common functionality.
      > > 
      > > This CL contains structural changes without any logical changes % small bug
      > > fixes.
      > > 
      > > R=machenbach@chromium.org
      > > CC=yangguo@chromium.org,sergiyb@chromium.org
      > > 
      > > Bug: v8:8174, v8:8769
      > > Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
      > > Reviewed-on: https://chromium-review.googlesource.com/c/1445881
      > > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
      > > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
      > > Cr-Commit-Position: refs/heads/master@{#59361}
      > 
      > Bug: v8:8174, v8:8769
      > Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
      > Reviewed-on: https://chromium-review.googlesource.com/c/1454485
      > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
      > Commit-Queue: Tamer Tas <tmrts@chromium.org>
      > Cr-Commit-Position: refs/heads/master@{#59369}
      
      TBR=machenbach@chromium.org,sergiyb@chromium.org,tmrts@chromium.org
      
      Change-Id: I8f5650b5f46be299c004e2fa8b708fa2c17a4dc2
      No-Presubmit: true
      No-Tree-Checks: true
      No-Try: true
      Bug: v8:8174, v8:8769
      Reviewed-on: https://chromium-review.googlesource.com/c/1454607Reviewed-by: 's avatarMichael Achenbach <machenbach@chromium.org>
      Commit-Queue: Tamer Tas <tmrts@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#59370}
      ea4412ad
    • Tamer Tas's avatar
      Reland "[test] refactor testsuite configuration" · 81eec150
      Tamer Tas authored
      This is a reland of 7f92ad0a
      
      Original change's description:
      > [test] refactor testsuite configuration
      > 
      > Every testsuite configuration consist of at least 30% code duplication.
      > 
      > The code age ranges from 10 years old to 5 years old. Implementing anything that
      > touches the testsuite code becomes a technical fight to the death.
      > 
      > This CL removes all the duplication by refactoring the common functionality.
      > 
      > This CL contains structural changes without any logical changes % small bug
      > fixes.
      > 
      > R=machenbach@chromium.org
      > CC=yangguo@chromium.org,sergiyb@chromium.org
      > 
      > Bug: v8:8174, v8:8769
      > Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
      > Reviewed-on: https://chromium-review.googlesource.com/c/1445881
      > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
      > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
      > Cr-Commit-Position: refs/heads/master@{#59361}
      
      Bug: v8:8174, v8:8769
      Change-Id: I8e7078cfb875ceb3777e57084e6f8dfac09693e7
      Reviewed-on: https://chromium-review.googlesource.com/c/1454485Reviewed-by: 's avatarMichael Achenbach <machenbach@chromium.org>
      Commit-Queue: Tamer Tas <tmrts@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#59369}
      81eec150
    • Tamer Tas's avatar
      Revert "[test] refactor testsuite configuration" · 97068800
      Tamer Tas authored
      This reverts commit 7f92ad0a.
      
      Reason for revert: https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20Win32/19148
      
      Original change's description:
      > [test] refactor testsuite configuration
      >
      > Every testsuite configuration consist of at least 30% code duplication.
      >
      > The code age ranges from 10 years old to 5 years old. Implementing anything that
      > touches the testsuite code becomes a technical fight to the death.
      >
      > This CL removes all the duplication by refactoring the common functionality.
      >
      > This CL contains structural changes without any logical changes % small bug
      > fixes.
      >
      > R=​machenbach@chromium.org
      > CC=​yangguo@chromium.org,sergiyb@chromium.org
      >
      > Bug: v8:8174, v8:8769
      > Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
      > Reviewed-on: https://chromium-review.googlesource.com/c/1445881
      > Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
      > Reviewed-by: Michael Achenbach <machenbach@chromium.org>
      > Cr-Commit-Position: refs/heads/master@{#59361}
      
      TBR=machenbach@chromium.org,sergiyb@chromium.org,tmrts@chromium.org,v8-reviews@chromium.org
      
      Change-Id: I473f0d4c6b9c0239923b8c03699dbc38b7f85030
      No-Presubmit: true
      No-Tree-Checks: true
      No-Try: true
      Bug: v8:8174, v8:8769
      Reviewed-on: https://chromium-review.googlesource.com/c/1454599
      Commit-Queue: Tamer Tas <tmrts@chromium.org>
      Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
      Reviewed-by: 's avatarClemens Hammacher <clemensh@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#59362}
      97068800
    • Tamer Tas's avatar
      [test] refactor testsuite configuration · 7f92ad0a
      Tamer Tas authored
      Every testsuite configuration consist of at least 30% code duplication.
      
      The code age ranges from 10 years old to 5 years old. Implementing anything that
      touches the testsuite code becomes a technical fight to the death.
      
      This CL removes all the duplication by refactoring the common functionality.
      
      This CL contains structural changes without any logical changes % small bug
      fixes.
      
      R=machenbach@chromium.org
      CC=yangguo@chromium.org,sergiyb@chromium.org
      
      Bug: v8:8174, v8:8769
      Change-Id: Iee299569caa7abdc0307ecf606136669034a28a2
      Reviewed-on: https://chromium-review.googlesource.com/c/1445881
      Commit-Queue: Sergiy Belozorov <sergiyb@chromium.org>
      Reviewed-by: 's avatarMichael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#59361}
      7f92ad0a
  9. 01 Feb, 2019 3 commits
  10. 21 Jan, 2019 2 commits
  11. 18 Jan, 2019 1 commit
  12. 19 Dec, 2018 1 commit
  13. 18 Dec, 2018 2 commits
  14. 17 Dec, 2018 1 commit
  15. 22 Nov, 2018 1 commit
  16. 30 Oct, 2018 2 commits
  17. 22 Oct, 2018 1 commit
  18. 16 Oct, 2018 1 commit
  19. 10 Oct, 2018 1 commit
  20. 02 Oct, 2018 1 commit
  21. 07 Sep, 2018 1 commit
    • Michael Achenbach's avatar
      Reland "[test] Increase coverage of d8_default test suites" · 0dba4b90
      Michael Achenbach authored
      This is a reland of 8ac91f6c
      
      Skips failing tests on gc stress and fixes predictable testing.
      
      Original change's description:
      > [test] Increase coverage of d8_default test suites
      >
      > NOTRY=true
      >
      > Bug: v8:7285,v8:8140
      > Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
      > Change-Id: I9cb216de302bc787189f8f12f5b254909b0f5773
      > Reviewed-on: https://chromium-review.googlesource.com/1208496
      > Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      > Reviewed-by: Ross McIlroy <rmcilroy@chromium.org>
      > Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
      > Cr-Commit-Position: refs/heads/master@{#55682}
      
      Bug: v8:7285, v8:8140, v8:8141
      Change-Id: Ia7a437b874d5c8712f6def30382404e527145610
      Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
      Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel_ng
      Cq-Include-Trybots: luci.v8.try:v8_linux_optional_rel_ng
      Cq-Include-Trybots: luci.v8.try:v8_linux_gc_stress_dbg
      Cq-Include-Trybots: luci.v8.try:v8_mac64_gc_stress_dbg
      Reviewed-on: https://chromium-review.googlesource.com/1209762Reviewed-by: 's avatarSergiy Byelozyorov <sergiyb@chromium.org>
      Reviewed-by: 's avatarYang Guo <yangguo@chromium.org>
      Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#55701}
      0dba4b90
  22. 10 Aug, 2018 3 commits
    • Michael Achenbach's avatar
      Reland "[test] Add logic to run tests on Android" · b2537f21
      Michael Achenbach authored
      This is a reland of 4c094342
      
      Original change's description:
      > [test] Add logic to run tests on Android
      > 
      > This adds a new command abstraction for running commands on Android
      > using dockered devices on swarming.
      > 
      > The new abstraction handles pushing all required files to the device.
      > The logic used for pushing and running is reused from the perf runner.
      > 
      > This adds only the mjsunit test suite. Others will be handled in
      > follow up CLs. The suite logic is enhanced with auto-detection of files
      > to be pushed to devices, for e.g. load or import statements.
      > 
      > Some test cases need an extra resource section for specifying required
      > files.
      > 
      > Remaining failing tests are marked in the status files for later
      > triage.
      > 
      > Bug: chromium:866862
      > Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
      > Reviewed-on: https://chromium-review.googlesource.com/1150153
      > Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
      > Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      > Cr-Commit-Position: refs/heads/master@{#55041}
      
      Bug: chromium:866862
      Change-Id: Icf7e04c75d4abeab7254d10ba21240e46b0022ae
      Reviewed-on: https://chromium-review.googlesource.com/1170643Reviewed-by: 's avatarSergiy Byelozyorov <sergiyb@chromium.org>
      Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#55059}
      b2537f21
    • Sergiy Byelozyorov's avatar
      Revert "[test] Add logic to run tests on Android" · 2071051e
      Sergiy Byelozyorov authored
      This reverts commit 4c094342.
      
      Reason for revert: Unfortunately this broke all perf builders.
      
      Original change's description:
      > [test] Add logic to run tests on Android
      > 
      > This adds a new command abstraction for running commands on Android
      > using dockered devices on swarming.
      > 
      > The new abstraction handles pushing all required files to the device.
      > The logic used for pushing and running is reused from the perf runner.
      > 
      > This adds only the mjsunit test suite. Others will be handled in
      > follow up CLs. The suite logic is enhanced with auto-detection of files
      > to be pushed to devices, for e.g. load or import statements.
      > 
      > Some test cases need an extra resource section for specifying required
      > files.
      > 
      > Remaining failing tests are marked in the status files for later
      > triage.
      > 
      > Bug: chromium:866862
      > Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
      > Reviewed-on: https://chromium-review.googlesource.com/1150153
      > Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
      > Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      > Cr-Commit-Position: refs/heads/master@{#55041}
      
      TBR=machenbach@chromium.org,yangguo@chromium.org,sergiyb@chromium.org
      
      Change-Id: If80129810586b709dab762c9b5724888e15daec2
      No-Presubmit: true
      No-Tree-Checks: true
      No-Try: true
      Bug: chromium:866862
      Reviewed-on: https://chromium-review.googlesource.com/1170962Reviewed-by: 's avatarSergiy Byelozyorov <sergiyb@chromium.org>
      Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#55058}
      2071051e
    • Michael Achenbach's avatar
      [test] Add logic to run tests on Android · 4c094342
      Michael Achenbach authored
      This adds a new command abstraction for running commands on Android
      using dockered devices on swarming.
      
      The new abstraction handles pushing all required files to the device.
      The logic used for pushing and running is reused from the perf runner.
      
      This adds only the mjsunit test suite. Others will be handled in
      follow up CLs. The suite logic is enhanced with auto-detection of files
      to be pushed to devices, for e.g. load or import statements.
      
      Some test cases need an extra resource section for specifying required
      files.
      
      Remaining failing tests are marked in the status files for later
      triage.
      
      Bug: chromium:866862
      Change-Id: I2b957559f07fdcd8c1bd2f7034f5ba7754a31fb7
      Reviewed-on: https://chromium-review.googlesource.com/1150153Reviewed-by: 's avatarSergiy Byelozyorov <sergiyb@chromium.org>
      Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#55041}
      4c094342
  23. 25 Jul, 2018 1 commit
  24. 20 Feb, 2018 1 commit
    • Michael Achenbach's avatar
      [test] Abort testing immediately on interrupt and sigterm · 4a2d9b1b
      Michael Achenbach authored
      This is a partial revert of:
      https://crrev.com/c/890938 and https://crrev.com/c/893982
      
      Before this CL, the test runner blocked on ongoing tests in order to
      process their results after an internal timeout. However, the logic
      required for this feature was overly complicated and prevented an
      acceptable implementation for fast aborts. Furthermore, also the fuzzers
      suffered from timeouts on swarming due to hanging tests.
      
      Instead, we now abort immediately on internal timeout (used on
      fuzzers), SIGINT (Ctrl-C) and SIGTERM. Ongoing tests are immediately
      terminated and their results are disregarded. On SIGTERM and SIGINT,
      we return with non-zero exit codes, and zero on internal timeout.
      
      This will also properly return json output, when the external hard
      timeout is reached on swarming (causes SIGTERM).
      
      TBR=sergiyb@chromium.org
      
      Bug: v8:7423, chromium:813065
      Change-Id: Ib20f835f58a0970693bdd3b21dc5d766d8e115d8
      Reviewed-on: https://chromium-review.googlesource.com/924852Reviewed-by: 's avatarMichael Achenbach <machenbach@chromium.org>
      Commit-Queue: Michael Achenbach <machenbach@chromium.org>
      Cr-Commit-Position: refs/heads/master@{#51399}
      4a2d9b1b
  25. 01 Feb, 2018 2 commits
  26. 31 Jan, 2018 2 commits