Skip to content

ZTS: make zpool_iostat_interval_all teardown deterministic - #18776

Merged
behlendorf merged 1 commit into
openzfs:masterfrom
MorganaFuture:iostat-export-flaky
Jul 17, 2026
Merged

ZTS: make zpool_iostat_interval_all teardown deterministic#18776
behlendorf merged 1 commit into
openzfs:masterfrom
MorganaFuture:iostat-export-flaky

Conversation

@MorganaFuture

@MorganaFuture MorganaFuture commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Motivation and Context

cli_root/zpool_iostat/zpool_iostat_interval_all is intermittently failing
and is currently suppressed in tests/test-runner/bin/zts-report.py.in
(#18273).

The test runs zpool iostat -T u 0.1 in the background and compares
its output, parsed into a sequence of "chunks", against a fixed expected
sequence as pools are created, imported, exported and destroyed. Every step
changes the visible pool list by exactly one pool except the teardown, which
uses a single zpool export -a.

export -a exports the pools one after another rather than atomically, so
there is a brief window in which one pool is already gone and the other is
not. At the 0.1s sampling interval iostat occasionally catches that
intermediate single-pool state and emits an extra chunk that is not in the
expected sequence, and the test fails. Whether a sample lands in the window
depends on timing, which is why it shows up as a flake.

Description

Export the two pools explicitly, one at a time, and add the intermediate
single-pool state to the expected output — the same one-pool-at-a-time shape
the imports and destroys elsewhere in the test already use. The teardown
transition becomes deterministic and the spurious extra chunk can no longer
appear.

With the race removed the test's entry is dropped from the
zts-report.py.in "maybe" list.

How Has This Been Tested?

Looped the teardown in a VM (150 iterations each):

  • With zpool export -a, iostat emitted a spurious extra single-pool chunk
    during teardown in 17 of 150 runs — the failure mode from ZTS: zpool_iostat_interval_all #18273.
  • With the two explicit ordered exports, every run produced the expected
    POOLBOTH -> POOL2 -> NOPOOL teardown; no spurious chunk in any run.

Opening as a draft to get a full CI run on the de-suppressed test.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • My code follows the OpenZFS code style requirements.
  • I have run the ZFS Test Suite with my changes.
  • All commit messages are properly formatted and contain Signed-off-by.

@github-actions github-actions Bot added the Status: Work in Progress Not yet ready for general review label Jul 10, 2026
@MorganaFuture
MorganaFuture force-pushed the iostat-export-flaky branch 2 times, most recently from 46a8f83 to edb26fc Compare July 10, 2026 12:05
behlendorf
behlendorf approved these changes Jul 10, 2026
View reviewed changes

@behlendorf behlendorf left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that nicely explains things.

@MorganaFuture
MorganaFuture marked this pull request as ready for review July 11, 2026 16:30
@github-actions github-actions Bot added Status: Code Review Needed Ready for review and testing and removed Status: Work in Progress Not yet ready for general review labels Jul 11, 2026
@behlendorf

behlendorf commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

@MorganaFuture when you get a chance can you rebase this and resolve the conflict.

@behlendorf behlendorf added Status: Revision Needed Changes are required for the PR to be accepted and removed Status: Code Review Needed Ready for review and testing labels Jul 14, 2026
@MorganaFuture
ZTS: make zpool_iostat_interval_all teardown deterministic
417ee75 
zpool_iostat_interval_all runs "zpool iostat" in the background at a
0.1s interval and compares its output, parsed into a sequence of
chunks, against a fixed expected sequence as pools are created,
imported, exported and destroyed. Every step changes the visible pool
list by exactly one pool except the teardown, which used a single
"zpool export -a".

export -a exports the pools one after another rather than atomically,
so there is a brief window in which one pool is already gone and the
other is not. At the 0.1s sampling interval iostat occasionally catches
that intermediate single-pool state and emits an extra chunk that is
not in the expected sequence, and the test fails. Whether a sample
lands in the window depends on timing, which is why it shows up as a
flake.

Export the two pools explicitly, one at a time, and add the
intermediate single-pool state to the expected output, the same way
the imports and destroys elsewhere in the test already change one pool
at a time. The teardown transition is now deterministic.

Drop the test from the zts-report.py.in "maybe" list.

Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
Closes openzfs#18273
@github-actions github-actions Bot removed the Status: Revision Needed Changes are required for the PR to be accepted label Jul 17, 2026
@MorganaFuture

MorganaFuture commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Done

@behlendorf behlendorf added the Status: Accepted Ready to integrate (reviewed, tested) label Jul 17, 2026
@behlendorf
behlendorf merged commit 817fc37 into openzfs:master Jul 17, 2026
27 of 30 checks passed
lundman pushed a commit to openzfsonosx/openzfs-fork that referenced this pull request Jul 30, 2026
@MorganaFuture @lundman
ZTS: make zpool_iostat_interval_all teardown deterministic
064e669 
zpool_iostat_interval_all runs "zpool iostat" in the background at a
0.1s interval and compares its output, parsed into a sequence of
chunks, against a fixed expected sequence as pools are created,
imported, exported and destroyed. Every step changes the visible pool
list by exactly one pool except the teardown, which used a single
"zpool export -a".

export -a exports the pools one after another rather than atomically,
so there is a brief window in which one pool is already gone and the
other is not. At the 0.1s sampling interval iostat occasionally catches
that intermediate single-pool state and emits an extra chunk that is
not in the expected sequence, and the test fails. Whether a sample
lands in the window depends on timing, which is why it shows up as a
flake.

Export the two pools explicitly, one at a time, and add the
intermediate single-pool state to the expected output, the same way
the imports and destroys elsewhere in the test already change one pool
at a time. The teardown transition is now deterministic.

Drop the test from the zts-report.py.in "maybe" list.

Reviewed-by: Brian Behlendorf 
Reviewed-by: Alexander Motin 
Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
Closes openzfs#18273
Closes openzfs#18776
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Aug 12, 2026
@MorganaFuture @tonyhutter
ZTS: make zpool_iostat_interval_all teardown deterministic
8d80560 
zpool_iostat_interval_all runs "zpool iostat" in the background at a
0.1s interval and compares its output, parsed into a sequence of
chunks, against a fixed expected sequence as pools are created,
imported, exported and destroyed. Every step changes the visible pool
list by exactly one pool except the teardown, which used a single
"zpool export -a".

export -a exports the pools one after another rather than atomically,
so there is a brief window in which one pool is already gone and the
other is not. At the 0.1s sampling interval iostat occasionally catches
that intermediate single-pool state and emits an extra chunk that is
not in the expected sequence, and the test fails. Whether a sample
lands in the window depends on timing, which is why it shows up as a
flake.

Export the two pools explicitly, one at a time, and add the
intermediate single-pool state to the expected output, the same way
the imports and destroys elsewhere in the test already change one pool
at a time. The teardown transition is now deterministic.

Drop the test from the zts-report.py.in "maybe" list.

Reviewed-by: Brian Behlendorf 
Reviewed-by: Alexander Motin 
Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
Closes openzfs#18273
Closes openzfs#18776
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Aug 13, 2026
@MorganaFuture @tonyhutter
ZTS: make zpool_iostat_interval_all teardown deterministic
d37f629 
zpool_iostat_interval_all runs "zpool iostat" in the background at a
0.1s interval and compares its output, parsed into a sequence of
chunks, against a fixed expected sequence as pools are created,
imported, exported and destroyed. Every step changes the visible pool
list by exactly one pool except the teardown, which used a single
"zpool export -a".

export -a exports the pools one after another rather than atomically,
so there is a brief window in which one pool is already gone and the
other is not. At the 0.1s sampling interval iostat occasionally catches
that intermediate single-pool state and emits an extra chunk that is
not in the expected sequence, and the test fails. Whether a sample
lands in the window depends on timing, which is why it shows up as a
flake.

Export the two pools explicitly, one at a time, and add the
intermediate single-pool state to the expected output, the same way
the imports and destroys elsewhere in the test already change one pool
at a time. The teardown transition is now deterministic.

Drop the test from the zts-report.py.in "maybe" list.

Reviewed-by: Brian Behlendorf 
Reviewed-by: Alexander Motin 
Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
Closes openzfs#18273
Closes openzfs#18776
pull Bot pushed a commit to A-Archives-and-Forks/openzfs that referenced this pull request Sep 26, 2026
@MorganaFuture @lundman
ZTS: make zpool_iostat_interval_all teardown deterministic
9b91da0 
zpool_iostat_interval_all runs "zpool iostat" in the background at a
0.1s interval and compares its output, parsed into a sequence of
chunks, against a fixed expected sequence as pools are created,
imported, exported and destroyed. Every step changes the visible pool
list by exactly one pool except the teardown, which used a single
"zpool export -a".

export -a exports the pools one after another rather than atomically,
so there is a brief window in which one pool is already gone and the
other is not. At the 0.1s sampling interval iostat occasionally catches
that intermediate single-pool state and emits an extra chunk that is
not in the expected sequence, and the test fails. Whether a sample
lands in the window depends on timing, which is why it shows up as a
flake.

Export the two pools explicitly, one at a time, and add the
intermediate single-pool state to the expected output, the same way
the imports and destroys elsewhere in the test already change one pool
at a time. The teardown transition is now deterministic.

Drop the test from the zts-report.py.in "maybe" list.

Reviewed-by: Brian Behlendorf 
Reviewed-by: Alexander Motin 
Signed-off-by: MorganaFuture <103630661+MorganaFuture@users.noreply.github.com>
Closes openzfs#18273
Closes openzfs#18776
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@behlendorf behlendorf behlendorf approved these changes
@amotin amotin amotin approved these changes

Assignees

No one assigned

Labels

Status: Accepted Ready to integrate (reviewed, tested)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@MorganaFuture @behlendorf @amotin