Tests fail in clean fork

[tfendin]

It seems like the tests test_git_tag and test_git_stable requires the tags from the git repo. When cloning a new fork the tags doesn't come along and thus the tests fail, see log below.

================================================================================== FAILURES ==================================================================================
________________________________________________________________________________ test_git_tag ________________________________________________________________________________

    @skipif_no_git
    def test_git_tag():
        '''Compare version numbers in main script and git tag'''
        skip_development_versions()
        app_version = read_version(BCPP, lines=5)
        command = 'git log -1 --format=%D --'.split() + [BCPP]
        output = run(command, stdout=PIPE)
        match = VERSION.search(output)
        if not match:
>           raise AssertionError('version pattern not found in {}'.format(' '.join(command)))
E           AssertionError: version pattern not found in git log -1 --format=%D -- bash_completion

tests/test_release.py:72: AssertionError
______________________________________________________________________________ test_git_stable _______________________________________________________________________________

    @skipif_no_git
    def test_git_stable():
        '''Check that git stable tag points to the latest CHANGELOG entry'''
        skip_development_versions()
>       stable = read_git_ref('stable')

tests/test_release.py:81:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_release.py:53: in read_git_ref
    return run(command, stdout=PIPE)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

command = ['git', 'show-ref', '-s', '--heads', '--tags', 'stable'], ka = {'stdout': -1}
process = <Popen: returncode: 1 args: ['git', 'show-ref', '-s', '--heads', '--tags', '...>, exit_code = 1, output = ''

    def run(command, **ka):
        '''Backwards compatible subprocess runner'''
        process = Popen(command, **ka)
        exit_code = process.wait()
        output = process.stdout.read().decode().strip()
        if exit_code:
>           raise RuntimeError('command returned {}: {}'.format(exit_code, ' '.join(command)))
E           RuntimeError: command returned 1: git show-ref -s --heads --tags stable

tests/test_release.py:33: RuntimeError
========================================================================== short test summary info ===========================================================================
FAILED tests/test_release.py::test_git_tag - AssertionError: version pattern not found in git log -1 --format=%D -- bash_completion
FAILED tests/test_release.py::test_git_stable - RuntimeError: command returned 1: git show-ref -s --heads --tags stable
================================================================== 2 failed, 33 passed, 2 skipped in 14.50s ==================================================================

[sio]

I understand the error, but I don't know what we can do about it. Do you have any suggestions?

Test suite rightly warns you about missing git tags in what appears to be a release source tree. You can append a "-dev" suffix to the version number in bash_completion file - then these tests will be skipped because they're not applicable to development versions (see skip_development_versions()). This will not happen automatically upon checking out a fork of this repo, of course. Another option would be to mirror tags from the parent repo - which also does not happen automatically, as you noticed.

I don't think we can teach the test suite to detect if it's running from a clone of a fork instead of a clone of a parent repo. Checking git remote URL would be pretty ugly and invasive. Somewhat error prone too - with git decentralized nature a clone can come from an unexpected URL even when it does not mean the project was forked.

To summarize, I think our options are:

• Detect if test suite is running from a clone of a forked repo, or
• Improve documentation to mention this behavior, or
• Improve assertion error messages to minimize confusion

What do you think?

[tfendin]

I think we can extend skip_development_versions() so that it skips those tests if there are no tags in the repo. I'm currently stuck on mobile so I can't investigate if it's feasable.

[sio]

I'm not sure that this is a good idea. No tags is definitely a "brown bag release" kind of scenario, and test suite should be loud about that.

[sio]

How about introducing an environment variable to skip release tests? Like BCPP_TEST_SKIP_RELEASE_CHECKS - if set, it would act the same as -dev suffix in bcpp version.

In addition to that we could add a pipeline step that sets this variable if the pipeline does not belong to my repo. This would still not affect local checkouts of a forked repo, and release tests would still fail there, but at least it would be fully automatic in GitHub and more or less easy to skip manually.