PR#10: WIP: tests: Execute tests from upstream instead - rpms/standard-test-roles

rpms / standard-test-roles

#10 WIP: tests: Execute tests from upstream instead

Merged 5 years ago by astepano. Opened 5 years ago by stefw.

rpms/ stefw/standard-test-roles tests-upstream into master

tests: Execute tests from upstream instead

Stef Walter • 5 years ago

485ae9d

.gitignore

file modified

		`@@ -1,3 +1,4 @@`
		`+ /tests/str/`
		`/standard-test-roles-0.2.tar.gz`
		`/standard-test-roles-0.3.tar.gz`
		`/standard-test-roles-0.4.tar.gz`

tests/avocado.yml

file removed

-24

		`@@ -1,24 +0,0 @@`
		`- # Tests for avocado role`
		`- - hosts: localhost`
		`- tags:`
		`- - classic`
		`- - container`
		`- roles:`
		`- - role: standard-test-avocado`
		`- tests:`
		`- - /bin/true`
		`- - role: standard-test-avocado`
		`- tests:`
		`- - /bin/false`
		`- ignore_errors: yes`
		`- tasks:`
		`- - name: Read test.log on test environment`
		`- set_fact:`
		`- test_log: "{{ lookup('file', artifacts + '/test.log') }}"`
		`- - name: Check for FAIL on test.log on test environment`
		`- fail: msg="Could not find expected text on test log"`
		`- when: >`
		`- (test_log.find('/bin/false: FAIL') == -1) or`
		`- (test_log.find('/bin/true: PASS') == -1)`
		`- # We should fail when we can not find FAIL string on log`
		`- # -1 means string not found`

tests/basic.yml

file removed

-38

		`@@ -1,38 +0,0 @@`
		`- - import_playbook: prepare.yml`
		`-`
		`- # Tests for basic role`
		`- - hosts: localhost`
		`- tags:`
		`- - atomic`
		`- - classic`
		`- - container`
		`- roles:`
		`- - role: standard-test-basic`
		`- tests:`
		`- - test-basic-simple`
		`- - test-basic-parameters:`
		`- dir: ./`
		`- run: echo "check parameters on basic role" \| grep "check parameters on basic role"`
		`- required_packages:`
		`- # Test if we can install required packages`
		`- - "{{req_pkg}}"`
		`- tasks:`
		`- - import_tasks: shared-tasks/artifacts_test_env.yml`
		`- - import_tasks: shared-tasks/artifacts_test_runner.yml`
		`-`
		`- # Make sure the role behaves correctly if test fails`
		`- - hosts: localhost`
		`- tags:`
		`- - atomic`
		`- - classic`
		`- - container`
		`- roles:`
		`- - role: standard-test-basic`
		`- tests:`
		`- - test-basic-fail`
		`- ignore_errors: yes`
		`- tasks:`
		`- # 'verify_failed_test' tasks should run after 'test-basic-fail'`
		`- - import_tasks: shared-tasks/verify_failed_test.yml`
		`- - import_tasks: shared-tasks/artifacts_test_env.yml`
		`- - import_tasks: shared-tasks/artifacts_test_runner.yml`

tests/beakerlib.yml

file removed

-35

		`@@ -1,35 +0,0 @@`
		`- - import_playbook: prepare.yml`
		`-`
		`- # Tests for beakerlib role`
		`- - hosts: localhost`
		`- tags:`
		`- - atomic`
		`- - classic`
		`- - container`
		`- roles:`
		`- - role: standard-test-beakerlib`
		`- tests:`
		`- - test-beakerlib-simple`
		`- required_packages:`
		`- - "{{req_pkg}}"`
		`- tasks:`
		`- - import_tasks: shared-tasks/artifacts_test_env.yml`
		`- - import_tasks: shared-tasks/artifacts_test_runner.yml`
		`-`
		`-`
		`- # Make sure the role behaves correctly if test fails`
		`- - hosts: localhost`
		`- tags:`
		`- - atomic`
		`- - classic`
		`- - container`
		`- roles:`
		`- - role: standard-test-beakerlib`
		`- tests:`
		`- - test-beakerlib-fail`
		`- ignore_errors: yes`
		`- tasks:`
		`- # 'tests_verify_failed_test' tasks should run after 'test-beakerlib-fail'`
		`- - import_tasks: shared-tasks/verify_failed_test.yml`
		`- - import_tasks: shared-tasks/artifacts_test_env.yml`
		`- - import_tasks: shared-tasks/artifacts_test_runner.yml`

tests/inventory.yml

file removed

-14

		`@@ -1,14 +0,0 @@`
		`- - hosts: localhost`
		`- tags:`
		`- - always`
		`- tasks:`
		`- - name: Get inventory with empty input`
		`- command: merge-standard-inventory`
		`- environment:`
		`- # Reset vars to default values`
		`- TEST_SUBJECTS:`
		`- register: inventory`
		`- delegate_to: localhost`
		`- - name: Assert merge-standard-inventory produces correct JSON inventory (empty)`
		`- fail:`
		`- when: inventory.stdout != '{}'`

tests/prepare.yml

file removed

-13

		`@@ -1,13 +0,0 @@`
		`- # Preparation steps`
		`- - hosts: localhost`
		`- vars:`
		`- # Must be the same as in str-common/defaults/main.yml`
		`- artifacts: "{{ lookup('env', 'TEST_ARTIFACTS')\|default('./artifacts', true) }}"`
		`- remote_artifacts: /tmp/artifacts/`
		`- tasks:`
		`- - import_tasks: shared-tasks/artifacts.yml`
		`- tags:`
		`- - always`
		`- - import_tasks: shared-tasks/req-pkg.yml`
		`- tags:`
		`- - always`

tests/rhts.yml

file removed

-33

		`@@ -1,33 +0,0 @@`
		`- - import_playbook: prepare.yml`
		`-`
		`- # Tests for rhts role only runs on classic tag`
		`- - hosts: localhost`
		`- tags:`
		`- - classic`
		`- roles:`
		`- - role: standard-test-rhts`
		`- tests:`
		`- - test-rhts-simple`
		`- required_packages:`
		`- # Test if we can install required packages`
		`- - "{{req_pkg}}"`
		`- tasks:`
		`- - import_tasks: shared-tasks/artifacts_test_env.yml`
		`- - import_tasks: shared-tasks/artifacts_test_runner.yml`
		`-`
		`- # Make sure the role behaves correctly if test fails`
		`- - hosts: localhost`
		`- tags:`
		`- - classic`
		`- roles:`
		`- - role: standard-test-rhts`
		`- tags:`
		`- - classic`
		`- tests:`
		`- - test-rhts-fail`
		`- ignore_errors: yes`
		`- tasks:`
		`- # 'verify_failed_test' tasks should run after 'testi-rhts-fail'`
		`- - import_tasks: shared-tasks/verify_failed_test.yml`
		`- - import_tasks: shared-tasks/artifacts_test_env.yml`
		`- - import_tasks: shared-tasks/artifacts_test_runner.yml`

tests/shared-tasks/artifacts.yml

file removed

-9

		`@@ -1,9 +0,0 @@`
		`- # STR requires that artifacts dir must be absent`
		`- - name: "Check if {{artifacts}} is absent at test-environment."`
		`- stat:`
		`- path: "{{remote_artifacts}}"`
		`- register: st_te`
		`-`
		`- - fail:`
		`- msg: "Remove {{remote_artifacts}} before test run at test-environment."`
		`- when: st_te.stat.exists`

tests/shared-tasks/artifacts_test_env.yml

file removed

-9

		`@@ -1,9 +0,0 @@`
		`- # These tasks should run after as test that fails`
		`- - name: "Check if {{remote_artifacts}} was created properly on test environment"`
		`- shell: "ls {{remote_artifacts}}/test.log"`
		`-`
		`- - name: "Clean up {{remote_artifacts}} to not affect other tests on test environment"`
		`- file:`
		`- state: absent`
		`- path: "{{remote_artifacts}}/"`
		`-`

tests/shared-tasks/artifacts_test_runner.yml

file removed

-10

		`@@ -1,10 +0,0 @@`
		`- # These tasks should run after as test that fails`
		`- - name: "Check if {{artifacts}} was created properly on test runner"`
		`- shell: "ls {{artifacts}}/test.log"`
		`- delegate_to: localhost`
		`-`
		`- - name: "Clean up {{artifacts}} to not affect other tests on test runner"`
		`- file:`
		`- state: absent`
		`- path: "{{artifacts}}/"`
		`- delegate_to: localhost`

tests/shared-tasks/req-pkg.yml

file removed

-12

		`@@ -1,12 +0,0 @@`
		`- - import_role:`
		`- name: str-common-pkgs`
		`- tasks_from: inspect.yml`
		`-`
		`- - set_fact:`
		`- req_pkg: zsh`
		`-`
		`- - name: Remove a test package from test-environment`
		`- package:`
		`- name: "{{req_pkg}}"`
		`- state: absent`
		`- when: not is_atomic`

tests/shared-tasks/verify_failed_test.yml

file removed

-11

		`@@ -1,11 +0,0 @@`
		`- # These tasks should run after as test that fails`
		`- - name: Read test.log on test environment`
		`- shell: "cat {{remote_artifacts}}/test.log"`
		`- register: test_log`
		`-`
		`- - name: Check for FAIL on test.log on test environment`
		`- fail: msg="Could not find FAIL on test log"`
		`- when: test_log.stdout.find("FAIL") == -1`
		`- # We should fail when we can not find FAIL string on log`
		`- # -1 means string not found`
		`-`

tests/test-basic-fail/runtest.sh

file removed

-5

		`@@ -1,5 +0,0 @@`
		`- #!/bin/bash`
		`- echo "Running a command that should fail"`
		`-`
		`- false`
		`-`

tests/test-basic-simple/runtest.sh

file removed

-5

		`@@ -1,5 +0,0 @@`
		`- #!/bin/bash`
		`- PACKAGE="bash"`
		`-`
		`- echo "Using $PACKAGE version:"`
		`- rpm -q $PACKAGE`

tests/test-beakerlib-fail/runtest.sh

file removed

-14

		`@@ -1,14 +0,0 @@`
		`- #!/bin/bash`
		`-`
		`- # Include Beaker environment`
		`- . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`-`
		`- PACKAGE="bash"`
		`-`
		`- rlJournalStart`
		`- rlPhaseStartTest "Simple beakerlib test"`
		`- # Running a command that should fail`
		`- rlRun "false"`
		`- rlPhaseEnd`
		`- rlJournalPrintText`
		`- rlJournalEnd`

tests/test-beakerlib-simple/runtest.sh

file removed

-12

		`@@ -1,12 +0,0 @@`
		`- #!/bin/bash`
		`- # Include Beaker environment`
		`- . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`-`
		`- PACKAGE="bash"`
		`-`
		`- rlJournalStart`
		`- rlPhaseStartTest "Simple beakerlib test"`
		`- rlRun "ls /"`
		`- rlPhaseEnd`
		`- rlJournalPrintText`
		`- rlJournalEnd`

tests/test-rhts-fail/Makefile

file removed

-34

		`@@ -1,34 +0,0 @@`
		`- export TESTVERSION=1.0`
		`-`
		`- BUILT_FILES=`
		`-`
		`- FILES=$(METADATA) runtest.sh Makefile`
		`-`
		`- .PHONY: all install clean`
		`-`
		`- run: $(FILES) build`
		`- ./runtest.sh`
		`-`
		`- build: $(BUILT_FILES)`
		`- test -x runtest.sh \|\| chmod a+x runtest.sh`
		`-`
		`- clean:`
		`- rm -f *~ $(BUILT_FILES)`
		`-`
		`-`
		`- include /usr/share/rhts/lib/rhts-make.include`
		`-`
		`- $(METADATA): Makefile`
		`- @echo "Owner: Bruno Goncalves <bgoncalv@redhat.com>" > $(METADATA)`
		`- @echo "Name: $(TEST)" >> $(METADATA)`
		`- @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`- @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`- @echo "Description: fail rhts test role" >> $(METADATA)`
		`- @echo "Type: Sanity" >> $(METADATA)`
		`- @echo "TestTime: 10m" >> $(METADATA)`
		`- @echo "Priority: Normal" >> $(METADATA)`
		`- @echo "License: GPLv2" >> $(METADATA)`
		`- @echo "Confidential: no" >> $(METADATA)`
		`- @echo "Destructive: no" >> $(METADATA)`
		`-`
		`- rhts-lint $(METADATA)`

tests/test-rhts-fail/runtest.sh

file removed

-14

		`@@ -1,14 +0,0 @@`
		`- #!/bin/bash`
		`-`
		`- # Include Beaker environment`
		`- . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`-`
		`- PACKAGE="bash"`
		`-`
		`- rlJournalStart`
		`- rlPhaseStartTest "Simple beakerlib test"`
		`- # Running a command that should fail`
		`- rlRun "false"`
		`- rlPhaseEnd`
		`- rlJournalPrintText`
		`- rlJournalEnd`

tests/test-rhts-simple/Makefile

file removed

-34

		`@@ -1,34 +0,0 @@`
		`- export TESTVERSION=1.0`
		`-`
		`- BUILT_FILES=`
		`-`
		`- FILES=$(METADATA) runtest.sh Makefile`
		`-`
		`- .PHONY: all install clean`
		`-`
		`- run: $(FILES) build`
		`- ./runtest.sh`
		`-`
		`- build: $(BUILT_FILES)`
		`- test -x runtest.sh \|\| chmod a+x runtest.sh`
		`-`
		`- clean:`
		`- rm -f *~ $(BUILT_FILES)`
		`-`
		`-`
		`- include /usr/share/rhts/lib/rhts-make.include`
		`-`
		`- $(METADATA): Makefile`
		`- @echo "Owner: Bruno Goncalves <bgoncalv@redhat.com>" > $(METADATA)`
		`- @echo "Name: $(TEST)" >> $(METADATA)`
		`- @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`- @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`- @echo "Description: simple rhts test role" >> $(METADATA)`
		`- @echo "Type: Sanity" >> $(METADATA)`
		`- @echo "TestTime: 10m" >> $(METADATA)`
		`- @echo "Priority: Normal" >> $(METADATA)`
		`- @echo "License: GPLv2" >> $(METADATA)`
		`- @echo "Confidential: no" >> $(METADATA)`
		`- @echo "Destructive: no" >> $(METADATA)`
		`-`
		`- rhts-lint $(METADATA)`

tests/test-rhts-simple/runtest.sh

file removed

-12

		`@@ -1,12 +0,0 @@`
		`- #!/bin/bash`
		`- # Include Beaker environment`
		`- . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`-`
		`- PACKAGE="bash"`
		`-`
		`- rlJournalStart`
		`- rlPhaseStartTest "Simple beakerlib test"`
		`- rlRun "ls /"`
		`- rlPhaseEnd`
		`- rlJournalPrintText`
		`- rlJournalEnd`

tests/tests.yml

file modified

+18 -5

		`@@ -1,5 +1,18 @@`
		`- ---`
		`- - import_playbook: inventory.yml`
		`- - import_playbook: basic.yml`
		`- - import_playbook: beakerlib.yml`
		`- - import_playbook: avocado.yml`
		`+ - hosts: localhost`
		`+ roles:`
		`+ - role: standard-test-beakerlib`
		`+ tags:`
		`+ - classic`
		`+ repositories:`
		`+ - repo: "https://pagure.io/standard-test-roles.git"`
		`+ dest: "str"`
		`+ version: "2.17"`
		`+ tests:`
		`+ - str/tests/test-basic-fail`
		`+ - str/tests/test-basic-simple`
		`+ - str/tests/test-beakerlib-fail`
		`+ - str/tests/test-beakerlib-simple`
		`+ - str/tests/test-rhts-fail`
		`+ - str/tests/test-rhts-simple`
		`+ tasks:`
		`+ - include_tasks: str/tests/inventory-tasks.yml`

stefw commented 5 years ago

Instead of all the tests living in this repository, execute
them from the sourcesc tarball instead.

In order to do this, we must extract the source tarball before
we invoke the playbook. This is done by way of a local inventory
script. That inventory prepares the local host to be executing
the playbooks on the local host.

stefw commented 5 years ago

This depends on https://pagure.io/standard-test-roles/pull-request/274 being merged first.

stefw commented 5 years ago

@astepano I would like to request a new release of upstream standard-test-roles ... so that we can start pulling the tests in (now that they're included upstream).

astepano commented on line 26 of tests/inventory 5 years ago

"help" parameter is a bit strange.
It should be: https://docs.ansible.com/ansible/2.4/ansible-inventory.html#cmdoption-ansible-inventory-list
It should be something like : "Outputs a list of hosts for Ansible inventory"

astepano commented on line 19 of tests/inventory 5 years ago

@stefw, could you please help to understand please.

Suppose we merge this PR to master.
I build a package for F28.
I merge changes from 'master' to f28 branch.
Q: should I change --release=master to --release=f28 after merge?
This will require that CI-Pipeline has fedpkg installed. But, even it has it, STI doesn't make any promises that it will be installed. Not sure that this is good idea to call fedpkg from Ansible inventory script. This is very sophisticated approach and hard to understand logic.
I would propose go with FMF. And use 'shared tests' approach. @psss could you please point to good example?

Thank you!

stefw commented 5 years ago

@stefw, could you please help to understand please. 1. Suppose we merge this PR to master. I build a package for F28. I merge changes from 'master' to f28 branch. Q: should I change --release=master to --release=f28 after merge? 2. This will require that CI-Pipeline has fedpkg installed. But, even it has it, STI doesn't make any promises that it will be installed. Not sure that this is good idea to call fedpkg from Ansible inventory script. This is very sophisticated approach and hard to understand logic. I would propose go with FMF.

Sure thing. Do you have an example of how to do that?

How long do you think it'll take to come up with an example that uses FMF? If more than a day or two, we should get this in now, and then transition to FMF later. Will this take more than a couple hours to implement?

It's pretty urgent that we didn't have integration tests upstream for standard-test-roles.

psss commented 5 years ago

Here are some examples of executing tests from an external repo:

Some more examples can be found here:

https://fedoraproject.org/wiki/CI/Share_Test_Code

The setup is easy: Just include the "repositories" variable in the
playbook and then provide the list of tests for execution manually
or use an FMF filter to discover relevant tests automatically. For
standard-test-roles it could look like this:

repositories:
- repo: "https://pagure.io/standard-test-roles.git"
  dest: "str"
tests:
- str/test-basic-fail
- str/test-basic-simple
- str/test-beakerlib-fail
- str/test-beakerlib-simple
- str/test-rhts-fail
- str/test-rhts-simple

Hope this helps.

astepano commented 5 years ago

@psss thank you very much for the example
@stefw please let know is this approach sounds good for this particular case.

stefw commented 5 years ago

@psss @astepano The above does not yet work for this particular use case. We need to checkout a specific version of the tests that exactly matches the the source code being packaged. How do I specify exactly which tag to checkout ... and get that tag from the spec file version number?

psss commented 5 years ago

I see. The specific version can be specified as well:

- role: standard-test-beakerlib
  repositories:
  - repo: "https://src.fedoraproject.org/tests/shell.git"
    dest: "shell"
    version: "devel"

But this still does not resolve matching commit hash from which the package has been built.

stefw commented 5 years ago

Adding "2.17" to the version seems to work.

@astepano Can you release 2.18 of standard-test-roles and see if we can pull the tests into here?

@psss What I don't understand is how to run non beakerlib tests in standard-test-roles. For example the inventory.yml test. How can we accomplish that part?

psss commented 5 years ago

@psss What I don't understand is how to run non beakerlib tests in standard-test-roles. For example the inventory.yml test. How can we accomplish that part?

For running arbitrary shell scripts you can use the basic role:
https://fedoraproject.org/wiki/CI/Standard_Test_Roles#Basic

tests:
- script:
    dir: path/to/script
    run: ./venv.sh

Just provide dir with path to the directory and run with the script itself.

stefw commented 5 years ago

So does this come after the beakerlib + repository lines?

psss commented 5 years ago

I guess, if you need to use both standard-test-basic and standard-test-beakerlib you will have to combine both roles. But I have no experience how that will exactly work and which role would be applied first. I would expect they are executed in the same order as provided. @astepano will probably know more.