Adds tests according to the CI wiki [0] specifically the standard test interface in the spec [1].

The playbook includes Tier1 level test cases that have been tested in the following contexts and is passing reliably: Docker, and Classic. There are no tests for Atomic Host at this time due to package dependancies. Test logs are stored in the artifacts directory.

The following steps are used to execute the tests using the standard test interface:

Docker
sudo ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) TEST_SUBJECTS=docker:docker.io/library/fedora:26 TEST_ARTIFACTS=$PWD/artifacts ansible-playbook --tags container tests.yml

Classic
sudo ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) TEST_SUBJECTS="" TEST_ARTIFACTS=$PWD/artifacts ansible-playbook --tags classic tests.yml

[0] https://fedoraproject.org/wiki/CI
[1] https://fedoraproject.org/wiki/Changes/InvokingTests

Sample output:

[root@localhost artifacts]# cat test.upstream-testsuite.log
posix on
:: [ 20:21:40 ] :: [ WARNING ] :: POSIX mode detected and switched off
:: [ 20:21:40 ] :: [ WARNING ] :: POSIX mode detected and switched off
:: [ 20:21:40 ] :: [ WARNING ] :: Please fix your test to have /bin/bash shebang
:: [ 20:21:40 ] :: [ WARNING ] :: Please fix your test to have /bin/bash shebang

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [ LOG ] :: Setup
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ BEGIN ] :: Running 'useradd byacctestuser'
:: [ BEGIN ] :: Running 'useradd byacctestuser'
:: [ PASS ] :: Command 'useradd byacctestuser' (Expected 0,9, got 0)
:: [ BEGIN ] :: Running 'chown byacctestuser:byacctestuser /tmp/tmp.YH1vQgtmno'
:: [ BEGIN ] :: Running 'chown byacctestuser:byacctestuser /tmp/tmp.YH1vQgtmno'
:: [ PASS ] :: Command 'chown byacctestuser:byacctestuser /tmp/tmp.YH1vQgtmno' (Expected 0, got 0)
byacc-1.9.20170201-1.fc26.x86_64
:: [ PASS ] :: Checking for the presence of byacc rpm
:: [ 20:21:40 ] :: Package versions:
:: [ 20:21:40 ] :: Package versions:
:: [ 20:21:40 ] :: byacc-1.9.20170201-1.fc26.x86_64
:: [ 20:21:40 ] :: byacc-1.9.20170201-1.fc26.x86_64
gcc-7.2.1-2.fc26.x86_64
:: [ PASS ] :: Checking for the presence of gcc rpm
:: [ 20:21:40 ] :: Package versions:
:: [ 20:21:40 ] :: Package versions:
:: [ 20:21:40 ] :: gcc-7.2.1-2.fc26.x86_64
:: [ 20:21:40 ] :: gcc-7.2.1-2.fc26.x86_64
glibc-devel-2.25-12.fc26.x86_64
:: [ PASS ] :: Checking for the presence of glibc-devel rpm
:: [ 20:21:40 ] :: Package versions:
:: [ 20:21:40 ] :: Package versions:
:: [ 20:21:40 ] :: glibc-devel-2.25-12.fc26.x86_64
:: [ 20:21:40 ] :: glibc-devel-2.25-12.fc26.x86_64
glibc-headers-2.25-12.fc26.x86_64
:: [ PASS ] :: Checking for the presence of glibc-headers rpm
:: [ 20:21:41 ] :: Package versions:
:: [ 20:21:41 ] :: Package versions:
:: [ 20:21:41 ] :: glibc-headers-2.25-12.fc26.x86_64
:: [ 20:21:41 ] :: glibc-headers-2.25-12.fc26.x86_64
make-4.2.1-2.fc26.x86_64
:: [ PASS ] :: Checking for the presence of make rpm
:: [ 20:21:41 ] :: Package versions:
:: [ 20:21:41 ] :: Package versions:
:: [ 20:21:41 ] :: make-4.2.1-2.fc26.x86_64
:: [ 20:21:41 ] :: make-4.2.1-2.fc26.x86_64
sed-4.4-1.fc26.x86_64
:: [ PASS ] :: Checking for the presence of sed rpm
:: [ 20:21:41 ] :: Package versions:
:: [ 20:21:41 ] :: Package versions:
:: [ 20:21:41 ] :: sed-4.4-1.fc26.x86_64
:: [ 20:21:41 ] :: sed-4.4-1.fc26.x86_64
rpm-build-4.13.0.2-1.fc26.x86_64
:: [ PASS ] :: Checking for the presence of rpm-build rpm
:: [ 20:21:41 ] :: Package versions:
:: [ 20:21:41 ] :: Package versions:
:: [ 20:21:41 ] :: rpm-build-4.13.0.2-1.fc26.x86_64
:: [ 20:21:41 ] :: rpm-build-4.13.0.2-1.fc26.x86_64
:: [ BEGIN ] :: Running 'TMP=$(mktemp -d)'
:: [ BEGIN ] :: Running 'TMP=$(mktemp -d)'
:: [ PASS ] :: Command 'TMP=$(mktemp -d)' (Expected 0, got 0)
:: [ BEGIN ] :: Running 'chmod a+rx /tmp/tmp.J7uat6VrxO'
:: [ BEGIN ] :: Running 'chmod a+rx /tmp/tmp.J7uat6VrxO'
:: [ PASS ] :: Command 'chmod a+rx /tmp/tmp.J7uat6VrxO' (Expected 0, got 0)
:: [ BEGIN ] :: Running 'pushd /tmp/tmp.J7uat6VrxO'
:: [ BEGIN ] :: Running 'pushd /tmp/tmp.J7uat6VrxO'
/tmp/tmp.J7uat6VrxO /usr/local/bin/upstream-testsuite
:: [ PASS ] :: Command 'pushd /tmp/tmp.J7uat6VrxO' (Expected 0, got 0)

Yum-utils package has been deprecated, use dnf instead.
See 'man yum2dnf' for more information.

Enabling updates-source repository
Enabling fedora-source repository
http://download-i2.fedoraproject.org/pub/fedora/linux/updates/26/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates
Trying other mirror.
:: [ BEGIN ] :: Running 'chmod a+r byacc.src.rpm'
:: [ BEGIN ] :: Running 'chmod a+r byacc.src.rpm'
:: [ PASS ] :: Command 'chmod a+r byacc*.src.rpm' (Expected 0, got 0)

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [ LOG ] :: Test
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ BEGIN ] :: Running 'su - byacctestuser -c 'bash /tmp/tmp.YH1vQgtmno /tmp/tmp.J7uat6VrxO''
:: [ BEGIN ] :: Running 'su - byacctestuser -c 'bash /tmp/tmp.YH1vQgtmno /tmp/tmp.J7uat6VrxO''
+ _BASEDIR=/tmp/tmp.J7uat6VrxO
++ mktemp -d
+ _TMP=/tmp/tmp.ThCTS97GS9
+ rpm '--define=_topdir /tmp/tmp.ThCTS97GS9' -Uvh /tmp/tmp.J7uat6VrxO/byacc-1.9.20170201-1.fc26.src.rpm
Updating / installing...
byacc-1.9.20170201-1.fc26 warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root

warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
+ rpmbuild '--define=_topdir /tmp/tmp.ThCTS97GS9' -bc /tmp/tmp.ThCTS97GS9/SPECS/byacc.spec
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.woGdVv
+ umask 022
+ cd /tmp/tmp.ThCTS97GS9/BUILD
+ cd /tmp/tmp.ThCTS97GS9/BUILD
+ rm -rf byacc-20170201
+ /usr/bin/gzip -dc /tmp/tmp.ThCTS97GS9/SOURCES/byacc-20170201.tgz
+ /usr/bin/tar -xof -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd byacc-20170201
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ find . -type f -name '.c' -print0
+ xargs -0 sed -i 's/YYSTACKSIZE 500/YYSTACKSIZE 10000/g'
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.xv6XXH
+ umask 022
+ cd /tmp/tmp.ThCTS97GS9/BUILD
<snip build output>
+ exit 0
+ pushd /tmp/tmp.ThCTS97GS9/BUILD/byacc-20170201
/tmp/tmp.ThCTS97GS9/BUILD/byacc-20170201 ~
+ make check
/bin/sh ./test/run_test.sh ./test
Mon Jan 8 20:22:14 UTC 2018
testing flags help -z
...ok ./test/yacc/help.output
...ok ./test/yacc/help.error
testing flags big_b -B
...ok ./test/yacc/big_b.output
...ok ./test/yacc/big_b.error
testing flags big_l -L
...ok ./test/yacc/big_l.output
...ok ./test/yacc/big_l.error
testing flags nostdin - nosuchfile.y
...ok ./test/yacc/nostdin.output
...ok ./test/yacc/nostdin.error
testing flags no_opts -- nosuchfile.y
...ok ./test/yacc/no_opts.output
...ok ./test/yacc/no_opts.error
testing flags no_b_opt -b
...ok ./test/yacc/no_b_opt.output
...ok ./test/yacc/no_b_opt.error
testing flags no_b_opt1 -bBASE -o nosuchfile.c nosuchfile.y
...ok ./test/yacc/no_b_opt1.output
...ok ./test/yacc/no_b_opt1.error
testing flags no_p_opt -p
...ok ./test/yacc/no_p_opt.output
...ok ./test/yacc/no_p_opt.error
testing flags no_p_opt1 -pBASE -o nosuchfile.c nosuchfile.y
...ok ./test/yacc/no_p_opt1.output
...ok ./test/yacc/no_p_opt1.error
testing flags no_graph -g -o nosuchfile.c nosuchfile.y
...ok ./test/yacc/no_graph.output
...ok ./test/yacc/no_graph.error
testing flags no_verbose -v -o nosuchfile.c nosuchfile.y
...ok ./test/yacc/no_verbose.output
...ok ./test/yacc/no_verbose.error
testing flags no_output -o nosuchfile.output nosuchfile.y
...ok ./test/yacc/no_output.output
...ok ./test/yacc/no_output.error
testing flags no_output1 -onosuchfile.output nosuchfile.y
...ok ./test/yacc/no_output1.output
...ok ./test/yacc/no_output1.error
testing flags no_output2 -o
...ok ./test/yacc/no_output2.output
...ok ./test/yacc/no_output2.error
testing flags no_defines -d -o nosuchfile.c nosuchfile.y
...ok ./test/yacc/no_defines.output
...ok ./test/yacc/no_defines.error
testing flags no_include -i -o nosuchfile.c nosuchfile.y
...ok ./test/yacc/no_include.output
...ok ./test/yacc/no_include.error
testing flags no_code_c -r -o nosuchfile.c nosuchfile.y
...ok ./test/yacc/no_code_c.output
...ok ./test/yacc/no_code_c.error
testing ./test/calc.y
...ok ./test/yacc/calc.error
...ok ./test/yacc/calc.output
...ok ./test/yacc/calc.tab.c
...ok ./test/yacc/calc.tab.h
testing ./test/calc1.y
...ok ./test/yacc/calc1.error
...ok ./test/yacc/calc1.output
...ok ./test/yacc/calc1.tab.c
...ok ./test/yacc/calc1.tab.h
testing ./test/calc2.y
...ok ./test/yacc/calc2.error
...ok ./test/yacc/calc2.output
...ok ./test/yacc/calc2.tab.c
...ok ./test/yacc/calc2.tab.h
testing ./test/calc3.y
...ok ./test/yacc/calc3.error
...ok ./test/yacc/calc3.output
...ok ./test/yacc/calc3.tab.c
...ok ./test/yacc/calc3.tab.h
testing ./test/code_calc.y
...ok ./test/yacc/code_calc.error
...ok ./test/yacc/code_calc.output
...ok ./test/yacc/code_calc.tab.c
...ok ./test/yacc/code_calc.tab.h
...ok ./test/yacc/code_calc.code.c
testing ./test/code_debug.y
...ok ./test/yacc/rename_debug.error
...ok ./test/yacc/rename_debug.output
...ok ./test/yacc/rename_debug.c
...ok ./test/yacc/rename_debug.h
...ok ./test/yacc/rename_debug.i
testing ./test/code_error.y
...ok ./test/yacc/code_error.error
...ok ./test/yacc/code_error.output
...ok ./test/yacc/code_error.tab.c
...ok ./test/yacc/code_error.tab.h
...ok ./test/yacc/code_error.code.c
testing ./test/empty.y
...ok ./test/yacc/empty.error
...ok ./test/yacc/empty.output
...ok ./test/yacc/empty.tab.c
...ok ./test/yacc/empty.tab.h
testing ./test/err_syntax1.y
...ok ./test/yacc/err_syntax1.error
...ok ./test/yacc/err_syntax1.output
...ok ./test/yacc/err_syntax1.tab.c
...ok ./test/yacc/err_syntax1.tab.h
testing ./test/err_syntax10.y
...ok ./test/yacc/err_syntax10.error
...ok ./test/yacc/err_syntax10.output
...ok ./test/yacc/err_syntax10.tab.c
...ok ./test/yacc/err_syntax10.tab.h
testing ./test/err_syntax11.y
...ok ./test/yacc/err_syntax11.error
...ok ./test/yacc/err_syntax11.output
...ok ./test/yacc/err_syntax11.tab.c
...ok ./test/yacc/err_syntax11.tab.h
testing ./test/err_syntax12.y
...ok ./test/yacc/err_syntax12.error
...ok ./test/yacc/err_syntax12.output
...ok ./test/yacc/err_syntax12.tab.c
...ok ./test/yacc/err_syntax12.tab.h
testing ./test/err_syntax13.y
...ok ./test/yacc/err_syntax13.error
...ok ./test/yacc/err_syntax13.output
...ok ./test/yacc/err_syntax13.tab.c
...ok ./test/yacc/err_syntax13.tab.h
testing ./test/err_syntax14.y
...ok ./test/yacc/err_syntax14.error
...ok ./test/yacc/err_syntax14.output
...ok ./test/yacc/err_syntax14.tab.c
...ok ./test/yacc/err_syntax14.tab.h
testing ./test/err_syntax15.y
...ok ./test/yacc/err_syntax15.error
...ok ./test/yacc/err_syntax15.output
...ok ./test/yacc/err_syntax15.tab.c
...ok ./test/yacc/err_syntax15.tab.h
testing ./test/err_syntax16.y
...ok ./test/yacc/err_syntax16.error
...ok ./test/yacc/err_syntax16.output
...ok ./test/yacc/err_syntax16.tab.c
...ok ./test/yacc/err_syntax16.tab.h
testing ./test/err_syntax17.y
...ok ./test/yacc/err_syntax17.error
...ok ./test/yacc/err_syntax17.output
...ok ./test/yacc/err_syntax17.tab.c
...ok ./test/yacc/err_syntax17.tab.h
testing ./test/err_syntax18.y
...ok ./test/yacc/err_syntax18.error
...ok ./test/yacc/err_syntax18.output
...ok ./test/yacc/err_syntax18.tab.c
...ok ./test/yacc/err_syntax18.tab.h
testing ./test/err_syntax19.y
...ok ./test/yacc/err_syntax19.error
...ok ./test/yacc/err_syntax19.output
...ok ./test/yacc/err_syntax19.tab.c
...ok ./test/yacc/err_syntax19.tab.h
testing ./test/err_syntax2.y
...ok ./test/yacc/err_syntax2.error
...ok ./test/yacc/err_syntax2.output
...ok ./test/yacc/err_syntax2.tab.c
...ok ./test/yacc/err_syntax2.tab.h
testing ./test/err_syntax20.y
...ok ./test/yacc/err_syntax20.error
...ok ./test/yacc/err_syntax20.output
...ok ./test/yacc/err_syntax20.tab.c
...ok ./test/yacc/err_syntax20.tab.h
testing ./test/err_syntax21.y
...ok ./test/yacc/err_syntax21.error
...ok ./test/yacc/err_syntax21.output
...ok ./test/yacc/err_syntax21.tab.c
...ok ./test/yacc/err_syntax21.tab.h
testing ./test/err_syntax22.y
...ok ./test/yacc/err_syntax22.error
...ok ./test/yacc/err_syntax22.output
...ok ./test/yacc/err_syntax22.tab.c
...ok ./test/yacc/err_syntax22.tab.h
testing ./test/err_syntax23.y
...ok ./test/yacc/err_syntax23.error
...ok ./test/yacc/err_syntax23.output
...ok ./test/yacc/err_syntax23.tab.c
...ok ./test/yacc/err_syntax23.tab.h
testing ./test/err_syntax24.y
...ok ./test/yacc/err_syntax24.error
...ok ./test/yacc/err_syntax24.output
...ok ./test/yacc/err_syntax24.tab.c
...ok ./test/yacc/err_syntax24.tab.h
testing ./test/err_syntax25.y
...ok ./test/yacc/err_syntax25.error
...ok ./test/yacc/err_syntax25.output
...ok ./test/yacc/err_syntax25.tab.c
...ok ./test/yacc/err_syntax25.tab.h
testing ./test/err_syntax26.y
...ok ./test/yacc/err_syntax26.error
...ok ./test/yacc/err_syntax26.output
...ok ./test/yacc/err_syntax26.tab.c
...ok ./test/yacc/err_syntax26.tab.h
testing ./test/err_syntax27.y
...ok ./test/yacc/err_syntax27.error
...ok ./test/yacc/err_syntax27.output
...ok ./test/yacc/err_syntax27.tab.c
...ok ./test/yacc/err_syntax27.tab.h
testing ./test/err_syntax3.y
...ok ./test/yacc/err_syntax3.error
...ok ./test/yacc/err_syntax3.output
...ok ./test/yacc/err_syntax3.tab.c
...ok ./test/yacc/err_syntax3.tab.h
testing ./test/err_syntax4.y
...ok ./test/yacc/err_syntax4.error
...ok ./test/yacc/err_syntax4.output
...ok ./test/yacc/err_syntax4.tab.c
...ok ./test/yacc/err_syntax4.tab.h
testing ./test/err_syntax5.y
...ok ./test/yacc/err_syntax5.error
...ok ./test/yacc/err_syntax5.output
...ok ./test/yacc/err_syntax5.tab.c
...ok ./test/yacc/err_syntax5.tab.h
testing ./test/err_syntax6.y
...ok ./test/yacc/err_syntax6.error
...ok ./test/yacc/err_syntax6.output
...ok ./test/yacc/err_syntax6.tab.c
...ok ./test/yacc/err_syntax6.tab.h
testing ./test/err_syntax7.y
...ok ./test/yacc/err_syntax7.error
...ok ./test/yacc/err_syntax7.output
...ok ./test/yacc/err_syntax7.tab.c
...ok ./test/yacc/err_syntax7.tab.h
testing ./test/err_syntax7a.y
...ok ./test/yacc/err_syntax7a.error
...ok ./test/yacc/err_syntax7a.output
...ok ./test/yacc/err_syntax7a.tab.c
...ok ./test/yacc/err_syntax7a.tab.h
testing ./test/err_syntax7b.y
...ok ./test/yacc/err_syntax7b.error
...ok ./test/yacc/err_syntax7b.output
...ok ./test/yacc/err_syntax7b.tab.c
...ok ./test/yacc/err_syntax7b.tab.h
testing ./test/err_syntax8.y
...ok ./test/yacc/err_syntax8.error
...ok ./test/yacc/err_syntax8.output
...ok ./test/yacc/err_syntax8.tab.c
...ok ./test/yacc/err_syntax8.tab.h
testing ./test/err_syntax8a.y
...ok ./test/yacc/err_syntax8a.error
...ok ./test/yacc/err_syntax8a.output
...ok ./test/yacc/err_syntax8a.tab.c
...ok ./test/yacc/err_syntax8a.tab.h
testing ./test/err_syntax9.y
...ok ./test/yacc/err_syntax9.error
...ok ./test/yacc/err_syntax9.output
...ok ./test/yacc/err_syntax9.tab.c
...ok ./test/yacc/err_syntax9.tab.h
testing ./test/error.y
...ok ./test/yacc/error.error
...ok ./test/yacc/error.output
...ok ./test/yacc/error.tab.c
...ok ./test/yacc/error.tab.h
testing ./test/expr.oxout.y
...ok ./test/yacc/expr.oxout.error
...ok ./test/yacc/expr.oxout.output
...ok ./test/yacc/expr.oxout.tab.c
...ok ./test/yacc/expr.oxout.tab.h
testing ./test/grammar.y
...ok ./test/yacc/grammar.error
...ok ./test/yacc/grammar.output
...ok ./test/yacc/grammar.tab.c
...ok ./test/yacc/grammar.tab.h
...ok ./test/yacc/grammar.dot
testing ./test/ok_syntax1.y
...ok ./test/yacc/ok_syntax1.error
...ok ./test/yacc/ok_syntax1.output
...ok ./test/yacc/ok_syntax1.tab.c
...ok ./test/yacc/ok_syntax1.tab.h
testing ./test/pure_calc.y
...ok ./test/yacc/pure_calc.error
...ok ./test/yacc/pure_calc.output
...ok ./test/yacc/pure_calc.tab.c
...ok ./test/yacc/pure_calc.tab.h
testing ./test/pure_error.y
...ok ./test/yacc/pure_error.error
...ok ./test/yacc/pure_error.output
...ok ./test/yacc/pure_error.tab.c
...ok ./test/yacc/pure_error.tab.h
testing ./test/quote_calc.y
...ok ./test/yacc/quote_calc.error
...ok ./test/yacc/quote_calc.output
...ok ./test/yacc/quote_calc.tab.c
...ok ./test/yacc/quote_calc.tab.h
...ok ./test/yacc/quote_calc-s.error
...ok ./test/yacc/quote_calc-s.output
...ok ./test/yacc/quote_calc-s.tab.c
...ok ./test/yacc/quote_calc-s.tab.h
testing ./test/quote_calc2.y
...ok ./test/yacc/quote_calc2.error
...ok ./test/yacc/quote_calc2.output
...ok ./test/yacc/quote_calc2.tab.c
...ok ./test/yacc/quote_calc2.tab.h
...ok ./test/yacc/quote_calc2-s.error
...ok ./test/yacc/quote_calc2-s.output
...ok ./test/yacc/quote_calc2-s.tab.c
...ok ./test/yacc/quote_calc2-s.tab.h
testing ./test/quote_calc3.y
...ok ./test/yacc/quote_calc3.error
...ok ./test/yacc/quote_calc3.output
...ok ./test/yacc/quote_calc3.tab.c
...ok ./test/yacc/quote_calc3.tab.h
...ok ./test/yacc/quote_calc3-s.error
...ok ./test/yacc/quote_calc3-s.output
...ok ./test/yacc/quote_calc3-s.tab.c
...ok ./test/yacc/quote_calc3-s.tab.h
testing ./test/quote_calc4.y
...ok ./test/yacc/quote_calc4.error
...ok ./test/yacc/quote_calc4.output
...ok ./test/yacc/quote_calc4.tab.c
...ok ./test/yacc/quote_calc4.tab.h
...ok ./test/yacc/quote_calc4-s.error
...ok ./test/yacc/quote_calc4-s.output
...ok ./test/yacc/quote_calc4-s.tab.c
...ok ./test/yacc/quote_calc4-s.tab.h
* testing ./test/varsyntax_calc1.y
...ok ./test/yacc/varsyntax_calc1.error
...ok ./test/yacc/varsyntax_calc1.output
...ok ./test/yacc/varsyntax_calc1.tab.c
...ok ./test/yacc/varsyntax_calc1.tab.h
+ popd
~
+ set -xe
:: [ PASS ] :: Command 'su - byacctestuser -c 'bash /tmp/tmp.YH1vQgtmno /tmp/tmp.J7uat6VrxO'' (Expected 0, got 0)