Tempest¶

Tempest Version release wise:¶

• Ocata : 16.1.0
• Pike : 17.2.0
• Queens : 18.0.0
• Master : master

Required resources before running Tempest¶

The following resources are needed to be created, only if Tempest is run manually.

• If Tempest is run against undercloud, then source the stackrc file:

  $ source stackrc
  
  $ export OS_AUTH_URL="$OS_AUTH_URL/v$OS_IDENTITY_API_VERSION"
  

• If Tempest is run against overcloud, then source the overcloudrc file:

  $ source overcloudrc
  

• Create Member role for undercloud/overcloud, it will be used by tempest tests:

  $ openstack role create --or-show Member
  

• Create a public network having external connectivity, will be used by tempest tests when running tempest tests against overcloud

  • Create a public network:

    $ openstack network create public --router:external=True \
                              --provider:network_type flat \
                    --provider:physical_network datacenter
    

  • Create/Attach subnet to it:

    $ openstack subnet create ext-subnet \
                    --allocation-pool \
      start=192.168.24.150,end=192.168.24.250 \
                      --no-dhcp \
                      --gateway 192.168.24.1 \
                      public 192.168.24.0/24
    

  • Export public network id:

    $ public_net_id=$(openstack network show {{ public_net_name }} -f value -c id)
    

Installing Tempest RPM and its plugins¶

Install openstack-tempest:

$ sudo yum -y install openstack-tempest

Install tempest plugins

• Find out what are the openstack services configured on overcloud/undercloud.
• Then install the respective plugins on undercloud using yum command.

Getting the list of tempest rpms and tempest plugins installed on undercloud:

$ rpm -qa | grep tempest

Tempest workspace¶

Create a tempest workspace:

$ tempest init tempest_workspace

tempest_workspace directory will be created automatically in the location where the above command is executed. It will create three folders within tempest_workspace directory.

• etc - tempest configuration file tempest.conf will resides here.
• logs - tempest.log file will be here
• tempest_lock - It holds the lock for tempest workspace.
• .stestr.conf - It is used to load all the tempest tests.

List tempest workspaces:

$ tempest workspace list

The tempest workspace information is found in ~/.tempest folder.

Generating tempest.conf using discover-tempest-config¶

For running Tempest a tempest configuration file called tempest.conf needs to be created. Thanks to that file Tempest knows the configuration of the environment it will be run against and can execute the proper set of tests.

The tempest configuration file can be generated automatically by discover-tempest-config binary, which is provided by python-tempestconf package installed by openstack-tempest rpm. discover-tempest-config queries the cloud and discovers cloud configuration.

All the below operations will be performed from undercloud.

$ source stackrc

$ cd <path to tempest workspace>

$ discover-tempest-config --out etc/tempest.conf \
  --image <path to cirros image> \
  --debug \
  --create \
  auth.use_dynamic_credentials true \
  auth.tempest_roles Member \
  network-feature-enabled.port_security true \
  compute-feature-enabled.attach_encrypted_volume False \
  validation.image_ssh_user cirros \
  validation.ssh_user cirros \
  compute-feature-enabled.console_output true

$ source overcloudrc

$ discover-tempest-config --out etc/tempest.conf \
  --deployer-input ~/tempest-deployer-input.conf \
  --network-id $public_net_id \
  --image <path/url to cirros image to use> \
  --debug \
  --remove network-feature-enabled.api_extensions=dvr \
  --create \
  auth.use_dynamic_credentials true \
  auth.tempest_roles Member \
  network-feature-enabled.port_security true \
  compute-feature-enabled.attach_encrypted_volume False \
  network.tenant_network_cidr 192.168.0.0/24 \
  compute.build_timeout 500 \
  volume-feature-enabled.api_v1 False \
  validation.image_ssh_user cirros \
  validation.ssh_user cirros \
  network.build_timeout 500 \
  volume.build_timeout 500 \
  object-storage-feature-enabled.discoverability False \
  service_available.swift False \
  compute-feature-enabled.console_output true \
  orchestration.stack_owner_role Member

Always save the state of resources before running tempest tests¶

In order to be able to use tempest utility to clean up resources after running tests, it’s needed to initialize the state of resources before running the tests:

$ tempest cleanup --init-saved-state

It will create saved_state.json file in tempest workspace containing all the tenants and resources information present on the system under test. More about the feature can be found in Tempest documentation <https://docs.openstack.org/tempest/latest/cleanup.html>

List tempest plugins installed on undercloud¶

Since we install the required tempest plugins on undercloud, use tempest command to find out:

$ tempest list-plugins

List tempest tests¶

Go to tempest workspace and run the following command to get the list:

$ cd <path to tempest workspace>
$ tempest run -l

To grep a list of specific tests like all compute tests:

$ tempest run -l | grep compute

Running Tempest tests¶

tempest run utility is used to run tempest tests. It will use the configs defined in tempest.conf to run tests against the targeted host.

• For running all api/scenario tempest tests:

  $ tempest run -r '(api|scenario)'
  

• For running smoke tests for basic sanity of the deployed cloud:

  $ tempest run --smoke
  

• For running specific tempest plugin tests like: keystone_tempest_plugin tests:

  $ tempest run --regex '(keystone_tempest_plugin)'
  

• Running multiple tests:

  $ tempest run --regex '(test_regex1 | test_regex2 | test_regex 3)'
  

• Use --black-regex argument to skip specific tests:

  $ tempest run -r '(api|scenario)' --black-regex='(keystone_tempest_plugin)'
  

  The above will skip all keystone_tempest_plugin tests.

Using whitelist file for running selective tests¶

Writing long test regex seems to be boring, let’s create a simple whitelist file and use the same with tempest run to run those specific whitelist tests.

• Create a whitelist.txt file in tempest workspace:

  $ touch whitelist.txt
  

• Append all the all tests in a newline which we want to run in whitelist.txt file:

  $ cat whitelist.txt
    keystone_tempest_plugin.*
    # networking bgpvpn tempest tests
    networking_bgpvpn_tempest.tests*
  

  Note

  Use # to add comments in the whitelist/blacklist file.

• Running tempest tests present in whitelist file:

  $ tempest run -w <path to whitelist file>
  

Using blacklist file to skipping multiple tests¶

If we want to skip multiple tests, we can blacklist file for the same.

• Create a skip_test.txt file in tempest workspace:

  $ touch skip_test.txt
  

• Append all the all tests in a newline which we want to skip in skip_test.txt file:

  $ cat whitelist.txt
    keystone_tempest_plugin.*
    # networking bgpvpn tempest tests
    networking_bgpvpn_tempest.tests*
  

• Use -b optuon with tempest run to skip/blacklist tests:

  $ tempest run -w <path to whitelist_file> -b <path to skip tests>
  

Running Tempest tests serially as well as in parallel¶

• All test methods within a TestCase are assumed to be executed serially.

• To run tempest tests serially:

  $ tempest run --serial
  

• Run the tests in parallel (this is the default):

  $ tempest run --parallel
  

• Specify the number of workers to use when running tests in parallel:

  $ tempest run -r '(test_regex)' --concurrency <numbers of workers>
  

• The default number of workers is equal to the number of CPUs on the system under test.

Generating HTML report of tempest tests¶

• In order to generate tempest subunit files in v2 format, use --subunit flag with tempest run:

  $ tempest run -r '(test_regex)' --subunit
  

• Generating html output from it:

  $ subunit2html .stestr/<run number file> tempest.html
  

• subunit2html command is provided by python-subunit rpm package.

Where are my tempest tests results?¶

Once tempest run finishes, All the tests results are stored in subunit file format under .stestr folder under tempest workspace.

• 0,1,<list of tempest run> files contains the tempest run output.
• failing contains the list of failed tests with detailed api responses.
• All the tests executions api responses is logged in tempest.log file in tempest workspace.

Status of Tempest tests after tempest run¶

After the execution of tempest tests, It will generate 3 status

• PASSED: The test successfully run.
• FAILED: The test got failed due to specific reasons.
• SKIPPED: If a tempest tests is skipped, it will give a reason why it is skipped.

Cleaning up environment after tempest run¶

More about this feature can be found in Tempest documentation <https://docs.openstack.org/tempest/latest/cleanup.html>

• Get a report of resources and tenants which got created/modified after tempest tests run:

  $ tempest cleanup --dry-run
  

  It will create a dry_run.json file in tempest workspace.

• Cleaning up the environment:

  $ tempest cleanup
  

• We can force delete the tempest resources and as well as associated admin tenants:

  $ tempest cleanup --delete-tempest-conf-object
  

Prepare the tempest container¶

• Change to /home/stack directory:

  $ cd /home/stack
  

• Download a container:

  $ docker pull docker.io/tripleomaster/centos-binary-tempest:current-tripleo-rdo
  

• Create directories which will be used for exchanging data between the host machine and the container:

  $ mkdir container_tempest tempest_workspace
  

• We’ll use container_tempest as a source of files for the container, so let’s copy there all needed files:

  $ cp stackrc overcloudrc tempest-deployer-input.conf container_tempest
  

• List available images:

  $ docker image list
  

  or:

  $ docker images
  

  you should see something like:

  REPOSITORY                                      TAG                     IMAGE ID            CREATED             SIZE
  docker.io/tripleomaster/centos-binary-tempest   current-tripleo-rdo     881f7ac24d8f        10 days ago         1.09 GB
  

How to execute commands within the container?¶

In order to make it easier, create an alias as follows:

$ alias docker-tempest="docker run -i \
    -v "$(pwd)"/container_tempest:/home/stack/container_tempest \
    -v "$(pwd)"/tempest_workspace:/home/stack/tempest_workspace \
    docker.io/tripleomaster/centos-binary-tempest:current-tripleo-rdo \
    /bin/bash"

When mounting the directories, make sure that absolute paths are used.

• If you want to check available tempest plugins in the container, run:

  $ docker-tempest -c "tempest list-plugins"
  

• For getting a list of tempest related rpms installed within the tempest container run:

  $ docker-tempest -c "rpm -qa | grep tempest"
  

Generate tempest.conf and run tempest tests within the container¶

• Let’s create a tempest script which will be later executed within the container in order to generate tempest.conf and run tempest tests:

  $ cat <<'EOF'>> /home/stack/container_tempest/tempest_script.sh
  # Set the exit status for the command
  set -e
  
  # if you want to run tempest against overcloud, overcloudrc file needs
  # to be sourced and in case of undercloud it's stackrc
  # NOTE: the files need to be copied to /home/stack/container_tempest
  # directory in order to have it accessible from the container
  source /home/stack/container_tempest/overcloudrc
  
  # Create a tempest workspace, use the shared directory so that the files
  # in it are accessible from the host as well.
  tempest init /home/stack/tempest_workspace
  
  # change directory to tempest_workspace
  pushd /home/stack/tempest_workspace
  
  # export TEMPESTCONF environment variable for easier later usage
  export TEMPESTCONF="/usr/bin/discover-tempest-config"
  # Execute the discover-tempest-config in order to generate tempest.conf
  # Set --out to /home/stack/tempest_workspace/tempest.conf so that the
  # tempest.conf file is later accessible from host machine as well.
  # Set --deployer-input to point to the tempest-deployer-input.conf
  # located in the shared directory.
  $TEMPESTCONF \
    --out /home/stack/tempest_workspace/etc/tempest.conf \
    --deployer-input /home/stack/container_tempest/tempest-deployer-input.conf \
    --debug \
    --create \
    object-storage.reseller_admin ResellerAdmin
  
  # Run for example smoke tests
  tempest run --smoke
  
  EOF
  

  Note

  • Apart from arguments passed to python-tempestconf showed above, any other wanted arguments can be specified there. See Generating tempest.conf using discover-tempest-config.

  • Instead of running smoke tests, other types of tests can be ran, see Running Tempest tests section.

  • Always save the state of resources before running tempest tests.

  • If you already have a tempest.conf file and you want to just run tempest tests, omit TEMPESTCONF from the script above and replace it with a command which copies your tempest.conf from container_tempest directory to tempest_workspace/etc directory:

    $ cp /home/stack/container_tempest/tempets.conf /home/stack/tempest_workspace/etc/tempest.conf
    

• Set executable privileges to the tempest_script.sh script:

  $ chmod +x container_tempest/tempest_script.sh
  

• Run the tempest script from the container as follows:

  $ docker run -i \
      -v "$(pwd)"/container_tempest:/home/stack/container_tempest \
      -v "$(pwd)"/tempest_workspace:/home/stack/tempest_workspace \
      docker.io/tripleomaster/centos-binary-tempest:current-tripleo-rdo \
      /bin/bash \
      -c 'set -e; /home/stack/container_tempest/tempest_script.sh'
  

• In case you want to rerun the tempest tests, clean tempest workspace first:

  $ sudo rm -rf /home/stack/container_tempest
  $ mkdir /home/stack/container_tempest
  

  Note

  It’s done with sudo because tempest in containers creates the files as root.