Overcloud Major Upgrade Workflow and CLI¶

openstack overcloud upgrade prepare $ARGS¶

The entry point for the the upgrade CLI commands, prepare, run and converge, is given in the python-tripleoclient setup.cfg. All three are also defined in the same file, overcloud-upgrade.py.

The ‘prepare’ Heat stack update does not apply any TripleO configuration and is exclusively used to generate the Ansible playbooks that are subsequently invoked to deliver the upgrade.

As you can see the UpgradePrepare class inherits from DeployOvercloud. The reason for this is to prevent duplication of the logic concerned with validating the configuration passed into the prepare command (all the -e env.yaml files), as well as updating_the_swift_stored_plan with the overcloud configuration.

The prepare_env_file is automatically prepended to the list of environment files passed to Heat (as specified by prepare_command_prepends). It contains resource_registry and parameter_defaults which are intended to be in effect during the upgrade.

As a result the UpgradePrepare class inherits all the Deploy_parser_arguments, including --stack and -e for the additional environment files. We explicitly set the update_plan_only argument so that the Heat stack update does not get executed by the parent class and returns after completing all the template processing.

Instead, the Heat stack update is performed by a mistral workflow. On the client side the hook is in the update method defined in package_update.py. This invokes the package_update_plan mistral workflow in tripleo-common. The package_update_plan workflow has a number of tasks, one of which invokes the heat stack update using the update_stack_action.

Back on the tripleoclient side, we use base_wait_for_messages to listen for messages on the Zaqar_queue that is used by the mistral workflow.

The operator must include all environment files previously used with the overcloud deploy command. It is especially important that the operator includes the environment file containing the references for the target version container images. See the operator_docs for pointers to how that file is generated and for reference it will look something like

parameter_defaults:
  DockerAodhApiImage: 192.168.24.1:8787/queens/centos-binary-aodh-api:current-tripleo-rdo
  DockerAodhConfigImage: 192.168.24.1:8787/queens/centos-binary-aodh-api:current-tripleo-rdo
  DockerAodhEvaluatorImage: 192.168.24.1:8787/queens/centos-binary-aodh-evaluator:current-tripleo-rdo
  DockerAodhListenerImage: 192.168.24.1:8787/queens/centos-binary-aodh-listener:current-tripleo-rdo

Once the Heat stack update has been completed successfully and the stack is in UPDATE_COMPLETE state, you can download the configuration ansible playbooks using the config download cli

[stack@521-m--undercloud ~]$ source stackrc
(undercloud) [stack@521-m--undercloud ~]$ openstack overcloud config download --config-dir MYCONFIGDIR
The TripleO configuration has been successfully generated into: MYCONFIGDIR/tripleo-gep7gh-config

and you can inspect the ansible playbooks which are used by the upgrade run before executing them.

openstack overcloud upgrade run $ARGS¶

Unlike the first step in the workflow, the upgrade prepare, the UpgradeRun class does not inherit from DeployOvercloud. There is no need for the operator to pass all the environment files and configuration here. The template processing and update of the stack and swift stored plan have already taken place. The ansible playbooks are ready to be retrieved by config download as demonstrated above. The upgrade run operation thus will simply execute those ansible playbooks generated by the upgrade prepare command, against the nodes specified in the parameters.

Either --nodes or --roles parameters are used to limit the ansible playbook execution to specific nodes. Both --roles and --nodes are used by ansible with the tripleo-ansible-inventory. This creates the ansible inventory based on the Heat stack outputs, so that for example Controller and overcloud-controller-0 are both valid values for the ansible-playbook --limit parameter.

As documented in the run_user_docs and the nodes_or_roles_helptext, the operator must use --roles for the controllers. Upgrading the controlplane, one node at a time is currently not supported, mainly due to limitations in the pacemaker cluster upgrade which needs to occur across all nodes in the same operation. The operator may use --roles for non controlplane nodes or may prefer to specify one or more specific nodes by name with --nodes. In either case the value specified by the operator is simply passed through to ansible as the limit_hosts parameter.

The --ssh-user and all other parameters are similarly collected and passed to the ansible invocation which starts on the client side in the run_update_ansible_action method call. The --skip-tags parameter can be used to skip certain ansible tasks with the ansible-skip-tags ansible-playbook parameter. The allowed --skip-tags values are restricted to a predefined set, validated against MAJOR_UPGRADE_SKIP_TAGS. Finally, the --playbook parameter as the name suggests is used to specify the ansible playbook(s) to run. By default and as you can see in the definition, this defaults to a special value ‘all’ which causes all-upgrade-playbooks-to-run. The value of all_playbooks in that previous reference, is stored in the MAJOR_UPGRADE_PLAYBOOKS constant.

As with the upgrade prepare, for upgrade run a mistral workflow is used to perform the ‘main’ operation, which in this case is execution of the ansible playbooks. On the client side the update_nodes_workflow_invocation is where mistral is invoked and takes as workflow input the various collected parameters described above. You can see that the update_nodes_workflow which lives in tripleo-common has parameters defined under the ‘input:’ section which correspond to the openstack overcloud upgrade run parameters.

There are two main tasks in the update_nodes_workflow, the download-config_action which is invoked in a first ‘download_config’ task, and the ansible-playbook_action action which is invoked in the ‘node_update’ task. This is ultimately where ansible-playbook-is-executed with processutils.execute.

Finally back on the client side we listen for messages on the run_zaqar_queue before declaring the upgrade-run-success!

openstack overcloud external-upgrade run $ARGS¶

The external-upgrade run command is used to upgrade the services whose deployment (and upgrade) procedure is not tied to execution on particular overcloud nodes. The deployment/upgrade procedures are thus executed from the undercloud, even though a full overcloud inventory is available for use.

The external upgrade playbook first executes external_upgrade_tasks and then external_deploy_tasks. The execution happens within the same Ansible play, so facts from external_upgrade_tasks are carried over to external_deploy_tasks. This is a mechanism which will allow you to amend what your deploy tasks do based on whether an upgrade is being run or not.

Often it’s not desirable to run the tasks for all services at the same time, so external-upgrade run supports --tags argument to limit which tasks are run.

The mechanisms of external-upgrade and external-update commands and Ansible tasks are the same, but two commands and task hooks are provided because generally in OpenStack we distinguish minor update vs. major upgrade workflows. If your service only has one type of upgrade, you can make the external_update_tasks the same as external_upgrade_tasks by using YAML anchors and references.

openstack overcloud upgrade converge $ARGS¶

The UpgradeConverge class like the UpgradePrepare class also inherits from the DeployOvercloud class thus getting all of its parameters and template processing. The operator needs to pass in all Heat environment files used as part of the upgrade prepare including the container images file.

The main objective of the upgrade converge operation is to unset the upgrade specific parameters that have been set on the overcloud Heat stack as part of prepare. These are unset using the converge_env_file which is included in the list of client_converge_env_files passed to the Heat stack update.

The ‘converge’ applies all TripleO configuration against all overcloud nodes and thus serves as a sanity check that the overcloud was successfully upgraded, since the same configuration will already have been applied. The ‘converge’ will also leave the Heat stack in a good state for subsequent updates, for instance scaling to add nodes.

As these values are set in parameter_defaults a Heat stack update is required against the overcloud Heat stack to explicitly unset them. In particular and as pointed out in the operator_converge_docs until converge has completed, any operations that require a Heat stack update will likely fail, as the ‘noop’ of the DeploymentSteps in the prepare_env_file in particular means none of the usual docker/puppet/* config is applied. Setting something with parameter_defaults means it is used until explicitly unset via parameter_defaults as that value will override any other default value specified via the tripleo-heat-templates.

Unlike the prepare command there is no mistral workflow here and instead we rely on the parent DeployOvercloud class to invoke the converge_heat_stack_update and so the implementation is also simpler.

Upgrade CLI developer workflow¶

This section will give some examples of a potential developer workflow for testing fixes or in-progress gerrit reviews against python-tripleoclient, tripleo-common or tripleo-heat-templates for the upgrade workflow. This may be useful if you are working on an upgrades related bug for example.