Introduction
Automated CI/CD pipelines are no longer a ‘nice-to-have’ in today’s software landscape, they should be considered mandatory by any enterprise. With different developers and engineers moving in and out of enterprise teams, organizations must ensure that their code is tested, deployed efficiently and safely, and able to be rolled back in the event of any issue. The use of automation in a robust CI/CD pipeline can take all of these critical tasks off the minds of your admins, engineers, developers, and project managers and allow them to focus on mission-critical tasks to keep your business running smoothly and growing simultaneously.
GitHub Actions is GitHub’s built-in CI/CD option and comes with an extensive library of custom steps available that developers are constantly expanding upon to increase its ease of use. In this post, we’ll lay out a plan on how to approach using GitHub and GitHub Actions to their fullest potential to provide a modular and reusable pipeline for any enterprise.
Define A Branching Strategy
The first step of building an automated pipeline for your organization is to decide what branching strategy will be used. There are many branching strategies that all have their advocates, but factors like your team size, the complexity of projects, and how often you release to Production all play a key role in deciding which strategy is right for you. Regardless of what your branching strategy is, the steps provided in this blog will show you an approach to creating a secure, reusable, and easily maintained pipeline to deliver your releases on time.
For the CI/CD pipeline in this example, we are using the traditional GitFlow (GF). GitFlow has taken a back seat to other branching strategies like trunk-based development (TBD) in recent years, but it still has its advantages and is still widely used. GF is generally seen as a more complex branching strategy vs. TBD, but by leveraging a well thought out and robust pipeline, the aim is to remove a lot of that complexity from your daily workflow by automating it in the pipeline. If you would like a refresher on GitFlow, here is the original blog post that introduced it to the world in 2010.
Following GF’s recommended approach, we will have two long-living branches named main and develop, and developers will submit work on feature/, fix/, and hotfix/ branches. This approach also allows us to treat hotfixes slightly differently which will allow us to get a quick fix deployed to production while staying within the lanes of the overall strategy. Below we can see the differences between a normal workflow and how hotfixes are handled.
Global Configurations
In this section we will look at a few items that will be utilized by all repositories to make each project’s workflow easily maintained and not too granular. Recognizing which items should be set at a global level allows for easy maintenance when secrets have to be rotated or you want a different functionality to be applied to all repositories that utilize the workflow.
Creating a GitHub Personal Access Token
To perform steps like creating and deleting a release-* branch, while also protecting said branch, you will need an admin to create a Personal Access Token for use in the pipeline. Follow the steps provided by GitHub to create a Personal Access Token making sure to select the organization that will be housing your application repositories as the Resource Owner and give Repository Access to All Repositories. For necessary permissions, under Repository Permissions change Actions, Administration, and Content to Read & Write.
Organization-level Properties
Organization-level properties are values that you want all of your repositories to be able to access and reference that will not change in each application’s configuration. Within these properties are Secrets and Variables. Variables are items that are fine to be left in plain text and will show within the logs of the pipeline as plain text. Secrets are items you want to keep hidden at all times and only allow someone to enter them once. Once a secret has been entered and saved, it can no longer be viewed in plain text, but a new value can overwrite it. The Personal Access Token created in the last step is a good example of a value that would be included in organization-level Secrets. To reference Secrets and Variables in a GitHub Actions workflow document use secrets.exampleSecretName and vars.exampleVarName, respectively. Additionally, you will typically have to provide the reference to secrets inside a GitHub Action Expression that has the syntax of ${{}} and can be further expanded upon in the GitHub Action Documentation.
Repository Settings
Repository settings are either more fine-grained than organization-level settings, or are not available at the organization level. These settings will allow one to set controls for who can deploy to specific target environments and also add useful automations like ensuring unit tests are successful before a pull request is even peer-reviewed.
Creating Environments
Inside your application’s repository settings, you will need to create an environment for each deployment environment you have on your destination server(s). These environments have three main objectives:
- Controlling who can deploy to each environment
- Preventing a user from deploying their work to higher environments like production without a second reviewer
- Overwriting any organization-level properties with an environment or application-specific value, if necessary; for example, each environment can set the same Secrets and Variables that were mentioned in the organization-level settings, but if the key names match, the environment property will take precedence
Typically these will include a development, staging, and production environment with staging and production having restrictions on who can approve deployments. However, with the development environment, we will not be applying deployment restrictions but will create the environment for any future use of environment-specific secrets or variables. Refer to the GitHub documentation on using environments for deployments for more information, but in our case, we will create an environment called DEV (again, with no restrictions), UAT, and PROD. UAT and PROD will both have the same settings, which can be seen below.
Creating Protected Branches
Since we are using GitFlow, our long-living branches that trigger deployments will be develop and main, and they should be protected to ensure no one is directly committing to them or altering them without a peer review. Additionally, we will also want to protect release-* branches as they will also trigger the pipeline to test and deploy any updates that would be added to the release as an outcome of user acceptance testing.
Creating Protected Branches
Since we are using GitFlow, our long-living branches that trigger deployments will be develop and main, and they should be protected to ensure no one is directly committing to them or altering them without a peer review. Additionally, we will also want to protect release-* branches as they will also trigger the pipeline to test and deploy any updates that would be added to the release as an outcome of user acceptance testing.
Workflows
Now that our organization and repository are set up correctly, we can move on to the actual automation work by adding workflows to our project’s repository. GitHub Actions looks for YAML files in the repository’s directory <root>/.github/workflows/. It’s important to note that subdirectories are not supported at this time so the file names are the only organization you can have for your workflow YAML files. How you specify triggers in the YAML files will determine when the workflow will run. You can have your workflows run on a schedule, a push, or a pull request to any branch. However, to create modular and reusable workflows, the way we want to trigger an action is called workflow_call, which means it will run when called from another workflow.
Jobs needed to build a functioning CI/CD pipeline can usually be logically grouped into 4 separate categories: testing/building, deploying to a target environment, cleaning up branches, and creating a tag and/or release of the most recent production deployment. With these four groups identified, we can create YAML files (build-test.yml, deploy.yml, branch-cleanup.yml, and create-tag.yml) with the steps necessary to complete all of those jobs and then invoke those files with a main.yml file. main.yml will handle all of the orchestration and supply any necessary inputs. This setup allows us to reuse items like deploy.yml for every environment that we wish to deploy to by simply supplying the changing input parameters.
While main.yml will have all the necessary items for completing the deployment, we want to make this process reusable for future similar projects that would benefit from using the same deployment process. For this, we need to ensure main.yml is also using the workflow_call trigger and then we can invoke main.yml from another repository that would like to kick off a workflow. To maximize this reusability we move all of our workflows into its own repository called shared-workflows, again under the .github/workflows/ directory. Then for every new repository that is created and that would like to utilize our workflows, all that is needed is a simple YAML file pointing to the main.yml file in shared-workflows repository and triggers how often it is invoked. In our example the application is a MuleSoft application, so the suggested name would be something like mule-deploy.yml.
This approach creates a setup where we can modularize each task that we want to complete in its configuration file, invoke those configuration files from an orchestration file, and finally allow any new project to invoke the orchestration file on its schedule.
Full Example
If you want to check out the full example of this implementation you can check the sample project GitHub and the shared-workflow repository here.