Developers are hired to solve business problems, but in the world of software development there often end up being many menial, repetitive tasks. There are multiple issues with developers performing these tasks manually...
Developers are hired to solve business problems, but in the world of software development there often end up being many menial, repetitive tasks. There are multiple issues with developers performing these tasks manually:
• They are not providing direct value to the business when distracted with these tasks
• Humans are prone to error when doing the same thing over and over (as we become familiar, we become over-confident and make silly mistakes)
• The tasks are not challenging to the mind of a developer which can cause boredom and a lack of job satisfaction
Fortunately, many of these repetitive tasks are nothing more than a set of well-defined steps that need to be executed in a specific sequence, often depending on the output of the previous step. These are perfect candidates for automation.
The following war cry sums it up well: “If it’s important and repetitive, then automate it!”.
1. Foundations of software development automation
An extremely important practice is required to enable automation. This practice is source control, or sometimes called version control.
Source control allows a team to track changes made to the source code as features are developed. Developers can see a comprehensive history of changes made as well as jump around that history to revert to a specific point in time.
These practices also allow for the “branching” of code which enables developers to make sweeping changes without affecting the work of others. There are various strategies around branching, each with their own pros and cons, but the two most popular are:
1) Trunk-based or master-based development: All work is continuously merged into the main branch, even if the feature is not yet complete.
2) Feature branches: Work is done on a separate branch and only merged into the main branch once complete.
Besides the primary benefits of a source control system, maintaining the source of an application in a centralised, version-controlled repository is the key to unlocking many of the automations mentioned below.
2. Types of automation
There are many kinds of tasks that are performed across all software projects, there are also unique tasks per specific project. Most of these tasks can be automated. Let’s have a look at some generally accepted best practices for automation in the life of building a solution.
• Continuous integration
Eventually, most software projects grow large, and the development teams expand to cope with the demand for new features and maintenance. This results in many parallel branches of the code existing at the same time and in various states.
These branches will have to be merged together (and back into the main branch) at some point. There is a high risk of merging errors, as well as logic bugs when this code comes together, especially if the branches have lived alone for any extended period. The solution is to merge all branches regularly and to make sure everything is always working all the time. This process is known as “continuous integration,” and is defined by Martin Fowler as “a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including automated tests) to detect integration errors as quickly as possible.”
Another advantage of this automated integration is a consistent build. We have a well-defined automated process that produces binaries to be deployed onto some environment. This removes the risk of a developer having some specific setup that no-one else can replicate and resulting in only that developer being able to build for a release. We also remove the commonly used “but it works on my machine” excuse. If it doesn’t build on the build server, it doesn’t go to production.
• Automated testing
An extremely important part of continuous integration is the execution of all automated tests. These tests should be written by the developers as a safety net that gives them confidence to make changes without accidentally breaking something else in the code base.
In the diagram, the bottom 3 levels can be automated with varying degrees of difficulty.
Unit tests are the most straight-forward to automate as they should be small self-contained tests that only confirm the functionality of a single piece of code under very specific conditions. These tests should have no outside dependencies (or all dependencies are mocked out) so that they can be executed over and over with no side-effects.
Integration tests are used to ensure that blocks of functionality can talk to each other successfully. This may mean accessing a database or file system, or calling a service over the network. These types of tests may also check the flow of data through the entire system. Automated integration tests are harder to set up, as they often require a valid database to access (with valid test data pre-populated) or real services to call.
Acceptance tests exist to make sure that the end-to-end solution being built meets the requirements specified by the business. In systems with a user interface, these often take the form of scripted steps through a specific flow, recording and verifying along the way that the system is behaving as expected. For non-UI systems, these tests may include loading production-like data into the system and checking the output. Acceptance tests require a deployed instance of the application to be executed on.
Whatever the type of test, running them manually would put a huge strain on the team and severely affect delivery. Executing the tests on every code check-in (or once a day for heavier integration and acceptance tests) provides confidence in the changes being made with very little effort from the team required (beyond writing the tests when functionality is built).
• Automated deployment/continuous delivery
Deploying an application into an environment can take many forms. In most cases there is a file copy involved, and often there are multiple steps involved before and after the file copy. These steps could theoretically be documented as a recipe for someone to follow when deploying, but there is still room for error when a human gets involved with simple, repetitive tasks.
An application could also be deployed to various testing environments on its way through the production environment. To get feedback quickly, it’s a good idea to deploy the latest changes to a test environment as they are made. Doing all these deployments manually would be a huge time sink for the team.
At a minimum, deployments should be executed by a script. In an ideal world, that script should be controlled by a system that can put permissions and auditing around when a script is executed, who executed it and what the results of that execution were.
• Provisioning and scaling
In this era of cloud-based computing, it is also possible to automate the provisioning of resources for the consumption of our application. We can spin up an entirely new environment just to run a set of tests and then tear it down again when done.
We may have a system that provides functionality for adding new customers each with their own dedicated resources. These resources can be automatically provisioned on demand.
Once an application is released into the wild, it is extremely important to ensure that it continues running well and that any errors or unexpected behaviour is picked up immediately. Systems should be put in place that alert the team to anomalies, instead of requiring someone to watch dashboards all day or sift through thousands of system generated error reports. Artificial intelligence can also play an important role in identifying unusual trends.
3. Automation tooling
As with many tools available in the software development world, there is a wide variety of tools and systems that will help a team accomplish its automation goals.
• Scripting tools
Scripts allow the chaining together of commands into a single parameterised command. Initially, the team would execute the scripts themselves until they feel confident they are performing as intended.
• DevOps tools
The term DevOps is used to describe a set of practices that combine software development and IT operations teams in order to shorten the cycle of delivering working software and realising business value. Automation falls into these practices and there are many software solutions available in this space.
A team should invest in these systems so that they are able to automate and control the execution of the scripts (or extend them with further capabilities) and provide additional functionality such as permissions, logging, remote execution, scheduling, etc.
There are many players in this field and the choice of system would depend on many factors:
o What technologies are used to build the application?
o Are there operating system constraints on the build tools?
o Does the organisation have preferred software vendors?
o Is it suitable to execute builds and tests in the cloud, or must it be on-prem?
Read the full From Here to There publication