In this post we'll cover QA and Test Automation at a high level. We'll break down industry buzz-words like Unit and Integration Testing, as well as call attention to popular vendors in the industry currently.
Note: We will not dive into performance or security and application testing in this post, look for this info in future posts!
If you've ever heard terms like "Test Automation," "The Testing Pyramid," "Smoke and Sanity test," and more and were curious what these terms mean and why they matter, this training is for you.
- Sales teams can be use this for training so they can be more knowledgeable when asked questions like, "How does this help me with shifting left?"
- Software Engineers that are being asked to do more (DevOps) should be familiar with these terms as they may become responsible for creating and maintaining them
- Managers who are trying to learn more about how Test Automation can impact their organization
What is QA and Test Automation?
QA, or Quality Assurance, is the idea of determining and maintaining a level of quality in your final product that is being consumed by your users. This term shouldn't be too new, as it's used in many industries from manufacturing to the software industry. For the purposes of this training, QA will be referring to software quality testing.
When discussing QA, a few other terms are usually brought up. User Acceptance Testing (UAT) is getting stakeholder sign off (or sometimes end user approval) for your product. Manual Testing is exactly as it sounds, manually going through your product or WebApp and looking for defects. Automated Testing refers to automating the process of testing quality to improve accuracy and efficiency.
Test Automation is the idea of using expected outcomes versus actual outcomes to automatically test your application. This will be the core form of testing we'll discuss in this article.
Why do I Care About Testing?
QA is paramount to ensuring your software works properly. When experts discuss failing early and increasing deployment confidence we are talking about quality. The sooner a defect or bug can be found in the Software Development Lifecycle (SDLC), the lower the cost of fixing the defect will be in terms of time and money. This is also what we mean by reducing technical debt, which is a term used to describe the cost of reworking your code (which includes bugs, poor initial implementation, speed and complexity, etc).
Having a strong test automation setup also allows you to gain benefits in terms of shipping faster and even helping to document your code. With a higher level of quality and confidence in your code changes, you can ship faster to the end users knowing that there is less risk of a major bug. With well written tests, you are able to see exactly what a specific function or domain of the application should be doing, which serves as documentation in one way.
Finally, QA and test automation can help prevent potential lawsuits. There have been many instances where quality assurance and testing could have potentially prevented lawsuits and even death. The following are a few examples:
- Therac-25: A bug in radiation treatment software for cancer patients caused a 100x dosage to be administered. 4 patients died, and 6 others were affected
- Panama City Cancer Institute: Similar to the above, there was a bug in radiation software that caused a double dosage of radiation to patients. 20 people were affected. and 8 died. An interested bit of this one is the two developers responsible for the bug were held liable criminally for negligence
- Patriot Missile System: A rounding error in a function caused mis-timing, meaning the missile system would miss it's target mid-air. There was a missile attack and Patriot Missile Systems should have prevented any damage by intercepting the incoming missiles, however this bug made it miss. This resulted in a military barracks being hit with 28 soldiers killed and another 100 wounded
- Almost Nuclear War: The USSR had an early reaction system during the cold war to detect missiles from known US missile sites. A bug caused a false report of 5 US ballistic missiles launching from a US territory. Luckily, the duty officer decided it was a mistake as the US would've launched more than just 5 missiles if it was real. Retaliation would have been the USSR launching their own nuclear missiles back toward US targets
What are the Different Types of Tests?
Alright, by now I hopefully convinced you that quality is important, and maintaing QA in an automated fashion helps to facilitate this process. Let's take a look at the most common testing terms when it comes to Software Testing.
A Unit Test tests the smallest unit of code in isolation. A unit is often a single function, or at most a single class. These tests focus solely on the code, and ensuring functions behave as they should in isolation.
These should be the foundation of your test strategy. They are cheaper to maintain, provide the earliest form of bug detection, and should be the majority of your total tests, with the number of unit tests being greater than any other test. A good rule of thumb is to write at least one new test for every bug you fix to help ensure it doesn't happen again.
Integration Testing (a.k.a. Functional or Service Testing)
An Integration Test tests units of code together. These still focus on the code only, and are also referred to as Functional or Service testing. There should be fewer integration tests than unit tests to keep the maintenance cost and execution time.
Integrations tests may also test your applications interactions with external systems, such as databses, other applications and services, network calls, etc. Because of this, they may also test API endpoints, though there are a few strategies involved here to consider so you do not have flakey taests (tests that fail when they should have passed).
To test an API, you likely want to remove the actual API call itself as network issues, an external service outage, or other issue could prevent your tests from passing. To solve this problem, a concept of fixtures and mocks were developed.
These terms can mostly be used interchangeably, and refer to stubbing out a fake "response" from an API endpoint. This means a JSON file with fake data that resembles the payload you would expect to receive from the API service. The benefit of this is clear in that you avoid rate limits, increase test execution time, and can test with known data for your assertions. Unfortunately, this does leave an opening for the API itself to have made a breaking change, and because of this some companies opt to build a separate service to maintain a mock and fixture library with the "latest" payload.
User Interface (UI) and End to End (E2E) Testing
UI and End to End Testing is the final category we'll discuss today. These tests run after your Unit and Integration tests, and are the first tests that finally look at your application from the user's perspective.
Because of this, this form of testing is also known as black-box (or closed box) testing. This means that the tests verify functionality as a user, without any knowledge of the underlying code execution paths, etc. This could be verifying page elements react as expected on a WebApp, forms have proper field validation, you can checkout after adding items to a cart, etc.
These tests should cover your common procedures from start to finish, such as registering a new account and logging in, or going through the checkout flow on your eCommerce site.
Because of the nature of these tests, however, they are very expensive to maintain, take longer to write, are prone to more false failures, and take longer to execute than unit and integration tests. Therefore, these should be the smallest set of tests in your testing strategy.
A Quick Note on Performance and Security Testing
Future posts will go more in depth on these subjects, however I wanted to at least give a high-level definition of each of these types of tests.
- Performance testing (or load testing) provides a way to determine how your application performs under pressure. For a WebApp, this could mean a lot of users navigating your site and monitoring page load times, server errors, etc.
- Security testing refers to things like penetration tests, SQL-injection, and general application security. This can include dependency management and scanning, container scanning, and more.
What is the Testing Pyramid?
I've mentioned throughout the post hints toward a Testing Pyramid, so let's dive into that now.
The Testing Pyramid is a metaphor to describe the number of tests in each category you should have in your testing strategy. Sticking to this metaphor, we see the call-outs I made earlier in that you should have more unit tests tahn integration tests, and more integration tests than UI tests.
If you keep this balance within your organization, you'll keep costs down in terms of maintenance, execution time, fragility (flakiness) of the tests, and more.
Finally, you may be asking what the Manual Tests thought bubble is for. Many companies do not want to rely solely on automation testing for their QA. Therefore, they employ manual testers who verify behavior of the application by interacting with it themselves. This can be called User Acceptance Testing (UAT) at times, and helps to verify potential issues that the automation tests would overlook such as a banner being too large, a button's label not being readable, etc. There's a high-cost associated to manual testing, so this should be limited in a healthy test strategy.
Unfortunately, you'll find many enterprises with this pyramid upside-down. They often have a huge number of UI tests, minimal integration tests, and sometimes little to no unit tests. These organizations often set 8 or more hours per sprint for each developer on a team to be dedicated to bug fixes and interrupters from quality issues in production. The tests are flakey due to how many rely on the UI itself, and therefore there is little confidence in the test automation.
To correct a testing pyramid means to slowly roll out culture change. Future posts will go more into the culture aspect of this, however a brief summary would include adding tests to all new bug fixes and features, and as bandwidth permits, begin replacing UI tests with a series of unit and integration tests to catch the bugs earlier in the software development process.
What are Smoke and Sanity Tests?
Smoke and Sanity tests verify the "happy paths" within your application.
Smoke test suites specifically cover basic functionalities in order to fail early. For example, if there's a bug in your checkout form, there's no reason to wait for an E2E test to run that tests the entire flow from start to finish. Smoke tests are also known as regression tests, in that you are also verifying the new feature being added or the bug fix being implemented.
Sanity test suites, are a more breadth version of smoke testing and as such, the terms are often interchanged. Sanity checks that the bug fixes in your previous build have not come back, and can assist in confirming new feature functionality.
A typical continuous integration (CI) pipeline would look something like this:
- Run smoke suite
- Run sanity suite
- Run unit tests
- Run integration tests
- Run UI and E2E tests
- Perform additional manual testing as needed
As you can see, this process allows the basic functionality to be checked before running the longer running test suites. As you move through these steps, there should be less tests to run to offset the extra time it takes to execute the more complex tests (UI tests for example).
What are Popular Tools and Frameworks?
When it comes to tools and frameworks to build your automation tests, there are a lot, and I mean a LOT, of options. Here are a few so you can at least recognize them and connect them to test automation. Future articles may cover these more in depth.
Tools and Test Frameworks
You'll find a tool and/or framework for almost every programming language in existence (if not every language). Some common ones are:
- Selenium, Apache JMeter, Jest, Moq, XUnit, NUnit, JUnit, RSpec, minitest, TestNG, and many more
Running Automated Tests
The CI/CD approaches and capabilities in most of the automated test runner vendors listed below are similar today. They check out your code, build, run tests, and may deploy to a test environment for UI and manual testing. Here are some common vendors in the industry:
- GitHub Actions, GitLab CI/CD, Atlassian Bamboo, Azure Pipelines, AWS CodePipeline, Google Cloud Build, Jenkins (CloudBees), TravisCI, CircleCI, TeamCity, and many others
In the future, I'll write articles on GitHub Actions, and potentially other providers as well. Comment below if you're interested in anything specific.
What does "Shifting Left" mean?
As DevOps has become more prominent, and agile methodologies and practices have taken over in the modern Software Development Lifecycle (SDLC), a common phrase being touted is the concept of shifting left.
Shifting left is a buzzword referring to detecting build failures and bugs earlier in the SDLC process. The diagram below shows a typical SDLC chain, whether this is waterfall or agile, these steps are similar.
While testing does occur at all the stages, shifting left means putting more emphasis on testing earlier in the process, and closer to the developer. This lowers the cost of maintenance, increases defect detection, increases efficiency, and allows a course correction to happen before too much time is spent moving in the wrong direction. Finally, this supports the testing pyramid, in that more unit tests would allow you to fail earlier in this chain.
To the managers and upper-management (Directors, CTOs, etc) that want to implement this in your organization, please remember the cultural adoption required for this kind of shift. As you move left, you are putting more of these responsibilities on your individual contributors (developers). Look for future articles where I discuss roll-out plans and ways to approach these topics as you introduce it to your organization.
What is Code Coverage?
Another popular term thrown around in the testing world is the idea of Code Coverage (or test coverage) which is a measurement of the effectiveness and breadth of your automation testing. Code coverage is built into many popular test runners, and the metric is represented in percentage form, i.e. 97% code coverage.
These tools calculate code coverage by comparing the test execution path through your code, versus the available paths within your code. For every if condition, you should test both true and false conditions, therefore the test coverage reflects if you do indeed cover both paths in your tests.
What are some Popular Vendors and Tools for Code Coverage?
As test automation and DevOps have come to be mainstream in almost all organizations, many vendors have started providing their own code coverage tools that provide a variety of extra features on top of the basic coverage analysis many test runners have. The following is a list of some of the more popular vendors in this category. Future articles may go into depth on these (let me know in the comments if you'd like that), but for now just know they provide test/code coverage.
- Popular vendors include: Code Climate, CodeCov, Coveralls, SonarQube, Codacy, and many others.
Throughout this article, I've covered many of the most popular terms in the software quality / test automation world. Here are some additional terms often used when discussing software testing, please see the relevant links for more information. Future articles may include an indepth look at theses.
- TDD: Test Driven Development
- tl;dr - Writing a test first before you write the code, so you can verify the original functionality as you build the code. Also known as red-green testing (fail/pass)
- DDD: Domain Driven Development/Design
- tl;dr - Writing your code in ways that adhere to specific domains they operate in, often tied to specific business domains
- BDD: Behavior Driven Development
- A BDD example popular in Ruby
- tl;dr - Developing your application with specific functionality and behaviors in mind. This could mean speaking with a Business Analyst or other stakeholder more often as a developer, to ensure you are building something that meets their requirements, versus waiting for acceptance later in the development life cycle
- Regression Testing: Ensuring new features and bug fixes didn't break anything in the application (older code, previous features, previous bugs, etc). This ensures your new work didn't undo other work, cause more bugs, etc.
- A/B Testing: Often used in marketing and UI. This technique is often used to show a portion of your users (demographics, location, or just a 50%/50% split) one version of a UI/advertisement and the other portion of your users see another version of a UI/advertisement.
- A/B testing allows you to test something like a new product page to see if the new layout offers better conversions (people actually purchasing versus leaving the page).
- A/B testing examples could include Reddit, where it seems most people prefer the "classic" reddit, and therefore click to enable "classic" which is tracked. Reddit as a company can determine how many users have adopted the new UI, take feedback, make improvements, etc.
If you want to reduce costs of maintenance and ship more confidently while deploying faster, you should be employing a strong test automation suite. Following the testing pyramid will keep these tests managemable and cost effective. This process helps you to shift left, meaning errors are found faster.
If you are interested in some of the specifics I called out such as the QA bugs that nearly caused nuclear war, the testing pyramid, more about specific test types, etc. Please refer to the following: