When to Start Automation Testing for your Project?

• Testing Types in Web Development
• Types of Testing of Web Service Parameters
• Manual and Automated Testing
• Benefits of Automation Testing
• When to Automate Testing
• Selection of Test Types for Different Web Projects
• Conclusion

Testing is a vital component of web development, playing a pivotal role in software quality assurance. Modern web projects can encompass a multitude of testing types, including manual testing and automated testing services, both of which I will go into in more detail in this article. The complexity of the system you’re developing generally dictates the range and depth of the testing required, but ask a tester and they’ll tell you can never have too many tests.

As an IT entrepreneur, my approach to testing is pragmatic. I consider all aspects of the problem and avoid unnecessary expenditure whenever possible. For this reason, I advocate for the use of automated tests in large projects, while reserving manual testing for smaller ones. In this article, I will explain my rationale behind these recommendations, rooted in my own experience and best practices within the field of web development.

Testing Types in Web Development

Testing is essential for software quality assurance, as its primary goal is to identify software failures and trigger timely fixes. There are several types of software testing, each with their unique classification principles. Let’s start by breaking this testing landscape down. 

Testing levels

The International Software Testing Qualifications Board (ISTQB) defines four testing levels:

1. Unit testing

Unit testing is a method designed to verify the functionality of specific sections of code. It’s crucial to understand that developers create unit tests as they work on the software code. Several tests may be written and applied to ensure that a single function operates as expected. 

In web development, unit tests primarily deal with server-side aspects, and the objective of unit tests is to confirm the proper functioning of the backend, or the logical layer. They are used to validate, for instance, individual features or modules of a website or web application, independent of the rest of the system. 

My advice is that you should always assess the expediency of implementing unit tests for each function and component on a case-by-case basis, as their creation can be time-consuming. Typically, the time spent constructing unit tests can range from 30% to 100% of the time used for writing backend code.

Due to this, developers initially focus on covering the most critical portions of the code with unit tests. This often includes areas such as payment processing or user registration. Only then, if required, are less critical sections addressed with unit testing.

2. Integration testing

This form of testing is crucial to confirm that distinct units of code function seamlessly as a cohesive entity. It involves examining the interactions between different components to identify any potential issues arising from their integration. The primary emphasis of integration testing lies in the interfaces between components, however bear in mind that establishing a testing environment for integration testing of complex systems can often be a challenging and time-consuming task.

3. System testing 

With system testing, you test a fully integrated system.

4. Acceptance testing

Acceptance testing is a critical phase wherein you verify if the website or web application aligns with the requirements outlined in project documentation compiled at the beginning of the project. Rather than focusing solely on technical aspects, this phase emphasizes testing web resources against the client’s expectations. As such, the process involves simulating a realistic user experience to gain insight into actual usage scenarios. The acceptance testing phase typically includes:

• User acceptance testing (UAT)
• Operational acceptance testing (OAT)
• Contractual and regulatory acceptance testing, etc.

The bottom line is that acceptance testing is crucial for assessing the frontend of the system, as it provides a user’s perspective on its functionality and usability. The goal is not merely to confirm operational status but to ensure an optimal user experience.

Moreover, it’s worth highlighting that this phase of testing demands meticulous attention to detail and significant expertise. During acceptance testing, the aim is not just to «click around the layout to check if the design is satisfactory,» but to meticulously scrutinize the entire project’s quality. The objective is to maximize convenience and user-friendliness.

Functional and non-functional testing

Testing methodologies can typically be categorized according to what is being examined:

• Specific code functions or user actions (functional testing)
• Broader system parameters (non-functional testing).

For instance, performance, scalability, and security tests are categorized as non-functional. The outcomes from non-functional requirements testing offer insights into the quality level of a website or web application.

Static and dynamic testing 

We also consider if testing is static or dynamic, which depends on whether program code has been executed. 

• Static testing does not require code execution. Such testing is mostly suitable for documentation verification. 
• For dynamic testing, you need to compile and run the program code. Both functional and non-functional testing can be performed during code execution.

Regression testing

Regression testing is a type of functional testing that is particularly critical for web projects that frequently undergo changes, especially those that incorporate new features. During regression testing, testers search for defects following major modifications to the code. It is common knowledge that software alterations often result in the recurrence of old errors and the emergence of new bugs. Occasionally, individual features or entire modules of the web system that previously functioned seamlessly may cease to operate effectively.

Regression testing can be quite comprehensive and labor-intensive as it necessitates detailed inspections. Often, as part of regression testing, specialists rerun past test sets and verify if previously rectified errors have resurfaced.

The impracticality of expending excessive time manually repeating the same tests during regression testing has emerged as one of the strongest arguments for test automation. Further details on the automation of regression testing, among other types of testing, will be discussed later.

Types of Testing of Web Service Parameters

This overview will not encompass all forms, but rather, will concentrate on the primary ones:

1. Security testing

This kind of testing assesses a website or web application’s capacity to safeguard sensitive data from unauthorized access. Remember, when it comes to ensuring security and conducting the corresponding tests, there can be no such thing as «too much». Consequently, the purpose of security testing goes beyond simply determining whether a web service is protected; it’s also aimed at evaluating the extent of such protection.

The security testing process could include the following components:

• Vulnerability Testing
• Penetration Testing
• Compliance with web development standards (such as OWASP)
• Authentication and Authorization Testing.

Choice of security tests is guided by the unique features of the project, simulation of probable threats and attack types, among other considerations. For example, where a web service interfaces with a database, a SQL injection test might be suitable. This test determines if it’s possible to construct a user-controlled SQL query without proper input validation. By employing such testing procedures, potential vulnerabilities can be proactively discovered and warded off.

Furthermore, the choice of security standards to adhere to is made by both clients and web project teams. This decision significantly influences the scope and focus of security testing. Fundamental security standards are ingrained in the frameworks utilized in web development. Projects that prioritize security often go beyond basic requirements and adopt more rigorous standards. Apart from the aforementioned OWASP, prevalent standards include ISO/IEC 27034-3, ISO/IEC 27001, ISO/IEC 15408-1:2022, PCI Security Standards for the payment industry, and others.

2. Performance  testing

Performance testing is conducted to discover how swiftly a website responds to user requests and how much traffic it can withstand before crashing, measuring key metrics such as website loading time and system response time. 

This type of testing provides a wealth of insightful information about the current condition of the web service, its performance, stability, and resource consumption across various user scenarios and load states. In essence, performance testing serves as a ‘moment of truth’ for a website or app, revealing the operability and stability of the web resource under a range of conditions.

In the context of website performance testing, we aim to validate several characteristics, including scalability and load tolerance, among other factors. 

Performance testing encompasses a variety of tests:

• Load Testing – To assess response times to a range of requests, ensuring that the web service performs optimally under typical user load.
• Stress Testing – Executed under loads significantly higher than the usual user traffic, this test reveals the true limits of performance, stability, and exception handling.
• Stability/Reliability Testing — As the name suggests, this test examines the stability and performance of a web service during extended operation under normal load.
• Volume Testing — This involves evaluating the performance of the web service under conditions of increasing data volumes.

Scalability Tests are also a vital component of performance testing; designed to determine whether performance indicators improve as the number of available resources expands.

Performance testing further incorporates Configuration Testing or Portability Testing. These testing procedures are designed to evaluate the performance of the web system under various operating systems, hardware, and software configurations.

3. Usability testing 

The purpose of usability testing is to make sure that the user interface is easy to use and understand. Accordingly, various situations and entire scenarios are simulated that reproduce typical user behavior.

4. Accessibility testing

Software testing is changing in response to societal and IT sector demands. Accessibility testing is implemented to confirm the accessibility of web resources for users with disabilities or impairments.

5. Compatibility testing 

The incompatibility of a web system with other software is a common flaw that often causes failures. For web resources, compatibility with operating systems and web browsers, including mobile browsers, is vital.

Specialists use compatibility testing to ensure that the developed web service works on various operating systems and web browsers, including those designed for mobile devices.

6. Documentation testing

Such non-functional testing is required to verify the completeness, validity, and correctness of the web system documentation. To learn more about software documentation, you should read one of my previous articles.

Manual and Automated Testing

I have outlined numerous forms of testing and the principles upon which they are classified, so now let’s  consider how we can further separate testing based on the degree of automation involved. Automated testing services may already be familiar to you.

In essence, testing is divided based on whether the process is automated or not into two types:

• Manual testing —  when testers perform tests manually 
• Automated testing — accomplished using specific software, where the execution of tests and the comparison of actual results vs. expected ones are conducted automatically.

As with any complex area of activity, automated testing is further subdivided into several types: 

1. Code-driven testing – This type of testing is performed at the software modules, classes, and libraries level. Automated unit tests, which are developed by programmers, fall into this category.
2. Graphical user interface testing — this form of testing is done with the support of dedicated test automation frameworks, where user actions are generated. The response of the web service to these actions is monitored and compared against the project specification.
3. Automated API (Application Programming Interface) testing — Specialists often utilize specific frameworks for this kind of testing, typically for interfaces designed to interact with other software.

This wraps up our introduction to classification and various types of testing. Let’s now shift our focus to automated testing, delving into some practical tips for your web development projects.

Keep in mind that while automated testing can save a significant amount of time, effort, and cost when used in appropriate contexts, achieving complete, or 100%, automation in testing web projects remains impractical for quality assurance purposes. There remains a crucial need for a skillful balance between automated and manual testing, both of which have their own set of advantages and disadvantages.

As a rule of thumb, larger projects often incorporate both manual and automated testing. The greater the project’s size, the more likely you’ll need to increase your reliance on automated testing due to its cost-effectiveness. I will illustrate this point with an example later.

Let’s examine the reasons for conducting manual testing. This method is particularly useful in situations where testing needs to commence immediately, yet there are no existing automated tests. For instance, verification of minor system changes can begin right away without the time-consuming setup of automated tools. 

Here are some specific instances where manual testing is either the only or the most suitable choice:

• User Interface (UI) testing — Comprehensive testing of the UI can only really be done manually. Manual QA engineers can detect interface defects and compatibility issues across popular browsers, including mobile versions.
• User Experience (UX) testing, including usability testing — Manual testing is often the best approach for many UX aspects. A manual tester’s report mirrors potential user feedback, covering a broad spectrum of possible oversights, from errors in domain logic to intricate details about a website or web app’s user-friendliness and comprehensibility.

One area where manual verification provides distinct advantages is in Exploratory testing. This form of testing, which does not rely on a predetermined test plan, aims to uncover unexpected issues that automated testing scripts will very likely overlook. Manual testing allows the exploration of various app features through complex and unforeseen user scenarios, beyond the reach of standard test cases.

Experienced test engineers can even devise improvised scenarios, with ad-hoc testing offering substantial assistance.

However, manual testing does come with its drawbacks:

• Human error — Some defects may be missed.
• Laborious repetition — Often, it is simpler to execute a series of standard automated tests than manually retesting a web project after even minor modifications.
• Limitations on load testing — Simulating a high volume of users manually is not feasible.
• High time consumption — A large number of manual tests can significantly slow down a web project.
• Unsuitability for large websites or applications — Imagine the number of manual testers required, the effort needed to organize their testing, and the extensive work involved in processing the results.

There’s a lot to factor in when considering the cost implications of manual vs. automated testing.

Firstly, whilst the bulk of manual testing costs are derived from the salaries of QA engineers, in fact, in the short term, manual testing often proves to be more cost-effective than its automated counterpart. This is because the process of developing automated testing tools, using automated testing services, and providing the necessary training for their use can all be quite costly. However, in the long term, especially when repetitive tasks are involved, automated testing will very likely end up being much more cost-efficient.

Let’s look at some scenarios that highlight the costs associated with different testing types. 

Typically, the hourly rate for a manual QA engineer is roughly half of what you’ll pay for an AQA engineer; who is, in essence, more of a programmer than a tester.

So, let’s consider you’re developing a web project that can be tested by a single manual QA engineer in 160 hours. Every major update to the project requires retesting via regression testing. If we assume you’ll release ten updates over the next year, you’ll need 1600 hours of manual testing to ensure a high-quality web system.

In contrast, an AQA engineer could potentially create automated tests to meet the same objectives. In such a scenario, the workload would be about 200 hours. Yet, with each project update, team members could run these automated tests with a simple click.

While these 200 hours of AQA engineer work might be substantially more expensive than 160 hours of a manual QA specialist, it’s a one-time investment, and the potential savings are self-evident.

On the other hand, if you have a small web project that, once developed, won’t undergo changes for the next few years, automated testing is not necessary. In fact, the cost of implementing it would be disproportionate to your project budget.

Ultimately, your choice of testing methods should depend on the size and unique characteristics of each project. We will delve into the specifics of how to make these decisions in the following sections.

The accompanying image will further illustrate the intricate nature of testing.

Each type of testing serves a unique purpose in web development, and each is integral in maintaining high-quality standards for websites and web applications. By conducting a comprehensive set of tests, developers and customers can ensure these digital platforms meet user expectations.

And it’s important to emphasize that different types of testing are not mutually exclusive; they’re not alternatives where you’re limited to using one or the other. Typically, multiple types of testing are used concurrently, complementing rather than replacing one another. The correct application of each type of testing can significantly enhance the quality of a web project. Major industry players such as Google or Facebook employ an extensive range of the testing types we’ve discussed across all their projects.

In my experience, automated tests become beneficial for projects that involve 800 to 1000 development hours or more. But, occasionally, even smaller projects may benefit from automated testing services. However, this usually applies to specific project categories, such as those involving data parsing from other websites.

Benefits of Automation Testing

Let’s continue our discussion on automation testing, by asking ourselves, what are some automated testing benefits?  In short, automating test processes not only reduces the possibility of human error but also saves significant working time. Cost reduction and improving project quality can also be counted among the advantages of test automation. 

Automated testing is particularly valuable in scenarios where the following factors are critical:

• Time Efficiency — Manual testing of extensive applications can be a lengthy and tedious process. In contrast, automated scripts, once written, can be executed swiftly with a single click. Consequently, you can obtain test results almost instantly, just a few minutes after the test begins. 
• Cost-Effectiveness — As explained earlier, automation substantially reduces manual labor costs. This is especially beneficial in the context of repetitive tests and regression testing. The ability to reuse test scripts for each subsequent project update is a significant cost-saving advantage.
• Load testing —  Automated testing allows you to swiftly simulate a large number of users, which would be impossible to achieve manually.
• Enhanced quality — Automated tests minimize the risk of overlooking defects and reduce human error, leading to more accurate testing.
• Safety margin in security testing — I’ve emphasized throughout this article that automation of testing critical components of a web system is highly recommended, and that’s particularly relevant to security testing. The ability to augment security testing with automated scripts that are repeatedly used and detect errors reliably is an undoubted advantage of test automation.

Now let’s consider why automated testing is important. Consider a large-scale project that is constantly evolving and undergoing changes. It’s inevitable that bugs will surface. Over time, these bugs may even multiply. Often, updating one part of the project inadvertently impacts another area, but the team may not immediately notice this.

It is scenarios like this as to why automated tests were specifically developed, where they prove exceptionally effective in testing interfaces and loads. Whenever a project is updated, automated tests are initiated and produce an error report within a few minutes. Based on this report, your project team can promptly rectify the errors, and there is no need for time-consuming manual retesting of the entire project. Therefore, the larger the project, the more frequently automated tests can be utilized, making for obvious automated testing benefits.

When to Automate Testing

So, you already know that as your project grows, you’ll need to think about applying automated tests. But when is the precise moment to incorporate automation testing? In a nutshell, you should introduce automated tests as early as possible for large-scale projects that are frequently updated and constantly necessitate regression testing.

A surefire indicator that it’s time to employ automated testing services is when there’s a high volume of bugs that persistently remain unresolved, creating an endless string of issues. Although there may be circumstances where a high number of bugs could be attributed to other quality problems in development (which we’ve discussed in a previous article), in most instances, a significant accumulation of bugs signals that the project is already in dire need of automated tests. 

Automated testing might not be as essential for websites that aren’t frequently updated, and it’s beneficial to evaluate each project on an individual basis. However, for large web resources that are regularly upgraded, the value of automated testing services is usually undeniable.

For example, if you wanted to set up a software delivery pipeline for your web system, you would need continuous testing. Such testing can only be automated. By conducting continuous testing, you can get immediate feedback on risks and defects in preparation for the next release.

Selection of Test Types for Different Web Projects

Different types of testing in software are better suited to different projects depending on their nature and scope. 

First and foremost, it’s essential to understand the unique characteristics and constraints of each type of testing. Professional project teams typically comprise experts who possess extensive experience in web systems verification.

Project priorities and specifications guide the selection of appropriate testing procedures and their application extent, all aimed at ensuring optimal development quality. Consequently, an understanding emerges about the optimal mix of types of software testing required for the project.

When and how do you choose automated testing over manual testing? Knowing that these processes flow better when you automate will help you know when to automate testing: 

• Regression testing 
• Performance testing, including load testing
• So-called white-box testing using unit testing tools  
• Code coverage analysis.

However, automation doesn’t justify the efforts invested in all testing processes. Here are scenarios where implementing automated testing can be overly challenging or simply not worthwhile:

• Unstable systems — If a web system is still under development, with frequent changes and no stable versions, automating testing is inadvisable. As such, it is not recommended to automate testing for web products in their early development stages.
• Infrequently used test scripts — The benefits of seldom-used automated scripts do not justify their creation cost.
• Analysis of software code and documentation, etc.

Small projects or infrequently updated ones typically rely solely on manual testing. This typically involves a tester clicking on each page individually and running them through specific services like PageSpeed Insights.

Such tests are mandatory for all projects as developers alone cannot effectively test their work. For medium-sized projects, a combination of manual and automated tests is often employed to maintain quality control.

For larger websites or web applications, it’s advisable to utilize both manual and automated tests, including unit tests. Moreover, it’s essential to separately test security, load, and other priority aspects of the project.

How to switch from manual to automation testing in a web project 

So, we’ve now well-established that the choice between manual and automated testing largely depends on the project size, and that for smaller web projects, manual testing usually suffices. However, as a project expands, the reliance solely on manual testing can risk both the development timeline and overall quality.

This brings us to the question of how and when we should transition from manual testing to automation. Here are the key steps involved in this process:

1. Identify the scope of the test automation — Begin by thoroughly understanding your product’s requirements and testing scope. Identify test cases or scenarios that are most suitable for automation, considering factors such as usage frequency and runtime. Special emphasis should be placed on critical functionality.
2. Choose Suitable Automation Tools — Select an automation toolset that matches your project and infrastructure requirements. Your chosen tools should support the project’s environment, language, and component reuse. It may be beneficial to consult a team experienced in various testing types across different projects to guide your selection of automation testing services.
3. Prepare the test environment — Automated web testing requires several components, such as servers and databases. It is essential to ensure these are properly set up before initiating the automated testing process.
4. Formulate a Test Strategy — Develop a comprehensive test strategy and automation plan, which includes defining testing goals, cases, schedules, and team members.
5. Develop Test Scripts and Run Automated Tests — Create test scripts, making sure they can generate the desired outcomes. These scripts should be reusable and modifiable throughout the testing process. Once ready, execute the tests and verify the results.
6. Evaluate the Results — Post testing, it’s crucial to evaluate the results and identify the root cause of any issues that might have arisen. This may involve troubleshooting, detecting defect causes, and modifying scripts to rectify these problems.
7. Incorporate Automated Testing into Continuous Integration Pipelines — Automated testing becomes most efficient when integrated with a continuous integration tool like Jenkins, which can autonomously execute tests and inform developers of the results.
8. Maintain Automated Tests — Continually update test scripts to reflect any changes made in the project, ensuring consistent results even with the introduction of new features.

The transition of a web project to automated testing requires careful planning, evaluation, and implementation. Ensure that all steps are followed closely to maintain project continuity, reduce errors in code, and achieve shorter release cycles.

How to reduce testing time: practical tips

The question of how to reduce testing time so as not to delay releases is always a concern of web development project founders and managers. Here are some practical steps you can take in this respect during automated testing:

1. Prioritize test cases — Start by identifying and automating the most critical test cases. In automated testing, and indeed in other types of testing, determining code coverage—the proportion of tested source code—is essential. Start with automating the main features, then move on to lower-priority ones. Gradually build up your automated tests this way. If the budget is tight, concentrate on covering the project’s crucial parts with automated tests. By automating core functionalities first, you save both time and resources.
2. Use reusable test scripts — These scripts offer the advantage of easy modification with minimal effort when changes are needed in the project.
3. Implement Parallel Testing — Parallel testing lets you run multiple tests simultaneously, significantly reducing test execution time. This method can be employed to run automated tests across different devices or browsers concurrently, thus reducing the total test cycle duration.
4. Optimize Test Environment Setup — By optimizing your test environment setup, you can save considerable time. Implement virtualization or containerization techniques to expedite the setup process.
5. Use Advanced Automation Testing Tools — Automation testing services and tools like Selenium can drastically cut down the time needed to automate test cases. These tools come with pre-built configurations, including browser and device settings, and offer user-friendly script editors.
6. Implement continuous testing — Continuous testing allows you to test features as they are developed or retested, speeds up the feedback loop, and detects bugs earlier in the development process. This method cuts the overall testing time and the number of defects that need to be fixed.
7. Use cloud-based testing — Cloud-based testing allows teams to run tests on a remote server, freeing up resources and reducing the time required to execute tests.

Optimizing testing, like any aspect of web development, calls for experienced judgment and adherence to best practices. At SECL Group, we are here to provide guidance on this topic and advise on how to reduce wasted time in automated testing.

Quality Assurance Team

There are many types of testers in IT. To exemplify this, let’s take a look at our project for the Kia Corporation, which involved: 

• Manual QA/QC Engineer — this team member is primarily tasked with testing new feature releases and executing user scenarios, including those for fresh designs.
• Automated QA (AQA) Engineer — This role entails automating as many test cases as possible, with the primary aim of ensuring that none of the existing functionality is compromised following any updates.
• Programmers — They are responsible for writing unit tests for all the code they develop, ensuring that all features function as expected post each release.

In accordance with IT Quality Assurance standards, a project of this scale typically necessitates a team of about 15 members. Here’s a practical tip for project planning: On average, you should aim for a ratio of one QA specialist for every 4-5 developers.

Conclusion

As I wrote at the beginning, you might feel that there can never be too many tests. But an experienced project team will be able to select the best combination of tests for your developed web system based on its specifics. 

Initiating automated testing as early as possible is the ideal approach, especially for frequently updated projects. Even if you didn’t implement it from the outset, it’s not a major concern, and many projects initially forgo automated testing to conserve resources. However, as a project expands, these teams invariably adopt automated testing services. Autotests can significantly enhance the quality of your system and solve the problem of an endless string of bugs.

We recommend consulting with experts to determine the feasibility and potential benefits of implementing automated tests for your specific project. It’s crucial to identify the optimal timing when automated testing services will yield the maximum return on investment.

Feel free to contact us for a no-obligation discussion on the necessity and intricacies of using automated tests. We will back our recommendations with empirical evidence, tailoring our advice to the specific needs of your project. This includes suggesting which types of tests to employ, indicating which are essential and which are optional. We can also assist in estimating your testing budget and provide a timeline for addressing any persistent bugs that may seem insurmountable. Our experts are proficient in automated testing and are dedicated to ensuring the superior quality of your web project.