How to Effectively Prepare “Test Bed” and Minimize the Test Environment Defects
Test Bed / Test Environment Setup Challenges and Best Practices:
On several occasions the testers find their defects getting rejected for environment issues or they find themselves constantly replicating the defects for the similar reasons. While opening the most number of defects must certainly be one of the personal benchmarks for every tester, most testers must also emphasize on having the most number of valid defects.
How is this achieved? Apart from the other aspects like planning a variety of test scenarios and understanding the line item thoroughly,a good amount of time must be invested in setting up the test bed or test environment. Secondly, despite having an estimated amount for test cases planning, testers must also focus their energies on creating effective test data.
Personally, being a part of the audit process, I have observed that the most number of valid defects are found when a good amount of effort has been invested in creating the test bed or test environment in a correct manner and when the tester has a thorough understanding of the kind of environment needed. Also the kind of test data supplied to the test environment can exposure some very serious flaws in the code/ feature under test which can severely impact the quality of the product.
This article talks about what exactly the test bed entails: It being a two-step process of test environment setup and test data setup:
Part 1) The earlier part of the article will discuss general process of test environment setup, most commonly faced setup problems faced by tests and pointers to keep in mind while creating a test bed in lieu of these challenges.
Part 2) Having said so much with respect to the test bed collectively in this article, it was worth to throw some light on the test environment maintenance aspects as well. The latter part of the article discusses the second part of test bed setup which involves the test data the approach to set it up and some effective test data management techniques.
With a constant “big bang” in software development and testing, there is an increasingly large focus on adopting various methodologies to make the overall Quality Assurance process transparent, efficient and adequate. Various Quality audits are conducted across organizations to ensure that the performance of testing team can be suitably evaluated and has measurable results with the metrics identified at the initialization of the testing cycle. These results make it possible to identify where a particular team stands in terms of ensuring optimal quality for the software they test. These reports also help the team understand the opportunities for improvement based on the observations made during the audit.
Needless to mention, a very obvious metric for any test team would be with respect to the total number of defects opened versus thenumber of defects that are valid. Hence one of the questions that obviously pop up are – What is the basis of trying to discover any defect? Stated another way, what is the foundation on which a defect can be found?
The answer is unanimous – test bed and / or test environment setup. There are quality benchmarks set within teams to reduce the defects which are rejected as a test setup error/ User error, invalid configurations or in some cases the defects that arise as escapes from a particular team due to unavailable configurations, untested configurations.
Let’s make a start with taking a closer look on defining what a test bed or test environment is.
What is a test bed and test environment?
In a very generic sense a test-bed could be defined as a kind of development environment whereby the implementers of code or modules have the freedom of testing their modules without any disturbances from the testing team, in absolute confinement. However a test bed is not only specific to a development team. In perspective of a test team or a tester, since the test bed is nothing but a platform identified for software / product testing, it is also interchangeably called a test environment. Any test bed or test environment would have to be configured in accordance to meet the identified test goal for the application/ product / software under test. In certain situations a test bed would be the collation of the test environment and the test data it operates with.
Components of a test environment:
Any test would have its specific test environment requirements, but in a very broad sense any test bed / test environment will comprise of the hardware, software, and the networking pieces to support the required configuration at the minimum to drive and conduct the particular test.
Commonly faced Challenges with respect to a test environment
It’s a well known fact that a reasonable amount of a tester’s time is consumed by environment problems, which in turn affect the productivity and test schedules. Although the kind of challenges vary for each and every test team, some of them may be common.
Some key challenges that are commonly faced are:
#1. Remote environment:
The test assets or environments are mostly placed geographically in sites that are remote to the teams. This is one of the most commonly faced challenges for test teams as in case of any problems that may arise pertaining to hardware, firmware, software, networking , etc.. The teams consuming the assets would need to heavily rely on the support teams in the location where the assets are present.
In the same lines if some asset needs a firmware upgrade or a build upgrade, again the test team may need the support of the support teams owning the environment by opening support tickets. This may also hog up considerable testing time and delay schedules especially in cases of time-zone differences.
#2. Combined usage between teams:
Most often than not, development and test teams use the same environment assets. Although the general norm defines that development, test and production environments must be separate, in actuality this ideal scenario is very rarely achieved. It becomes extremely cost unfriendly for organizations to procure separate resources for each team. Hence most organizations mandate the common usage of environment between development and test. Added to this, if the development and test resources contend for the usage of the same assets at the same time, it leads to chaos and disagreements within members.
#3. Ineffective planning for resource usage for integration:
In some cases for scenarios needing end to end testing whereby there is an integration of two or more components to function together, again there maybe a requirement to have the common usage of resources between test teams. Ineffective planning with respect to usage is a large contributor of environment turning instable, besides conflict between teams. The most evident effect of this is that an issue that is noticed for a particular once or twice may produce a completely different behavior in the following runs for the same scenario. If a defect is already opened for this, there are high changes of it not being accepted by the development as a valid candidate for a fix.
#4. Complex test configuration:
The test bed or test environment configuration is sometimes too complex. This will pose several challenges as the test team will need the required skills to understand the needed configurations. At times there is a lack of knowledge base available for the tester to be able to come up with the required configuration. In such cases the testers may themselves induce an error in the test-bed by configuring it incorrectly. This would greatly impact the test case and results it produces.
#5. Elaborate setup time:
In certain other times, for each test case the test setup may be far too elaborate each test case identified. This could be due to a wide variety of co-existent technologies that need to be coupled together or multiple components to work together in cases of integration testing. In these cases, each of the components have to be working perfectly to ensure consistent results, as one component may form an input to the next.
Some suggested best practices for setting up a test environment
We’ve taken a look at the broad outline of challenges that a tester is faced with before or while beginning the test execution. Most of us have faced one or more of these issues at some point during our project milestones. These challenges have existed and will possibly continue to exist in varying degrees because an idealistic situation doesn’t exist.
Given that setup challenges are part and parcel of a tester’s job and are inevitable, here are some suggestions on how to effectively prepare the setup for testing. This could help minimizing the defects that may originate out of setup issues.
Tip 1: Understand the test requirements thoroughly and educate yourself
Always start with the basics and with the most obvious! When a specifications document or a use case document is rolled out by the development team, the invariable step for the test team is to understand the line item requirements and then prepare a test case document detailing the test cases. While the test planning is being carried out, it is the best practice to also include the detailed test environment information in the test case document. No guesses to the fact that the tester will then spend sometime analyzing what test environment maybe required and accordingly the needed configurations. This can be achieved by talking to the development team / architects in order to build a good knowledge base. This would not only save sometime in the execution cycle, but will also help a tester allocate his execution time effectively between simple and complex tests.
Personally, a good outcome of this is that many of us discovered setup issues (that would inherently prevent consistent test execution) in the very beginning of the cycle, which rendered us time to channelize and acquire the required help to get those issues fixed – thus not extending the test cycle beyond unacceptable periods. Another positive impact this would have is that, this would greatly improve the test team’s knowledge and would prevent unnecessary defects. Although this practice summarizes almost all the practices that are inherently needed to cope with the test setup challenges mentioned above, it’s still worth to make a mention about the other tips.
Tip 2: Checking the connectivity
Another most important checkpoint is to make sure that the resources or assets you intend to use for testing are reachable. In case the system needs to be run integrated with other machines, check their connectivity with each other by using ping or telnet.
Also if the systems need to interact with each other and are behind firewalls, make sure they can authenticate through these firewalls using Basic security Options (BSO) and check for any proxies as well. In case you notice that some machines are not reachable or need BSO authentication, appropriate service requests can be raised to fulfill the requirement to the support team. This is particularly useful when the environment is in remote locations and will also avoid escalations with respect to the machines and systems. In case the test team requires access to any resource or repository, this would help in pro-actively determining the same.
Tip 3: Checking the network and / or storage
This is almost an extension to the previous tip and would need a certain other more checks with greater technical depth. Make sure that testing you require has the needed bandwidth and if your testing needs an internet connection. Also make sure you find a way to verify that the network topology between systems and resources are correct.
Secondly if your testing goal implies the need of any storage make sure that there is storage and network connectivity. Mostly it is an administrator’s responsibility to have this in place, however it is also a great value add to have some working and functional knowledge of the same.
Tip 4: Check for the required hardware and software, licenses
Many a times it so happens that testers begin execution on the systems without checking the needed hardware and software that maybe required. As a result of this many a times a tester realizes almost during the testing cycle that certain functionality is available only on a higher level of hardware or software / firmware. At that time the tester will flag a blocker in his test effort which eats up considerable testing time. Hence it is a priceless practice to have a checkpoint to make a note of the hardware and software that is needed priorly.
Many times there maybe a downtime involved in upgrading the hardware / software, which all boils down to Tip 1 where a tester must involve in proactive planning regarding the hardware. Some of the software’s might require licenses which may require approvals and actions from the legal team. This being a process driven action, might again take a number of days to be fulfilled, which needs to be planned.
Tip 5: Browsers and versions
The testing you do has to mirror what an end user will perform. He could be testing on a particular browser on the latest versions of all browsers. Hence it’s mandatory to identify the different kind of browsers that would be used for testing and have them installed in your own local test setup. Secondly, also identify what versions of browsers need to be used for testing. A good practice would be to start with a browser of the lower version, thereby ensuring backward compatibility and then upgrade to the latest version.
Tip 6: Planning out the use of the test environment.
Given the fact that the test team will never have a situation of having their own test resources, systems and assets – it’s one of the major milestones in the test planning to have an effective usage of test resources. This is particularly required when more than one team has to access the same set of resources either because of an end to end scenario which comprises of two or more components working together, or a situation where the test setup is too elaborate or complex to be replicated very easily and there could be multiple members within the same team having their test own goals with the same setup.
A good practice would be to work out a time-sharing approach whereby a certain team or person uses it for the earlier half and the remaining people for the latter half. There can be sometime in between that will be a common where each of them can run independent tests that will not hamper the other. Doing this will not only reduce the chaos and conflicts within the members, but will also ensure behavioral stability of the environment for a longer duration.
Tip 7: Automation tools and their configurations
As we know, every line item in testing will have a few repetitive tests that will be a part of the regression cycle which will need to be automated. The test team must identify what kind of automation they would like to do and the necessary tools for it. Although this necessary doesn’t need to be a part of the environment preparation, I would still list this as a best practice to have the automation tools identified and configured accordingly. This would completely depend on the tester’s discretion on when he wants to perform this activity as this is not a mandatory factor to ensure test readiness.
Conclusion:
These tips and tricks can form a good yardstick and footprint to ensure test environment readiness for testing. Doubtlessly, each team is faced with their own unique set of challenges and the tips above can be adapted and customized to suit their own respective needs.
In fact the source for jotting out this whole skeleton of tips comes from one of my assignments where I faced severely complex setup issues and took me almost a year to even begin testing! Although the limitations in the test environment were out of my control, I felt a lot of those issues could have been reported earlier if I had applied these tips. I have been applying it for every assignment that comes my way since then and this skeleton has helped me a lot in pro-actively finding setup issues and channelize my efforts to get them resolved.
About the author:This article is written by Sneha Nadig. She is working as a Test lead with over 7 years of experience in manual and automation testing projects.
In part 2 of this article we will see test environment setup and maintenance process and test data preparation and management tips. Meanwhile feel free to post your test bed preparation queries in comments.
No comments:
Post a Comment