engineers build framework or rectify the current one or upgrade the current framework.
The objective of the automation are:
- Test execution can be completed faster
- consistent and effectiveness
- Re usability makes better ROI.
- Reduce the cost of testing
Few basic criteria for Automation:
- Tool must support UAT
- Multiple releases should be carried out
- Less manual intervention
- Project should be in spiral model.
- UAT should be stable
- budget should be there to carry out expensive automation
Importance of framework –
- Flexible
- Reusable
- Easy Maintenance
- Better ROI
Examples of automation:
- sending bank statement
- credit salary information
- Any manual process which is repetitive and tedious.
For automation , we need to pick up a tool and script using underlying programming language. Example of the tools-UFT/QTP,Selenium,Silktest,RFT,Test partner.
Automation Testing is activated if there are two conditions
1. if you are doing regression testing for single or multiple dataset
2. if your application is stable enough to perform the test repeatedly
Steps for Automation:
- Collect the acceptance testcases/ regression testcases
- Record and code the script
- Improve the testscript in order to reliability.
- Parameterize the hard coded values
- Separate the test data from code by moving variables to a separate files
- Create utility functions
So creating a basic framework needs the following things covered:
- Prototype components
- Test designing
- Create test interpreter
- Create AUT specific test drivers
- Add repository
- Add logger,reports capabilities
- Create test documentation.
Phases of Automation Testing:
- Development phase
- Execution phase
- Maintenance phase
Why Automation fails?
Coverage:
Coverage is the most important feature of any Automation success. In my view a simple framework creation which can cover more is much more acceptable than a complex framework which covers less. So decide correctly by which approach you can get more coverage.When we get a big flow to automate , generally seen very less time is given to make it done.Management often misunderstood the term automatic and automated. Allocated time is not sufficient as the development team pushes the patching window or crosses the deadlines.This time an automation test engineers thinks to find out all the reusable components so that the development time becomes less.We use a lot of loop to do same activity again and again. We try to automate what ever is possible. Our objective is to check an outline to check the overall completeness.Smart enough to complete the task???
Mostly No!!!
standard become important as more people will be developing in a very short span of time. Getting feedback from the framework users will help for latter acceptance. Automation Test development is a dedicated effort, there is nothing called free lunch.(often management asks for free or extra time of a tester to automate the flow.) Yes,we can automate in a shorter span of time. but shall we??
No!!. we must not automate poor tests.Efforts put into automating high value,high quality tests will improve the test execution and provides greater pay back. Similarly, simply automating manual testcases may not give us better ROI or a best automation suite.Select the tests judiciously.
Don’t try to automate very complex flows at first.Automation should go in a slow but steady pace.Identify the proper testcase while automating the complex flows. If you automate chaos,all you will get a faster chaos.Smoke testcases are designed to check health of the builds,releases. This type of tests saves hundreds of hours for testers and developers. Smoke testcases are good place to start the automation development, as they run almost everyday,on every checkin,or on every build.This is a less value add for testers error prone and boring.
Extended environment and multitool Support:
Most of designed Automation is not environment neutral. They support only one Operating system or one tool specific. Please build a framework that will work for multiple OS or in multiple tool. This can be achieved through designing a loosely coupled system.
The second most important issue is data issue.Most of the time either data is not present or wrong data causes the execution to stop.Even using manual testdata sometimes gives wrong output. The third point may be data-hell condition. When we create a lot of modular scripts and try to create data for modular approach we create same data again and again.Say I need to change a set of data there is no mechanism to make the this latent change reflect everywhere. We have to open the child script datasheet one by one and change. there is no consistency at all between the data. These causes problem when running the scripts.
What I mean that any test can be operated from any machine.Let us improve the framework little bit more. I can copy the master driver which will automatically create the folder structure add up the necessary dependent files and will be ready to run any testcases.
Business Process:
An Automation process is very much useful when a non technical person can create an automation testcase and execute the same. Basic building block of such kind of framework is to create business process. A non technical person might not understand your piece of code but very well understand the Business process. He should not bother about the under laying code.
Ease of Functionality can be achieved when we can give user a platform where user can plug in and plug out any extra feature he/she required/not required respectively.Like Manager , business analysts and functional people can control global testing without knowing the tool.This will truly become Skill Utilization too.The more we make our framework user friendly , by providing Easy controlled and controllable actions ,More it will be a go to tool for all.This feature will allow business user to scope a set of testcases or scenario for a specific business flow.
Easy Reporting and Self documentation:
This is one more improvement that can be implemented is the self documentation of steps performed. Gradually this feature will give proper navigation step validation and data validation in future.
New framework should support video recording of each and every testcase execution report. Based on the requirements, framework should support the pass video or fail video along with steps and logs.
Re usability:
Try less coding and try to reuse most of the possible combinations. This will reduce ambiguity. Maintenance is much more easy when you have less lines of code. Redundant lines sucks a lot in later phase.Most of the time after a successful completion of development , our scripts got stuck.This is the time when we have very less no of time to complete the task. 80% people say that development of the scripts are not proper. Even I found a instance where “for” loop sucks if it got stuck in between of the loop.Understanding the loop also a big factor for the execution master who is in charge of execution. sometimes we think if we can write statements instead of loop.Basically we look for consistency of testscripts at this point.We need to structure our automation regression test for most successful execution.The main objective of regression suite to give confidence and assurance,not detecting bugs.
Maintain Standards:
Standards can be of different types:
- Coding standards
- Naming standards
- HIPPA or SOX guideline standards
- Folder Structure standards
- Library Architecture
- Easy plugin adapters
- Easy interface for framework
- Distributed Nature standards
Please use this whenever possible. Try to make a practice. This will reduce ambiguity inside the code. It is easy to understand when we have a proper Naming convention or we follow a proper guidelin.. Also we can use proper commenting to get consistent test script.Framework should support/Accept only the consistent script.When we are preparing a data driven framework or testing a script for multiple data set, we require lot more data files to be processed. Framework must support a plug ins for data for driving the script or data pool to supply during testing of same scenario for multiple data.When a test engineer wants the same test to be executed on different data.Framework should support him with easy excel plugins of data and also give flexibility to repeat the test how many times needed with different data.
This is continuous from the above point.How a non technical person will interact with our automation code? They need an interface to design the test. Quality center is a classic example of interface. But that is costly.Framework should have an interface where functional people can use that and prepare their test.
Framework should have flexibility to design and execute the test in different machines without any problem. Design and execution should be in Distributed mode.
How to put automation Standards?
Automation process starts with capture and playback. It is the beginning. We need to upgrade our script with validations,Synchronization points, conditional points need to be added. We have upgraded from mouse based co-ordinate testing to proper element testing. Slowly our code base increases.
Apart from our regular test engineers,automation will be used by several other engineers(even functional test engineers too).It is important to have some standards in place.More and more standards , if placed,automation will be streamed lined and this will reduce human dependency and human error.This will ensure the same meaning for all. Reduces the code duplicity.Which means the reduce the effort of coding ,higher ROI. A structure if well documented and standardised yields more results and quality scripts than a quick undocumented and dirty approach. it will help to scale up the automation.Expensive tool is not necessarily needed or the best tool need not be picked. With the available tools, if we are able to put little standards , brings wonders.
Another important aspect of automation introduction in agile to provide coverage so that development team have safety net around them to change the code.Regression testing is only successful if we have such coverage and safety features around development.
Continuous integration using jenkins etc and test a build in automatic way will ensure the hygiene of development code. One can opt for any opensource tool to automate the testing with supporting infrastructure.
Quality of automation testing needs to be established.Automation should have many folds. The major aspect again smoke or monkey way of testing the new build. Even the basic security ,sanity and deep drive into application may come handy in changing environment.
By putting framework in place we can get quality scripts,reusable code,stability and extend ability. Putting standards,guidelines automation will be matured enough to go in unattended mode. Else management needs to incur the monitoring cost.
The standards can be divided into three sections.
- Development standards
- Cosmetic Section
- Infrastructure Section
Development Standards:
- Naming conventions for variables –camel casing.
- Naming convention for global variable-must start with g
- Naming convention for function name,class name follow starting of word with capital.also function needs to start with the return type.
- Documentation and open discussion-A great discussion often leads to great findings. Follow the principle -If you don’t know ask!!
- How to guide to write a test script
- Standards for naming an object
- Data file construction
- Constant should be all upper case
Cosmetic section mainly talk about the spacing. proper loop structure spacing. It is said code must be human readable first and then machine readable. Header is an important aspect of cosmetic section. Each function or class should have a header section to talk about the same. Comment must be included to make the code human understandable.
here is a set of does not don’ts
- use of tagging gives better search facility.tagging can be mentioned in the commented area.
- only one statement per line
- maximum of 80 to 100 characters per line so that testers do not need to scroll right to see the content.
- indent sub items’’
- use of proper parenthesis of using operators.
- turn on option explicit statement this will force variables to have certain values instead of junk,null values
- parameters needs to be defined as pass by value or pass by reference
- keywords like if then etc should be in the same line.
Naming convention:
Local- Use camel casing-small letter “l” in the at the first letter of the variable.Like lFirstName. It denotes that it is a local variable.
Global-Same, if the scope is defined by global just use a small “g” at the beginning of the variable.Like-gCustomerNumber it denotes that the variable is global and accessed from anywhere.
CONST- Use all capital words for variable that are constant in nature.Like-CONST CDATE.
Functions should start with FN_ followed by function name like- FN_getDate()
Similarly for Action is QTP should be names as AN_Action name like-AN_CreateCustomer
Append a small r before function and action if they are reusable in nature.
like-
rFN_getDate() or rAN_CreateCustomer
The script name can be varied from framework to framework. Best practice is to give a module name before the script name. Like-CO_CreateCustomer(Co stands for Customer operation). It must be capital letters.
Function Library should start with FN followed by function library name and ends with a lib.like-FN_CreateOrder_lib
For a small project we may go with .mtr but when project goes big it is better to use .tsr object repository.ORs should start with the Module name like-CO_Customer.tsr
a structured approach is always needed to work properly in a team together. It is important to structure the test assets(name,testdata file,scripts,testcases,config file etc). Another best option to work with multiple environments is to use relative folder options.
Second approach is to share the function lists created by different automators. functions should be as generic as possible.
The best way to achieve this is to use
- One global function library
- Use business specific library
Store all the structure in the version control system.Extra folders could be-
- Documentation-Related to framework
- General-Function to initialize the framework
- Language-Functions specific to any language
Advanced features/Recovery Scenario:
Any technical change should be caught by framework itself and handle properly.In this way also try to capture the delayed fuse bugs like -memory leak wild pointers etc.While advancing forward , framework should have ability to handle the unexpected exceptions. Every tool has its own way of handling exceptions.Use that to configure recovery scenario.Please handle known issues with code itself. As the more intelligence you will put into a framework..it will slowdown the process. It is kind of attaching recovery Scenario to the test.
Ease of debugging, logging and feedback system:
An automation should give a easy debugging feature so that the code executed by any test engineers or a customer can debug the code easily. One of the best method could be linear code generation.
Always listen or give proper value to the feedback of the users.They may pin point lot of relevant process and practice issues.Even they may advice standards.This is very important for long term success.
Look for better ROI:
Easy Refactoring and update:
The maintenance process of a framework should be defined previously. The process to update a script /asset/code has to have a prior approval process and version control mechanism.Refactoring is an essential part of automation testing in order to restructure of test code or test data for ..
- Increase internal efficiency
- Stability
- Support upgraded technology
- Support change in functionality
- Increase maintainability.
When the script is old and running for say 3/4 years .It is extremely difficult without a proper comment or explanations.At this time we try to understand from code itself what it is doing. Remember the more number of lines of code is costly during maintenance .
In agile, we often automate some flows which are unstable. Actually, there is no value add while automating unstable functionalities as maintenance overhead is huge.
This is the basic building block for an automation framework. Please visit Part-2 for more advanced feature.A better communication between testers and automation engineers is necessary to ensure framework design meets testing objective. Any test framework design effort is very serious factors for project development. A good planing ,scheduling and budgeting is required. It is important to get feedback from testing team. This may lead to greater learning and idea generation. there may be many cycles before perfection.Accept the short fall and work towards it.If the project is complex ,big and we are at a tight schedule.We need to scale back .Communication is the key here.
While designing , we need to make sure-
- Find relevant information,code,examples from internet. It is available in different form.
- The automation many be developed by a group and maintained by another group. Transparency is the key.
- While designing the framework, we need to take care business issues,people and process issues and technical issues.
