If you are new to automated website testing it might seem pretty hard to get a decent overview about what there is that you should know, what different options you have for your automated web application testing and what else there is to consider. In this post I will try to give a good overview of the different approaches and the pros and cons of each. I hope at the end you will know which choice of automated web testing is the best one for you. If you have any corrections or improvements – feel free to comment. If you want your companie’s tool added to one of the tool lists, please send and email.
If you are not interested in all the information around this topic: Go directly to the List of Automated Web Testing Tools.
Topics of this Post:
- What is Automated Website Teting?
- Automated Website Regression Testing
- General approaches
- Frameworks – For Programmers
- Big difference: Selenium and the Selenium IDE
- Automated Web Testing Tools Comparison
- Other types of automated web testing
What is Automated Website Testing
It does not exist an official definition of “automated website testing” but most people refer to functional testing of websites and web applications when talking about this. The more features a website has the more time it takes to verify on a regular basis that every feature is still working as it is supposed to. To check manually whether or not all the features of your website are still working you probably use the same pattern for each feature again. E.g. to check whether your login does work, you first you go to the login page, second you put in your username and password, third you hit the submit button and fourth you check whether your login was successful. So instead of taking the same steps every week or so again, automated website testing is an approach to automate these time consuming tasks. For every feature for which you want to make sure that it is still working correctly, you define a test case which you can run every month, every week or every day.
The important difference to common software testing approaches is that you actually test the real frontend/gui of your web application in an browser while common software testing will test on a lot more abstract level functions with a certain input if the output is as expected. So if all your testcases pass you know that your different parts of your software work as they used to but you don’t know if the actual user interface works as it is supposed to.
Automated Website Regression Testing
Most websites continuously change their content, outlook and features on a regular basis. That is necessary to keep up with your competitors.
Those changes might be due to only editing some of your content or adding a WordPress-Plugin but especially if you work in an agile environment with software deployments on a regular basis regression testing becomes more and more time consuming. After every change of your web application you need to know not only if the new feature works as supposed to but also if all the previous developed features still work fine, too. To test the whole application after every deployment – that is regression testing. The first part of this post is about how to automate this time consuming work which is what we call automated website testing. Later in this post we will also go over the other types of automated web testing. Have a look at the table of contents and jump directly to the topic that you are interested in, if you wish.
General approaches on automated website testing
There are several different approaches on how to automate your web application testing. Each one has its own pros and cons and which one is best depends on your needs, your technical knowledge and resources.
- For programmers: If you are a skilled programmer you are most flexible if you just write your own code either with Selenium (available for different programming languages) or JavaScript. Your scripts can be executed on your own devices or you can use cloud services.
- Tools and software: There is a huge variety of tools for automated web application testing available. Many of those tools are also used by writing scripts, so are also basically made for programmers. Other tools can be used without any programming skills. One of those tools should be your choice if you don’t want to write your own code. These tools all have different advantages and disadvantages, scroll down to see a comparison of different tools and pick the one that fits your needs best.
Automated Web Testing Framework
No matter what framework you choose for your web testing approach you need to have quite a bit of programming skills and technical knowledge. Also you need to have devices to execute your test. If you decide to run the tests on the same computer that you are working on this can be very annoying and time consuming.
Automated Website Testing with Selenium
This for programmers and for those who want to have total control. With Selenium your are completely flexible on how and what you want to test. With this framework you can automate almost every browser on any device you wish. The downside of this approach is that you need to provide all the hardware and you need to have good programming and technical skills to get your tests running.
If the mentioned downsides won’t bother you, Selenium is probably the way to go for you. Chrome, IE/Internet Explorer, FireFox, Safari on Mac, Ihpone or Ipad, Android and more are available for your tests. Every browser and device has its own hazards to connect with Selenium, but eventually you will work it out.
Selenium is available for the following programming languages:
- Java>/li>
- C#
- Ruby
- Python
- JavaScript
Run Selenium scripts in the cloud
If you want the flexibility of writing your own Selenium scripts but don’t want to provide your own device lab, this one might be for you. There are service providers who offer you to run your selenium scripts in the cloud, either on virtual machines or on real devices. The pros and cons are basically the same as if you run selenium on your own machines except that you don’t need to provide the hardware (but instead pay the cloud service).
Here you find a list of cloud services who let you run your Selenium Scripts in the cloud on virtual machines or on real devices:
Your company also offers this service? Don’t hesitate to contact me, so I can complete this list.
Automated Website Testing using JavaScript
JavaScript Frameworks: There are different libraries, frameworks and tools available that allow you to write your tests directly in JavaScript. Basically this approach has the same advantages and disadvantages as writing your tests directly in Selenium: You are very flexible but need very well developed technical skills and programming knowledge and also you need to provide the hardware yourself.
Big difference: Selenium and the Selenium IDE
The Selenium IDE is also very well known for automated browser tests. It is a free Extension / Add-On for the Browser Mozilla Firefox. The upside here is, that you can also use id without programming knowledge. Either you use the built in recorder to set up your test cases or you define them by using the GUI of the Extension where you can define every single step of your test case with command, target and value. Some technical understanding of html and css is recommended. The major downside of this approach is that you can only test with Firefox while web applications should always be tested at least on the most common browsers and devices.
Automated Web Testing Tools Comparison
| Name | Cloud or Local Install | Virtual Machine or Real Device? | Cross Browser Testing* | Responsive Testing* | Multiple Domains* | Chrome on Desktop | IE / Internet Explorer | Mozilla Firefox | Mobile | Screenshot Comparison | Automated Testing Schedules | Comment |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FRET | Cloud | Real devices | yes | yes | yes | yes | yes | yes | yes | yes | yes | Basic understanding of html/css is helpful to start with this tool. |
Your company also offers this service? Don’t hesitate to contact me, so I can complete this list.
* For Cross browser testing, responsive testing and multiple domain testing you will see a “yes” if you can run one test definition on various browsers, a set of predefined resolutions and multiple domains. If you can test on different browsers, different resolutions and different domains but for each you have to define a new test case or you have to manually resize your browser for every run on a different resolution it is not considered efficient cross browser/rensponsive or multiple domain testing.
For Programmers: Usage with programming/scripting skills
Your company offers a tool for this purpose? Don’t hesitate to contact me via eMail, so I can complete this list.
Other types of automated web testing
Now I will come to some types of testing that might some people call a type of automated web testing and others might argue that they are just similar in what they are called but actually totally different types of testing.
Automated Website Usability Testing
Usability Tesing or UX-Testing is ment to find flaws in your web application that hinder your user in performing their tasks or make it harder for them to understand your website structure or content. This might a dynamic dropdown navigation that your users don’t understand or links that don’t look like links and are never klicked. Ususally you would perform this type of testing by inviting a random group of test users and give them tasks to complete on your website. Usuall a moderator is sitting next to the test persons to ask questins or giving a little guidance at certain poinst. This is why this type of usability testing also is refered to as “moderated usability testing”. If set up correctly you most definately will find tons potential to optimize your website.
Now, let’s get back to “automated” usability testing. I would argue that you can fully automize usability testing. There are tools which claim to test the usability of a website fully automized. For example this tool from Google: XXX. In my opinion though it doesn’t help a lot for usability testing. May be those tools can help you to identify some total basic issues like to smal fonts or links that are to close together for good mobile usability but those tools will also attest websites good usability even if there are huge flaws and furthermore those tools will not give you any good hints how to improve your usability. You need real users to see what the usability issues for real users are on your website. Although, there are services and tools to help you scale or semi-automize your usability testing efforts by letting people remotely over the internet perform your usability tests. So, when people talk about “Automated Website Usability Testing”, most of them really refer to “Unmoderated Remote Usability Testing”. You can either do this type of testing all on your own by sending manually test-links and tasks to hand picked test users or you use an only service that for example even offers registered test users from a huge crowd. The results are usally provide as screen videos and completed questionaires.
Automated Website Usability Testing Tools
Your company offers a tool for this purpose? Don’t hesitate to contact me, so I can complete this list.
Automated Web Performance Testing
In this type of web testing you measure the performance of your web application over time. How long do your users need to wait for your server? How long does it usually take to load your website completely? Does your website performance vary over time or is it pretty consistent.
To perform this type of testing your will usually use one of the available tools which request your web application on a regular basis, record all different kind of performance KPIs for you to analyze and send out performance alerts if the performance KPIs reach critical marks.
Automated Web Performance Testing Tools
Your company offers a tool for this purpose? Don’t hesitate to contact me, so I can complete this list.
Automated Web Accessibility Testing
As the name of this type of testing already gives away, it tests the accessibility of your website. So, really you could say it is a really basic part of web performance testing. I don’t know of any automated performance testing tools, that do not include monitoring your website accessiblity since this is a vital part of your website performance.
Although, if you really only want to test your web application’s accessibility, you might be able to find tools for a smaller price than the performance testing tools.
Your company offers a tool for this purpose? Don’t hesitate to contact me, so I can complete this list.
Automated Website Load Testing
Website Load Testing has nothing to do with measuring the load time of your web site. Usually people refer to a test where they test how much load their website or server can take. E.g. large ecommerce sites would perform a load test before a relaunch to make sure that the new platform will withstand the expected amount of visits per second. Don’t perform this kind of test if your webserver is not managed by yourself. If your website is hosted by a third party web hoster you definately will want to talk to him before starting a load test – otherwise you could get him and yourself in trouble.
Your company offers a tool for this purpose? Don’t hesitate to contact me, so I can complete this list.
Automated Web Service Testing
A web service is basically an interface which is used to communicate over the web. Usually a web service is an URL that can be called with a certain set of parameters and you will get a response often in XML-format. This type of interface is very common for communication over the internet and web since it is totally indepenent of certain software and operation system. Often it is used with the HTTP-protocol. Testing a web service is a lot closer to usual software unit testing: You call a functin (the web service) and you check whether the response is the expected response or not.
To automate those tests there are two common approaches:
- With specialized tools (see list below)
- Using traditional software testing: For example you can set up in Java a JUNIT test-suite and for every test case you can use Java implementations of calling a web service an checking the result.
Automated Web Service Testing Tools
Your company offers a tool for this purpose? Don’t hesitate to contact me, so I can complete this list.
Automated Website Security Testing
To check the security of you website fully automated is probably better than nothing and may be you will find plenty of security issues using this approach. Still, a tool will not really “understand” your website. So an security expert who actually understands your website will know a lot better where the critical parts are and where there could be security issues. That means, if you run a website where security is very critical you should consult an expert to manually check your website. If you run a website that is mainly for information purposes like a blog or a company website just for information about the company one of the following tools for automated website security testing should be sufficiant.
Your company offers a tool for this purpose? Don’t hesitate to contact me, so I can complete this list.
Automated Web Api Testing
To be completed…