Managing Web Projects #10 – Testing the Product
Testing is the most important part of the whole web production process.
I don’t care what anybody else tells you – listen to me, not them!
At a company I once worked for (who shall remain nameless for obvious reasons!), we released a rather large website update. In normal circumstances our procedures were excellent (well, they had to be to maintain ISO 9001:2000 status) but in this case we were running late. Very late. So late in fact that the project manager decided that the version of the website we had on our development server was fine and we didn’t need to bother with loading onto the staging server for the client to see (they had signed all the content off already but that’s not the point), the project manager also decided that we didn’t need to run it in our testing suite. Yes, we were posh and had a “suite”. Oh, okay, I admit it, it was a workbench but we called it a suite in front of clients! Anyway, the project manager said “Launch the site, we’ll lose money!”, we argued about testing and the project manager stuck their fingers in their ears, so we did what we were told and published the site.
Whoops! What a mistake
3 years worth of financial announcements were lost, the site now only worked on Internet Explorer 4 (and most people who hit it were running Netscape at the time – yes, it was that long ago!), half of the images were missing (including the company logo, very good) and the site was riddled with 404 errors.
We still don’t really know how it happened, in theory there should have been no missing/deleted content but there was. If we’d have taken a copy of the live site, put it on a staging server (that was identical to the live server) and run the update we would have seen these problems and they would have been avoided. As it was, poor muggins here had to stay in the office until midnight repairing the site from backups. Fun!
That’s just one (worse case) scenario of what can go wrong when you don’t test a site properly. Other problems that you may encounter may include:
- Browser incompatibility
- Accessibility issues
- Broken links/images
- Server errors
- Database errors
- Component errors
- Incorrect/old content
- Corrupted files
Plus much more – scary stuff, huh?
Okay, you’ve convinced me, I don’t want to mess up my biggest client’s website. What do I do?
Testing can be broken down into four key areas:
- User Testing – Testing whether certain data entered into a form breaks it, are there spelling mistakes (surprisingly common!), do all the links work, is all the page content there etc.
- Accessibility Testing – Is it (if specified in the TRS) accessible in all browsers, does the code validate, are there alt tags on all images, what happens if stylesheets are turned off etc.
- Server Testing – Can the server handle the bandwidth, are the correct components/databases installed, are the versions the same as the development server, does it have permission to send email through the company firewall etc.
- Database Testing – Can it handle X number of concurrent users, does the web developer have the correct permissions to run queries, is it on the correct port etc.
The amount of testing that is required will obviously vary depending on the size and type of website. For example, a small 10 page brochureware site can be validated quite quickly with only User Testing, Accessibility and perhaps email permissions on the server (for the website’s contact form) needing to be tested. For larger sites, the testing process may even be longer than the actual development process!
Okay, I’m off to test the site….
Really? How?
Sure, link checking, code validation and certain accessibility issue are easy to test and can be largely by automated routines/programs but that’s not enough.
How will you know what conditions you are going to test against? What are the expected results if a user enters a valid or invalid email address into a form? How will you know if the site works in Opera if you have only looked at it in Internet Explorer?
A lot of people rush the testing process (or ignore it entirely), but as you have seen from my earlier example it’s something that can’t be ignored or rushed.
Here are the main points to consider before beginning the testing process:
- Where will the site be tested?
- What machines are available to test the site on, do they have the correct Operating Systems/Browsers on them
- What conditions are you testing against?
- What is supposed to happen when a user does something correct or incorrect?
- How will the bugs be tracked/fixed/retested?
- How will the testing be signed off?
Depending on the type of site you have, the above list may change but I think that it’s the bare minimum you should consider when thinking out your testing plan. I’ll go over each point below and discuss what you need to consider and what to look out for.
Where will the site be tested?
Whilst I have been lucky in the past to have access to a “suite” of different machines with different Operating Systems/Browsers on, sometimes its not possible to have this luxury.
Many people over look this step as they feel they can’t test sites in this way which isn’t strictly true. There are plenty of software emulators that can show you how your site will run (or look) on different platforms, operating systems and browsers. Be aware though, that whilst these programs/websites are quite useful they may not necessarily give an accurate representation of how the site will actually responds in a real-world scenario.
If you have access to the various pieces of software you need (or can download trial versions with the features you required enabled), you can install VMWare (or equivalent Virtual Machine software) which allows you to create a virtual “desktop” on an existing PC. This software keeps your own PC, software and configuration clean and gives you a “clean” install of your chosen Operating system.
If neither of these options work for you, don’t forget the power of the Internet! If you website is publicly accessible, there are many online forums and user groups where you can post your URL and other like-minded web developers will test your website in various environments. Be aware though that they might have certain software on their machines that may affect the way your website runs, or just be petty as you’ve built a nicer site than them!
Note though, with the last option, you would need to have this specified in the contract or have some form of agreement from the client – the last thing they want is their biggest competitor seeing their new site before it’s released!
Operating Systems/Browsers
Different sites can, and invariably will, react differently with different browsers and operating systems.
I have to hold my hand up here. A website we designed and built for a rather large multi-national ran perfectly in every browser on Windows (3.1, 95, 98 & 2000 – I told you I’d been doing this job for a while!) but did we test it on a Mac? Did we heck! Consequently what we had to do was some nasty JavaScript browser sniffing and write another 3 versions of the homepage depending on whether the browser was Mac/Netscape, Mac/I.E 4 or Mac/I.E 5 – 4 home pages altogether – Madness! Fortunately the rest of the site was fine (the homepage had a kind of funny design) but it really hammered home the importance of testing on all browser/platforms that will be (or could be) used (which of course you will have set out in the TRS, right?!)
So, the moral of the story is Test, Test, Test!
Testing Conditions
I’ve lost count of the number of times I’ve been to a website and type something like “It’s not cricket” into the site search and the results page breaks thanks to the single quote – very good!
Worse still are the snotty emails you receive from marketing types because someone’s stuck the word “Blah” into their email signup form whcih has now mucked up their carefully crafted mailing list.
Setting testing conditions for each page/form element etc. allows you to verify the type of data the user is allowed to input (which is particularly important if you want to avoid database SQL Injection attacks), require a certain type of data format (e.g. a valid email address) or specific user action.
You can also use testing conditions to see what will happen if users follow different paths to find certain information e.g. searching for it or clicking through links.
You should test for broken links, which can be done by automated tools, number of clicks taken to reach a certain page (the ideal maximum is 3 clicks), broken images, code validation and general accessibility.
Forms should be validated against desired data type and have set testing scripts e.g. what are the results if a user enters the following in an email field:
- myname@xyz.com (valid)
- myname@xyz
- Blah
- myname@xyzcom
- ‘
- “%><%=Response.Write(PathToDatabase)%><% (Or some other SQL Injection Code)
This then leads on to the next step in the testing process – what the users expect to happen after they’ve done something
What is supposed to happen? – Managing users’ expectations
If I go and randomly stick single quotes in every form field I can what is the site supposed to do? Will it strip out the quote? Make it safe for the database? Fall over and die? If I enter an invalid email address will it tell me or will it just carry on regardless just to annoy marketing?
The more a user knows what is expected of them, the better their experience of the site will be.
This doesn’t only apply to things like form field inputs and database error messages, but also to things like general navigation (e.g. click the “Home link” to get back to the homepage rather than the link named “Flibbertyjibbert” – and I’ve seen that happen!) and helpful 404 pages.
Don’t forget that your “Average Joe” user may have a completely different view of what they are expecting to happen rather than what does actually happen (and you’re biased as you know what’s going to happen as you built the site) so focus groups and real-world user testing are very important at this stage.
For example, you may write a testing script asking Jim-Bob to locate information about Product XYZ. Jim-Bob looks at what he has to find, sticks “Product XYZ” into the site search engine and then spends 15 minutes trawling through page after page of useless results before throwing the PC across the room in frustration.
In the debrief, you ask Jim-Bob what made him so annoyed:
“I couldn’t find the product in the search results!” He cries
“The search results only finds product by Code, not by name.”
“That Sucks!”
And you can bet your bottom dollar that if Jim-Bob was annoyed so will 99% of your other visitors. Of course, you may have set out in the TRS that the search was always only supposed to search on product code (not very user friendly I must say) but if that’s the case, tell the user! A simple message at the top of the search results page is all that it would have take to allay Jim-Bob’s frustration for 5 minutes at least!
Writing scripts that ask people to look for specific information allows you to see how the site will be used by your target audience, there’s no point in shipping it out if no-one can use it.
The other important point to this exercise is the debrief. Whilst the user is testing the site, give them a form to fill in which they can note down the good and bad points of their experience with the site. When the testing is complete, sit down with them and talk through the points and the actual emotions of their experience, you’ll be surprised how enlightening phrases like “I felt frustrated…”, “I really liked…” can be.
You can also use the debrief to figure out why a user did something in a certain way, and if you feel it will be a more widespread issue, do something about it. If you think it’s just that specific user then re-test and see if anything needs changing.
To get you started, some example test scripts could be:
- Find information on Widget 123
- When was the company founded? How many employees does it have?
- What’s our best selling product?
- Register for our email newsletter
Of course, the above is quite simple and in no way all encompassing but it will give you an idea of what to ask your testers.
Bug Tracking/Fixing/Re-Testing
Once you’ve identified any errors or usage problems with the site, these need to be entered into your bug tracking system which you set up during the development process (you did set it up didn’t you?), they need to be prioritised, assigned, fixed and retested – after all, you may have made things worse!
When retesting, I personally am in two minds as to whether to use the original tester or not. On the one hand, they may recreate the previous test and then say “This is brilliant” (or “hey, this still sucks”), on the other they may choose to go down an entirely new route to fulfil the test script (after all, last time the way they did it sucked), which may make the changes null and void.
I would recommend re-testing with the original tester and another “impartial” tester to make sure the changes haven’t screwed up everything and you’ll also get a much clearer picture of what the changes mean.
How will the testing be signed off?
Once you have internally tested the site and are happy with it, then comes the fun part – Client testing!
In my experience, this can go one of two ways a) the client doesn’t care, looks at two pages and is happy or b) checks every individual link, tries to break the forms, changes screen resolution etc.
It’s important to know what criteria the client will be testing the site to so you can ensure it works (!), and also to make sure that their testing matches up with the TRS. To use an earlier example, there’s no use them testing the search form for “Product XYZ” and complaining they can’t find it when the TRS clearly states the search is on product codes rather than names.
So, make sure they’re aware of the limitations, point them back to the TRS and let them test their little socks off. After all, your rigorous testing means the site will pass with flying colours, right?!
Once the client is happy then you can get a testing sign-off. This could be a simple email stating that the site passed their tests, or a list of all tests done with their associated comments. Whichever one you prefer, make sure you get it! You don’t want the client coming back to you in 6 months time screaming that the email form accepts anything instead of just email addresses – which of course is where you whip out your TRS and Contract and blow a big raspberry at them .
If the client is happy then you can finally get sign-off on the project, but nine times out of ten, you’ll enter the dreaded “Change Request Zone” which, if you’re not careful can turn into a never ending series of rewrites and become your own personal limbo!
Seriously though, if managed properly then change requests can be quite a simple (and almost painless) process which I’ll cover in more detail in the next article.
- Follow Katy Whitton on:
“The only problem I find with testing is that it soon gets boring. I’ve been playing around with Selenium ( http://www.openqa.org/selenium/) recently as it seems fairly easy to write test cases and then run them automatically.I’ve not used it in anger yet, but thought I’d mention it in case it’s of use to anyone wanting to test websites.”
“Thanks Adrian! It certainly has a lot of features worth looking into :)”
“Anything can get boring if you don’t like doing it. I’m all about testing, not only do I like breaking things, I also like to help prevent things getting broken in the first place.”