If we want to go faster, we can just get AI to test it, can't we? Read on to discover why we shouldn't.

AI Horror Stories

We've heard the horror stories that often stem from an over-reliance on AI and giving it permissions or an ability to run in a production environment that it probably should not have had.

The AI then goes on to delete code, drop database tables, or do something that it should definitely not do but has a major negative impact on the application and the business.

I'm not really focusing on those aspects here, this article is about whether using AI to test AI-generated code or human built stuff is a good idea or not.

Faster Faster Faster

Generating code with AI or using an AI to assist with writing code enables software teams to build faster. New features are churned out faster, the roadmap moves along at an ever quicker pace, new releases are more frequent and the momentum builds, 'look how fast we are moving'.

Testing, traditionally seen as holding back teams that want to go faster, can now just be speeded up by using AI. No more annoying humans getting in the way of releasing dozens of super new features as quickly as possible. Just get the AI to test the stuff that the AI generated - perfect, right?

Take a step back for a minute, is faster always better? Is churning out piles of half-baked new features really benefitting your product or is it causing your users to grow increasingly tired and frustrated of the frequent problems and bad execution that contributes to a worsening experience?

Is it possible for the business to go faster by actually slowing the pace of releases down, allowing for more careful and thoughtful work to take place and doing some thorough testing?

Usefulness Of AI

I'm not suggesting for one minute that AI is not a very useful tool. We use AI in an increasing amount at WebDepend, as a tool to help us in the testing of websites and applications.

And that's what AI is, it's a tool that people use to help them in their work. AI does not replace testers or testing, just as it does not replace developers, engineers, product managers or designers.

We are also building testing tools with AI. Partly to learn for ourselves what AI is good at and what it's not good at. And also to build things that are genuinely useful for our team, after they have been tested thoroughly to find any issues.

What Is Testing Anyway?

Software Testing, as defined by Wikipedia, 'is the act of checking whether software meets its intended objectives and satisfies expectations.'

You can give the AI tool the intended objectives of the thing you want it to check, and the AI will most likely also pick up some signals from other material it has been given or has access to, which enable it to carry out tests that seem entirely reasonable and may find some legitimate bugs or issues that require investigation.

However, in the above Wikipedia definition, the part the AI tool cannot do is whether the software satisfies expectations. The AI tool has no way of assessing whether the project being tested satisfies expectations or not, as it has no method of understanding and so cannot gain the understanding of whether a specified expectation or an implied expectation has been satisfied.

An AI tool following a series of prompts in the form of test objectives, or a set of test cases, is providing some basic value but is not an assessment of the application nor is it testing. Furthermore, AI tools may raise 'issues' that are not issues at all because the AI tool does not understand the implied expectations of the project being tested.

Similar to test automation, having AI carry out a series of checks can be a part of testing but should not be the whole test effort.

Responsibility

Who is actually responsible for what gets built and released to production?

If an AI generates the code, then the AI goes on to check that code works as intended or whether the generated code meets the objectives and then the AI pushes that code to production, is the AI responsible for what ultimately ends up in production?

The answer is, of course not, the AI cannot be responsible as it is only a tool. AI is as responsible as your Jira or your GitHub. The humans using the AI are responsible.

When AI is used in the development process and also used to check what has been built, there is a responsibility to check that what has been built by the AI is what the business wants.

There is also responsibility towards the users and the customers of the business to check that what has been built is something they can adequately use, that satisfies their expectations and does not cause problems for them.

Because AI cannot be responsible for anything, does the responsibility therefore reside with the development team? The answer is that the development team is also not responsible, as they are instructed to work using the tools and processes provided and defined by the management. The management of the business are ultimately responsible for what gets deployed to production.

But Everything Seems To Work Fine

If everything seems to work fine with the AI doing the testing then what is the problem?

Well, the whole point of testing is to cut out the 'seems to' part. Anything can 'seem to' work correctly in the first impression. Testing is about digging around underneath the glossy exterior to look for the problems that lie there - and problems do indeed lie there, lots of them.

What AI is good at is announcing confidently that all checks passed, or all test cases are green. AI does not understand what it is testing or why, it has no concept of expectations that the website visitors or application users have when they try to use the application. It does have an ability to mimic what understanding looks like and that can seem like the AI is testing thoroughly and doing a good job at it.

There is a duty for someone to go beyond what the results are from the AI's checks to carry out testing and look for issues.

Conclusion

So should you use AI to conduct all the testing for a project or not?

For simple things that don't matter that much, such as personal projects then feel free to use AI as a checker. I would still recommend that you also do some testing though to make sure you have carried out some kind of assessment.

But for a business, which has a responsibility to its users and customers and to rules and regulations, AI can be used as a tool to do some checks but should not be used to test everything. There needs to be someone involved in carrying out testing that goes beyond what the AI is checking and also challenges the AI's findings.

At WebDepend, we're starting to use AI to carry out some base-level checks, things that are well defined and fairly easy to report back on. One example is an accessibility scanner tool that we've developed, which scans a website and reports back on accessibility issues.

Even for this, where the accessibility tool is expected to find less than half of the accessibility issues, there should be a person checking the report over, investigating the issues and then carrying out the rest of the accessibility testing that the automated tool cannot cover.