Crowdtesting usually describes the fee-based testing of software, by a broad mass of free and independent tester through the internet. It is something comparable to an outsourced measure for quality assurance. And it can be used very flexibly, right from the very first click dummy in the process of software development. Primarily, the goal is to test application software on all relevant end devices and operating systems.

The term crowdtesting is based on the term crowdsourcing (crowd and outsourcing), which was introduced by Jeff Howe in his article “The Rise of Crowdsourcing” from 2006 [1]. There were first attempts to define the test method between 2009 and 2011, accompanied by an amalgamation of contributions to the topic.

The professional use of this test method as a service was developed at the end of 2010 and was met with approval in the start-up scene at the beginning. With the growing number of devices on the market, crowdtesting is becoming increasingly more important for companies developing application software, especially for the mobile development.

Figure 1 Shows the three most important participants of crowdtesting.

The provider and his platform

Numerous providers are available for a crowdtest operation. They either are dedicated exclusively to this service or they offer it as an additional service. Popular representatives include the following providers [2]:

• RapidUsertests (Berlin, Germany)
• Applause (Framingham, USA)
• Passbrains (Zurich, Swiss)
• BugFinders (Cheltenham, Great Britain)
• 99tests (Bangalore, India)
• Testbirds (Munich, Germany)

With 150 000 registered testers and 380 000 devices in all parts of the world, Testbirds is one of the world’s biggest crowdtesting platforms. Due to the testers being scattered around the world, they can test application software in different languages and on a wide variety of devices.

The provider and his platform act as a central communication point between the testers and the client:

• Providers and testers communicate over a web-based platform, which provides all necessary functionalities. Depending on the requirements, a project manager on the provider side, for example in quality assurance, supports the results which are provided by the testers and he also serves as a first point of contact for other open issues.
• Providers and clients use the web platform for communication and also for the possibility to arrange direct agreements. Usually, this feature is carried out by the testmanager of the client. Under certain framework conditions, agreements and commitments do contain restraints, devices or operating systems, through which the tests are executed. This way, the testmanager is accompanying the crowdtesting process from begin to end.
• Testers and contractors, in this case the developers of the client, only rarely get into contact – if they do, then it’s indirectly – about found defects (errors), via contributions on the platform.

The tester and his heterogeneity

Testers do represent all social classes, interest groups, cultural and religious affiliations, age classifications, and people with different technical affinities, whether these are professional or just for fun in regards to the topic. Thus, there are theoretically no restrictions on who can qualify as a tester. In practice, of course, there are basic conditions that must be fulfilled, for instance a person’s business capability.

These persons have the possibility to register on a crowdtesting platform, until a test object is ordered. If the testers meet the criteria and bring the appropriate test environment (device) with them, interested parties can then request a test run. In practice, testers are classified according to, for example, their experience or their frequent participation and are then subsequently selected. Then, they must document and record any error, which they find a commission, accordingly.

The testers receive a payment for the services, which they do preform. These payments do vary from provider to provider and usually are orientated on the number of defects found and tiered according to risk classes. Other factors, which can affect payment, are the experience of the testers and the duration of their platform membership.

The client and his software product

All persons and companies who develop or have application software developed are eligible as clients. The main focus is either on finding errors or on user and market analysis, through the feedback of usability suggestions. Furthermore, the client is hoping for a multitude of advantages, which will be considered separately in the following.

The advantages of crowdtesting for the client

The Internet of Things (IoT for short) [3], and especially the test coverage which is to be achieved on the countless mobile devices, are making crowdtesting more popular. Even large software companies with a powerful QS team are no longer in the position to satisfactorily supply the market of devices with quality. The availability of software on many different devices is becoming increasingly important and marks a significant competitive advantage. In the long run, you will no longer be in the comfortable position to exclude devices, and thus important market shares, in order to leave them to the competition. This raises the question as to how many devices, with different specifications and random operating system, can be tested, which will become obsolete again tomorrow.

Summarized this does result in six features which clearly speak to the use of crowdtests (Fig.2).

Flexibility:

It is technically possible to place a crowdtest during the entire software life cycle. Nevertheless, it has to be considered whether this does make sense or not, because not every phase is a suitable candidate. The classical use of crowdtests is the functional testing in search of errors. In earlier phases a crowdtest can be applied to click dummies in order to perform usability tests or market analyses. The crowd shows a high temporal flexibility in the execution of the tests, which can be carried out at any time. For example, the client can view the defects found over the weekend without any time delay at the beginning of the week and can then proceed to rectification. The tests can be carried out either explorative, i.e free of any specifications, or with specifications and up to very strict basic conditions.

Cost reduction:

Basically, it can be assumed that all necessary devices, their versions and designs, which are relevant for the market, are represented in a large crowd. Therefore the customer can save himself the provisioning of an extensive equipment pool and thus the costs of the administration, maintenance and the versioning. Also, the outsourcing of the test operation does free up internal resources in the form of employees, who are otherwise permanently being bound.

Real market conditions:

An end user should test the application, unaltered and without being bound to a place and time. This best reflect the real market. And to ensure neutrality in testing, the tester should be independently employed. A crowdtester meets these requirements.

He’s testing, whenever it is convenient: Whether that may be on the sofa, or while he is walking, or when he is on the bus under conditions of a fluctuating reception. It does reflect feedback from real end users in real scenarios.

High diversification of end devices:

In theory, crowdtesting is distinguished by the availability of all the end devices and their combinatorics with other factors of influence such as operating systems, browser and display size. Hence, this makes it possible to test and ultimately to guarantee the compatibility of software with as many end devices as possible. The strength of crowdtesting is that it is possible to test comprehensively. For traditionally used methods (regression-, smoke- or functions tests etc.), this is no longer affordable and almost impossible, in the times of IoT.

Experienced and unbiased testers:

The testers experience is quite broad. Some earn their living with their experience, while other only do it sporadically, for fun or just out of sheer curiosity. This is the strikes of balance between experience-oriented testing and the explorative, naive approach. The impartiality of the testers, which the International Software Testing Qualifications Board demands in its ISTQB standard [4], is guaranteed. Also, the testers are not involved in the development process.

Fast and precise feedback:

The documentation of defects is done via a template and complemented with video recordings or images.  In the case of video recordings the defects can be reproduced quickly and unambiguously. Direct feedback to the testers is platform-dependent and possible, for example, through a chat or a comment function. In rare cases, the personal contact data is stored. Since the recordings are created during the testing, they are available on the platform almost simultaneously. This way it possible to improve the usability during the development phase.

Method meets reality

The advantages of crowdtesting do inevitably correlate with the provider and his platform. Therefore, we are going to consider the following, connected disadvantages, their causes and measures for compensation, from practice. Make note of the following points, because they are the very basics of every successful crowdtest operation.

No acceptance without usability:

The usability of the platform and the allocation of the most important and necessary functions are critical for the use and acceptance of the platform.

A good provider does have a modularized platform which enables a fast configuration, in order to provide only the most necessary of functions for a customer request. Furthermore, the interface must be fun and easily understandable. Because a lengthy registration process for new users is the first indicator of a lack of usability. If the application has too many functions, extensive paths of navigation and never ending drop down menus, then it is strongly advised to not use this platform.

A defect-documentation free of Interpretation:

If there is no possibility of direct feedback, then a documentation of defects should be possible without interpretation. For a fast reproduction of reported defects, they must be documented without interpretation. Here it is recommended to record the creation of the defect, for example, with a short video. The reason behind this procedure is that only in the rarest of cases, there is a direct feedback possibility to the testers, for example, in the form of a telephone number. The platform provides a query function, e.g. a forum, in most cases though. However, the earliest an answer can be expected in this forum is after three hours, no matter how urgent the actual question might be.  No tester is obliged to be available on demand outside of their testing hours. And it’s not unheard of that a written documentation turns out be incomplete. In the case of no possible direct communication with the tester, it should be checked, whether the documentation is possible and feasible as a video. This approach does prevent a lot of frustration during the reproduction and correction.

Use with caution:

Crowdtests are promoted as being flexible all-purpose weapons which they are not. Therefore, it makes sense to examine the usefulness and area of application. A crowd test is a good idea, if the company structure and the prevailing development model can integrate it. For example, the test can be excellently integrated as an additional QS method in the V-model. In agile development models with the goal of 100% test automation, an explorative crowdtest at the end of development does make sense. We’re talking about the top of the test pyramid. The prevailing methodology should continue to be applied. Although, a widespread use is not recommended, as the costs incurred are better invested in other test methods. Furthermore, the test form is very well suited for a usability test, a code review or a functional test in the final stage. This is less true for module or integration tests. It is recommended to check to what extent crowdtesting can supplement the existing quality assurance, but not replace it. A reputable provider, who has a long-term customer loyalty in mind, will analyze the circumstances together with the customer and will also point out suitable application possibilities.

Restricted real market conditions:

Real market conditions are only conditional. Who waits for events to be paid for when the Internet connection is bad? The primary goal of the tester is to find defects. After all this for what he gets payed for, unless he is somehow remunerated for his working hours. Therefore, the testers want to navigate through the application as quickly as possible in order to generate inputs that cause the application to stumble. This works best with a good Internet connection and in a quiet environment. If an on-the-go application is about to be tested real life conditions, then you should be aware of the point that the crowdtest is usually performed at home, in order to avoid performance issues.

Many testers, many duplicates:

Many testers mean quite a lot of duplicates – so a good support by the provider is absolutely essential. The testers do not know each other, neither are they aware of the tests results. Therefore, the probability is very high that a found error will be recorded several times. Here, the provider is obliged to accompany the crowdtest with a test manager that ensures high-quality documentation and results without duplicates. The scope of the provider’s services should be examined and renegotiated if necessary.

Additional work through crowdtests:

You live and breathe Scrum and avoid unnecessary extra work true to the Agile Manifesto?  Crowdtests require planning and coordination and thus generate additional costs. Especially for agile development methods, with a high test automation (test-driven development) [5], a crowdtest as an explorative test can be of great use. However, this does require a coordinator or a test manager, who does not always belong into the philosophy (here for example, Scrum). With well thought out platforms and good interfaces for other tools, good providers minimize the planning and coordination effort for the client with the provider. It is impossible to avoid appointing a person responsible for coordination. Another recommendation is, to ask for a demonstration close to the project in order to get an insight into the processes and to be able to better estimate additional expenses.

Does the crowd meet the requirements?

A crowd is only as versatile and flexible as the mass it embodies. The application will be tested today in German and tomorrow in Spanish, Chinese and Russian with twenty of the most popular devices and operating systems? Not every crowd can keep up with every scale. A provider with 100 000 testers worldwide is usually much more diverse and flexible, can react better to changes and continue to meet customer requirements with high scaling. It is a good advice to be informed about the size of the crowd and to select a suitable provider. Also, it must not be the cheapest one, who won’t fit again tomorrow. Here, the advice is to plan in medium terms to avoid changing the provider.

To avoid effort in defect management:

The interfaces to the most common application lifestyle management systems (ALM) do minimize the effort and facilitate the further processing [6]. The defects documented on the platform are rarely corrected immediately and go through one or the other processing status in the regular process of the company. Maintenance and tracking are then carried out in the tool specified by the company, e.g. HP ALM [7] or DOORS [8], so that the defects created on the platform are transferred to the system. This process is very time consuming without an interface to the system, because the defects are retightened manually. It must be ensured that the provider deploys an interface to the customer’s system or exports a format which can be imported from the in-house system.

Final advice for the usage

It is almost an impossible to test the quality with crowdtests alone – they are simply not suitable for this task. However, the method is highly recommended as a supplementary and flexibly plannable quality assurance method. Especially with regard to the countless devices and their volatility, crowdtesting does avoid an elaborate and expensive test pool. Explorative tests, usability tests and market analyses are the great strengths of the method. It is difficult to give a general recommendation for the use of crowdtests, because their use depends on many individual factors. An examination of the usefulness is necessary in each case by the client; a reputable provider advises for this independently and supports prospective customers with the selection.

The success of the application stands or falls with the provider and its platform. Here the wheat separates from the chaff. Therefore, the scope of the service and support must be observed and checked. It does make sense to take a close look at the supplier, to have a detailed demonstration of the application and the processes shown and to use the method first in an uncritical project. Users will quickly notice if the requirements are not met or if the effort involved in processing the defects found has been estimated unrealistically.

Conclusion and outlook

All things considered, a well-placed crowdtest with a carefully selected provider will have an impact on the quality of a software solution.

I am firmly convinced that crowdtesting will bring about far-reaching changes in quality assurance and will see significant growth. Existing test centers which have been outsourced to near-, off- or smart shore projects for cost reasons will become significantly less important. After all, they are only a reflection of the old, internal structures and processes, in a different place, at a different time, at the same or even higher expense and at slightly lower costs.

All these arguments will no longer meet the requirements of modern and flexible quality assurance in the future. Because crowdtesting is neither tied to one place nor to one time, causes significantly fewer processes and thus saves time and costs. In the foreseeable future, outsourced test centers will either become crowdtesting platforms themselves or they will disappear completely, as crowdtesting platforms are already successfully offering test center services.