A Literature Review on Software Testing Techniques for Smartphone Applications

-Smartphone applications are getting popular and have become a necessity. There numerous smartphone applications ranging from entertaining to gaming and from utility to mission-critical. Almost everything on the web is now in hands of Smartphone users, which makes this domain very important and its quality should not be compromised. Achieving the desired quality is not an easy task for the mobile platform as it has its limitations. To produce a quality app, developers and testers need to test and assess the app in numerous ways to ensure the best trait of the application. In this concern, some efficient and mature techniques are required to test smartphone applications. In this study, the techniques, approaches, and models to assess mobile apps covering major prospects and angels to test mobile apps are identified. Our focus is on assessing the existing techniques and to evaluate them on standard validation parameters.

INTRODUCTION Smartphone applications are getting popular and have become a necessity. From banking to healthcare or from gaming and utility to mission-critical, there is a huge pool of smartphone applications [1,2]. Achieving the desired quality is not an easy task for the mobile platform as it has its limitations such as processor, battery etc. To produce a quality app, developers and testers need to validate it in numerous ways to ensure the best trait of the application [3]. In this concern, some efficient and mature techniques are required to test mobile applications. Mobile applications have their quirks and challenges regarding testing, such as the high number of different events that need to be tested [4]. Security is also an important aspect of smartphone applications [5]. These challenges mostly rely on the mobile platform, but some challenges arise due to the interoperability of the mobile platform to other platforms like the web, third party systems, and the cloud. In this study, the techniques, approaches, and models to assess mobile apps covering major prospects and angels to test mobile apps are identified. Our aim is to assess the existing techniques and to evaluate them on standard validation parameters.

II. RESEARCH METHODOLOGY
This review was conducted according to the guidelines proposed by [6]. For this purpose, we have formulated a search string and executed this search string on IEEE Explore, ACM and Science Direct to identify research studies published from 2007 to 2020. By reviewing the title and the abstract we have initially selected 98 research papers for full text reading. From this initial database of research studies, we selected 19 research papers to include in this study which were aimed at testing smartphone applications. The database of the selected research papers covers journal and conference papers about testing techniques for smartphone applications. Most of the research studies are primarily focused on theoretical reports, case studies, field studies, and experience reports.

A. Security and Malware Testing
A study presented APSET for detecting the intent-based vulnerabilities of Android applications. APSET takes vulnerable patterns proposed by domain experts and system specifications [5]. The major contribution of this study is test case generation via the automatic generation of partial specifications from applications. APSET also detects issues in the data on the basis on intent mechanism. APSET was tested on over 70 Android applications and it detected 62 vulnerabilities which could be exploited by hackers or attackers to crash the application. This tool is founded over the modelbased testing technique along with the support of the ioSTS model to generate patterns besides the reverse engineering of system class diagram and specification to generate test cases for the application under test. The limitation of the proposed tool is that it only works over intent-based loopholes and pitfalls. APSET cannot be used for any other vulnerability. A novel hybrid technique was proposed to detect malware in Android applications based on static and dynamic analysis [7]. The proposed technique requires efficient data processing (pattern generation and detection). System calls are used to generate patterns for malware and normal apps which are done through the support of Android OS manipulation. The advantage of this approach is that it does not spend much time as it takes in the static analysis, neither it consumes a lot of resources as does the dynamic analysis. Some disadvantages include that it needs to constantly gather new malware and benign apps to keep detection accuracy because new types of malware continue to emerge. This approach is based on the difference between malware and benign apps in runtime system calls. More normal and malware patterns are always needed. Second, due to limited computing and storage resources, it is not suitable to perform large-scale data processing in the mobile phone. To achieve real-time detection, this approach requires the mobile phone to have powerful computing capabilities and sufficient data storage. Authors in [8] presented a test automation technique for mobile platforms based on observation, extraction, and abstraction of the running SUT by using its GUI widget. The abstraction used in this tool is used to create a scalable state machine model using eventbased test coverage criteria, which automatically creates test cases for the SUT. The study demonstrates that the generated test cases were effective and useful in detecting serious issues and defects in the applications. Using the mentioned approaches, the authors developed a fully automatic tool for the detection of bugs and errors in Android applications. This study also compares two famous Android application testing tools: Monkey and Dynodroid. They configured Monkey and Dynodroid to test the same four mobile applications which they tested with Mobiguitar under the same parameters and inputs. An experiment demonstrated that Monkey and Dynodroid both did not find all of the exceptions which their proposed tool did.

B. Cloud-based Testing
A systematically review the state of the knowledge of the empirical studies is presented in [9]. The study focuses on mapping the testing techniques for mobile applications. Additionally, the study emphasizes the need for testing metrics to be included and adhere to address mobile application testing lifecycle conformance. The major lags in the mentioned techniques for a smartphone application testing lie in the automation of testing. According to the authors, this is an emerging and future of mobile and other testing, but very few of them implemented this technique over complex applications. Automated testing techniques perform well over small to medium and simple mobile applications, but very little work is done over the implementation and analysis of this technique over complex mobile applications which put a question mark over its credibility and reliability.

C. Test Automation
Authors in [10] proposed a novel framework for comparing automatic testing techniques for smartphone applications. The salient characteristics of every technique were picked to develop the parameters. A comparison was done among online testing techniques and in result, a general framework was proposed based on the Unified Online Testing Algorithm. The authors found that the random technique is more effective than an active learning technique, but it is approximately 100 times more expensive due to the nature of the experimental setup. Active learning was cheaper and more efficient than the random one in some cases, but in most of the cases the random testing technique was better in execution but more expensive in cost. Authors in [11] presented an experimental study to analyze and evaluate the MBT approach in modeling, concretization, and execution of automated tests in mobile applications. Along with the usage of MBT they adopted the Event Sequence Graph (ESG) to design their test model. For the implementation of the test cases, they used the Robotium framework. This study evaluated the perks and shortcomings of using MBT as an approach to automate test execution for this particular platform. Several perks and challenges were identified including: automatic generation of test cases, capacity to detect faults, improvement in test quality, test time and cost reduction, and evolution of test models. The challenges in the usage of MBT in the mobile application that are related to this domain are: difficulties in test modeling and particularities in the concretization of test cases in mobility context and in-depth knowledge of Robotium.
The design and implementation of the mobile TaaS system called MTaaS is presented in [12]. MTaaS is an infrastructure for mobile application testing on the cloud which provides large scale remote mobile application testing for the Android platform. Some issues regarding resource allocation and sharing were discussed as this study focuses on the cloud to implement the proposed system where resource allocation and management is a very important factor to be handled. This issue is resolved through a hybrid model that aims to improve the system performance while reducing cost. The system was tested on different real time scenarios and use cases. MTaaS was compared to Perfecto Mobile, TestDroid, YiceYun, Testin, and UTest under the same parameters and inputs. The results showed that MTaaS performed satisfactorily. Further, some limitations were discussed like the security and privacy of the user's data, security threats to multiple virtual machines, and intrusion detection. Another issue that was highlighted was the lack of standards in mobile test environments and test automation for mobile application testing with the addition to the lack of well-defined test models and coverage criteria for cloud-based application testing. Authors in [13] presented a tool named SIT to test self-adaptive applications. SIT framework stands on Abstract Trace (AT) and Trace Segments (TS). Test case generation was achieved by sampling-based test generation. As per experimental results, the SIT improved defect detection significantly. SIT was evaluated and analyzed over three online available context-sensitive and self-adaptive applications: Robot car, Phone Adaptor and SECONDO. which means if an application is running in a particular environment, that environment should not be easily changed or manipulated during testing. The improvement room is also available in the proposed approach in a way that assertions that were used to define program failures were composed manually in the application. It might be possible to derive these assertions automatically from specifications.
In [14], a GUI based automated testing tool called SPAG-C is proposed which uses a record-replay technique to perform GUI testing. SPAG-C used event-based and smart wait function to eliminate the uncertainty of the reply process and by using GUI layout information to verify the results testing process produces. The experimentation showed that SPAG-C maintains the accuracy to 99.5% in addition to the reduction of the time required to record test cases by verifying the process automatically with the increased reusability of the test oracles without compromising the accuracy. The proposed tool was compared with Histogram, SURF, and Template matching tools and their techniques and results proved that SPAG-C outperformed them. The novelty of the SPAG-C is that for some apps only a small portion of the screen is changed when the app responds to an event. For such apps the tester can select the region of interest in the screen of the device and then SPAG-C will verify it which saves time to fully verify the screens once again. As a limitation, there is the problem that applications with non-deterministic GUI cannot be tested by SPAG-C. This is because the camera cannot extract the necessary elements from the screen for later comparison and matching as AUT screen keeps changing in video playing applications or gaming apps. This approach works on the image capturing mechanism so the camera which is used to capture screenshots of all UI could be affected by external factors of the environment like light, exposure etc. A controlled environment is required to execute this procedure. Authors in [15] presented an empirical study in which MBT was applied to mobile applications to examine the effectiveness of their approach over mobile systems and applications. The study uses EFSM to model a mobile app and implemented a commandline tool "Kelevra" along with Appium, an automation tool. Gestures, clicks and keyboard inputs are examples of methods that can then be applied to the retrieved UI element objects. The MBT approach was also previously tested over an app "GMSEC", which is quite simple as compared to "Quiz-Up". The experimental setup exhibits that MBT pays well off over the effort it requires. Experiments show that applying MBT found non-trivial bugs and defects in Quiz-Up which was already being tested, proving the effectiveness of this approach towards mobile systems or applications. A possible extension to this would be to minimize the manual steps even more. Another option would be to implement a language to describe a SUT and its possible outcomes. For mobile applications, we can describe the UI elements and patterns in a particular view or scene under the test. The constructed textual description could then be translated into a model representation such as the EFSM. The key limitation observed is that the proposed technique is suitable for testing small-sized apps.
Authors in [16] presented an adaptation model for testing mobile applications which is comprised of two sub-parts, Mobile analyzer and Test Mobile [16]. The study was inspired by a framework used by web app testing, Reweb and Testweb, a tool for analysis, testing and restructuring application. Refactoring is a key step in this proposed technique as refactoring minimizes code size without affecting the functionality of an app. The major focus of this study was to minimize the test effort by minimizing the test cases of an app, which can be achieved through refactoring AUT (Application Under Test), so that the transitions and paths of applications can be minimized which ultimately results in less test cases and efforts to test an app without compromising the coverage. Based on this model, test cases were generated. At this point, applications entered the second module, Test Mobile, and generated test cases were selected upon defined criteria. After test case selection, the expected outcome and run time output were compared after the execution, and the result decided whether the test passed or failed. Reweb and Testweb were initially used for web domain applications and they were designed for a particular domain so they performed well. The advantage of this technique is that it minimizes test case space without compromising coverage. Further advancement in this technique can be made by automating the process of refactoring which eases its use and lessens human involvement and effort on refactoring. Authors in [17] presented an MBT approach for test case generation for smartphone applications. State machines were used for modeling and test case generation. Test cases were generated through SPIN, an automatic tool for test case generation through model usage. The proposed approach used XML based transformations to translate the test cases to some executable form to activate the applications under test. For experimentation, Facebook and YouTube applications were used. The proposed approach was tested with other model-based approaches and techniques which include APSET, MobiGUITAR, and SwiftHand. This approach uses the view state machine model to model any application, which involves excessive mathematics and prior working knowledge of state machines, as these models become very complex in large size programs. The second issue is that there is a lot of involvement of third-party tools which are assisting in the basic three steps of this approach.
Authors in [18] proposed a grey box approach for automatically reverse engineering GUI-models of mobile applications. Their system, Orbit, uses a finite state machine for model generation and the results of an empirical evaluation on several real apps were presented. At first, the authors used static analysis of the application's source code to extract the set of user actions supported by each widget in the app GUI. Then, a dynamic crawler was used to reverse engineer a model of the app, by systematically exercising the extracted actions on the live app. Orbit was evaluated on 3 parameters, test coverage, time consumption, and precision. It was compared with Monkey, MobiGUITAR, and Android GUI Ripper in experimental steps. A limitation in this approach is that it requires a lot of manual work, consisting in manually selecting attributes of the executable components to compose the visual observable states for the GUI.
Authors in [19] proposed a technique to test mobile applications using reverse engineering and pattern recognition of mobile AUT. The process was based on the automatic and dynamic exploration of the apps' GUI. To support the dynamic exploration, static analysis was also conducted to verify how they are executed. GUI widgets were examined through their execution and calls. The achieved patterns decided the errors and bugs, if found. The major contribution of this work is a reverse engineering approach to identify occurrences of behavioral patterns in mobile applications and a dynamic, run time, and on the go testing approach based on the application of test patterns associated with behavioral patterns identified in the mobile application. The proposed approach was found to perform better than the MBT approach as most of the GUI MBT approaches use reverse engineering for model gain. A drawback of this study is that the pattern behavior judgment is not easy and can be a tedious task if done manually. Since this study reverse engineers the app under test and analyzes its patterns statically, this static task of pattern recognition and extraction is not easy as apps get more complex day by day. Future work in this study could be the automation of pattern identification and recognition, which would reduce and minimize execution time and minimize human error.
Authors in [20] proposed test adapters to test GUIs through automated testing of industrial applications. Test generations through test adapters can be applied at the unit, integration and system test level. Authors in [21] proposed the A3E approach to use static, taint style, data flow analysis on the app to develop a high level control flow graph that captures legal transitions on the app's screen. The experiment on 25 popular apps achieved 59.39-64.11% activity coverage and 29.53-36.46% method coverage. Model-based software testing can be used to automatically produce test cases from a formal model describing the SUT. In addition to conventional test automation, it may increase the quality of testing and reduce the resources needed. A case study was presented to illustrate the ability of a MBT to produce long-term test cases and run parallel tests on multiple smartphone devices in [22]. Authors in [23] introduced a novel method designed to identify irregular network traffic activities in a multimedia app and how different user experiences will lead to unexpected traffic patterns. It makes the generation of a test suite composed of a huge number of test cases that can be executed and measured with adequate automation. Instead of random interactions, this test suite represents realistic user behaviors, which may reveal unforeseen consequences to users.

IV. CRITICAL EVALUATION
A systematic literature review was performed in order to find, analyze, and classify papers which focused on testing mobile applications. The aim of the current study was to provide practitioners and researchers a clear view of the state of the art so that they can easily find existing solutions pertinent to their issues. The papers have been classified based on the focused area, testing techniques/models, tool(s) (used and proposed), platform/OS, tested applications domain, and validation methods and parameters. A summary of the approaches and techniques is presented in Table I. V. CONCLUSION In this study, several techniques and approaches in addition to models and tools were reviewed for smartphone app testing. We found a pool of testing strategies to assess the quality of mobile apps of various natures. While analyzing and evaluating the reviewed approaches for mobile app testing, we found the model-based approach more convenient and promising to test mobile apps because of its appealing approach to model the overall design of a system [24]. The main advantage of using the model-based approach is that the testers are well satisfied with the recused efforts and the level of test cases it helps produce [25]. Another advantage to adopt model-based testing is that it supports several automated tools for test case generation and execution. In this study, we have presented an abstract model for testing smartphone apps which support semi-automated testing. A potential future work in this context could be to evolve a fully automated approach in the form of a testing tool which takes an app as an input and generates test cases according to the chosen model. The proposed tool may possess more than one model for modeling that app and would present the test results after executing the generated test cases in accordance with the selected model. The proposed approach would significantly lessen the human efforts required to test mobile apps and would also minimize the chances of human error.