Multicriteria Decision Making ( MCDM ) Methods for Ranking Estimation Techniques in Extreme Programming

It is essential to use multicriteria decision making (MCDM) methods to evaluate human judgments, for decision problems requiring the measuring of tangible and intangible criteria. Among the MCDM techniques, the analytic hierarchical process (AHP) and its extended version, the analytic network process (ANP) are the most powerful methodologies for ranking options and alternatives. They have been utilized by many scientists and researchers in numerous fields, especially for complex engineering problems. Both tools allow leaders to structure their issues numerically utilizing individual judgments. In this article, it is suggested that the MCDM can be useful in agile processes where complicated decisions happen routinely. This paper shows the ranking of the extreme programming (XP) estimation methods using AHP and ANP in educational and industrial environments. Keywords-analytic hierarchy process; analytic network process; extreme programming; planning game; estimation techniques; user stories


INTRODUCTION
Developers face uncertainties when designing software projects.It is important to minimize these uncertainties.In extreme programming practices, there are certain activities that aid this process, one of them being the planning game practice.In this practice, XP team needs to discuss the system requirements, known in agile methodology as user stories.
Author in [1] characterizes user stories as "short descriptions of functionality told from the perspective of a user that are valuable to either a user of the software or the customer of the software".These client stories are noteworthy in light of the fact that they make it simple to draw a general structure regarding the system.User stories contain different activities such as writing and breaking stories down into tasks.User stories can be prioritized based on the customer business value.Moreover, one of the most important activities is estimating user story effort and cost.During iteration planning meetings, estimating processes are used to establish the iteration plan.This occurs by assigning stories to each iteration considering their priorities.Story points are used to measure the effort required to implement a story.Story points can determine the cost and many-sided qualities of the requirements.Developers usually break down the user stories into small tasks to be able to estimate effort and time.In this paper, the researcher conducted two case studies in educational and industrial environments to introduce the analytic hierarchy process (AHP) and analytic network process (ANP) to assist the XP team ranking the estimation methods.There are four common estimation methods have been evaluated in this paper: Planning poker, expert opinion, analogy, and disaggregation.

II. ESTIMATION METHODS WITH EXTREME PROGRAMMING
Agile methodology is based on collaboration among team members and therefore, estimations are not obtained by individuals.All team members share estimations, and "estimates are best derived collaboratively by the team, which includes those who will do the work" [2].It is important during estimations of user stories to appropriately manage stories so that they can be estimated clearly.For example, one could combine related stories into a group, which is called theme to clearly estimate them as one item.There are several methods that can be used to software estimation.Some methods mathematically obtain estimations by concentrating on historical data [3].Other methods estimate effort by measuring the size of the task.Expert opinion is one that works by asking an expert about each story.Based on his/her experience, the expert gives an estimate.Another technique that can be used to estimate user stories is analogy.This technique can involve a triangulation process, which depends on comparing the story that is being estimated with two other stories [4].Disaggregation is another method that is used to split large stories into smaller ones to ease the process of estimation.Some studies suggest that the best technique that can be used to estimate stories is planning poker.This merges the three abovementioned techniques of expert opinion, analogy, and disaggregation.All team members are involved in planning poker to estimate stories and this involves several steps and stages, which are explained in more detail in [2].
Authors in [5] introduced several areas that are useful to estimate using expert opinion.These areas are: 1) areas where it is difficult to find empirical data, and 2) when it is difficult to www.etasr.com

Alshehri et al.: Multicriteria Decision Making (MCDM) Methods for Ranking Estimation Techniques …
estimate because of the lack of problem understanding.These areas are considered reasons for widely applying of the expert opinion method in software estimation.Authors conducted an industrial experiment to evaluate the reliability of using the expert opinion method in cost estimation.The study aimed to enhance cost estimation in a medium size software organization."The intention of the organization in supporting this study was to assess its cost estimation capability in the bid phase of a project and, if required, to use the results as a catalyst for change of their estimation practices" [5].In this study, an on-line survey was distributed among thirty employees.The response rate of the survey was 47%, and the average experience of the participants was 7.5 years.The questions focused on two areas, variability and calibration.The study contained questions asking about several issues, for example, authors asked about the possibility of having the same estimate from various groups of estimators.After presenting the results, the authors concluded that the expert opinion method obstacles are inconsistencies and overconfidence.This issue causes unreliable software estimation.However, they suggested that integrating the expert opinion method with other techniques like Wideband-Delphito makes it more reliable.Author in [6] conducted a survey in 364 organizations and found that only 51 estimated efforts by the use of models.However, the model was not better than the non-model users in estimation.But when they relied on the expert opinion, they believed that was better than the use of estimation models.
Authors in [7] investigated effort estimation in 32 software projects in different Iranian software companies.Questionnaires were distributed to collect data from these companies' employees and their project estimation.The study shows that most of these companies depend on small teams (of five people and less) because of the small scale of developed products, and the simplicity of managing small teams.Also, the most used development processes in product development are rational unified process and XP.The distributed questionnaire included various parts such as the type of estimation method used in a project and the degree of differences between estimated and real data.On this study, expert opinion and analogy method were the most common estimation methods.Among all these projects, the percentages of using analogy method and expert opinion were 29% and 25% respectively.The study concluded stating that these two methods are simple to be applied to similar projects and provide rapidly estimation.The major defect, however, is the dependence on expert judgment, which may be less reliable.Authors in [8] conducted a study to evaluate the accuracy of planning poker estimation method.13 students were formalized in teams to develop a web-based student records information system.All students received the same user stories, and asked to implement them in three sprints.By using planning poker method, students estimated the user stories and "the estimates provided by each team member during the first round were averaged to obtain the statistical combination for further comparison" [8].In the same time, a few experts were given the same user stories to provide their estimations.The results of this study show that planning poker method lead the students to have over optimistic estimation, while the expert's estimation was closer to the actual effort.The study addressed that planning poker method is less beneficial when it is applied by less experienced developers.
Authors in [9] investigated XP practice development in an IBM group.They concluded that the XP product has improved pre-release and post release quality.The XP team noted enhancements in their effort estimation, schedule and productivity.Also, customer satisfaction was very high with the XP product since developers delivered more than what was planned.Authors in [10] estimated effort by applying artificial neural network (ANN) and case-based reasoning.They used data set from the Australian Software Metrics Association.ANN estimated the development effort within 25% of the actual effort in more than 75% of the projects.In [11] authors emphasized several challenges in current estimation methods.Time duration is one of the challenges due to the differences in the story points and the velocity of the team.

III. THE ANALYTICAL NETWORK AND HIERARCHY PROCESS
Author in [12] defined ANP as "a multi-criteria theory of measurement used to derive relative priority scales of absolute numbers from individual judgments (or from actual measurements normalized to a relative form) that also belong to a fundamental scale of absolute numbers".Similarly, AHP is defined as a systematic approach for problems that include the thought of different criteria in a hierarchical model.AHP reflects human thinking by grouping the elements of a problem requiring complex and multi-aspect decisions [13].Both concepts were developed in [14] as means of finding an effective and powerful methodology that can deal with complex decision-making problems.AHP comprises the following steps: (1) Structure the hierarchy model for the problem by breaking it down into a hierarchy of interrelated decision elements.(2) Define the criteria or factors and construct a pairwise comparison matrix for them.Each criterion is compared with the other criteria of the same level in respect of their importance to the main goal.(3) Construct a pairwise comparison matrix for alternatives with respect to each objective in separate matrices.( 4) Check the consistency of the judgment errors by calculating the consistency ratio.(5) Calculate the weighted average rating for each decision alternative and choose the one with the highest score.
In the ANP method, dependencies among various criteria are considered making it different from the AHP."In fact the ANP uses a network without the need to specify levels.As in, the AHP, dominance or the relative importance of influence is a central concept.In the ANP, one forms a judgment from the fundamental scale of the AHP by answering two kinds of questions regarding strength of dominance: (a) Given a criterion, which of the two elements is more dominant with respect to that criterion, (b) which of the two elements influences a third element more, with respect to a criterion" [15]?In pairwise comparisons, entered values mirror the relative effect among elements with respect to a control criterion.These entered values are based on the importance of each criterion.The network structure consists of different clusters, and these clusters contain various nodes or elements.These clusters are connected to each other based on the relative influences among the nodes.In [16], author developed a www.etasr.com

Alshehri et al.: Multicriteria Decision Making (MCDM) Methods for Ranking Estimation Techniques …
numerical scale for assigning the weight for criteria or alternative by giving a value between 1 (equal importance) and 9 (extreme importance), see Table I.IV.METHODOLOGY The primary goal of this exploration paper is to examine how the AHP and the ANP can help in selecting the most suitable estimation method.The case study analysis strategy, which is clarified in [17], is the examination methodology.The following research questions give more concentration to the exploration contextual analysis:  Could the AHP and ANP process support the XP team to select the appropriate estimation method to estimate each user story?
 How can the AHP and ANP impact the improvement group's correspondence and efficiency?
From the previous questions, the units of examination for our investigation were inferred.The primary goal is to rank the most common estimation methods used in XP approach.

A. Data Collection Sources
Toward the start of each utilization of AHP and ANP in XP, we investigated their benefits and abilities by introducing the related criteria and XP areas.Data is gathered from looking past examinations and other reviews.Also, information triangulation is obtained to expand the legitimacy of the investigations.The major data source for the educational case study was a project conducted during the winter semester of 2016 at the University of Regina, while the data source for the industrial case study collected from two development teams located in Canada and Saudi Arabia.The data sources in this research are:  Questionnaires given to the participants during their development activities.
 Comments received from the developers in both studies.
 Open-ended interviews with the participants.

B. Proposed Criteria for Selecting Estimation Methods
To rank estimation methods, it is important to identify the criteria that influence the estimation process.These criteria are compared with each other (to show the interdependences), and with alternatives.The estimation methods are compared with respect to the criteria to show the feedback relation in the selection process.In this case study, four estimation criteria were proposed.These criteria are:  Accuracy: Which estimation technique gives the most accurate estimation?
 Simplicity: What is the simplest estimation method to understand and to apply?
 Collaboration: Which estimation method has the highest degree of collaboration between the team members?
 Time: Which estimation method saves more time when estimating the user stories?
V. ANP AND AHP CASE STUDIES ANP and AHP methods were used to rank the various estimation techniques in XP.In the following sections, the ANP and AHP evaluations, structures and process are detailed.

A. ANP in Educational Environment
The educational case study was conducted at the University of Regina, at 2016.It took around 12 weeks.The case study included 12 graduate participants with an additional participant as a client.These students had intermediate knowledge of XP process and practices, and different programming levels.The participants' backgrounds included various programming languages like C++, Java, and PHP.The participants were organized into two teams, the first team used the ANP method to make their decisions in the mentioned areas, and the second team followed the traditional XP method.Both teams were asked to develop a project called "Professors' Availability Managing System" complete with a set of requirements.The project was developed in 5 iterations, allowing two weeks for each.At the end of the project, the two teams implemented all system requirements.Assistance materials that focused on planning game practices were given to the participants to ensure their understanding [18].These materials involved estimating user stories, writing user stories, and making programming commitments.The ANP team was given white papers, several presentations, and other important materials about the ANP to allow them to apply it in their development.Team 1 practiced on several pairwise comparisons and increased their understandings of the ANP structure.At the end, the researcher handed out a survey to the participants to collect more data about the participants' perspectives.

B. AHP in Industrial Environment
Eight experts in IT industry were quested to participate in the evaluation.The experts were in two companies in two countries, Canada and Saudi Arabia.Three of them were working in a governmental project in Canada.The rest of the participants were working in different projects in Saudi Arabia.Their experience in project development ranged from 10 to15 years.They have been through many projects involved in managing teams and projects.The participants also were familiar with the activities of tasks estimations process and a good understanding of scrum and agile methodologies

VI. ANP AND AHP STRUCTURES FOR SELECTING THE ESTIMATION METHODS
Organizing the problem [19] is the initial phase in both ANP and AHP.The ANP network contains a criteria cluster, an alternative cluster and the goal.The main cluster is the "alternatives" cluster, which contains the four afore mentioned estimation methods to select the most preferable one.The www.etasr.com

Alshehri et al.: Multicriteria Decision Making (MCDM) Methods for Ranking Estimation Techniques …
"objective" cluster referrers to the "criteria" cluster which interacts with the "alternatives" cluster.In AHP, the problem is structured as a hierarchy that includes three levels.The top level is the main objective: selecting the estimation technique.The second level contains the criteria (accuracy, simplicity, collaboration, and time).The third level has the alternatives (planning poker, expert opinion, disaggregation and analogy).The appropriate ANP and AHP tables were created and gotten by every one of the participants.They were requested to round out the pairwise comparisons in view of the scale of [14] depicted in Table I.

VII. PROCESS OF THE PAIRWISE COMPARISON
In both studies, participants had an opportunity to experience the estimation methods through real projects to see the real impact on the team and the velocity of the development.Then they were required to rank the most favorite estimation methods based on the four criteria.For this aim, AHP sheets containing tables were given to the participants in the industry and ANP tables were given to the students in the educational case study.Members were asked to analyze the criteria utilizing the Saaty scale from 1-9.The members were asked these inquiries: After the criteria evaluations finished, members needed to rank all the estimation strategies in view of every criterion every time.(example: Focusing on the simplicity, which is more vital: taking the expert opinion or doing the planning poker and by how much?).Additionally, all the accompanying correlations were directed in view of every criterion [example: (Planning Poker-Expert Opinion), (Planning Poker-Disaggregation), (Planning Poker-Analogy).(Expert Opinion-Disaggregation), (Expert Opinion-Analogy), (Disaggregation-Analogy)]. Similar comparisons rehashed until the point when the participants assessed all estimation techniques in light of every criterion.

A. ANP in Educational Case Study
The sorting of the estimation methods in view of all criteria, is outlined as follows: Expert opinion was first (35.36%),followed by planning poker (33.64%).Analogy was third (15.35%) and disaggregation (5.48%) was fourth.Table II shows the results.Figure 1 demonstrates the significance of every measure: accuracy (40.86%), collaboration (32.49%), simplicity (15.29%), and time (11.34%).Each participant in team 1 evaluated the estimation techniques using the ANP pairwise comparisons.Super Decisions Software was used to determine the aggregation judgements for team 1. Team 2 took after the conventional technique in their decisions and in this manner they were requested to archive each progression in their procedure of settling on choices.Team 1 results show that expert opinion method was given the highest rank among the four alternatives.Planning poker came second followed by analogy, then disaggregation.Table II exhibits the relative weight of each one as a percentage.The accuracy criterion received the highest importance among the criteria, followed by collaboration, simplicity, and time.Team 2 results show that the analogy method was given the highest rank among the other estimation methods.By asking team 2 which was the most important factor for selecting an estimation method, they gave accuracy the top score.

Remarks:
 Considering all criteria, the expert opinion method was the most preferable estimation technique by team 1. Analogy was selected as the best estimation method by team 2.
 Both teams considered accuracy the most significant criterion.
 Team 1 ranked the estimation methods by considering each criterion individually.The findings were: the planning poker method was ranked the highest in terms of accuracy, expert opinion placed as the top selection when focusing on time and simplicity, while analogy was ranked the highest in terms of collaboration.

B. AHP in Industrial Case Study
Sorting the estimation techniques by experts in industry outlined as follows: Planning poker (33.76%) was first, expert opinion (29.03%) second, disaggregation (20.89%) ranked third and analogy (16.30%) was fourth.Table III shows the results.Figure 4 demonstrates the significance of every criterion. Expert opinion ranked with the highest score when the time and the simplicity were the most important criteria.
 Disaggregation ranked in the second position after the planning poker when considering accuracy and collaboration.
 Expert opinion technique ranked last two times, for the accuracy and collaboration criteria.It is clearly that relying on one person means the team will lose the benefits of the collaboration, but the surprising result was regarding the accuracy criterion.The question if the experience can help to get an accurate result arises.It could be true sometimes.Yet, the result in this case study indicates the companies are getting more afraid to follow the opinion of one person.Nowadays, the trend in the IT companies is to minimize the approach of "one person" and replace it with a "team" approach.
 The analogy technique did not attract the experts, it ranked last with "simplicity" and "time" criteria and third with "accuracy" and "collaboration" criteria.
 Looking closer to the estimation techniques and considering every criterion separately as it is appeared in Table IV, it is confirmed that the expert developers ranked the expert opinion in the top place regarding the simplicity and time.
On the other hand, the planning poker sorted at the top in accuracy and collaboration attributes.

IX. CONCLUSIONS
This research paper assessed the application of two of the common methods of MCDM methods, AHP and ANP, in order to benefit practitioners to choose a method for solving a specific problem.These methods are advantageous as they reduce the burden of the decision-maker.After conducting two case studies in two different environments with varying experience and development culture the MCDM methods were found as an appropriate and beneficial tool that gave the development team a good understanding for selecting a suitable estimation method.The results of using ANP in educational study and AHP in industrial study showed that the expert opinion is the most preferable tool in the estimation process.The most experienced members could easily convince the other team members of their preferences.Junior developers may fear to express their opinions, even though relying on individuals is not always recommended for building motivated and energetic XP teams.The cases studies also emphasized on the importance of approaching "accuracy" when the team is estimating any development activities.


Which is more imperative: simplicity or accuracy and by what amount? Which is more imperative: simplicity or time and by what amount?Which is more imperative: simplicity or collaboration and by what amount? Which is more imperative: accuracy or time and by what amount?Which is more imperative: accuracy or collaboration and by what amount? Which is more imperative: time or collaboration and by what amount?

Fig. 4 .
Fig. 4.Importance of the criteria for estimation techniques by experts.

etasr.com Alshehri et al.: Multicriteria Decision Making (MCDM) Methods for Ranking Estimation Techniques … consistency
rates.Figures2 and 3demonstrate the distinctions in group velocity between the two groups. www.

TABLE III .
Remarks: The planning poker technique was the first choose by all experts.

TABLE I .
ESTIMATION METHODS BASED ON EACH OF THE CRITERIA