A Survey on the Adaption of CMS in Pakistani Universities

The use of Campus Management Systems (CMSs) has increased dramatically in Pakistani universities, a huge amount of money has been invested in its development and deployment. The CMS provides an integrated platform for managing academic activities, controlling process flows and provide online access to related information. It improves the efficiency and effectiveness of the universities and eventually improves the quality of teaching. However, it is very important to consider the attitudes and perceptions of faculty members and students towards the adoption of CMS because they may affect the acceptance of this technology. The aim of this study is to investigate and highlight the user satisfaction level about the CMS quality according to ISO/IEC 9126 standard. This work uses cross-sectional design as a primary research method, and data from 105 students and faculty members were collected from a Pakistani public sector university using a questionnaire survey. The response of the respondents illustrates that the system functionality is good, reliable, usable, and efficient. However, some improvements are necessary in some areas such as understandability, learnability, operability, and attractiveness. Keywords-software quality; software functionality, CMS

INTRODUCTION Information Technology applications play a significant role in any organization or company [1]. They are controlling robots, helping in healthcare and commerce [2][3][4], they have also reached higher education in recent years under the name of CMS [5] which basically are Enterprise Recourse Planning (ERP) programs for institutions [6,7]. The CMS improves the efficiency and effectiveness of the overall organization and ultimately improves the quality of teaching and learning. It shows the holistic picture of university processes [7]. In developing countries, public and private sector universities are facing competition in the higher education market, and want to use information and communication technology (ICT) as an edge on other institutions [8]. In the ICT landscape system, student-oriented processes are important because they reduce the operational cost, which is done when students independently manage their academic activities such as enrollment or acquiring information such as attendance and grades. Through the integration of CMS and Web, these services are available any time. The importance of CMS has also been recognized by the accreditation of institutions such as the Higher Education Commission (HEC) of Pakistan which initiated the HEC-funded deployment of CMS on different public sector universities [9]. Various universities in Pakistan are implementing CMS to handle their academic processes Assessment and evaluation are conducted to improve the efficiency and effectiveness of any system. There are different kinds of evaluations such as (1) formative, which is performed during the development and (2) summative, which is performed after the completion of the system. It focuses on system's effectiveness and if it meets the original requirements. This work evaluates the CMS in four characteristics: functionality, reliability, usability, and efficiency. These characteristics are basically a part of ISO/IEC 9126-1 standard [10,11] which focuses on 6 characteristics of software quality. However, the remaining two, portability and maintainability are not directly related to the end-users (students and faculty members). Only CMS features related to students and faculty such as student records, gradebook, student self-service, and faculty selfservice were considered in the current survey.

A. Functionality
The functionality of the system concentrates the complete function list that should be provided by it. Table I shows the subcharacteristics of functionality and the respected questions asked to the end-users.

B. Reliability
Reliability concerns the probability of error-free operation of the software and its ability to maintain its service delivery under specified circumstance. Table II shows the subcharacteristics of reliability and the respected questions asked to the end-users.

C. Usability
Usability is concerned about the software system's ease of use and whether it is learnable. In usability, we focus on the user interface and how much the software system interface is easy to operate, easy to learn, and attractive to the end-users. Table III shows the sub characteristics of usability and the respected questions asked to the end-users.

D. Efficiency
The efficiency concerns the resources' utilization when performing the required operations such as memory, amount of disk space, processor, network, etc. Table IV shows the subcharacteristics of efficiency and the respected questions asked to the end-users.

A. Research Method
There are various methods for evaluating software quality but selecting the correct one for evaluation depends on complexity and functionality. This study used the qualitative method for data collection and in order to understand the participants' attitudes and experience. It is used in order to evaluate ideas, experience, and beliefs of the participants regarding the quality of the CMS.

B. Participants
In the survey, data were collected from N=105 students and faculty members. In the sample, 74% participants were male and 24% were female, while 20% were faculty members and 80% were students, out of which 59% were in their 4th year, 34% in their 3rd year and the rest were in their 2nd year, while the 1st year students were not selected due to their little experience on CMS.

C. Data Collection Tools
In this study, the evaluation form included two sections. The first section included demographic information, such as gender, age, and department, and the 2nd section consisted of the survey questions. The results showed that this method of assessment is effective in pointing various software quality issues in the CMS.

D. Data Analysis
Data were analyzed with SPSS to find how much the CMS realized the four considered characteristics of the ISO/IEC 9126-1. The feedbacks of all sub-characteristic were combined, and the basic mean was obtained.

E. Validity and Reliability
In this study, for ensuring reliability, the data were collected through according to ethical research. In order to reach the utilization in the study, the faculty and students of Pakistani university who experienced CMS were selected.

F. Results and Discussion
In this section, the data collected from the questionnaire survey are presented and analyzed statistically. In this work, a questionnaire was spread among faculty members and students who were familiar with the CMS. In addition, interviews with some faculty members and students were conducted in order to validate the gaps in the questionnaire. Figure 1 shows the detailed feedback regarding the quality characteristics of the CMS. Four items were combined to determine this factor, and the results are shown in Table V. The minimum and maximum scores were 27 and 160, respectively. As it can be seen from Table V, the overall mean is 84. So, it can be concluded that the faculty and students were satisfied with the functionality of the CMS. Table VI and Figure 2 show the detail feedbacks on how the functionality of the CMS works. As it can be seen in Table VI, around 51.5% of the participants' responses fall in the range of agree and strongly agree which shows satisfaction on the functionality of the studied system. It is worth mentioning that 23.8% of the participants were neutral on their response on the functionality of the system and 25.1% of the participants were unhappy with the functionality of CMS and gave disagree and strongly disagree feedback. The reliability of the system was examined and the results are shown in Table V. The minimum and maximum scores are 35 and 98 respectively and the mean is 63. It can be concluded that the faculty and students were not very much satisfied with the systems' reliability. As it can be seen from the results in Table VI and Figure 3, 42.98% of the participants' responses fall in the range of agree and strongly agree which shows that the participants are moderately satisfied with the system's efficiency. It is worth mentioning that 23.14% of the participants are neutral on their response and 33% consider the CMS unreliable and give feedback as disagree and strongly disagree. The impact on the control of the system upon the freedom of use was examined and the results are shown in Table V. The minimum and maximum scores were 50 and 126 and the mean was 84 which is near to minimum. As it can be seen from the results in Table VI and Figure 4, only 37% of the participants' responses fall in the range of agree and strongly agree which shows that the users were unsatisfied with the usability features. It is worth mentioning that 13% of the participants are neutral on their response on that feature and 49.2% of them consider that the CMS is not easy to use and does not provide easy learning and gave feedback as disagree and strongly disagree. Finally, the capability of the system to provide efficient usage was examined and the results are shown in Table V. The minimum and maximum scores were 17 and 62 and the mean was 42. By this it can be concluded that the participants were moderately satisfied with the efficiency of the CMS. Table VI and Figure 5 present the details of how the system efficiency responds to the user. As it can be seen from the results in Table VI, 45.98% of the participants' responses fall in the range of agree and strongly agree which shows that the participants are moderately satisfied with the system's efficiency, 25.23% of the participants are neutral on their response and only 29% of them consider the CMS as not efficient and give feedback as disagree and strongly disagree.

III. CONCLUSION AND RECOMMENDATIONS
This paper presented an evaluation of CMS in four dimensions: functionality, reliability, usability, and efficiency. These characteristics are a major part of ISO/IEC 9126-1. From the evaluation, it was observed that there were no issues related to the functionality of the system regarding suitability, security, accuracy, and interoperability, with 51.5% of the participants' responses falling in the range of agree and strongly agree. In addition, the results show that participants were moderately satisfied with the reliability of the system regarding maturity, recoverability, and fault tolerance, while around 43% of the participants agree and strongly agree with the reliability features of the CMS. The participants were not very much satisfied from the usability features of the CMS and only 37% of them agreed or strongly agreed with the effectiveness of usability features such as understandability, learnability, operability, and attractiveness. Finally, it was observed that the participants were also moderately satisfied with the efficiency and around 43% of them agreed and strongly agreed with the questions regarding the efficiency features. As a conclusion, it is observed that most participants are satisfied with the CMS quality characteristics, but it would be better to enhance its usability features.