Performance Enhancement of Distributed Processing Systems Using Novel Hybrid Shard Selection Algorithm


  • Praveen M. Dhulavvagol School of Computer Science and Engineering, KLE Technological University, India
  • Sashikumar G. Totad School of Computer Science and Engineering, KLE Technological University, India
Volume: 14 | Issue: 2 | Pages: 13720-13725 | April 2024 |


Distributed processing systems play a crucial role in query search operations, where large-scale data are partitioned across multiple nodes using shard selection algorithms. However, the existing shard selection algorithms pose significant challenges, such as shard ranking, shard cut-off estimation, high latency, low throughput, and high processing costs. These limitations become more pronounced as the data size increases, affecting the efficiency and effectiveness of search operations. To address these challenges, the novel Hybrid Shard Selection Algorithm (HSSA) is proposed as a solution in this paper, designed specifically to enhance the effectiveness and efficiency of search operations within distributed processing systems. HSSA employs an advanced sharding approach that adeptly navigates and targets pertinent shards based on specific queries. This not only curtails search-related overhead but also enhances operational efficiency. Through rigorous testing using the Gov2 dataset, the HSSA algorithm has proven its merits. When set against well-established algorithms like CORI, Rank-S, and SHiRE, HSSA stands out, registering remarkable gains in average throughput by 21%, 16%, and 12%, while also slashing latency by 14.2%, 9.4%, and 8.2%, respectively. The insights gained from this research underscore HSSA's capability to effectively bridge the gaps inherent in traditional shard selection strategies. Furthermore, its exemplary efficacy with datasets of varied sizes amplifies its relevance for practical integration within distributed processing landscapes.


sharding, cluster, indexing, partitioning, allocation


Download data is not yet available.


N. Venkateswaran and S. Changder, "Simplified data partitioning in a consistent hashing based sharding implementation," in TENCON 2017 - 2017 IEEE Region 10 Conference, Penang, Malaysia, Aug. 2017, pp. 895–900.

A. Kulkarni, A. S. Tigelaar, D. Hiemstra, and J. Callan, "Shard ranking and cutoff estimation for topically partitioned collections," in Proceedings of the 21st ACM international conference on Information and knowledge management, New York, NY, USA, Jul. 2012, pp. 555–564.

J. Kamal, M. Murshed, and R. Buyya, "Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications," Future Generation Computer Systems, vol. 56, pp. 421–435, Mar. 2016.

H. R. Mohammad, K. Xu, J. Callan, and J. S. Culpepper, "Dynamic Shard Cutoff Prediction for Selective Search," in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, New York, NY, USA, Mar. 2018, pp. 85–94.

P. M. Dhulavvagol, V. H. Bhajantri, and S. G. Totad, "Performance Analysis of Distributed Processing System using Shard Selection Techniques on Elasticsearch," Procedia Computer Science, vol. 167, pp. 1626–1635, Jan. 2020.

Z. Dai, C. Xiong, and J. Callan, "Query-Biased Partitioning for Selective Search," in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, New York, NY, USA, Jul. 2016, pp. 1119–1128.

A. Kulkarni and J. Callan, "Selective Search: Efficient and Effective Search of Large Textual Collections," ACM Transactions on Information Systems, vol. 33, no. 4, pp. 17:1-17:33, Dec. 2015.

P. M. Dhulavvagol, S. G. Totad, and S. Sourabh, "Performance Analysis of Job Scheduling Algorithms on Hadoop Multi-cluster Environment," in Emerging Research in Electronics, Computer Science and Technology, Singapore, 2019, pp. 457–470.

N. C. Kundur, B. C. Anil, P. M. Dhulavvagol, R. Ganiger, and B. Ramadoss, "Pneumonia Detection in Chest X-Rays using Transfer Learning and TPUs," Engineering, Technology & Applied Science Research, vol. 13, no. 5, pp. 11878–11883, Oct. 2023.

E. Rodrigues and R. Morla, "Run Time Prediction for Big Data Iterative ML Algorithms: a KMeans case study," Oct. 2017.

M. Ali, N. Q. Soomro, H. Ali, A. Awan, and M. Kirmani, "Distributed File Sharing and Retrieval Model for Cloud Virtual Environment," Engineering, Technology & Applied Science Research, vol. 9, no. 2, pp. 4062–4065, Apr. 2019.

N. Jayakumar and A. M. Kulkarni, "A Simple Measuring Model for Evaluating the Performance of Small Block Size Accesses in Lustre File System," Engineering, Technology & Applied Science Research, vol. 7, no. 6, pp. 2313–2318, Dec. 2017.


How to Cite

P. M. Dhulavvagol and S. G. Totad, “Performance Enhancement of Distributed Processing Systems Using Novel Hybrid Shard Selection Algorithm”, Eng. Technol. Appl. Sci. Res., vol. 14, no. 2, pp. 13720–13725, Apr. 2024.


Abstract Views: 39
PDF Downloads: 71

Metrics Information

Most read articles by the same author(s)