QQ客服 官方微博 反馈留言

	When computing meets heterogeneous cluster: Workload assignment in graph computation
	Xue, Jilong ; Yang, Zhi ; Hou, Shian ; Dai, Yafei
	2015
英文摘要	In order to process very large graphs, existing graph processing systems, such as Pregel and Giraph, usually partition and distribute the graph computation on large number of nodes (i.e., workers). However, due to the heterogeneity of computing clusters (e.g., nodes with various bandwidth or CPU resource), blindly increasing the number of workers for a job may even degrade the overall performance. In this paper, we address the question of how to distribute the graph computation over the heterogeneous cluster to maximize performance. Based on the practical constraints of current systems, we address this problem in two scenarios. For systems using hash-based partition method (for avoiding the overhead of indexing and searching vertex), we propose a coarse-grained mechanism to greedily select suitable worker set to execute the job. For systems allowing arbitrary graph partition, we further propose a heterogeneity-aware streaming graph partitioning model that can assign workload in fine-grained level. We implement the scheduling mechanisms as a general middleware which can be easily adopted in existing graph computing systems. Our experiments on both university lab cluster (46 machines) and EC2 cluster (100 instances) show that, the proposed framework can significantly improve the execution performance. Compared with the default configurations (i.e., using the whole set of workers and hash-based graph partition), our framework can reduce the overall execution time by 55.9% for lab cluster and 44.7% for EC2 cluster respectively. ? 2015 IEEE.; EI; 154-163
语种	英语
出处	3rd IEEE International Conference on Big Data, IEEE Big Data 2015
DOI标识	10.1109/BigData.2015.7363752
内容类型	其他
源URL	[http://ir.pku.edu.cn/handle/20.500.11897/436404]
专题	信息科学技术学院
推荐引用方式 GB/T 7714	Xue, Jilong,Yang, Zhi,Hou, Shian,et al. When computing meets heterogeneous cluster: Workload assignment in graph computation. 2015-01-01.

个性服务

查看访问统计

相关权益政策

暂无数据

收藏/分享

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。