Optimizing distributed build time in large industrial build cluster

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


MPhil Thesis Defence


Title: "Optimizing distributed build time in large industrial build cluster"

By

Mr. Bangyan DU


Abstract:

Large IT companies encounter significant compilation demands due to their 
extensive codebases, particularly when managing and hosting them in a 
multirepo approach. For instance, WeChat has over 1,500 code repositories 
totaling more than 800 GB, covering various services. With more than 800 
frontline developers making over 4,000 code sub- missions daily, the average 
compilation demand exceeds 20,000 requests. Distributed build clusters 
encounter two main challenges: resource scheduling and the absence of dis- 
tributed preprocessing. Firstly, effective resource scheduling is crucial 
for optimizing the use of existing cluster to compile more C++ files 
quickly. The second challenge stems from the lack of distributed 
preprocessing in a build cluster, primarily due to the high costs associated 
with uploading all related header files from the local master machine to the 
re- mote slave machines for a given C++ source file. To address the first 
issue, we propose a resource prediction model that estimates the compilation 
resources required for each C++ source file. This model facilitates the 
allocation of adequate remote slave machines and helps eliminate typical 
errors like Out of Memory errors and Deadline Exceeded errors in the build 
cluster. For the second issue, we present a two-stage distributed 
preprocessing solution and further minimize the huge number of system calls 
of GCC C++ compiler to help reduce the preprocessing time on the local 
master machine. Our experience report not only sheds light on WeChat’s build 
cluster but also offers insights into effective practices of build time 
reduction for large-scale IT companies.


Date:                   Tuesday, 27 May 2025

Time:                   1:00pm - 3:00pm

Venue:                  Room 2128B
                        Lift 19

Chairman:               Dr. Shuai WANG

Committee Members:      Prof. Charles ZHANG (Supervisor)
                        Dr. Ziyi LIU