Build Performance Improvement (Optimization)
Last October, I wrote about optimizing Yocto build performance in a post on Yocto’s website and briefly mentioned some of the work we did at the company, such as applying clang and ccache.
In this article, I aim to share insights based on my experience on improving build performance in embedded software development. This should provide a broad perspective on enhancing performance for builds involving operating systems like Yocto Linux, QNX, and Android.”
As multiple vehicle ECUs are integrated into a single unit, high-performance SoCs are required, running multiple operating systems on a hypervisor-based platform. Building each of these operating systems takes a long time, and using the cloud can be quite costly. Our current project is like this, and we are continuously working on improving performance.
Yesterday, we had a meeting with a consulting company, and our shared interest was optimizing cloud costs. Naturally, improving build performance leads to cost optimization, but policy considerations also play a significant role.
Honestly, when we look at the project, there are many areas where cloud infrastructure costs can be optimized. The same goes for managing the team’s AWS cloud, where we have actually managed to reduce costs significantly.
Methods for Improving Build Performance
1. Use Latest Generation CPUs: Naturally, the latest generation CPUs offer faster single-core speeds than previous generations. Currently, AWS supports up to the 7th generation of Intel CPUs.
2. Use Optimal AWS Instance Types: This can lead to significant cost savings. For example, instance types that start with “m” are general-purpose and have more RAM, making them more expensive compared to compute-optimized types that start with “c.” Choosing the right instance type can result in substantial cost reductions.
3. Consider Using AMD or Graviton: AMD is often cheaper than Intel and sometimes offers better build performance. However, older AMD Ryzen generations can occasionally cause segmentation faults during parallel builds, so they were not previously recommended. If testing shows these issues no longer occur, it could be worth considering AMD for cost optimization. Graviton, based on the ARM64 architecture, is cheaper but may not support all builds, requiring additional evaluation.
4. Switch Compilers (gcc to clang): Clang offers advantages such as build profiling, tool support, and parallel linker support, making it highly recommended for components with long linking times. We experienced significant reductions in build times, such as from 5 hours to 15 minutes, and from 1 hour 30 minutes to 20 minutes for certain components. You might want to read this article for more information.
5. Use Caching: Incremental builds are highly recommended. For Yocto Linux, using a combination of the provided shared state cache and ccache is advised. Bazel also offers robust caching options and is recommended. For C/C++ code based on CMake or Make, applying ccache is suggested.
6. Consider Using S3 Buckets: Many use EFS for shared storage in AWS, but considering S3 buckets can result in significant cloud cost savings.
7. Retention Policies: Decide on a project-specific policy for how long caches or artifacts are retained. Cloud storage costs can be higher than expected.
8. Streamline Dependencies: Remove unnecessary build dependencies and optimize component dependencies architecturally.
9. Optimize MR (Merge Request) / PR (Pull Request) Builds: Ideally, every MR/PR should build all images and targets, but this can be costly. It’s important to manage this effectively according to the project’s needs.
10. Optimize Random Pipeline Issues: Random build errors lead developers to retrigger builds, incurring cloud costs if using cloud resources.
11. Optimize Yocto Recipes: Poorly configured recipes can prevent Yocto cache (Shared State Cache) hits. Regularly review and update these configurations.
12. Optimize Build Triggers: Set triggers to initiate builds only when necessary to reduce unnecessary builds. For example, configure builds to trigger only when there are code changes.
In addition to these methods, other strategies can be applied according to the project’s needs. I plan to continue documenting ideas that seem beneficial as the work progresses. However, be cautious not to focus so much on optimization and cost reduction that essential development aspects are neglected, as this may lead to additional costs elsewhere. Compare the pros and cons carefully when implementing these strategies.