Debugging Yocto Shared State Cache
The shared state cache (SSTATE) in the Yocto Project is a mechanism used to increase the build speed and efficiency of the build system. This caching system allows for the reuse of previously built artifacts, so the entire build process does not have to be repeated every time.
The above illustration explains Bazel’s Remote Cache, which is similar to Yocto’s Shared State Cache or PREMIRROR. These systems store cache data in a remote, shareable location that is accessed during builds to improve speed by reusing cached build artifacts when there are no changes.
However, sometimes the cache may not hit due to unchanged or incorrect dependencies, causing the build to restart from the source code. This can lead to a cascade of rebuilds for other components due to their dependencies, resulting in significantly increased build times. This article will briefly explain how to debug the Yocto Shared State.
Debugging Methods”
1. First, examine the overall cache hit ratio. After parsing at the start of the build, assess the situation from the sstate summary section as shown in the figure below. Then, extract the list of components for which the source is downloaded and built. You can identify components where tasks like do_configure and do_compile have been executed by checking the log file.
2. Verify whether the components identified in step 1 have a shared sstate cache. The cache is located in the ${TOPDIR}/sstate-cache directory. You can search for it using the combination of the recipe name and task name with the find command as shown below.
In the Yocto Project, the siginfo file is related to the build system’s caching mechanism. The Yocto Project uses various cache files to speed up and efficiently manage the build process. The siginfo file is one of these cache files and primarily serves the following purposes:
• Storing Signature Information: The siginfo file stores signatures of build tasks. This signature is a unique hash value generated based on the task’s inputs, outputs, and other related metadata. It allows the build system to determine which tasks have changed and which need to be rebuilt.
• Checking Reusability: If a build task has already been executed and its result is stored in the cache, a task with the same signature can reuse the cached result without being executed again. This significantly reduces build time.
• Supporting Build Debugging: The siginfo file helps the build system track the flow of the build process and assists in debugging when issues arise. Because signature information is recorded, it is easy to determine which tasks have changed and why they were rebuilt.
3. To debug the cache of the problematic recipe, use the siginfo file. Yocto provides tools such as bitbake-diffsigs and bitbake-dumpsig.
- bitbake-diffsigs compares the differences between two signature files (.siginfo). This helps identify which input or environment variable changes caused the build task to be re-executed. In the example below, the siginfo files of the curl compile task were compared, revealing that the hash value of do_configure changed, prompting a rebuild. You can trace issues step by step this way.”
- bitbake-dumpsig is one of the utilities provided by the BitBake tool in the Yocto Project. It is used to dump the contents of a build task’s signature file, decoding and displaying detailed information in a way that is easy for developers to understand. This utility helps analyze and comprehend the various components contained within the signature file.”
Summary
To debug, navigate to the sstate-cache directory and locate the siginfo file for the relevant recipe, often focusing on the prepare_recipe_sysroot task. The reason for looking at prepare_recipe_sysroot is that, when using diffsigs, it frequently points to other recipes that affect the build. By tracing these issues step by step, you can often find clues to the problem. Additionally, using the bitbake -DDD option can gather more debugging information during the build process, but be aware that this may produce an overwhelming amount of messages, making it challenging to identify meaningful data.
Additional Information
In the Yocto Project, ensuring hash concurrency in the Shared State (SState) is crucial. Maintaining data consistency is especially important when multiple threads or processes access the build environment simultaneously. To achieve this, Yocto uses the hash equivalence setting. This configuration helps effectively manage changes in the hash values of build tasks and ensures concurrency.
By setting up Hash Equivalence, you can guarantee hash concurrency when accessing the cache from different environments. I recommend reading the attached PDF from the Yocto Project Summit 2023. I plan to cover this topic in more detail in a future article.”