New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FS#2597 - ccache hit-rate very low #7435
Comments
yousong: Please share with us the content of ccache.conf It's also useful to refer to the ccache.conf prepared by buildbot: https://git.openwrt.org/?p=buildbot.git;a=blob;f=scripts/ccache.sh;h=8756bc25b1ba8969143d54d03e4fd8b4db5942ef;hb=HEAD |
mike-meshplusplus: The ccache.conf file is the default one generated when ccache starts using a newly initialized CCACHE_DIR.
max_size = 5.0G
My expectation was that the OpenWRT build root would generate a ccache.conf that was, maybe not perfect, but suitable for building with OpenWRT. |
jow-: I gave up on using ccache for the official build setup, it ended up introducing more problems than it solved. I'd even be in favor of removing ccache support completely. |
mike-meshplusplus: What problems did it introduce? Right now I'm seeing build times in the 1-2 hour range for my project, and it's quite a pain. I've changed my ccache.conf file similar to how https://git.openwrt.org/?p=buildbot.git;a=blob;f=scripts/ccache.sh;h=8756bc25b1ba8969143d54d03e4fd8b4db5942ef;hb=HEAD does it, and am rebuilding. I'll be able to report back tomorrow on if the ccache.conf settings made any difference. |
mike-meshplusplus: Adding
Based on the build bot configuration, increased my hit rate substantially. Now i'm at 71%, out of 5 builds, which is much closer to what I expected.
cache directory /home/jonesmz/meshpp-workspace/other/.ccache
primary config /home/jonesmz/meshpp-workspace/other/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sat Nov 16 00:16:15 2019
stats zeroed Fri Nov 15 19:30:51 2019
cache hit (direct) 102568
cache hit (preprocessed) 7882
cache miss 43371
cache hit rate 71.80 %
called for link 9249
called for preprocessing 17086
multiple source files 140
compiler produced stdout 7
compiler produced empty output 828
compile failed 9531
preprocessor error 2719
bad compiler arguments 1317
unsupported source language 53
autoconf compile/link 28240
unsupported compiler option 222
no input file 4565
cleanups performed 0
files in cache 102939
cache size 1.3 GB
max cache size 5.0 GB
ccache version 3.7.2
|
mike-meshplusplus: Unfortunately, I'm not seeing any advantage to using ccache, even with the 78% hit rate. Builds are still taking approximately 80 minutes. |
rinsingpromptly: My 5¢: Running make like "CCACHE_DIR="any_dir" make" doesnt really help, as CCACHE_DIR gets overwritten in a few *.mk files. |
mike-meshplusplus: @darth Vaper Actually, I patched the OpenWRT build tree to force the CCACHE_DIR to live outside of the OpenWRT tree. I know that the CCACHE_DIR patch works It just doesn't help build times at all. |
mike-meshplusplus: The specific directory I'm changing it to in the patch below is just to a specific folder on my machine. Nothing special about it.
|
rinsingpromptly: I didn't bother to patch host-build.mk, because I only build for one target. However, I've added a patch to rules.mk
|
mike-meshplusplus: @darth Vaper Have you timed a build with an empty ccache dir, and with a primed ccache dir? |
mike-meshplusplus:
From 39b8f1f83906059645d40bb62ac89ae6f12beb26 Mon Sep 17 00:00:00 2001
From: Michael Jones
Date: Wed, 13 Nov 2019 15:40:18 -0600
Subject: [PATCH] Modify CCACHE_DIR
This patch results in the following empty-cache: Second build: Third build: Stats after the third build:
cache directory /home/jonesmz/meshpp-workspace/other/.ccache/
primary config /home/jonesmz/meshpp-workspace/other/.ccache//ccache.conf
stats updated Thu Mar 12 23:17:42 2020
cache hit (direct) 120296
cache hit (preprocessed) 9701
cache miss 50902
cache hit rate 71.86 %
called for link 10397
called for preprocessing 18846
multiple source files 120
compiler produced stdout 6
compiler produced empty output 804
compile failed 10545
preprocessor error 3208
bad compiler arguments 1542
unsupported source language 60
autoconf compile/link 31389
unsupported compiler option 222
unsupported code directive 18
no input file 6006
cleanups performed 0
files in cache 126698
cache size 1.8 GB
max cache size 5.0 GB
So despite the 71.86% hit-rate that ccache gets, I'm still seeing no meaningful speedup. This strongly indicates that there's some other bottleneck in the openwrt build process. A lack of parallelism, or overly intense processing by GNU Make. |
rinsingpromptly: 1st build cpu time used drops by 2/3 and i get a 100% speedup in total build time. I only do make clean, not make dirclean (why re-build the toolchain every time, right?) From your stats I can see that cpu time used gets down by 50%, perhaps a slow disk prevents further speed-ups. Are you using CONFIG_AUTOREMOVE? Perhaps there is some other bottleneck when building for more than one target... |
mike-meshplusplus: This is happening on a build server for my organization. So the whole build is being done from scratch. The only thing being kept around is the .ccache directory. |
mike-meshplusplus:
I've been using the CONFIG_CCACHE=y .config setting lately, and I notice a very low hit rate in the ccache statistics.
There are two relevant ccache stats, the stats for the host ccache, and the stats for the build tree ccache.
I've set the CCACHE_DIR variable to point to a location outside of the build tree (by modifying the make files appropriately), so that it can be used for multiple build trees.
Note: I do not have the host ccache set to inject itself into any compilation. It's opt-in only on my machine.
I've now ran the following script (slight psuedo code), about 10 times, and see the following results.
git clone myOpenWRTFork cd myOpenWRTFork make rm -rf myOpenWRTFork
and observe the following statistics.
cache directory /home/user/workspace/.ccache primary config /home/user/workspace/.ccache/ccache.conf secondary config (readonly) /etc/ccache.conf stats updated Thu Nov 14 13:45:36 2019 cache hit (direct) 44296 cache hit (preprocessed) 5983 cache miss 128029 cache hit rate 28.20 % called for link 10954 called for preprocessing 19306 multiple source files 160 compiler produced stdout 8 compiler produced empty output 962 compile failed 10957 preprocessor error 3105 bad compiler arguments 1524 unsupported source language 60 autoconf compile/link 32353 unsupported compiler option 259 no input file 5295 cleanups performed 0 files in cache 295717 cache size 3.4 GB max cache size 5.0 GB
ccache version 3.7.2
cache directory /home/user/.ccache
primary config /home/user/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Thu Nov 14 13:45:09 2019
stats zeroed Wed Nov 13 13:24:30 2019
cache hit (direct) 902
cache hit (preprocessed) 0
cache miss 32139
cache hit rate 2.73 %
called for link 219
called for preprocessing 424
preprocessor error 319
unsupported code directive 25
no input file 1637
cleanups performed 0
files in cache 95601
cache size 2.5 GB
max cache size 5.0 GB
ccache version 3.7.4
There's a couple of problems here:
An observation that I made while letting a build run is that the build tree also seems to sometimes call ccache, and sometimes not, when compiling packages for the target. It's not clear to me what the distinction is. Perhaps the package specific build system? Does ccache not work properly for, e.g. cmake projects?
The text was updated successfully, but these errors were encountered: