Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2597 - ccache hit-rate very low #7435

Open
openwrt-bot opened this issue Nov 14, 2019 · 14 comments
Open

FS#2597 - ccache hit-rate very low #7435

openwrt-bot opened this issue Nov 14, 2019 · 14 comments
Labels

Comments

@openwrt-bot
Copy link

mike-meshplusplus:

I've been using the CONFIG_CCACHE=y .config setting lately, and I notice a very low hit rate in the ccache statistics.

There are two relevant ccache stats, the stats for the host ccache, and the stats for the build tree ccache.

I've set the CCACHE_DIR variable to point to a location outside of the build tree (by modifying the make files appropriately), so that it can be used for multiple build trees.

Note: I do not have the host ccache set to inject itself into any compilation. It's opt-in only on my machine.

I've now ran the following script (slight psuedo code), about 10 times, and see the following results.

git clone myOpenWRTFork cd myOpenWRTFork make rm -rf myOpenWRTFork

and observe the following statistics.

cache directory /home/user/workspace/.ccache primary config /home/user/workspace/.ccache/ccache.conf secondary config (readonly) /etc/ccache.conf stats updated Thu Nov 14 13:45:36 2019 cache hit (direct) 44296 cache hit (preprocessed) 5983 cache miss 128029 cache hit rate 28.20 % called for link 10954 called for preprocessing 19306 multiple source files 160 compiler produced stdout 8 compiler produced empty output 962 compile failed 10957 preprocessor error 3105 bad compiler arguments 1524 unsupported source language 60 autoconf compile/link 32353 unsupported compiler option 259 no input file 5295 cleanups performed 0 files in cache 295717 cache size 3.4 GB max cache size 5.0 GB

ccache version 3.7.2

cache directory /home/user/.ccache
primary config /home/user/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Thu Nov 14 13:45:09 2019
stats zeroed Wed Nov 13 13:24:30 2019
cache hit (direct) 902
cache hit (preprocessed) 0
cache miss 32139
cache hit rate 2.73 %
called for link 219
called for preprocessing 424
preprocessor error 319
unsupported code directive 25
no input file 1637
cleanups performed 0
files in cache 95601
cache size 2.5 GB
max cache size 5.0 GB

ccache version 3.7.4

There's a couple of problems here:

  1. It's surprising that the host ccache is used at all (As stated before, I don't have my host's ccache set to inject itself, it's opt-in only). My expectation is that when CONFIG_CCACHE=y is set, one of the earliest packages to be compiled would be ccache, and then the CCACHE_DIR that the buildsystem sets would be used from then on. I suppose that using the host ccache while compiling the build tree ccache isn't unreasonable, but that doesn't really seem to be what's happening.
  2. A 30% cache hit rate after rebuilding an identical build tree 10 times is extremely low. Since I'm building exactly the same code, every time, after 10 builds, I would expect a 90% hit rate. 1/10th to populate the cache, and 9/10 times loading from the cache.

An observation that I made while letting a build run is that the build tree also seems to sometimes call ccache, and sometimes not, when compiling packages for the target. It's not clear to me what the distinction is. Perhaps the package specific build system? Does ccache not work properly for, e.g. cmake projects?

@openwrt-bot
Copy link
Author

yousong:

Please share with us the content of ccache.conf

It's also useful to refer to the ccache.conf prepared by buildbot: https://git.openwrt.org/?p=buildbot.git;a=blob;f=scripts/ccache.sh;h=8756bc25b1ba8969143d54d03e4fd8b4db5942ef;hb=HEAD

@openwrt-bot
Copy link
Author

mike-meshplusplus:

The ccache.conf file is the default one generated when ccache starts using a newly initialized CCACHE_DIR.

max_size = 5.0G

My expectation was that the OpenWRT build root would generate a ccache.conf that was, maybe not perfect, but suitable for building with OpenWRT.

@openwrt-bot
Copy link
Author

jow-:

I gave up on using ccache for the official build setup, it ended up introducing more problems than it solved. I'd even be in favor of removing ccache support completely.

@openwrt-bot
Copy link
Author

mike-meshplusplus:

What problems did it introduce?

Right now I'm seeing build times in the 1-2 hour range for my project, and it's quite a pain.

I've changed my ccache.conf file similar to how https://git.openwrt.org/?p=buildbot.git;a=blob;f=scripts/ccache.sh;h=8756bc25b1ba8969143d54d03e4fd8b4db5942ef;hb=HEAD does it, and am rebuilding. I'll be able to report back tomorrow on if the ccache.conf settings made any difference.

@openwrt-bot
Copy link
Author

mike-meshplusplus:

Adding

compiler_check = %compiler% -dumpmachine; %compiler% -dumpversion

Based on the build bot configuration, increased my hit rate substantially. Now i'm at 71%, out of 5 builds, which is much closer to what I expected.

cache directory /home/jonesmz/meshpp-workspace/other/.ccache primary config /home/jonesmz/meshpp-workspace/other/.ccache/ccache.conf secondary config (readonly) /etc/ccache.conf stats updated Sat Nov 16 00:16:15 2019 stats zeroed Fri Nov 15 19:30:51 2019 cache hit (direct) 102568 cache hit (preprocessed) 7882 cache miss 43371 cache hit rate 71.80 % called for link 9249 called for preprocessing 17086 multiple source files 140 compiler produced stdout 7 compiler produced empty output 828 compile failed 9531 preprocessor error 2719 bad compiler arguments 1317 unsupported source language 53 autoconf compile/link 28240 unsupported compiler option 222 no input file 4565 cleanups performed 0 files in cache 102939 cache size 1.3 GB max cache size 5.0 GB ccache version 3.7.2

Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2019 Joel Rosdahl

@openwrt-bot
Copy link
Author

mike-meshplusplus:

Unfortunately, I'm not seeing any advantage to using ccache, even with the 78% hit rate.

Builds are still taking approximately 80 minutes.

real 75m5.512s
user 452m9.871s
sys 86m50.363s

@openwrt-bot
Copy link
Author

rinsingpromptly:

My 5¢:
Every "make clean" deletes staging_dir/target-arch_libc_device
The ccache directory for each target is located in directory staging_dir/target-arch_libc_device/ccache , so running make clean will empty the ccache for the target...

Running make like "CCACHE_DIR="any_dir" make" doesnt really help, as CCACHE_DIR gets overwritten in a few *.mk files.
I modified rules.mk and include/package.mk in order to use a centralised CCACHE_DIR

@openwrt-bot
Copy link
Author

mike-meshplusplus:

@darth Vaper

Actually, I patched the OpenWRT build tree to force the CCACHE_DIR to live outside of the OpenWRT tree.

I know that the CCACHE_DIR patch works

It just doesn't help build times at all.

@openwrt-bot
Copy link
Author

mike-meshplusplus:

The specific directory I'm changing it to in the patch below is just to a specific folder on my machine. Nothing special about it.

From 6f2732310f8c8f0e0d71379be9140d31c8f02d72 Mon Sep 17 00:00:00 2001
From: Michael Jones mike@meshplusplus.com
Date: Wed, 13 Nov 2019 15:40:18 -0600
Subject: [PATCH] Modify CCACHE_DIR


include/host-build.mk | 2 +-
include/package.mk | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/host-build.mk b/include/host-build.mk
index a2a31ae048..05334aa121 100644
--- a/include/host-build.mk
+++ b/include/host-build.mk
@@ -132,7 +132,7 @@ define Host/Exports/Default
$(1) : export STAGING_PREFIX=$$(HOST_BUILD_PREFIX)
$(1) : export PKG_CONFIG_PATH=$$(STAGING_DIR_HOST)/lib/pkgconfig:$$(HOST_BUILD_PREFIX)/lib/pkgconfig
$(1) : export PKG_CONFIG_LIBDIR=$$(HOST_BUILD_PREFIX)/lib/pkgconfig

  • $(1) : export CCACHE_DIR:=$(STAGING_DIR_HOST)/ccache
  • $(1) : export CCACHE_DIR:=$(STAGING_DIR_HOST)/../../../../../../../../.ccache
    $(if $(HOST_CONFIG_SITE),$(1) : export CONFIG_SITE:=$(HOST_CONFIG_SITE))
    $(if $(IS_PACKAGE_BUILD),$(1) : export PATH=$$(TARGET_PATH_PKG))
    endef
    diff --git a/include/package.mk b/include/package.mk
    index 2473eecb92..e562b0dd81 100644
    --- a/include/package.mk
    +++ b/include/package.mk
    @@ -144,7 +144,7 @@ define Build/Exports/Default
    $(1) : export CONFIG_SITE:=$$(CONFIG_SITE)
    $(1) : export PKG_CONFIG_PATH:=$$(PKG_CONFIG_PATH)
    $(1) : export PKG_CONFIG_LIBDIR:=$$(PKG_CONFIG_PATH)
  • $(1) : export CCACHE_DIR:=$(STAGING_DIR)/ccache
  • $(1) : export CCACHE_DIR:=$(STAGING_DIR)/../../../../../../../../.ccache
    endef
    Build/Exports=$(Build/Exports/Default)

--
2.23.0

@openwrt-bot
Copy link
Author

rinsingpromptly:

I didn't bother to patch host-build.mk, because I only build for one target. However, I've added a patch to rules.mk
I do see significantly good cache hit rates and speedups with that, see patch below...

diff --git a/.gitignore b/.gitignore
index 6549af83be..d71f88606e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -28,3 +28,4 @@ TAGS*~
git-src
.project
.cproject
+/ccache
diff --git a/include/package.mk b/include/package.mk
index c541f6edf7..d8ebe4d01f 100644
--- a/include/package.mk
+++ b/include/package.mk
@@ -173,7 +173,7 @@ define Build/Exports/Default
$(1) : export CONFIG_SITE:=$$(CONFIG_SITE)
$(1) : export PKG_CONFIG_PATH:=$$(PKG_CONFIG_PATH)
$(1) : export PKG_CONFIG_LIBDIR:=$$(PKG_CONFIG_PATH)

  • $(if $(CONFIG_CCACHE),$(1) : export CCACHE_DIR:=$(STAGING_DIR)/ccache)
  • $(if $(CONFIG_CCACHE),$(1) : export CCACHE_DIR:=$(TOPDIR)/ccache)
    endef
    Build/Exports=$(Build/Exports/Default)

diff --git a/rules.mk b/rules.mk
index 80cb3d63f4..44e8e4ef07 100644
--- a/rules.mk
+++ b/rules.mk
@@ -297,6 +297,8 @@ ifneq ($(CONFIG_CCACHE),)
TARGET_CXX:= ccache_cxx
HOSTCC:= ccache $(HOSTCC)
HOSTCXX:= ccache $(HOSTCXX)

  • CCACHE_DIR:=$(TOPDIR)/ccache
  • export CCACHE_DIR
    endif

TARGET_CONFIGURE_OPTS =

@openwrt-bot
Copy link
Author

mike-meshplusplus:

@darth Vaper

Have you timed a build with an empty ccache dir, and with a primed ccache dir?

@openwrt-bot
Copy link
Author

mike-meshplusplus:

From 39b8f1f83906059645d40bb62ac89ae6f12beb26 Mon Sep 17 00:00:00 2001 From: Michael Jones Date: Wed, 13 Nov 2019 15:40:18 -0600 Subject: [PATCH] Modify CCACHE_DIR

include/host-build.mk | 3 ++-
include/package.mk | 3 ++-
rules.mk | 2 ++
3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/include/host-build.mk b/include/host-build.mk
index a2a31ae048..8f47975597 100644
--- a/include/host-build.mk
+++ b/include/host-build.mk
@@ -132,7 +132,9 @@ define Host/Exports/Default
$(1) : export STAGING_PREFIX=$$(HOST_BUILD_PREFIX)
$(1) : export PKG_CONFIG_PATH=$$(STAGING_DIR_HOST)/lib/pkgconfig:$$(HOST_BUILD_PREFIX)/lib/pkgconfig
$(1) : export PKG_CONFIG_LIBDIR=$$(HOST_BUILD_PREFIX)/lib/pkgconfig

  • $(1) : export CCACHE_DIR:=$(STAGING_DIR_HOST)/ccache
  • $(1) : export CCACHE_DIR:=$(TOPDIR)/../../../../../../.ccache
  • $(1) : export CCACHE_BASEDIR:=$(TOPDIR)
  • $(1) : export CCACHE_COMPILERCHECK:=%compiler% -dumpmachine; %compiler% -dumpversion
    $(if $(HOST_CONFIG_SITE),$(1) : export CONFIG_SITE:=$(HOST_CONFIG_SITE))
    $(if $(IS_PACKAGE_BUILD),$(1) : export PATH=$$(TARGET_PATH_PKG))
    endef
    diff --git a/include/package.mk b/include/package.mk
    index 2473eecb92..52e2640947 100644
    --- a/include/package.mk
    +++ b/include/package.mk
    @@ -144,7 +144,9 @@ define Build/Exports/Default
    $(1) : export CONFIG_SITE:=$$(CONFIG_SITE)
    $(1) : export PKG_CONFIG_PATH:=$$(PKG_CONFIG_PATH)
    $(1) : export PKG_CONFIG_LIBDIR:=$$(PKG_CONFIG_PATH)
  • $(1) : export CCACHE_DIR:=$(STAGING_DIR)/ccache
  • $(1) : export CCACHE_DIR:=$(TOPDIR)/../../../../../../.ccache
  • $(1) : export CCACHE_BASEDIR:=$(TOPDIR)
  • $(1) : export CCACHE_COMPILERCHECK:=%compiler% -dumpmachine; %compiler% -dumpversion
    endef
    Build/Exports=$(Build/Exports/Default)

diff --git a/rules.mk b/rules.mk
index a97b2d2155..299538f590 100644
--- a/rules.mk
+++ b/rules.mk
@@ -293,6 +293,9 @@ ifneq ($(CONFIG_CCACHE),)
TARGET_CXX:= ccache_cxx
HOSTCC:= ccache $(HOSTCC)
HOSTCXX:= ccache $(HOSTCXX)

  • export CCACHE_DIR:=$(TOPDIR)/../../../../../../.ccache
  • export CCACHE_BASEDIR:=$(TOPDIR)
  • export CCACHE_COMPILERCHECK:=%compiler% -dumpmachine; %compiler% -dumpversion
    endif

TARGET_CONFIGURE_OPTS = \

2.24.1

This patch results in the following

empty-cache:
real 89m59.435s
user 655m2.642s
sys 127m23.095s

Second build:
real 72m43.975s
user 376m30.853s
sys 82m32.405s

Third build:
real 71m34.378s
user 374m18.632s
sys 81m57.795s

Stats after the third build:

cache directory /home/jonesmz/meshpp-workspace/other/.ccache/ primary config /home/jonesmz/meshpp-workspace/other/.ccache//ccache.conf stats updated Thu Mar 12 23:17:42 2020 cache hit (direct) 120296 cache hit (preprocessed) 9701 cache miss 50902 cache hit rate 71.86 % called for link 10397 called for preprocessing 18846 multiple source files 120 compiler produced stdout 6 compiler produced empty output 804 compile failed 10545 preprocessor error 3208 bad compiler arguments 1542 unsupported source language 60 autoconf compile/link 31389 unsupported compiler option 222 unsupported code directive 18 no input file 6006 cleanups performed 0 files in cache 126698 cache size 1.8 GB max cache size 5.0 GB

So despite the 71.86% hit-rate that ccache gets, I'm still seeing no meaningful speedup.

This strongly indicates that there's some other bottleneck in the openwrt build process.

A lack of parallelism, or overly intense processing by GNU Make.

@openwrt-bot
Copy link
Author

rinsingpromptly:

1st build
make -j8 4510,89s user 628,02s system 647% cpu 13:13,86 total
2nd build failed, i simply ran make clean again
(happens reliably after clearing cache, but doesnt happen again, strangely enough)
3rd build
make -j8 1463,52s user 287,47s system 477% cpu 6:06,49 total

cpu time used drops by 2/3 and i get a 100% speedup in total build time.
This is a quad-core with ht, nvme ssd and 32gb ram. I am only building one target and only the packages I actually need on the device.

I only do make clean, not make dirclean (why re-build the toolchain every time, right?)

From your stats I can see that cpu time used gets down by 50%, perhaps a slow disk prevents further speed-ups. Are you using CONFIG_AUTOREMOVE? Perhaps there is some other bottleneck when building for more than one target...

@openwrt-bot
Copy link
Author

mike-meshplusplus:

This is happening on a build server for my organization.

So the whole build is being done from scratch. The only thing being kept around is the .ccache directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant