Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FS#2429 - Container Support Enabled by Default for X86 builds #7290

Closed
openwrt-bot opened this issue Aug 6, 2019 · 4 comments
Closed

FS#2429 - Container Support Enabled by Default for X86 builds #7290

openwrt-bot opened this issue Aug 6, 2019 · 4 comments
Labels
flyspray

Comments

@openwrt-bot
Copy link

najdanovicivan:

Hi, Will it be possible to make Release build for 19.07 for X86 and X86_64 have Kernel compiled with full support for LXC/Docker. The idea is to have kernel only for x86 based build compiled with support so that all of use who want to use containers on the router don't have to recompile the kernel and be able to use kmod packages from the official repo.

Here are the kernel modules which should be included.

CONFIG_KERNEL_AIO=y
CONFIG_KERNEL_BLK_CGROUP=y
CONFIG_KERNEL_BLK_DEV_BSG=y
CONFIG_KERNEL_BLK_DEV_THROTTLING=y
CONFIG_KERNEL_BLK_DEV_THROTTLING_LOW=y
CONFIG_KERNEL_CFQ_GROUP_IOSCHED=y
CONFIG_KERNEL_CGROUPS=y
CONFIG_KERNEL_CGROUP_CPUACCT=y
CONFIG_KERNEL_CGROUP_DEVICE=y
CONFIG_KERNEL_CGROUP_FREEZER=y
CONFIG_KERNEL_CGROUP_PERF=y
CONFIG_KERNEL_CGROUP_PIDS=y
CONFIG_KERNEL_CGROUP_SCHED=y
CONFIG_KERNEL_CPUSETS=y
CONFIG_KERNEL_DEVPTS_MULTIPLE_INSTANCES=y
CONFIG_KERNEL_DEVTMPFS=y
CONFIG_KERNEL_DEVTMPFS_MOUNT=y
CONFIG_KERNEL_DIRECT_IO=y
CONFIG_KERNEL_FANOTIFY=y
CONFIG_KERNEL_FHANDLE=y
CONFIG_KERNEL_FREEZER=y
CONFIG_KERNEL_IPC_NS=y
CONFIG_KERNEL_LXC_MISC=y
CONFIG_KERNEL_MEMCG=y
CONFIG_KERNEL_MEMCG_KMEM=y
CONFIG_KERNEL_MEMCG_SWAP=y
CONFIG_KERNEL_MEMCG_SWAP_ENABLED=y
CONFIG_KERNEL_MM_OWNER=y
CONFIG_KERNEL_NAMESPACES=y
CONFIG_KERNEL_NETPRIO_CGROUP=y
CONFIG_KERNEL_NET_CLS_CGROUP=y
CONFIG_KERNEL_NET_NS=y
CONFIG_KERNEL_PERF_EVENTS=y
CONFIG_KERNEL_PID_NS=y
CONFIG_KERNEL_POSIX_MQUEUE=y
CONFIG_KERNEL_PROC_PID_CPUSET=y
CONFIG_KERNEL_RESOURCE_COUNTERS=y
CONFIG_KERNEL_USER_NS=y
CONFIG_KERNEL_UTS_NS=y

I've already tested building docked from this feed https://gitlab.com/mcbridematt/openwrt-container-feed with custom build OpenWrt images. But if any additional kmod package is need you'll have to compile it as well. Additionally there is a great guide (in Russian) https://habr.com/ru/post/341370/ on how to use LXC on OpenWrt

I completely understand the reason for not having these kernel modules on consumer router devices due to storage space limitation but most of users that use X86 for the router have a lot more disk space available.

Personaly I'm using APU3 Board with 64GB SSD to run OpenWrt and I want to use containers to run HomeAssistant on it and also as a PHP developer I can use Docker to run my Development server directly on the router so that I can easily switch between working on desktop and laptop.
@openwrt-bot
Copy link
Author

Hauke:

In master a lot of these options are activated, see:
https://git.openwrt.org/fcb41decf6c622482b20af45a77e62db8d95046e

Is this sufficient for you?

@openwrt-bot
Copy link
Author

najdanovicivan:

I've tried running the dockerd on latest snapshot build and here is what I got

WARN[2019-08-16T10:08:27.908909643Z] Your kernel does not support swap memory limit
WARN[2019-08-16T10:08:27.909080026Z] Your kernel does not support cgroup cfs period
WARN[2019-08-16T10:08:27.909186235Z] Your kernel does not support cgroup cfs quotas
WARN[2019-08-16T10:08:27.909633089Z] Your kernel does not support cgroup blkio weight
WARN[2019-08-16T10:08:27.909874410Z] Your kernel does not support cgroup blkio weight_device
WARN[2019-08-16T10:08:27.910060147Z] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[2019-08-16T10:08:27.910215856Z] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[2019-08-16T10:08:27.910551990Z] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[2019-08-16T10:08:27.910681759Z] Your kernel does not support cgroup blkio throttle.write_iops_device

I suppose it's due to some of those flags missing

KERNEL_BLK_DEV_BSG=y
KERNEL_BLK_DEV_THROTTLING=y
KERNEL_BLK_DEV_THROTTLING_LOW=y
KERNEL_CFQ_GROUP_IOSCHED=y
KERNEL_CGROUP_PERF=y
KERNEL_DEVTMPFS=y
KERNEL_DEVTMPFS_MOUNT=y
KERNEL_MEMCG_SWAP=y
KERNEL_MEMCG_SWAP_ENABLED=y
KERNEL_PERF_EVENTS=y
KERNEL_PROC_PID_CPUSET=y

Can those be added as well for !SMALL_FLASH ?

@openwrt-bot
Copy link
Author

najdanovicivan:

I've used the prebuild docker binaries from https://download.docker.com/linux/static/stable/x86_64/ to test

I also tried to use https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh to check config but the build was missing configs module so I've transfered the configs.ko from SDK and loaded the module.

After that I was able to run the script and here is what I got

info: reading kernel config from /proc/config.gz ...

Generally Necessary:

  • cgroup hierarchy: single mountpoint! [/sys/fs/cgroup]
    (see https://github.com/tianon/cgroupfs-mount)
  • CONFIG_NAMESPACES: enabled
  • CONFIG_NET_NS: enabled
  • CONFIG_PID_NS: enabled
  • CONFIG_IPC_NS: enabled
  • CONFIG_UTS_NS: enabled
  • CONFIG_CGROUPS: enabled
  • CONFIG_CGROUP_CPUACCT: enabled
  • CONFIG_CGROUP_DEVICE: enabled
  • CONFIG_CGROUP_FREEZER: enabled
  • CONFIG_CGROUP_SCHED: enabled
  • CONFIG_CPUSETS: enabled
  • CONFIG_MEMCG: enabled
  • CONFIG_KEYS: enabled
  • CONFIG_VETH: enabled (as module)
  • CONFIG_BRIDGE: enabled
  • CONFIG_BRIDGE_NETFILTER: enabled (as module)
  • CONFIG_NF_NAT_IPV4: enabled (as module)
  • CONFIG_IP_NF_FILTER: enabled (as module)
  • CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
  • CONFIG_IP_NF_NAT: enabled (as module)
  • CONFIG_NF_NAT: enabled (as module)
  • CONFIG_NF_NAT_NEEDED: enabled
  • CONFIG_POSIX_MQUEUE: enabled

Optional Features:

  • CONFIG_USER_NS: enabled
  • CONFIG_SECCOMP: enabled
  • CONFIG_CGROUP_PIDS: enabled
  • CONFIG_MEMCG_SWAP: missing
  • CONFIG_MEMCG_SWAP_ENABLED: missing
  • CONFIG_LEGACY_VSYSCALL_NONE: enabled
    (containers using eglibc <= 2.13 will not work. Switch to
    "CONFIG_VSYSCALL_[NATIVE|EMULATE]" or use "vsyscall=[native|emulate]"
    on kernel command line. Note that this will disable ASLR for the,
    VDSO which may assist in exploiting security vulnerabilities.)
  • CONFIG_BLK_CGROUP: enabled
  • CONFIG_BLK_DEV_THROTTLING: missing
  • CONFIG_IOSCHED_CFQ: missing
  • CONFIG_CFQ_GROUP_IOSCHED: missing
  • CONFIG_CGROUP_PERF: missing
  • CONFIG_CGROUP_HUGETLB: missing
  • CONFIG_NET_CLS_CGROUP: enabled
  • CONFIG_CGROUP_NET_PRIO: missing
  • CONFIG_CFS_BANDWIDTH: missing
  • CONFIG_FAIR_GROUP_SCHED: enabled
  • CONFIG_RT_GROUP_SCHED: enabled
  • CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
  • CONFIG_IP_VS: enabled (as module)
  • CONFIG_IP_VS_NFCT: enabled
  • CONFIG_IP_VS_PROTO_TCP: enabled
  • CONFIG_IP_VS_PROTO_UDP: enabled
  • CONFIG_IP_VS_RR: enabled (as module)
  • CONFIG_EXT4_FS: enabled
  • CONFIG_EXT4_FS_POSIX_ACL: missing
  • CONFIG_EXT4_FS_SECURITY: missing
    enable these ext4 configs if you are using ext3 or ext4 as backing filesystem
  • Network Drivers:
    • "overlay":
      • CONFIG_VXLAN: enabled (as module)
      • CONFIG_BRIDGE_VLAN_FILTERING: enabled
        Optional (for encrypted networks):
        • CONFIG_CRYPTO: enabled
        • CONFIG_CRYPTO_AEAD: enabled
        • CONFIG_CRYPTO_GCM: enabled (as module)
        • CONFIG_CRYPTO_SEQIV: enabled (as module)
        • CONFIG_CRYPTO_GHASH: enabled (as module)
        • CONFIG_XFRM: enabled
        • CONFIG_XFRM_USER: enabled (as module)
        • CONFIG_XFRM_ALGO: enabled (as module)
        • CONFIG_INET_ESP: enabled (as module)
        • CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
    • "ipvlan":
      • CONFIG_IPVLAN: missing
    • "macvlan":
      • CONFIG_MACVLAN: enabled (as module)
      • CONFIG_DUMMY: enabled (as module)
    • "ftp,tftp client in container":
      • CONFIG_NF_NAT_FTP: enabled (as module)
      • CONFIG_NF_CONNTRACK_FTP: enabled (as module)
      • CONFIG_NF_NAT_TFTP: enabled (as module)
      • CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
  • Storage Drivers:
    • "aufs":
      • CONFIG_AUFS_FS: missing
    • "btrfs":
      • CONFIG_BTRFS_FS: enabled (as module)
      • CONFIG_BTRFS_FS_POSIX_ACL: missing
    • "devicemapper":
      • CONFIG_BLK_DEV_DM: enabled (as module)
      • CONFIG_DM_THIN_PROVISIONING: missing
    • "overlay":
      • CONFIG_OVERLAY_FS: enabled
    • "zfs":
      • /dev/zfs: missing
      • zfs command: missing
      • zpool command: missing

Limits:

  • /proc/sys/kernel/keys/root_maxkeys: 1000000

So the flash that should also be enabled for !SMAL_FLASH Are

KERNEL_MEMCG_SWAP
KERNEL_MEMCG_SWAP_ENABLED
KERNEL_BLK_DEV_THROTTLING
KERNEL_IOSCHED_CFQ
KERNEL_CFQ_GROUP_IOSCHED
KERNEL_CGROUP_PERF
KERNEL_CGROUP_HUGETLB
KERNEL_CGROUP_NET_PRIO
KERNEL_CFS_BANDWIDTH

Also enabling USE_FS_ACL_ATTR might bi use full

Also this should be usefull
CONFIG_IPVLAN
CONFIG_AUFS_FS
CONFIG_DM_THIN_PROVISIONING

Also as I have to manually load configs.ko it might be a good idea to incude CONFIG_IKCONFIG as well

I'have attached the ouput of check config script

@openwrt-bot
Copy link
Author

najdanovicivan:

Got it working. I was actually missing the cgroupfs-mount package

So with snapshot build there is already a working docker setup using docker-ce package

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flyspray
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@openwrt-bot