OpenWrt/LEDE Project

  • Status Closed
  • Percent Complete
    100%
  • Task Type Feature Request
  • Category Kernel
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Najdanovic Ivan - 06.08.2019
Last edited by Petr Štetiar - 16.09.2019

FS#2429 - Container Support Enabled by Default for X86 builds

Hi, Will it be possible to make Release build for 19.07 for X86 and X86_64 have Kernel compiled with full support for LXC/Docker. The idea is to have kernel only for x86 based build compiled with support so that all of use who want to use containers on the router don’t have to recompile the kernel and be able to use kmod packages from the official repo.

Here are the kernel modules which should be included.

CONFIG_KERNEL_AIO=y
CONFIG_KERNEL_BLK_CGROUP=y
CONFIG_KERNEL_BLK_DEV_BSG=y
CONFIG_KERNEL_BLK_DEV_THROTTLING=y
CONFIG_KERNEL_BLK_DEV_THROTTLING_LOW=y
CONFIG_KERNEL_CFQ_GROUP_IOSCHED=y
CONFIG_KERNEL_CGROUPS=y
CONFIG_KERNEL_CGROUP_CPUACCT=y
CONFIG_KERNEL_CGROUP_DEVICE=y
CONFIG_KERNEL_CGROUP_FREEZER=y
CONFIG_KERNEL_CGROUP_PERF=y
CONFIG_KERNEL_CGROUP_PIDS=y
CONFIG_KERNEL_CGROUP_SCHED=y
CONFIG_KERNEL_CPUSETS=y
CONFIG_KERNEL_DEVPTS_MULTIPLE_INSTANCES=y
CONFIG_KERNEL_DEVTMPFS=y
CONFIG_KERNEL_DEVTMPFS_MOUNT=y
CONFIG_KERNEL_DIRECT_IO=y
CONFIG_KERNEL_FANOTIFY=y
CONFIG_KERNEL_FHANDLE=y
CONFIG_KERNEL_FREEZER=y
CONFIG_KERNEL_IPC_NS=y
CONFIG_KERNEL_LXC_MISC=y
CONFIG_KERNEL_MEMCG=y
CONFIG_KERNEL_MEMCG_KMEM=y
CONFIG_KERNEL_MEMCG_SWAP=y
CONFIG_KERNEL_MEMCG_SWAP_ENABLED=y
CONFIG_KERNEL_MM_OWNER=y
CONFIG_KERNEL_NAMESPACES=y
CONFIG_KERNEL_NETPRIO_CGROUP=y
CONFIG_KERNEL_NET_CLS_CGROUP=y
CONFIG_KERNEL_NET_NS=y
CONFIG_KERNEL_PERF_EVENTS=y
CONFIG_KERNEL_PID_NS=y
CONFIG_KERNEL_POSIX_MQUEUE=y
CONFIG_KERNEL_PROC_PID_CPUSET=y
CONFIG_KERNEL_RESOURCE_COUNTERS=y
CONFIG_KERNEL_USER_NS=y
CONFIG_KERNEL_UTS_NS=y

I’ve already tested building docked from this feed https://gitlab.com/mcbridematt/openwrt-container-feed with custom build OpenWrt images. But if any additional kmod package is need you’ll have to compile it as well. Additionally there is a great guide (in Russian) https://habr.com/ru/post/341370/ on how to use LXC on OpenWrt

I completely understand the reason for not having these kernel modules on consumer router devices due to storage space limitation but most of users that use X86 for the router have a lot more disk space available.

Personaly I’m using APU3 Board with 64GB SSD to run OpenWrt and I want to use containers to run HomeAssistant on it and also as a PHP developer I can use Docker to run my Development server directly on the router so that I can easily switch between working on desktop and laptop.

Closed by  Petr ┼átetiar
16.09.2019 11:57
Reason for closing:  Fixed
Project Manager
Hauke Mehrtens commented on 07.08.2019 18:35

In master a lot of these options are activated, see:
https://git.openwrt.org/fcb41decf6c622482b20af45a77e62db8d95046e

Is this sufficient for you?

Najdanovic Ivan commented on 16.08.2019 10:13

I've tried running the dockerd on latest snapshot build and here is what I got

WARN[2019-08-16T10:08:27.908909643Z] Your kernel does not support swap memory limit
WARN[2019-08-16T10:08:27.909080026Z] Your kernel does not support cgroup cfs period
WARN[2019-08-16T10:08:27.909186235Z] Your kernel does not support cgroup cfs quotas
WARN[2019-08-16T10:08:27.909633089Z] Your kernel does not support cgroup blkio weight
WARN[2019-08-16T10:08:27.909874410Z] Your kernel does not support cgroup blkio weight_device
WARN[2019-08-16T10:08:27.910060147Z] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[2019-08-16T10:08:27.910215856Z] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[2019-08-16T10:08:27.910551990Z] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[2019-08-16T10:08:27.910681759Z] Your kernel does not support cgroup blkio throttle.write_iops_device

I suppose it's due to some of those flags missing

KERNEL_BLK_DEV_BSG=y
KERNEL_BLK_DEV_THROTTLING=y
KERNEL_BLK_DEV_THROTTLING_LOW=y
KERNEL_CFQ_GROUP_IOSCHED=y
KERNEL_CGROUP_PERF=y
KERNEL_DEVTMPFS=y
KERNEL_DEVTMPFS_MOUNT=y
KERNEL_MEMCG_SWAP=y
KERNEL_MEMCG_SWAP_ENABLED=y
KERNEL_PERF_EVENTS=y
KERNEL_PROC_PID_CPUSET=y

Can those be added as well for !SMALL_FLASH ?

Najdanovic Ivan commented on 16.08.2019 11:12

I've used the prebuild docker binaries from https://download.docker.com/linux/static/stable/x86_64/ to test

I also tried to use https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh to check config but the build was missing configs module so I've transfered the configs.ko from SDK and loaded the module.

After that I was able to run the script and here is what I got

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: single mountpoint! [/sys/fs/cgroup]
    (see https://github.com/tianon/cgroupfs-mount)
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
- CONFIG_MEMCG_SWAP_ENABLED: missing
- CONFIG_LEGACY_VSYSCALL_NONE: enabled
    (containers using eglibc <= 2.13 will not work. Switch to
     "CONFIG_VSYSCALL_[NATIVE|EMULATE]" or use "vsyscall=[native|emulate]"
     on kernel command line. Note that this will disable ASLR for the,
     VDSO which may assist in exploiting security vulnerabilities.)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_IOSCHED_CFQ: missing
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: missing
- CONFIG_EXT4_FS_SECURITY: missing
    enable these ext4 configs if you are using ext3 or ext4 as backing filesystem
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

So the flash that should also be enabled for !SMAL_FLASH Are

KERNEL_MEMCG_SWAP
KERNEL_MEMCG_SWAP_ENABLED
KERNEL_BLK_DEV_THROTTLING
KERNEL_IOSCHED_CFQ
KERNEL_CFQ_GROUP_IOSCHED
KERNEL_CGROUP_PERF
KERNEL_CGROUP_HUGETLB
KERNEL_CGROUP_NET_PRIO
KERNEL_CFS_BANDWIDTH

Also enabling USE_FS_ACL_ATTR might bi use full

Also this should be usefull
CONFIG_IPVLAN
CONFIG_AUFS_FS
CONFIG_DM_THIN_PROVISIONING

Also as I have to manually load configs.ko it might be a good idea to incude CONFIG_IKCONFIG as well

I'have attached the ouput of check config script

Najdanovic Ivan commented on 05.09.2019 18:32

Got it working. I was actually missing the cgroupfs-mount package

So with snapshot build there is already a working docker setup using docker-ce package

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing