Pine, Alpine linux based on OSTree
Choosing what OS runs on your servers is a matter of convenience and familiarity. Convenience means you want something that gives you as less troubles as possible, familiarity means that you would prefer not to learn additional things if you don't have to.
My servers are pets so I am ok manually issuing a few commands every once in a while, and don't require complete automation.
After trying out CoreOS for a year I switched to my own simplified distro based on alpine and ostree.
This version of alpine takes cues from flatcar and project-atomic and is supposed to be installed as a read-only root file-system with updates happening atomically, that is, either they succeed or the system swaps back to the previous state. For this to be possible to system has to always have at least two snapshots of the released file-system version, available on storage.
In what environments will the system run? I targeted OVZ and [KVM], but in general you can say containers and virtual machines with the main difference being that containers don't run their own kernel, in particular they don't have a boot process, they call directly into the init system (which for example in a Dockerfile it would be defined by the CMD or ENTRYPOINT statements), which is responsible to manage the tree of prcesses that will keep the container running (just like a normal session, if the init process dies, the container terminates). Also containers can't configure system knobs, and can have additional restrictions on capabilities.
How is the image built?
The prepare.sh script handles dependencies, most of which are the packages to offer common cli tools like coreutils, util-linux, binutils, utilities to operate with block devices like blkid, sfdisk, multipath-tools and file systems with xfsprogs and e2fsprogs. The squashfs-tools package is used at the end to compress the built root file system. A glib compatibility package is also installed by default because alpine is based on musl, the compatibility package works by providing some libraries built against .
File trees for both VMs and containers are build with respectively make.sh and make_ovz.sh. This is a simplified description of the steps
set the version to be built according to repository tag
create a directory for the new tree and clear the environment
create base target directories and directories required by ostree, like sysroot
create symlinks to conform with the filesystem hierarchy standard
copy custom services and custom configs
setup chroot with mounted system directories
bootstrap an alpine rootfs with base packages
apply minimal configuration to the rootfs like credentials, timezone, hostname.
copy services to be started by init
setup the boot configuration with the specified kernel image
optionally add custom kernel modules
perform cleanups
commit the rootfs [1] to the ostree repository
For containers, the sequence is the same, but configuration changes, because with a system not booted from a bootloader ostree has troubles verifying the environment, we have to apply some workarounds and setup some devices which are usually handled by the initramfs step. This is how OVZ or LXC templates are configured.
Once we have our ostree committed files tree build.sh or build-update.sh takes care of producing the artifact that will be distributed. The difference between the scripts is that the update version starts from a previous ostree repository, and also produces a delta artifact that a running system can apply on its ostree instance to perform upgrades. This is a simplified description of the build steps
if new build
create new partitions on a new image mounted as a loop device
else
mount sysroot and boot partitions of previous build
clean up previous ostree deployments (link farms) on mounted build
commit new built tree on mounted build
verify integrity and checksums
create new delta for upgrades
execute ostree deployment to regenerate boot configuration
remove old ostree commits
verify boot partition integrity
unmount new updated build image
generate image checksum and compress it (with squashfs for containers)
The partitions configuration is applied with a fdisk layout.cfg file which defines the partition sizes, we have one partition for the rootfs (~430M), the boot partition (~40M) and a swap partition (~40M). With containers with just skip mounting the previous build over a loop device, and just pull the new ostree commit over the old (extracted) ostree repository.
What am I bundling in this image (apart from installed packages)?
A small lib in functions.sh for common tasks executed within the shell
ostree based upgrade scripts with locking mechanism based on KV store
monitoring scripts for IO both local (iomon) and network (tcpmon)
setup scripts for container runtimes other than bubblewrap: podman, toolbox
What used to be and isn't anymore
sup: Sup was used for orchestration (deploy containers) and configuration of the host machine, but then I switched to ansible because there were long standing bugs in sup that were lacking a fix, I did not choose ansible from the start because I didn't want a python dependency to install on every server, eventually I settled with a secondary alpine chroot on the host machines, located in /opt/alp were I install additional less critical software. Today, however, I would again switch from ansible to pyinfra, because
it is python without boilerplate (ansible has pseudo DSL that causes more headaches than the ones it solves)
it executes its recipes with plain ssh commands, so there isn't a python dependency requirement on target hosts
containerpilot: The use case for container pilot is to manage complex dependencies between containers without shell scripts...it stopped getting updates from joyent and was put in maintenance mode, I also didn't like the memory requirements and memory usage would happen to constantly increase for long period of uptime. I switched it for simple shell scripts with consul, I might look into more proper alternatives if shell scripts start to grow too much. Heavier solutions like kubernetes, swarm or nomad were discarded from the get-go.
beegfs: I used to ship the kernel module necessary for beegfs but after a while it broke compatibility, and the fact that there wasn't a properly supported fuse module made be drop it altogether, I am currently not running a [DFS] on my servers, but the possibility of having a ready to go file-system to plug in a network is still appealing.
To install the image you can either upload it to the hosting provider and install from VNC, in case of virtual machines, but I usually hijack an existing installation, because it is always possible, well as long I have tested the setup script against version of the linux distribution, generally I use debian-8 or ubuntu-14, haven't tested other ones since these I have always found these to be available. The setup steps follows
ensure https support for downloads
download a busybox version
install a busybox link-farm
determine ipv4/ipv6 addresses for network config
ensure chroot capabilities
download the pine image and extract it
if VM
flash over target device
mount over a loop device
write the local network configuration over the to be flashed rootfs
if container
copy the init service (for containers) over the main root mount point /sbin/init
if VM
partition the left over disk with standard (xfs) partitions
unmount
verify partitions integrity
reboot
I made pine 5 years from time of writing and I am still using it, and I see no reasons to switch to anything else. Alpine as a linux distro is great, simple, and I have never experienced breakage. I can easily deploy on NATed servers which tend to offer ultra-low resources, actually I have a box running with just 64M of RAM, and still have all the features I need.
| [1] | root file system |
