= Tebako: an image packager image:https://github.com/tamatebako/tebako/actions/workflows/ubuntu.yml/badge.svg["Ubuntu amd64", link="https://github.com/tamatebako/tebako/actions/workflows/ubuntu.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/alpine.yml/badge.svg["Alpine", link="https://github.com/tamatebako/tebako/actions/workflows/alpine.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/macos.yml/badge.svg["MacOS amd64", link="https://github.com/tamatebako/tebako/actions/workflows/macos.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/windows-msys.yml/badge.svg["Windows msys", link="https://github.com/tamatebako/tebako/actions/workflows/windows-msys.yml"] image:https://api.cirrus-ci.com/github/tamatebako/tebako.svg?branch=main&task=ubuntu-aarch64["Ubuntu aarch64", link="https://cirrus-ci.com/github/tamatebako/tebako"] image:https://github.com/tamatebako/tebako/actions/workflows/lint.yml/badge.svg["lint", link="https://github.com/tamatebako/tebako/actions/workflows/lint.yml"] == Purpose Tebako is an executable packager. It packages a set of files into a DwarFS file system for read-only purposes. After packaging the file system into an image, Tebako produces a single executable binary that allows the user to execute a selected file from the packaged software from a point in the file system. The packaged binary should support: * Packaging a default DwarFS image inside the binary * Support signing of the binary on macOS (via notarization) In the future: * Downloading new DwarFS images to be stored in the local home directory * Allowing loading multiple DwarFS images in a stacked way * Supporting a COW mechanism that the newly written files are stored in a separate image that can be loaded on top of the read-only file systems. == Supported platforms The Tebako packager is tested on the following platforms: * Linux: Ubuntu 20.04; Alpine 3.17 * MacOS: macOS 12 (Monterey), 13 (Ventura), 14 (Sonoma) * Windows: 2019, 2022 (using MinGW ucrt64 toolchain) Please note that Windows support is in pre-release and is a subject of further testing and optimization == Supported Ruby versions The Tebako packager supports the following versions of Ruby for packaging: * 2.7.8 (Linux, MacOS) * 3.0.6 (Linux, MacOS) * 3.1.4 (Linux, MacOS, Windows) * 3.2.3 (Linux, MacOS, Windows) Support of specific version including minor release requires some effort, sometimes extensive but our goal is to be able to package all maintained Ruby releases. == Prerequisites === Ubuntu 20.04 ==== GNU C/C++ 10+ or Clang C/C++ 12+ [source,sh] ---- apt install -y gcc-10 g++-10 update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10 update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10 ---- or [source,sh] ---- apt install -y clang-12 update-alternatives --install /usr/bin/clang clang /usr/bin/clang-12 150 update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-12 150 ---- ==== CMake version 3.20+ Tebako relies on CMake 3.20+, which may not be available as a default package. If it is not available as default package it can be set up as follows: [source,sh] ---- apt-get remove --purge --auto-remove cmake apt-get update apt-get install -y software-properties-common lsb-release curl apt-get clean all curl https://apt.kitware.com/kitware-archive.sh | bash apt-get install cmake ---- ==== Other development tools and libraries [source,sh] ---- apt-get -y install sudo git curl build-essential pkg-config bison flex autoconf \ binutils-dev libevent-dev acl-dev libfmt-dev libjemalloc-dev libiberty-dev \ libdouble-conversion-dev liblz4-dev liblzma-dev libssl-dev libunwind-dev \ libboost-filesystem-dev libboost-program-options-dev libboost-system-dev \ libboost-iostreams-dev libboost-date-time-dev libboost-context-dev \ libboost-regex-dev libboost-thread-dev libbrotli-dev libdwarf-dev libelf-dev \ libgoogle-glog-dev libffi-dev libgdbm-dev libyaml-dev libncurses-dev \ libreadline-dev libncurses-dev libreadline-dev ruby-dev ruby-bundler \ libutfcpp-dev ---- === Alpine 3.17 [source,sh] ---- apk --no-cache --upgrade add build-base cmake git bash autoconf boost-static \ boost-dev flex-dev bison make binutils-dev libevent-dev acl-dev sed python3 \ pkgconfig lz4-dev openssl-dev zlib-dev xz ninja zip unzip curl libdwarf-dev \ libunwind-dev gflags-dev elfutils-dev libevent-static openssl-libs-static \ lz4-static xz-dev zlib-static libunwind-static acl-static tar libffi-dev \ gdbm-dev yaml-dev yaml-static ncurses-dev ncurses-static readline-dev \ readline-static p7zip ruby-dev gcompat gettext-dev gperf brotli-dev \ brotli-static jemalloc-dev fmt-dev xz-static ---- === macOS 12 (Monterey) [source,sh] ---- brew update brew install gnu-sed bash pkg-config bison flex binutils libffi gdbm zlib ncurses \ double-conversion boost jemalloc fmt glog libevent libsodium lz4 xz libyaml openssl@3 ---- === Windows (2019, 2022) The simplest approach is to use Ruby development environment provided by RubyInstaller, for example Ruby+Devkit 3.1.4-1. Once it is installed use the following commands: [source,sh] ---- ridk enable ucrt64 pacman -S git tar bison flex toolchain make cmake boost diffutils libevent double-conversion fmt glog dlfcn gtest autotools ncurses libyaml ---- == Installation === General Tebako is distributed as a Ruby gem [source,sh] ---- gem install tebako ---- === Quick setup on Ubuntu 20.04 on Docker Launch a container on the target platform: [source,sh] ---- # For x86_64 docker run -it --platform linux/x86_64 ubuntu bash # For Apple M1 docker run -it --platform linux/aarch64 ubuntu bash ---- In the container: [source,sh] ---- export DEBIAN_FRONTEND=noninteractive export TZ=Etc/UTC apt-get update apt-get install -y software-properties-common add-apt-repository -y ppa:ubuntu-toolchain-r/test apt-get install -y gcc-10 g++-10 apt-get install -y curl git ruby ruby-dev pkg-config bison flex make autoconf curl https://apt.kitware.com/kitware-archive.sh | bash apt-get install -y cmake apt-get -y install sudo git curl build-essential pkg-config bison flex autoconf \ binutils-dev libevent-dev acl-dev libfmt-dev libjemalloc-dev libiberty-dev \ libdouble-conversion-dev liblz4-dev liblzma-dev libssl-dev libunwind-dev \ libboost-filesystem-dev libboost-program-options-dev libboost-system-dev \ libboost-iostreams-dev libboost-date-time-dev libboost-context-dev \ libboost-regex-dev libboost-thread-dev libbrotli-dev libdwarf-dev libelf-dev \ libgoogle-glog-dev libffi-dev libgdbm-dev libyaml-dev libncurses-dev \ libreadline-dev libutfcpp-dev libncurses-dev libreadline-dev gcc-10 g++-10 \ ruby-dev ruby-bundler gem install tebako ---- == Usage === Commands ==== Installation [source,sh] ---- gem install tebako ---- ==== Press This command "presses" a Ruby project using the Tebako setup from the Tebako root folder (``). Please note that upon the first invocation of press command tebako collects required packages, builds the and creates packaging environment. This is a lengthly task that can take significant time, up to 1 hour. Upon the next invocation tebako will use previously created packaging environment. The press process itself takes minutes. You can manage setup of packaging environment manually; please refer to description of setup and clean cmmands below. [source] ---- tebako press \ [-p|--prefix=] \ [-R|--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>] \ -r|--root= \ -e|--entry-point= \ [-o|--output=] \ [-l|--log-level=] ---- Where: * ``, the Tebako setup folder (optional, defaults to current folder) * `Ruby` parameter defines Ruby version that will be packaged (optional, defaults to 3.1.4) * ``, a folder at the host source file system where project files are located * ``, an executable file (binary executable or script) that shall be started when packaged file is called * `output`, the output file name (optional, defaults to `/] \ [-R |--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>] ---- Where: * ``, the Tebako setup folder (optional, defaults to current folder) * `Ruby` parameter defines Ruby version that will be packaged (optional, defaults to 3.1.4) ==== Clean This command deletes tebako artifacts created by setup and press commands. Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own. [source] ---- tebako clean \ [-p|--prefix=] ---- Where: * ``, the Tebako setup folder (optional, defaults to current folder) [example] ==== [source,sh] ---- tebako clean --prefix='~/.tebako' ---- ==== ==== Clean ruby This command deletes tebako Ruby artifacts created by setup and press commands. Dwarfs libraries are not cleaned. Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own. [source] ---- tebako clean_ruby [-p|--prefix=] [-R|--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>] ---- Where: * ``, the Tebako setup folder (optional, defaults to current folder) * `Ruby` parameter defines Ruby version that will cleaned (optional, cleans all versions by default) [example] ==== [source,sh] ---- tebako clean_ruby --prefix='~/.tebako' ---- ==== ==== Build script hash Hash command will calculate tebako script hash that may be used as a cache key in CI/CD environment like GitHub Actions [source] ---- tebako hash ---- === Exit codes [cols,"a,a"] |=== | Code | Condition | 0 | No error | 1 | Invalid command line | 101 | `tebako setup` failed at configuration step | 102 | `tebako setup` failed at build step | 103 | `tebako press` failed at configuration step | 104 | `tebako press` failed at build step | 253 | Unsupported Ruby version | 254 | Unsupported operating systems | 255 | Internal error |=== == Image extraction Tebako provides an option to an extract filesystem from a package to local folder for verification or execution. [source,sh] ---- --tebako-extract [] ---- Where, * `` is optional and defaults to `source_filesystem` [example] ==== Extracting Tebako content from the `metanorma` package: [source,sh] ---- metanorma --tebako-extract temp-image ---- ==== The `--tebako-extract` option actually runs the following Ruby script: [source,ruby] ---- require 'fileutils' FileUtils.copy_entry '', ARGV[2] || 'source_filesystem' ---- == Ruby packaging specification This is high-level description of the Tebako Ruby packaging mechanism. This specification was inspired by the `ruby-packer` approach. NOTE: For various reasons, Tebako Ruby is a fully separate implementation, no line of code was copied from `ruby-packer`. Depending on the configuration files that are present in the root project folder, the Tebako Ruby packager support five different scenarios: [cols="a,a,a,a"] |=== | Scenario | `*.gemspec` | `Gemfile` | `*.gem` | 1 | No | No | No | 2 | No | No | One | 3 | One | No | Any | 4 | One | One | Any | 5 | No | One | Any | Error | No | No |Two or more | Error |Two or more| Any | Any |=== These scenarios differ in what files are packaged and where the entry point is located, as follows: [cols="a,a,a,a"] |=== | Scenario | Description | Packaging | Entry point | 1 | Simple ruby script | Copy `` with all sub-folders to packaged filesystem | `/local/` | 2 | Packaged gem | Install the gem with `gem install` to packaged filesystem | `/bin/` (i.e., binstub is expected) | 3 | Gem source, no `bundler` | . Build the gem using `gem build` command at the host . Install it with `gem install` to packaged filesystem | `/bin/` (i.e., binstub is expected) | 4 | Gem source, `bundler` | . Collect dependencies at the host with `bundle install` . Build the gem using `gem build` command . Install it with `gem install` to packaged file system | `/bin/` (i.e., binstub is expected) | 5 | Rails project | Deploy project to packaged filesystem using `bundle install` | `/local/` |=== == Trivia: origin of name "tamatebako" (玉手箱) is the treasure box given to Urashima Taro in the Ryugu, for which he was asked not to open if he wished to return. He opened the box upon the shock from his return that three hundred years has passed. Apparently what was stored in the box was his age. This packager was made to store Ruby and its gems, and therefore named after the said treasure box (storing gems inside a treasure box). Since "tamatebako" is rather long for the non-Japanese speaker, we use "tebako" (手箱, also "tehako") instead, the generic term for a personal box.