= Tebako: an image packager

image:https://github.com/tamatebako/tebako/actions/workflows/ubuntu.yml/badge.svg["Ubuntu amd64", link="https://github.com/tamatebako/tebako/actions/workflows/ubuntu.yml"]
image:https://github.com/tamatebako/tebako/actions/workflows/alpine.yml/badge.svg["Alpine", link="https://github.com/tamatebako/tebako/actions/workflows/alpine.yml"]
image:https://github.com/tamatebako/tebako/actions/workflows/macos.yml/badge.svg["MacOS amd64", link="https://github.com/tamatebako/tebako/actions/workflows/macos.yml"]
image:https://github.com/tamatebako/tebako/actions/workflows/windows-msys.yml/badge.svg["Windows msys", link="https://github.com/tamatebako/tebako/actions/workflows/windows-msys.yml"]

image:https://api.cirrus-ci.com/github/tamatebako/tebako.svg?branch=main&task=ubuntu-aarch64["Ubuntu aarch64", link="https://cirrus-ci.com/github/tamatebako/tebako"]

image:https://github.com/tamatebako/tebako/actions/workflows/lint.yml/badge.svg["lint", link="https://github.com/tamatebako/tebako/actions/workflows/lint.yml"]

== Purpose

Tebako is an executable packager. It packages a set of files into a DwarFS file
system for read-only purposes.

After packaging the file system into an image, Tebako produces a single
executable binary that allows the user to execute a selected file from the
packaged software from a point in the file system.

The packaged binary should support:

* Packaging a default DwarFS image inside the binary
* Support signing of the binary on macOS (via notarization)

In the future:

* Downloading new DwarFS images to be stored in the local home directory
* Allowing loading multiple DwarFS images in a stacked way
* Supporting a COW mechanism that the newly written files are stored
  in a separate image that can be loaded on top of the read-only file systems.

== Supported platforms

The Tebako packager is tested on the following platforms:

* Linux: Ubuntu 20.04; Alpine 3.17
* MacOS: macOS 12 (Monterey), 13 (Ventura), 14 (Sonoma)
* Windows: 2019, 2022 (using MinGW ucrt64 toolchain)

Please note that Windows support is in pre-release and is a subject of further testing and optimization

== Supported Ruby versions

The Tebako packager supports the following versions of Ruby for packaging:

* 2.7.8 (Linux, MacOS)
* 3.0.6 (Linux, MacOS)
* 3.1.4 (Linux, MacOS, Windows)
* 3.2.3 (Linux, MacOS, Windows)

Support of specific version including minor release requires some effort, sometimes extensive
but our goal is to be able to package all maintained Ruby releases.

== Prerequisites

=== Ubuntu 20.04

==== GNU C/C++ 10+ or Clang C/C++ 12+

[source,sh]
----
apt install -y gcc-10 g++-10
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10
----

or

[source,sh]
----
apt install -y clang-12
update-alternatives --install /usr/bin/clang clang /usr/bin/clang-12 150
update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-12 150
----

==== CMake version 3.20+

Tebako relies on CMake 3.20+, which may not be available as a default package.

If it is not available as default package it can be set up as follows:

[source,sh]
----
apt-get remove --purge --auto-remove cmake
apt-get update
apt-get install -y software-properties-common lsb-release curl
apt-get clean all
curl https://apt.kitware.com/kitware-archive.sh | bash
apt-get install cmake
----

==== Other development tools and libraries

[source,sh]
----
apt-get -y install sudo git curl build-essential pkg-config bison flex autoconf  \
   binutils-dev libevent-dev acl-dev libfmt-dev libjemalloc-dev libiberty-dev    \
   libdouble-conversion-dev liblz4-dev liblzma-dev libssl-dev libunwind-dev      \
   libboost-filesystem-dev libboost-program-options-dev libboost-system-dev      \
   libboost-iostreams-dev  libboost-date-time-dev libboost-context-dev           \
   libboost-regex-dev libboost-thread-dev libbrotli-dev libdwarf-dev libelf-dev  \
   libgoogle-glog-dev libffi-dev libgdbm-dev libyaml-dev libncurses-dev          \
   libreadline-dev libncurses-dev libreadline-dev ruby-dev ruby-bundler          \
   libutfcpp-dev
----

=== Alpine 3.17

[source,sh]
----
apk --no-cache --upgrade add build-base cmake git bash autoconf boost-static   \
   boost-dev flex-dev bison make binutils-dev libevent-dev acl-dev sed python3 \
   pkgconfig lz4-dev openssl-dev zlib-dev xz ninja zip unzip curl libdwarf-dev \
   libunwind-dev gflags-dev elfutils-dev libevent-static openssl-libs-static   \
   lz4-static xz-dev zlib-static libunwind-static acl-static tar libffi-dev    \
   gdbm-dev yaml-dev yaml-static ncurses-dev ncurses-static readline-dev       \
   readline-static p7zip ruby-dev gcompat gettext-dev gperf brotli-dev         \
   brotli-static jemalloc-dev fmt-dev xz-static
----

=== macOS 12 (Monterey)

[source,sh]
----
brew update
brew install gnu-sed bash pkg-config bison flex binutils libffi gdbm zlib ncurses \
  double-conversion boost jemalloc fmt glog libevent libsodium lz4 xz libyaml openssl@3
----

=== Windows (2019, 2022)

The simplest approach is to use Ruby development environment provided by RubyInstaller, for example Ruby+Devkit 3.1.4-1.

Once it is installed use the following commands:

[source,sh]
----
  ridk enable ucrt64
  pacman -S git tar bison flex toolchain make cmake
            boost diffutils libevent double-conversion
            fmt glog dlfcn gtest autotools ncurses libyaml
----

== Installation

=== General

Tebako is distributed as a Ruby gem

[source,sh]
----
gem install tebako
----

=== Quick setup on Ubuntu 20.04 on Docker

Launch a container on the target platform:

[source,sh]
----
# For x86_64
docker run -it --platform linux/x86_64 ubuntu bash

# For Apple M1
docker run -it --platform linux/aarch64 ubuntu bash
----

In the container:

[source,sh]
----
export DEBIAN_FRONTEND=noninteractive
export TZ=Etc/UTC

apt-get update
apt-get install -y software-properties-common
add-apt-repository -y ppa:ubuntu-toolchain-r/test
apt-get install -y gcc-10 g++-10

apt-get install -y curl git ruby ruby-dev pkg-config bison flex make autoconf
curl https://apt.kitware.com/kitware-archive.sh | bash
apt-get install -y cmake

apt-get -y install sudo git curl build-essential pkg-config bison flex autoconf \
   binutils-dev libevent-dev acl-dev libfmt-dev libjemalloc-dev libiberty-dev    \
   libdouble-conversion-dev liblz4-dev liblzma-dev libssl-dev libunwind-dev      \
   libboost-filesystem-dev libboost-program-options-dev libboost-system-dev      \
   libboost-iostreams-dev  libboost-date-time-dev libboost-context-dev           \
   libboost-regex-dev libboost-thread-dev libbrotli-dev libdwarf-dev libelf-dev  \
   libgoogle-glog-dev libffi-dev libgdbm-dev libyaml-dev libncurses-dev          \
   libreadline-dev libutfcpp-dev libncurses-dev libreadline-dev gcc-10 g++-10    \
   ruby-dev ruby-bundler

gem install tebako

----

== Usage

=== Commands

==== Installation

[source,sh]
----
gem install tebako
----

==== Press

This command "presses" a Ruby project using the Tebako setup from the Tebako root
folder (`<tebako-root-folder>`).
Please note that upon the first invocation of press command tebako collects required packages,
builds the and creates packaging environment. This is a lengthly task that can take significant
time, up to 1 hour.
Upon the next invocation tebako will use previously created packaging environment. The press process
itself takes minutes.
You can manage setup of packaging environment manually; please refer to description of setup and clean
cmmands below.

[source]
----
tebako press     \
  [-p|--prefix=<tebako-root-folder>] \
  [-R|--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]   \
  -r|--root=<project-root-folder>     \
  -e|--entry-point=<entry-point>      \
  [-o|--output=<packaged file name>] \
  [-l|--log-level=<error|warn|debug|trace>]
----

Where:

* `<tebako-root-folder>`, the Tebako setup folder (optional, defaults to current
folder)

* `Ruby` parameter defines Ruby version that will be packaged (optional, defaults to 3.1.4)

* `<project-root>`, a folder at the host source file system where project files
are located

* `<entry-point>`, an executable file (binary executable or script) that shall
be started when packaged file is called

* `output`, the output file name (optional, defaults to `<current folder>/<entry
point base name`)

* `log-level`, the logging level for tebako built-in memory filesystem driver (optional, defaults to `error`)

[example]
====
[source,sh]
----
tebako press \
  --root='~/projects/myproject' \
  --entry=start.rb \
  --output=/temp/myproject.tebako
----
====

==== Setup

Collects required packages, builds the and creates packaging environment. This is a lengthly
task that can take significant time, up to 1 hour.
Tebako supports several configurations at a single system given that their root
directories differ and nultiple Ruby versions within single configuration

This command is optional, tebako creates packaging environment automatically upon the first
invocation of press command.
However, if you plan to use tebako in CI/CD environment with caching it is highly recommended to build cache
based on ```tebako setup``` output. Building cache based on ```tebako press``` may create inconsistent environment upon restore.

[source]
----
tebako setup     \
  [-p |--prefix=<tebako-root-folder>] \
  [-R |--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]
----

Where:

* `<tebako-root-folder>`, the Tebako setup folder (optional, defaults to current
folder)

* `Ruby` parameter defines Ruby version that will be packaged (optional, defaults to 3.1.4)

==== Clean

This command deletes tebako artifacts created by setup and press commands.
Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own.

[source]
----
tebako clean \
  [-p|--prefix=<tebako-root-folder>]
----

Where:

* `<tebako-root-folder>`, the Tebako setup folder (optional, defaults to current
folder)

[example]
====
[source,sh]
----
tebako clean --prefix='~/.tebako'
----
====

==== Clean ruby

This command deletes tebako Ruby artifacts created by setup and press commands. Dwarfs libraries are not cleaned.
Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own.

[source]
----
tebako clean_ruby
  [-p|--prefix=<tebako-root-folder>]
  [-R|--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]
----

Where:

* `<tebako-root-folder>`, the Tebako setup folder (optional, defaults to current
folder)
* `Ruby` parameter defines Ruby version that will cleaned (optional, cleans all versions by default)

[example]
====
[source,sh]
----
tebako clean_ruby --prefix='~/.tebako'
----
====

==== Build script hash
Hash command will calculate tebako script hash that may be used as a cache key in CI/CD environment like GitHub Actions

[source]
----
tebako hash
----

=== Exit codes

[cols,"a,a"]
|===
| Code | Condition

| 0    | No error
| 1    | Invalid command line
| 101  | `tebako setup` failed at configuration step
| 102  | `tebako setup` failed at build step
| 103  | `tebako press` failed at configuration step
| 104  | `tebako press` failed at build step
| 253  | Unsupported Ruby version
| 254  | Unsupported operating systems
| 255  | Internal error

|===

== Image extraction

Tebako provides an option to an extract filesystem from a package to local
folder for verification or execution.

[source,sh]
----
<tebako-packaged-executable> --tebako-extract [<root folder for extracted filesystem>]
----

Where,

* `<root folder for extracted filesystem>` is optional and defaults to
  `source_filesystem`

[example]
====
Extracting Tebako content from the `metanorma` package:

[source,sh]
----
metanorma --tebako-extract temp-image
----
====

The `--tebako-extract` option actually runs the following Ruby script:

[source,ruby]
----
require 'fileutils'
FileUtils.copy_entry '<in-memory filesystem root>', ARGV[2] || 'source_filesystem'
----

== Ruby packaging specification

This is high-level description of the Tebako Ruby packaging mechanism.
This specification was inspired by the `ruby-packer` approach.

NOTE: For various reasons, Tebako Ruby is a fully separate implementation,
no line of code was copied from `ruby-packer`.

Depending on the configuration files that are present in the root project folder,
the Tebako Ruby packager support five different scenarios:

[cols="a,a,a,a"]
|===
| Scenario | `*.gemspec` | `Gemfile`  | `*.gem`

| 1        |     No    |   No     |   No
| 2        |     No    |   No     |   One
| 3        |    One    |   No     |   Any
| 4        |    One    |   One    |   Any
| 5        |     No    |   One    |   Any
| Error    |     No    |   No     |Two or more
| Error    |Two or more|   Any    |   Any

|===

These scenarios differ in what files are packaged and where the entry point is
located, as follows:

[cols="a,a,a,a"]
|===
| Scenario | Description | Packaging | Entry point

| 1
| Simple ruby script
| Copy `<project-root>` with all sub-folders to packaged filesystem
| `<mount_point>/local/<entry_point base name>`

| 2
| Packaged gem
| Install the gem with `gem install` to packaged filesystem
| `<mount_point>/bin/<entry_point base name>` (i.e., binstub is expected)

| 3
| Gem source, no `bundler`
|
. Build the gem using `gem build` command at the host
. Install it with `gem install` to packaged filesystem

| `<mount_point>/bin/<entry_point base name>` (i.e., binstub is expected)

| 4
| Gem source, `bundler`
|
. Collect dependencies at the host with `bundle install`
. Build the gem using `gem build` command
. Install it with `gem install` to packaged file system

| `<mount_point>/bin/<entry_point base name>` (i.e., binstub is expected)

| 5
| Rails project
| Deploy project to packaged filesystem using `bundle install`
| `<mount_point>/local/<entry_point base name>`

|===


== Trivia: origin of name

"tamatebako" (玉手箱) is the treasure box given to Urashima Taro in the Ryugu,
for which he was asked not to open if he wished to return. He opened the box
upon the shock from his return that three hundred years has passed. Apparently
what was stored in the box was his age.

This packager was made to store Ruby and its gems, and therefore named after
the said treasure box (storing gems inside a treasure box).

Since "tamatebako" is rather long for the non-Japanese speaker, we use "tebako"
(手箱, also "tehako") instead, the generic term for a personal box.