https://danyspin97.org/tags/suse-labs/Recent content in Suse-Labs on danyspin97's siteHugo -- gohugo.ioen-usWed, 13 May 2026 15:00:00 +0000https://danyspin97.org/talks/health-checker-suse-labs/Wed, 13 May 2026 15:00:00 +0000https://danyspin97.org/talks/health-checker-suse-labs/<p>class: center, middle name: start</p> <h3 id="error-recovery">Error recovery</h3> <h2 id="with-microos-and-systemd-bless-boot">with MicroOS and systemd-bless-boot</h2> <h2 id="purplefooterdanilo-spinellabrsuse-labs-ifab-fa-suse">.purple[.footer[Danilo Spinella<br>SUSE Labs ]]</h2> <p>class: left, middle</p> <h2 id="who-am-i">Who am I</h2> <h3 id="danilo-spinellabr_linux-research-engineer_brfuture-technologies-team---greenifab-fa-suse">Danilo Spinella</br><em>Linux Research Engineer</em></br>Future Technologies Team @ .green[]</h3> <hr> <h2 id="background-image-urlhttpsipinimgcom736xe4bbf6e4bbf6d987196ca49097a7604d734d89jpg">background-image: url(<a href="https://i.pinimg.com/736x/e4/bb/f6/e4bbf6d987196ca49097a7604d734d89.jpg">https://i.pinimg.com/736x/e4/bb/f6/e4bbf6d987196ca49097a7604d734d89.jpg</a>)</h2> <p>class: left, middle</p> <h1 id="contents">Contents</h1> <p>health-checker</p> <p>Boot Loader Specification</p> <p>Automatic Boot Assessment</p> <hr> <p>class: left, middle</p> <h1 id="health-checker">health-checker</h1> <hr> <p>class: left, middle</p> <h2 id="opensuse-microos-and-suse-micro">openSUSE MicroOS and SUSE Micro</h2> <p>Immutable system (by using <strong>btrfs snapshots</strong> and <strong>transactional-update</strong>)</p> <p>User applications installed in containers</p> <p>Reliability first by leveraging snapshots</p> <hr> <p>class: left, middle</p> <h2 id="snapshots">Snapshots</h2> <p>Changing the system (adding and removing a package / updating the system) <strong>creates a new snapshot</strong></p> <p>Every snapshot has a different boot entry: <code>rootflags=subvol=@/.snapshots/1/snapshot</code></p> <p><code>transactional-update</code> handles btrfs snapshots and changes</p> <hr> <p>class: left, middle</p> <h2 id="health-checker-1">health-checker</h2> <p>Check the status of the system during the boot process</p> <p><strong>Plug-in based</strong>, each plug-in checks a different system component</p> <p><strong>Fail-safe mechanism</strong>, reboot on error to a working snapshot</p> <hr> <p>class: left, middle</p> <h2 id="health-checker-fail-safe">health-checker fail-safe</h2> <p>If a boot entry is new, <strong>automatically reboot</strong> up to 3 times when the boot fails</p> <p><strong>Update</strong> the default entry to a working snapshot</p> <p>If the entry was working before: reboot one time, if it still fails, <strong>open an emergency shell</strong></p> <hr> <p>class: left, middle</p> <h2 id="health-checker-plugins-12">health-checker plugins (1/2)</h2> <p><strong>btrfs-subvolumes-mounted</strong>: Check all btrfs subvolumes from fstab are mounted</p> <p><strong>etc-overlayfs</strong>: Check that /etc overlayfs is mounted</p> <p><strong>logind</strong>: Check systemd-logind service status</p> <hr> <p>class: left, middle</p> <h2 id="health-checker-plugins-22">health-checker plugins (2/2)</h2> <p><strong>rebootmgr</strong>: Check rebootmgr service status</p> <p><strong>rpmdb-consistency</strong>: Verify RPM database integrity</p> <p><strong>tmp</strong>: Check /tmp mount status</p> <hr> <p>class: left, middle</p> <h2 id="plugin-example-logind-12">Plugin example: logind (1/2)</h2> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="k">case</span> <span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span> in </span></span><span class="line"><span class="cl"> check<span class="o">)</span> </span></span><span class="line"><span class="cl"> run_checks </span></span><span class="line"><span class="cl"> <span class="p">;;</span> </span></span><span class="line"><span class="cl"> stop<span class="o">)</span> </span></span><span class="line"><span class="cl"> stop_services </span></span><span class="line"><span class="cl"> <span class="p">;;</span> </span></span><span class="line"><span class="cl"> *<span class="o">)</span> </span></span><span class="line"><span class="cl"> <span class="nb">echo</span> <span class="s2">&#34;Usage: </span><span class="nv">$0</span><span class="s2"> {check|stop}&#34;</span> </span></span><span class="line"><span class="cl"> <span class="nb">exit</span> <span class="m">1</span> </span></span><span class="line"><span class="cl"> <span class="p">;;</span> </span></span><span class="line"><span class="cl"><span class="k">esac</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">exit</span> <span class="m">0</span> </span></span></code></pre></div><hr> <p>class: left, middle</p> <h2 id="plugin-example-logind-22">Plugin example: logind (2/2)</h2> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">run_checks<span class="o">()</span> <span class="o">{</span> </span></span><span class="line"><span class="cl"> systemctl is-enabled -q systemd-logind </span></span><span class="line"><span class="cl"> <span class="nb">test</span> <span class="nv">$?</span> -ne <span class="m">0</span> <span class="o">&amp;&amp;</span> <span class="k">return</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> systemctl is-failed -q systemd-logind </span></span><span class="line"><span class="cl"> <span class="nb">test</span> <span class="nv">$?</span> -ne <span class="m">1</span> <span class="o">&amp;&amp;</span> <span class="nb">exit</span> <span class="m">1</span> </span></span><span class="line"><span class="cl"><span class="o">}</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">stop_services<span class="o">()</span> <span class="o">{</span> </span></span><span class="line"><span class="cl"> systemctl stop systemd-logind </span></span><span class="line"><span class="cl"><span class="o">}</span> </span></span></code></pre></div><hr> <p>class: left, middle</p> <h1 id="boot-loader-specification-bls">Boot Loader Specification (BLS)</h1> <hr> <p>class: center, middle</p> <h2 id="linux-system">Linux system</h2> <p> <img src="https://danyspin97.org/img/health-checker-fosdem-26/linux.drawio.svg" alt=":resize 800"> </p> <hr> <p>class: middle, top</p> <h2 id="adding-another-system">Adding another system&hellip;</h2> <p>&ndash;</p> <p>The <em>EFI partition</em> must be <strong>unique</strong> on each disk</p> <p>&ndash;</p> <p>Which bootloader will be the default? Does it read the other partition <code>/boot</code> folder?</p> <hr> <p>class: middle, left</p> <h2 id="boot-loader-specification">Boot Loader Specification</h2> <p>Allow the boot loader menu entries to be <strong>shared</strong> between <em>multiple operating systems</em></p> <p>Standardize configuration between bootloader, firmware and system components</p> <p>Supports two simple formats, found in <code>/loader/entries</code></p> <hr> <p>class: left, middle</p> <h2 id="boot-loader-specification-type1-entry">Boot Loader Specification type#1 entry</h2> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># /boot/efi/loader/entries/opensuse-microos-6.18.4-5-default-1.conf </span> </span></span><span class="line"><span class="cl"><span class="na">title openSUSE MicroOS 1</span> </span></span><span class="line"><span class="cl"><span class="na">version 1@6.18.4-5-default</span> </span></span><span class="line"><span class="cl"><span class="na">sort-key opensuse-microos</span> </span></span><span class="line"><span class="cl"><span class="na">options quiet rootflags</span><span class="o">=</span><span class="s">subvol=@/.snapshots/1/snapshot systemd.show_status=yes console=ttyS0,115200 console=tty0 ignition.platform.id=qemu security=selinux selinux=1 root=UUID=8f71b02c-9905-b5e6-fafe-9efdb7c0ecd0</span> </span></span><span class="line"><span class="cl"><span class="na">linux /opensuse-microos/6.18.4-5-default/linux-03046d37a644bdf18e0d0dbe4677405926c6df24</span> </span></span><span class="line"><span class="cl"><span class="na">initrd /opensuse-microos/6.18.4-5-default/initrd-95afcf129ca424d5481026705a360d3bc92d5d75</span> </span></span></code></pre></div><hr> <p>class: middle, left</p> <h1 id="automatic-boot-assessment">Automatic Boot Assessment</h1> <hr> <p>class: middle, left</p> <h2 id="automatic-boot-assessment-1">Automatic Boot Assessment</h2> <p><strong>Revert back to the previous version</strong> of the OS or kernel when the system fails to boot</p> <p>Relies on <strong>Boot Loader Specification</strong> and support from system components</p> <hr> <p>class: left, middle</p> <h2 id="boot-counting">Boot counting</h2> <p>Add a number of boot attempts on new boot entries</p> <p>Entries with non-zero tries left are &ldquo;<em>indeterminate</em>&rdquo;</p> <p>Entries without boot counting are considered &ldquo;<em>good</em>&rdquo;</p> <p>Entries with zero tries left are &ldquo;<em>bad</em>&rdquo;</p> <hr> <p>class: center, middle</p> <p> <img src="https://danyspin97.org/img/health-checker-fosdem-26/boot-counting.drawio.svg" alt=":resize 1000"> </p> <hr> <p>class: left, middle</p> <h2 id="installing-new-entries">Installing new entries</h2> <p>The initial counter defined in <code>/etc/kernel/tries</code></p> <p><strong>kernel-install</strong> adds boot entries with the boot counter enabled</p> <p><strong>sdbootutil</strong> manages the boot counter when editing the boot entries</p> <hr> <p>class: left, middle</p> <h2 id="bootloader-support">Bootloader support</h2> <p>Put the entries with zero left tries at the bottom</p> <p>Adjust boot counter when an entry is booted</p> <p><strong>systemd-boot</strong> and <strong>grub2</strong><em>*</em> supports Automatic Boot Assessment, but support can be extended to any bootloader</p> <hr> <p>class: left, middle</p> <h2 id="grub2-12">grub2 (1/2)</h2> <p><em>BLS</em> support added by different set of patches (one by <em>Fedora</em> and one by <em>openSUSE</em>)</p> <p>Upstream <em>grub2 2.14</em> added initial <strong>incomplete BLS support</strong></p> <hr> <p>class: left, middle</p> <h2 id="grub2-22">grub2 (2/2)</h2> <p><em>Boot counting</em> and partial BLI support <strong>implemented via downstream</strong> patches in <em>openSUSE</em>*</p> <p>The patches are currently being reviewed upstream (<a href="https://gitlab.freedesktop.org/gnu-grub/grub/-/merge_requests/124">MR</a>)</p> <hr> <p>class: left, middle</p> <h2 id="boottarget">boot.target</h2> <p>Services that check for a successful boot (like health-checker)</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-systemd" data-lang="systemd"><span class="line"><span class="cl"><span class="na">RequiredBy</span><span class="o">=</span><span class="s">boot-complete.target</span> </span></span></code></pre></div><p>Services that run when the system boots successfully (like systemd-bless-boot)</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-systemd" data-lang="systemd"><span class="line"><span class="cl"><span class="na">Requires</span><span class="o">=</span><span class="s">boot-complete.target</span> </span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">boot-complete.target</span> </span></span></code></pre></div><hr> <p>class: left, middle</p> <h2 id="systemd-bless-boot">systemd-bless-boot</h2> <p>Remove the boot counter when the boot succeeds</p> <p>Get the status of the current boot entry</p> <p>Change the status of a boot entry</p> <hr> <p>class: left, middle</p> <h2 id="issues-with-automatic-boot-assessment">Issues with Automatic Boot Assessment</h2> <p><strong>No automatic reboot</strong> on failed boot - user must manually intervene</p> <p><strong>Default entry not updated</strong> - next boot attempts the same failed entry</p> <p><strong>Simple by design</strong> - bootloaders can&rsquo;t (and shouldn&rsquo;t) understand complex failure scenarios</p> <hr> <p>class: left, middle</p> <h2 id="health-checker-fail-safe-in-detail">health-checker fail-safe (in detail)</h2> <p>Checks the status of the boot entry by using <code>systemd-bless-boot</code></p> <p><strong>Update</strong> or <strong>remove</strong> the current default entry to use the bootloader&rsquo;s sorting</p> <p>If the entry has boot counting enabled, <strong>reboot automatically</strong> (can be disabled via kernel cmdline)</p> <p>If the entry was working before: reboot one time, if it still fails, <strong>opens an emergency shell</strong></p> <hr> <p>class: left, middle</p> <h2 id="thank-you-for-your-attention">Thank you for your attention</h2> <h1 id="questions-ifa-fa-question">Questions </h1> <hr> <p>template: start</p>