Home

Papers

Blog

Pictures

About Me

Contact

GitHub


Riscy Business
________________________________________________________________________________

RISC-V [0] is one of the newest instruction set architectures on the block. It's
not necessarily novel as an FOSS ISA, but it's certainly interesting for another
thing: it has broad support in toolchains with glibc/musl/uclibc all supporting
it, it has active support in the kernel, and a lot of applications are already
packaged in major distributions for it.

And yet there is no hardware for it.

Certainly Sifive has done some excellent work in getting a RISC-V board into the
public's hands, but a lot of people are priced out of it - being $1,000 USD at
launch has a tendency to do that. BeagleBoard was going to make it more
affordable with their own board, but that project evaporated over time. As a
result, large parts of development work have happened in Qemu, which make this a
really intriguing architecture to tackle. I never got the chance to get in on
the ground level with AMD64, which dramatically revolutionized computing. I
largely ignored ARM (and largely still do; I didn't get a Pi until just a few
months ago). But RISC-V fulfills a lot of my dreams - it's a sane ISA that
avails itself of a realm usually associated with software - a free and open
specification that anybody can use without royalty. Incredible.

Luckily, some (very affordable!) RISC-V boards have come out recently, and more
are on the way later this year. I took a risk on aliexpress (another first for
me) and picked up a very cheap board - Sipeed's Lichee dev board based on
Allwinner's D1 chip [1]. It's a more affordable version of the Nezha [2], which
I may pick up as well. If you can't tell, I'm very excited about this ISA!

With that, we come to the meat of this blog post. I work at Canonical as an IoT
Field Engineer. Embedded hardware should be my bread and butter. I have a lot of
experience dealing with the kernel and PID1 on x86 machines, but almost none
when it comes to embedded hardware - the only bootloader I really ever became
"familiar with" is GRUB, and I have refused to use that for at least four years
(EFI has been the future since 2014, and we all need to get on board with
EFISTUB). So this project marks a very swift broadening of a knowledge area I am
impoverished in: u-boot, and OpenSBI.

Code & notes for this project can be found at the below GitHub repository:
$/dilyn-corner/ubuntu-core-riscv64


[0.0] INDEX
________________________________________________________________________________

- The Setup - Why?                                                         [1.0]
- Gadget                                                                   [2.0]
- OpenSBI                                                                  [3.0]
- u-boot                                                                   [4.0]
- Kernel                                                                   [5.0]
- Board Bringup                                                            [6.0]


[1.0] The Setup - Why?
________________________________________________________________________________

I frequently embark on projects which excite me. I find learning about them far
more interesting when I am fundamentally motivated by an interest in a topic,
and I end up becoming all-too ravenous for every little detail about the task.
ARM was never exciting for me, and so I never bothered. But my job relies on
knowing embedded hardware well, so I need to become more intimately familiar
with specific facts relating to these small boards. RISC-V enables me to do this
in a way that won't bore me, and for that I am grateful.

All that being said I firmly believe in RISC-V, both as an architecture model
and as a hardware philosophy. Personally, I think RISC-V is the future of
embedded hardware, and potentially even usurping x86 as the dominant
architecture of the desktop (ARM folx have been saying this for years, and it's
certainly probably true in poorer countries; but almost nobody I've ever met in
America seriously uses an ARM chip for things outside of managing what color
their kitchen lights are).

As a result, I would love to see RISC-V appear more in my work. Canonical does
not officially support RISC-V in most ways; we have a server image available and
many snaps are built for the architecture, but we don't have a working Ubuntu
Core image publicly available for it, let alone one which we support.

So as a side project to experiment with bringing up a new board for Ubuntu Core,
I decided to explore RISC-V. It was actually far easier than expected, and I'll
tell you why!

First and foremost, an Ubuntu Core image is primarily composed of four snaps:
    1) base   (Core20 is the currently available base snap)
    2) snapd  (the snap which provides the underlying snap infrastructure)
    3) kernel (provides the kernel and needed hardware firmware, plus initrd)
    4) gadget (defines the partition structure and how to boot the device)

In addition to this, you also need a json-formatted model which defines the bits
that go into your image and ultimately identify it as an Ubuntu Core device -
see an example of such a file here [3]. This json file is validated and signed
by a GPG-style key registered to an Ubuntu SSO account, uniquely tying that
particular image to you - this is how, for instance, your SSH keys are put on
your device for easy headless access.

Creating a model is quite straightforward, and in our case requires minimal
modification!

Luckily for us, the core20 snap and the snapd snap are both available on the
snapstore [4] for RISC-V. Cross-building the base snap is potentially quite
tricky, though something I am interested in pursuing (ideally, I want to run a
very small version of Ubuntu Core, so this is a very tantalizing topic).

There is not however a readily available kernel and gadget snap for most
architectures. This means that we have to provide them ourselves. Not terribly
daunting; I've built plenty of kernels and I'm very good at debugging issues,
and how hard could a gadget really be?

Narrator: more tricky than you think.


[2.0] Gadget
________________________________________________________________________________

The most important feature of a (Ubuntu Core 20) gadget snap is how the boot
process is defined to flow. The defacto nonGRUB method relies on u-boot taking
on the role as bootloader. Luckily, configuring u-boot is very straightforward,
and should be quite familar if you've messed with the kernel before; a simple
make foo_defconfig; make menuconfig; make will yield a small u-boot binary you
can flash to a device.

u-boot itself exists to do essentially one thing: load the kernel. It does this
by doing a few key things:

* A device tree for the board is loaded into memory (so that all the board 
  hardware is properly discovered and defined)

* If an initramfs is required for booting the ROOTFS, it is also loaded into
  memory.

* The kernel is loaded into RAM

* The boot process is kicked off; the device tree is (potentially flattened),
  the kernel starts doing its thing (after relocating and freeing), and it
  passes off to the initrd to do whatever its purpose is.

In our case, the initrd bootstraps the Ubuntu Core system (it also rolls back
our system in cases of boot failures and ensures that our ROOTFS is properly
unsquashed... It does a lot, okay?).

We also need to include an environment file which determines how Core will boot;
this file is modified occassionally (automatically, not so much by us) so that
our system boots the right way at the right times (install mode, run mode, or
try mode), and also specifies our backup system image (in case something goes
terribly wrong and we have to nuke the whole install). 

Note that we don't have to rely on this file for booting; we could create an
image that always bootstraps itself, for instance; but what good would that be?
Who knows. There's probably a use-case.

The real tricky part is: where do you put all these objects in RAM?

This requires some knowledge about your objects - how big each of the kernel,
initrd, and device tree are, along with the free and addressable space on your
board.

In my case, I am testing using Qemu until my board arrives. Qemu is interesting
here. It can certainly emulate several boards, but I didn't want to mess around
with them -- I don't have any familiarity with their spec, and I'd rather not
toy around with a specific board: I want a more generic experience. So I settled
with the virt board Qemu has.

On this board, the addressable memory (assuming you specified -memory 2G) is
0x80000000 to 0xffffffff. Right away, you cannot read anything before the 2G
mark. Go ahead and try; md 0x50000 will result in a board reset. Additionally,
u-boot itself exists in a space shortly after the first available address,
meaning that putting anything in RAM before 0x80020000 can be quite risky!

Luckily, Qemu loads its own device tree for us (at the lovely address
0xff73bb00), and so we really only have to be cognizant of where we load other
things. Almost everything up to 0xff600000 is free to be copied into, and so we
can give our kernel and initrd a broad enough amount of space to be happy.

Ultimately, you can pick and choose your favorites. Use bdi and md to checkout
useful facts about your board and what lives where before you start trying.

As far as the gadget.yaml itself, I did very little altering to it. The content
of our seed partition contains two files, our config file detailing what mode to
boot in, along with a payload object (the bootloader). Otherwise, it's a very
bog standard gadget.yaml (see the Pi gadget for inspiration [5]).

What is this payload object, my imaginary interlocutor? I'm glad you asked!


[3.0] OpenSBI
________________________________________________________________________________

OpenSBI is the Open Source Supervisor Binary Interface for the RISC-V
architecture. The project is BSD-2 licensed and Western Digital owns the
copyright. It's a neat little project [6]! OpenSBI interfaces between an M-mode
firmware and a bootloader/hypervisor/OS (in S-mode). It's an open source
reference implementation of RISC-V's SBI spec, and works quite well for us. 

OpenSBI can be built in a few different ways, but I'll focus on one way that we
are interested in. OpenSBI can be built as an ELF binary which executes an
embedded binary payload on boot. This payload can either be the kernel itself
or, in our case, u-boot.

Building OpenSBI is quite easy, just a single line command:

ARCH=riscv \
    CROSS_COMPILE=riscv64-linux-gnu- \
    make \
    PLATFORM=generic \
    FW_PAYLOAD_PATH=/path/to/u-boot.bin

The generic platform here indicates that it will be run in Qemu; there are of
course other platform options [7].

You'll notice that this invocation requires the u-boot binary, so I suppose we
should talk about building that next!


[4.0] u-boot
________________________________________________________________________________

u-boot is the go-to bootloader for embedded hardware [8]. It has support for an
almost unholy amount of devices (don't try to open the configs/ directory in
your browser). The build process for it is very similar to how the kernel is
built, but luckily it doesn't have nearly as many symbols to configure (just
1200 SLOC). So carry over your knowledge from the kernel:

ARCH=riscv make foo_defconfig

ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- make menuconfig

We only really require a small amount of tweaking to get our Qemu virt board
running - because our gadget.yaml uses FAT, we enable CONFIG_ENV_IS_IN_FAT, and
specify some options which become available once that is enabled:

CONFIG_ENV_FAT_INTERFACE="virtio"    # Qemu's virt board does not support -sd
CONFIG_ENV_FAT_DEVICE_AND_PART="0:1" # disk zero, first partition (ubuntu-seed)
CONFIG_ENV_FAT_FILE="boot.scr"       # The file detailing *how* to boot

Once we have those config options set, 

ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- make

Build OpenSBI with the built u-boot.bin, and you are almost good to go!


[5.0] Kernel
________________________________________________________________________________

The kernel is pretty easy to build. Ubuntu Core uses a kernel that is roughly
similar to the regular Ubuntu kernel. Importantly, we need a kernel which
supports Apparmor. At time of writing, almost all of those patches have been
upstreamed. For now, we either need those patches or Ubuntu's kernel. I stole
the base config file I used to build the kernel from the server image.

However you go about acquiring this kernel doesn't so much matter. One could
lift it directly from the RISC-V server image or build it from scratch. But one
thing we certainly require for our kernel snap is an initrd. Again at time of
writing, there is no Ubuntu Core initrd snap available for RISC-V. In fact,
the tool used to create it (ubuntu-core-initramfs) is not available on RISC-V.

This means that we have to build it ourselves.
This also means that we have to build its dependencies ourselves, if they are
not available on RISC-V.

Only one such dependency is not available on RISC-V in the standard Ubuntu
repositories: systemd-bootchart. This project seems to have diverged from
systemd upstream and doesn't include any of the patches which were added to
systemd proper to provide support to RISC-V. Luckily, this isn't so difficult to
accomplish!

I won't go into immense detail here in exactly how to build Ubuntu packages from
source (there are a lot of resources available), but you can find some
instructions in the repository I've made for this project (reminder link): 
$/dilyn-corner/ubuntu-core-riscv64

Once you have your uc-initrd snap, the rest of the work is pretty
straightforward. My PR to add support to another Canonicler's project to add
RISC-V 64-bit kernel cross-building kernel support has been merged, and you can
find those plugins at $/kubiko/snapcraft-kernel-plugin to use in your kernel
snap builds.

One important thing to note: because we are booting using a virtualized drive,
we have to enable CONFIG_VIRTIO_FS in our kernel. Because we are also using the
FAT filesystem, we have to enable some codepages. These are generally built as
modules in the upstream kernel config, BUT our Core initrd does *not* include
the modprobe utility (at least not the way I built it), which means that we
won't be able to find our filesystem or mount it without those two options
builtin to our kernel. Just a word of warning.

Finally, with those aforementioned plugins and our snapcraft.yaml fully defined,
building the kernel is roughly as easy as:

snapcraft \
    --destructive-mode \
    --target-arch=riscv64 \
    --enable-experimental-target-arch

(Note that the gadget can be built in a similar way once OpenSBI and u-boot are
built).


[6.0] Board Bringup
________________________________________________________________________________

Once you have your built kernel and gadget snaps, you simply need to sign your
model assertion and build your image. That is also a pretty easy process, and I
won't document it here (again, see my repo for instructions or the official
documentation [9]). The final image can then be booted by qemu with a very
simple command:

qemu-system-riscv64 \
    -M virt -m 2G -smp 1 \
    -drive file=ubuntu-core.img,format=raw,if=virtio

If you've done everything correctly, you should be briefly greeted by the
OpenSBI screen before u-boot quickly kicks off its process and the kernel starts
doing its magical thing.

After a while, the initrd will start, and after about ten minutes snapd will
have finished bootstrapping the device and you'll be greeted by the usual
console-conf interface to provision the device with your Ubuntu SSO account's
SSH keys!

Some helpful notes that are good to know:

Press any key within two seconds of the machine booting (default time) to
interrupt u-boot so it doesn't automatically start your environment. This is
important to do to scope out the available memory addresses on whatever board
you're trying to bring up.

If you run into issues at the initrd step, you can add 'dangerous' to your
kernel commandline to enable the emergency shell systemd drops you into when
init fails -- these are masked by default on Core (for obvious reasons).

If you run into kernel panics that aren't related to being unable to find your
root device (make sure you built filesystem support *into* your kernel), you are
probably more than likely encountering a bug related to your hardware, some sort
of malformed device tree, or (in my case) OOM failures. Easiest way to resolve
OOM is to just reevaluate where things are loaded - I like to put the initrd
veeeeeeeerrryy far away from the kernel. On Qemu's virt board, the kernel is
moved to 0x80200000 before it is started. Keep that in mind.

Ultimately, this was a supremely fun project. I learned a lot of things I didn't
previously understand or know much about, and I feel far more competent when
dealing with more broad hardware setups than just AMD64. I'm excited to get my
hands on my RISC-V board and do all sorts of fiddling around with it.

And maybe I'll even pick up an ARM board.

___

[0] https://riscv.org
[1] https://www.aliexpress.com/item/1005003741287162.html?spm=a2g0o.9042311.0.0.16e84c4dzaUn0B
[2] https://www.aliexpress.com/item/1005002856721588.html?spm=a2g0o.store_pc_groupList.8148356.11.3afc6c1ezQ6plp
[3] https://github.com/snapcore/models
[4] https://snapcraft.io
[5] https://github.com/snapcore/pi-gadget/blob/20-arm64/gadget.yaml
[6] https://github.com/riscv-software-src/opensbi
[7] https://github.com/riscv-software-src/opensbi/tree/master/platform
[8] https://github.com/u-boot/u-boot
[9] https://snapcraft.io/docs


________________________________________________________________________________

Dilyn Corner (C) 2020-2022