Subsystem Programming Model guide

Please read the fundamentals first!

In Acacia, Subsystems are the vessel for modularising any and all Microkit code.

They are designed to allow you to express client-server relationships principally, but also support clientless systems or single-client systems.

To leverage Acacia confidently and with minimal friction, it is important to understand the model motivating the design of subsystems.

Contents

Theoretical model of client-server Subsystems

Walkthrough: making a simple client-free system

  1. Config Structs

  2. Using your Subsystem in a metaprogram

  3. Adding clients with build hooks

  4. Clients needing additional configuration

  5. Subsystems that depend on other subsystems

Acacia Build phase quick reference

Servers and clients

Our subsystem abstraction is principally oriented around client-server relationships because this has turned out to be the most natural way to express many constructs when doing system design atop the seL4 Microkit. For example, the seL4 Device Driver Framework (sDDF) and LionsOS both express their subsystems as some set of “server” Protection domains, MRs, Maps, IRQs, etc. connected to some number of client Protection domains.

Client-server is generally a common pattern in the design of microkernel OSes for good reason … it is flexible and is good at capturing IPC and priority relationships between programs and the (user-space) infrastructure they depend on.

Servers

In Acacia, we consider a server to be a grouping of any number of:

  • Protection Domains (or VMs), possibly with child Protection Domains,

  • Memory Regions and mappings into ProtectionDomains,

  • Channels,

  • Configuration data,

  • IRQs, IOPorts, IOMMU primitives (IOSpace/Maps), other Microkit objects,

  • High level Python constructs or data used to compose the system.

The server has some notion of clients. By default, Acacia assumes clients are ProtectionDomains (PDs). The Subsystem object itself is a representation of the server and any logic needed to compose it.

  • Subsystems have a default method add_client() which appends the client to an internal list clients.

  • Subsystems may override add_client() to take arbitrary parameters or even make non-PD objects clients.

  • If add_client() isn’t needed it can be disabled at construction.

  • Subsystems define a build hook to create any Microkit elements needed to attach the clients to the server. E.g. channels, memory regions.

Clients

Clients in Acacia are ProtectionDomains. We assume the server offers some facility at the granularity of a client PD.

E.g. for the sDDF network system:

  • Clients connect to a TX virtualiser (sending packets),

  • The RX virtualiser delivers packets to the client via a copier protection domain per-client,

  • Memory regions and channels are used to define asynchronous, lock-free queues to connect the client to the virtualiser and copier respectively.

In this case, adding a client PD with ethernet.add_client(pd, has_copier=True) will add the client to the ethernet subsystem’s clients list and tell it to create a copier PD, then connect the client to the TX virt + copier with queues.

Configuration data

To tell the compiled code everything that the infrastructure and client may need to know, we use Config Structs.

Every Subsystem defines a method called generate_config_structs which returns a list of these, defining the config to be patched into the server PDs as well as any clients, if needed.

Designing a subsystem

Before writing your subsystem, consider the Acacia Build process. It is very important to understand the order in which things are put together in order to have your subsystem work nicely!

For your subsystem, there are four critical stages you must consider.

  1. Initialisation - run the __init__() of your subsystem. This happens at declaration time and is considered a part of the metaprogram.

  2. Assembly phase - the metaprogram has finished running and you can assume that all clients are added, all manual allocation of vaddrs and IDs have completed, and any other subsystems we depend on exist.

  3. Allocation phase - if you needed to create physical memory regions without assigning them an address, they are allocated after the assembly phase. This is done automatically by acacia.System.

  4. Final phase - everything is frozen, config structs and XML can now be generated.

Basics: initialisation and config structs

In order to explain these, let’s define a super simple system. To make a subsystem, you must create a subclass of acacia.Subsystem. You must initialise the super class with super().__init__() by supplying the Acacia System instance that will be used and a name.

Subsystems are similar to the Microkit primitives in that they cannot exist without a parent System, since the System represents what will be in the Microkit XML file.

In practically all subsystems, you will define a minimum of the constructor and a config struct method.

In this case, our subsystem has a simple task:

  • Create some ProtectionDomain called secret_server, assuming we have some C code that compiled to secret_server.elf.

  • Take some secret number from the metaprogram and patch it into the secret server.

class YourSubsystem(Subsystem):
    def __init__(self, sdf: System, secret_number: int):
        super().__init__(sdf, "Your Subsystem Name!", clients_allowed=True)

        self.secret = secret_number
        self.server_pd = ProtectionDomain(sdf, "your_server", "your_server.elf")
        ...

    def generate_config_structs(self) -> List[ConfigStruct]:
        return [
            ConfigStruct(
            {
                "secret_number" : self.secret
            },
            target_file = self.server_pd.prog_image,
            section_name = "server_config"
            type_name = "secretServerConfig_t"
            )
        ]

We have no clients yet, but our sole PD acts as an isolated server. We will add client support in Adding Clients.

Defining your server infrastructure

The infrastructure is effectively “everything that is not a client or something used to connect the client to the infrastructure”.

You should always create as many of your infrastructure Microkit elements as possible in __init__(). There are a few reasons for this:

  • It keeps things simple, don’t defer work if you don’t need to.

  • It allows more flexibility - anything you create at __init__() can be directly manipulated in the metaprogram. This is handy for experimentation, e.g. changing priorities.

  • It makes it easy to add parts of your infrastructure as clients to other servers (more in server-server dependencies section below).

In some cases, it is not possible to make everything right away. The infrastructure might depend on your clients somehow, and you cannot know these at __init__. You should use a build hook to manage such cases. We will discuss this once we introduce clients in Adding Clients.

What are config structs?

Config struct? … this is our mechanism for patching data into compiled programs. See Config Structs guide.

Our secret server only needs one right now - a way to specify the secret number we pass in at __init__().

In this case, imagine that there is a corresponding C structure:

typedef struct secretServerConfig {
    uint32_t secret_number;
}

… and this is defined as a section in secret_server.c

__attribute__((__section__(".secret_config"))) secretServerConfig_t config;

Acacia will patch the data we define in ConfigStruct into this section, which can be accessed like a normal variable.

printf("The secret number is %u\n", config.secret_number);

All you need to do in Acacia is to specify the mapping between names of fields and their Python values and the type, section and file name to target it. The Python values which will be translated into C types. Not all Python types can be converted, see Serialisable Types.

    def generate_config_structs(self) -> List[ConfigStruct]:
        return [
            ConfigStruct(
            # Define struct field name -> python val map
            {
                "secret_number" : self.secret
            },
            # Which compiled ELF file do we patch?
            target_file = self.server_pd.prog_image,
            # Which section in the ELF?
            section_name = "server_config",
            # What is the name of the struct type, for sanity checking?
            type_name = "secretServerConfig_t"
        )
    ]
...

Using your subsystem

Once you have written your implementation (and any corresponding C/Rust/Pancake code + supporting Makefiles), you can express a metaprogram to build the system.

meta.py

sdf = System(aarch64, paddr_top=0x90000000, DeviceTreeBlob(""./board.dts"))
MySubsystem(sdf, 42)
sdf.make_config_struct("./build")           # Outputs config struct blobs to `./build`
sdf.write_xml_file("secret_server.system")  # Write Microkit XML file

In this case, we are setting secretServerConfig_t.secret_number to uint32_t 42. Running secret_server.system through Microkit and then executing the binary we see:

The secret number is 42

Adding clients

So we can now create simple infrastructure and configuration data. Let’s extend our secret server to have clients … each client will be connected with a Microkit Channel and can perform a protected procedure call to get the secret data.

We cannot do anything about our clients in __init__() since none will have been added yet, so we must instead utilise the pre-defined method add_client(self, client_pd: ProtectionDomain). This method is defined by Subsystem() and will simply add the PD to a list in the superclass called clients.

(Note: if the subsystem is initialised with allow_clients=False this method will be disabled.)

Instead, we must use a build hook … a method that is run at the assembly phase.

Build hooks

Build hooked methods are annotated with a special decorator to tell Acacia to run them at the assembly phase. You can have as many as you want, and Acacia will execute them top-to-bottom within a class. I.e. the one closest to the top of the file will run first. We can add a build hook to connect our clients.

class YourSubsystem(Subsystem):
    def __init__(self, sdf: System, secret_number: int):
        self.client_channels: Dict[ProtectionDomain, int]   # Keep a list of IDs for clients
        ...

    @build_hook
    def connect_clients(self):
        for c in self.clients:
            ch = Channel(
                self.sdf,
                Channel.End(c, can_notify=False, can_pp=True),
                Channel.End(self.secret_server),
            )
            # Store channel ID so client knows who to call
            self.client_channels[c] = ch.id_for_pd(c)

As you can see, we make a new connect_clients() method which is annotated as a build hook. It will now be run automatically once the assembly phase runs and Acacia begins to build this subsystem instance.

We create a channel from every client to the secret server, and we then store the numerical ID of the channel (i.e. what the client will microkit_ppcall()) in a dictionary.

Config structs for clients

We’ve got an ID for each client, now we must get it to them! We can extend our generate_config_structs to make more config structs for the clients.

    def generate_config_structs(self) -> List[ConfigStruct]:
        return [
            ConfigStruct(
            {
                "secret_number" : self.secret
            },
            target_file = self.server_pd.prog_image,
            section_name = "server_config"
            type_name = "secretServerConfig_t"
            )
        ] + [
            ConfigStruct(
                {
                    secret_ch : self.client_channels[c]
                },
                target_file = c.prog_image,
                section_name = "secret_ch",
                type_name = "secretClientConfig_t"
            )
            for c in self.clients
        ]

We do not need to worry about if the clients are connected, since Acacia will only ever call generate_config_structs after the assembly phase finishes.

Adding clients with extra config

If you need to have additional configuration data alongside your clients, you can override the add_client() method.

I.e. in your subsystem, define add_client(), making sure to still super() call the original method (unless you are sure you don’t need it).

def add_client(self, client: ProtectionDomain, client_needs_thing: bool):
    super().add_client(client)  # Add to clients list
    self.client_configs[client] = client_needs_thing
   ...

@build_hook
def connect_clients(self):
    for c in self.clients:
        ...

        # Check config here
        if self.client_configs[c]:
            ...

In most cases… you should do nothing other than keep a reference to the client config in the initialisation phase. Create all Microkit objects etc. in a build hook, not in add_client(). This is because…

  • The PD might not be in a final state yet, i.e. the user might modify it in the metaprogram still.

  • Other subsystems may also have it as a client and may need to run before us.

  • It’s much less confusing to isolate all connection logic to a build hook.

For an example, see the sDDF net.py class which overloads the method to add options for MAC addresses and turning the copier on or off.

Sidenote: build hooks and OO programming

Build hooks support inheritance and multiple inheritance. In these cases, they will run in Method Resolution Order, that is, in the same order as Python would precedence of methods.

  • If a parent class defines build hooks, they will always run before the child class’s build hooks do.

  • If a child overrides a parent’s build hook, it will run in the same order as any other build hook defined in the child.

  • If a child class overrides a parent’s build hook and doesn’t annotate it as one (i.e. no @build_hook), it will not run.

We recommend minimising inheritance with build hooks. We only support MRO in order to make it sane to use this where it might be unavoidable.

Subsystems depending on other subsystems

It is very possible that your subsystem (or some infrastructure inside of it) may need to use the facilities of another subsystem.

For example, let’s imagine that our secret server needs a security feature … it wants to use the sDDF timer driver to limit how often clients can check the secret.

In this case, we most follow one very important rule:

Always capture Subsystem dependencies and add infrastructure as clients in __init__()!

I.e.

class YourSubsystem(Subsystem):
    def __init__(self, sdf: System, secret_number: int, timer: sDDFTimer):
        super().__init__(sdf, "Your Subsystem Name!", clients_allowed=True)

        self.secret = secret_number
        self.client_channels: Dict[ProtectionDomain, int]   # Keep a list of IDs for clients
        self.server_pd = ProtectionDomain(sdf, "your_server", "your_server.elf")

        # Add the server as a client of the timer.
        timer.add_client(self.server_pd)
        ...

Acacia will always run the build hooks of Subsystems in the order they are declared. We have a tension here…

  • We want to take a reference to the other subsystem right away, to ensure the we have our dependency, but,

  • Our dependent subsystem will build before us, so we cannot safely add ourselves as a client in the assembly phase.

We will implement a dependency graph to order building of subsystems in the assembly phase in the future.

Build phases quick reference.

Initialisation phase

When? Your __init__() of your Subsystem, any time before Acacia begins assembly. Conceptually, “in the metaprogram”.

What to do? Set up all infrastructure that doesn’t need to know about clients

Constraints Clients are still unknown.

Assembly phase

When? As soon as the metaprogram calls a terminal action … i.e. trying to generate config structs or write the XML. This triggers Acacia to begin assembly.

Subsystems are assembled in the order they are declared, i.e.

FirstSubsystem(sdf)
SecondSubsystem(sdf)
ThirdSubsystem(sdf)
...
sdf.write_xml()

… will run the build hooks for the three subsystems in top-to-bottom order (first, second, third).

What to do? Create connections between clients and infrastructure, modify infrastructure that depends on clients, interact with other subsystems.

Constraints Manual allocation of map virtual addresses and manual allocation of IDs for IRQs, IOPorts, child PDs is disabled.

This is to prevent collisions between multiple Subsystems who target the same client … two subsystems cannot both demand their map has a vaddr of 0x20202020 since this would overlap.

Allocation phase

When? As soon as all Subsystems are assembled.

What to do? Nothing, Acacia will allocate any outstanding physical memory addresses that are not allocated yet.

Constraints Nothinig should run now.

Final phase

When? As soon as allocation completes.

What to do? Tell Acacia to emit XML or config structs to your build directory.

Constraints Subsystems can no longer have clients added. Microkit primitives can no longer be instantiated.

Things to avoid

If you run into any of the following, it probably means something has gone wrong in your design!

Problem

Reason

Attempting to access clients in __init__

Clients should only be added after init!

Hardcoding virtual addresses or IDs in a subsystem

Automatic mappings are generally better as hard coded ones will clash. Automatic allocation and config structs mean that it is not necessary except for physicall addresses in drivers.

Creating new Microkit elements in generate_config_structs

All Microkit elements are expected to exist before getting here. Acacia will attempt to crash if you allocate too late. Use build hooks instead.

Lots of build hooks

Confusing, and usually not needed. Build hooks run in top-to-bottom order of declaration