Subsystem Programming Model guide¶
Please read the fundamentals first!
In Acacia, Subsystems are the vessel for modularising any and all Microkit code.
They are designed to allow you to express client-server relationships principally, but also support clientless systems or single-client systems.
To leverage Acacia confidently and with minimal friction, it is important to understand the model motivating the design of subsystems.
Contents¶
Theoretical model of client-server Subsystems
Walkthrough: making a simple client-free system
Servers and clients¶
Our subsystem abstraction is principally oriented around client-server relationships because this has turned out to be the most natural way to express many constructs when doing system design atop the seL4 Microkit. For example, the seL4 Device Driver Framework (sDDF) and LionsOS both express their subsystems as some set of “server” Protection domains, MRs, Maps, IRQs, etc. connected to some number of client Protection domains.
Client-server is generally a common pattern in the design of microkernel OSes for good reason … it is flexible and is good at capturing IPC and priority relationships between programs and the (user-space) infrastructure they depend on.
Servers¶
In Acacia, we consider a server to be a grouping of any number of:
Protection Domains (or VMs), possibly with child Protection Domains,
Memory Regions and mappings into ProtectionDomains,
Channels,
Configuration data,
IRQs, IOPorts, IOMMU primitives (IOSpace/Maps), other Microkit objects,
High level Python constructs or data used to compose the system.
The server has some notion of clients. By default, Acacia assumes clients are ProtectionDomains (PDs).
The Subsystem object itself is a representation of the server and any logic needed to compose it.
Subsystems have a default method
add_client()which appends the client to an internal listclients.Subsystems may override
add_client()to take arbitrary parameters or even make non-PD objects clients.If
add_client()isn’t needed it can be disabled at construction.Subsystems define a build hook to create any Microkit elements needed to attach the clients to the server. E.g. channels, memory regions.
Clients¶
Clients in Acacia are ProtectionDomains. We assume the server offers some facility at the granularity of a client PD.
E.g. for the sDDF network system:
Clients connect to a TX virtualiser (sending packets),
The RX virtualiser delivers packets to the client via a copier protection domain per-client,
Memory regions and channels are used to define asynchronous, lock-free queues to connect the client to the virtualiser and copier respectively.
In this case, adding a client PD with ethernet.add_client(pd, has_copier=True) will add the client
to the ethernet subsystem’s clients list and tell it to create a copier PD, then connect
the client to the TX virt + copier with queues.
Configuration data¶
To tell the compiled code everything that the infrastructure and client may need to know, we use Config Structs.
Every Subsystem defines a method called generate_config_structs which returns a list of these, defining the config
to be patched into the server PDs as well as any clients, if needed.
Designing a subsystem¶
Before writing your subsystem, consider the Acacia Build process. It is very important to understand the order in which things are put together in order to have your subsystem work nicely!
For your subsystem, there are four critical stages you must consider.
Initialisation - run the
__init__()of your subsystem. This happens at declaration time and is considered a part of the metaprogram.Assembly phase - the metaprogram has finished running and you can assume that all clients are added, all manual allocation of vaddrs and IDs have completed, and any other subsystems we depend on exist.
Allocation phase - if you needed to create physical memory regions without assigning them an address, they are allocated after the assembly phase. This is done automatically by
acacia.System.Final phase - everything is frozen, config structs and XML can now be generated.
Basics: initialisation and config structs¶
In order to explain these, let’s define a super simple system.
To make a subsystem, you must create a subclass of acacia.Subsystem. You must
initialise the super class with super().__init__() by supplying the Acacia System instance that will be used and a name.
Subsystems are similar to the Microkit primitives in that they cannot exist without a parent System,
since the System represents what will be in the Microkit XML file.
In practically all subsystems, you will define a minimum of the constructor and a config struct method.
In this case, our subsystem has a simple task:
Create some ProtectionDomain called
secret_server, assuming we have some C code that compiled tosecret_server.elf.Take some secret number from the metaprogram and patch it into the secret server.
class YourSubsystem(Subsystem):
def __init__(self, sdf: System, secret_number: int):
super().__init__(sdf, "Your Subsystem Name!", clients_allowed=True)
self.secret = secret_number
self.server_pd = ProtectionDomain(sdf, "your_server", "your_server.elf")
...
def generate_config_structs(self) -> List[ConfigStruct]:
return [
ConfigStruct(
{
"secret_number" : self.secret
},
target_file = self.server_pd.prog_image,
section_name = "server_config"
type_name = "secretServerConfig_t"
)
]
We have no clients yet, but our sole PD acts as an isolated server. We will add client support in Adding Clients.
Defining your server infrastructure¶
The infrastructure is effectively “everything that is not a client or something used to connect the client to the infrastructure”.
You should always create as many of your infrastructure Microkit elements as possible in __init__(). There are a few reasons for this:
It keeps things simple, don’t defer work if you don’t need to.
It allows more flexibility - anything you create at
__init__()can be directly manipulated in the metaprogram. This is handy for experimentation, e.g. changing priorities.It makes it easy to add parts of your infrastructure as clients to other servers (more in server-server dependencies section below).
In some cases, it is not possible to make everything right away. The infrastructure might depend on your clients somehow, and you cannot know these at __init__.
You should use a build hook to manage such cases. We will discuss this once we introduce clients in Adding Clients.
What are config structs?¶
Config struct? … this is our mechanism for patching data into compiled programs. See Config Structs guide.
Our secret server only needs one right now - a way to specify the secret number we pass in at __init__().
In this case, imagine that there is a corresponding C structure:
typedef struct secretServerConfig {
uint32_t secret_number;
}
… and this is defined as a section in secret_server.c
__attribute__((__section__(".secret_config"))) secretServerConfig_t config;
Acacia will patch the data we define in ConfigStruct into this section, which can be accessed like a normal variable.
printf("The secret number is %u\n", config.secret_number);
All you need to do in Acacia is to specify the mapping between names of fields and their Python values and the type, section and file name to target it. The Python values which will be translated into C types. Not all Python types can be converted, see Serialisable Types.
def generate_config_structs(self) -> List[ConfigStruct]:
return [
ConfigStruct(
# Define struct field name -> python val map
{
"secret_number" : self.secret
},
# Which compiled ELF file do we patch?
target_file = self.server_pd.prog_image,
# Which section in the ELF?
section_name = "server_config",
# What is the name of the struct type, for sanity checking?
type_name = "secretServerConfig_t"
)
]
...
Using your subsystem¶
Once you have written your implementation (and any corresponding C/Rust/Pancake code + supporting Makefiles), you can express a metaprogram to build the system.
meta.py
sdf = System(aarch64, paddr_top=0x90000000, DeviceTreeBlob(""./board.dts"))
MySubsystem(sdf, 42)
sdf.make_config_struct("./build") # Outputs config struct blobs to `./build`
sdf.write_xml_file("secret_server.system") # Write Microkit XML file
In this case, we are setting secretServerConfig_t.secret_number to uint32_t 42. Running secret_server.system through Microkit and then executing the binary we see:
The secret number is 42
Adding clients¶
So we can now create simple infrastructure and configuration data. Let’s extend our secret server to have clients … each client will be connected with a Microkit Channel and can perform a protected procedure call to get the secret data.
We cannot do anything about our clients in __init__() since none will have been added yet, so
we must instead utilise the pre-defined method add_client(self, client_pd: ProtectionDomain).
This method is defined by Subsystem() and will simply add the PD to a list in the superclass called clients.
(Note: if the subsystem is initialised with allow_clients=False this method will be disabled.)
Instead, we must use a build hook … a method that is run at the assembly phase.
Build hooks¶
Build hooked methods are annotated with a special decorator to tell Acacia to run them at the assembly phase. You can have as many as you want, and Acacia will execute them top-to-bottom within a class. I.e. the one closest to the top of the file will run first. We can add a build hook to connect our clients.
class YourSubsystem(Subsystem):
def __init__(self, sdf: System, secret_number: int):
self.client_channels: Dict[ProtectionDomain, int] # Keep a list of IDs for clients
...
@build_hook
def connect_clients(self):
for c in self.clients:
ch = Channel(
self.sdf,
Channel.End(c, can_notify=False, can_pp=True),
Channel.End(self.secret_server),
)
# Store channel ID so client knows who to call
self.client_channels[c] = ch.id_for_pd(c)
As you can see, we make a new connect_clients() method which is annotated as a build hook.
It will now be run automatically once the assembly phase runs and Acacia begins to build this subsystem instance.
We create a channel from every client to the secret server, and we then store
the numerical ID of the channel (i.e. what the client will microkit_ppcall()) in a dictionary.
Config structs for clients¶
We’ve got an ID for each client, now we must get it to them! We can extend our generate_config_structs to make more config structs for the clients.
def generate_config_structs(self) -> List[ConfigStruct]:
return [
ConfigStruct(
{
"secret_number" : self.secret
},
target_file = self.server_pd.prog_image,
section_name = "server_config"
type_name = "secretServerConfig_t"
)
] + [
ConfigStruct(
{
secret_ch : self.client_channels[c]
},
target_file = c.prog_image,
section_name = "secret_ch",
type_name = "secretClientConfig_t"
)
for c in self.clients
]
We do not need to worry about if the clients are connected, since Acacia will only
ever call generate_config_structs after the assembly phase finishes.
Adding clients with extra config¶
If you need to have additional configuration data alongside your clients, you can
override the add_client() method.
I.e. in your subsystem, define add_client(), making sure to still super() call the original method (unless you are sure you don’t need it).
def add_client(self, client: ProtectionDomain, client_needs_thing: bool):
super().add_client(client) # Add to clients list
self.client_configs[client] = client_needs_thing
...
@build_hook
def connect_clients(self):
for c in self.clients:
...
# Check config here
if self.client_configs[c]:
...
In most cases… you should do nothing other than keep a reference to the client config in the initialisation phase. Create all Microkit objects etc. in a build hook, not in add_client(). This is because…
The PD might not be in a final state yet, i.e. the user might modify it in the metaprogram still.
Other subsystems may also have it as a client and may need to run before us.
It’s much less confusing to isolate all connection logic to a build hook.
For an example, see the sDDF net.py class which overloads the method to add
options for MAC addresses and turning the copier on or off.
Sidenote: build hooks and OO programming¶
Build hooks support inheritance and multiple inheritance. In these cases, they will run in Method Resolution Order, that is, in the same order as Python would precedence of methods.
If a parent class defines build hooks, they will always run before the child class’s build hooks do.
If a child overrides a parent’s build hook, it will run in the same order as any other build hook defined in the child.
If a child class overrides a parent’s build hook and doesn’t annotate it as one (i.e. no
@build_hook), it will not run.
We recommend minimising inheritance with build hooks. We only support MRO in order to make it sane to use this where it might be unavoidable.
Subsystems depending on other subsystems¶
It is very possible that your subsystem (or some infrastructure inside of it) may need to use the facilities of another subsystem.
For example, let’s imagine that our secret server needs a security feature … it wants to use the sDDF timer driver to limit how often clients can check the secret.
In this case, we most follow one very important rule:
Always capture Subsystem dependencies and add infrastructure as clients in
__init__()!
I.e.
class YourSubsystem(Subsystem):
def __init__(self, sdf: System, secret_number: int, timer: sDDFTimer):
super().__init__(sdf, "Your Subsystem Name!", clients_allowed=True)
self.secret = secret_number
self.client_channels: Dict[ProtectionDomain, int] # Keep a list of IDs for clients
self.server_pd = ProtectionDomain(sdf, "your_server", "your_server.elf")
# Add the server as a client of the timer.
timer.add_client(self.server_pd)
...
Acacia will always run the build hooks of Subsystems in the order they are declared. We have a tension here…
We want to take a reference to the other subsystem right away, to ensure the we have our dependency, but,
Our dependent subsystem will build before us, so we cannot safely add ourselves as a client in the assembly phase.
We will implement a dependency graph to order building of subsystems in the assembly phase in the future.
Build phases quick reference.¶
Initialisation phase¶
When?
Your __init__() of your Subsystem, any time before Acacia begins assembly. Conceptually, “in the metaprogram”.
What to do? Set up all infrastructure that doesn’t need to know about clients
Constraints Clients are still unknown.
Assembly phase¶
When? As soon as the metaprogram calls a terminal action … i.e. trying to generate config structs or write the XML. This triggers Acacia to begin assembly.
Subsystems are assembled in the order they are declared, i.e.
FirstSubsystem(sdf)
SecondSubsystem(sdf)
ThirdSubsystem(sdf)
...
sdf.write_xml()
… will run the build hooks for the three subsystems in top-to-bottom order (first, second, third).
What to do? Create connections between clients and infrastructure, modify infrastructure that depends on clients, interact with other subsystems.
Constraints Manual allocation of map virtual addresses and manual allocation of IDs for IRQs, IOPorts, child PDs is disabled.
This is to prevent collisions between multiple Subsystems who target the same client … two subsystems cannot both demand their map has a vaddr of 0x20202020 since this would overlap.
Allocation phase¶
When? As soon as all Subsystems are assembled.
What to do? Nothing, Acacia will allocate any outstanding physical memory addresses that are not allocated yet.
Constraints Nothinig should run now.
Final phase¶
When? As soon as allocation completes.
What to do? Tell Acacia to emit XML or config structs to your build directory.
Constraints Subsystems can no longer have clients added. Microkit primitives can no longer be instantiated.
Things to avoid¶
If you run into any of the following, it probably means something has gone wrong in your design!
Problem |
Reason |
|---|---|
Attempting to access clients in |
Clients should only be added after init! |
Hardcoding virtual addresses or IDs in a subsystem |
Automatic mappings are generally better as hard coded ones will clash. Automatic allocation and config structs mean that it is not necessary except for physicall addresses in drivers. |
Creating new Microkit elements in |
All Microkit elements are expected to exist before getting here. Acacia will attempt to crash if you allocate too late. Use build hooks instead. |
Lots of build hooks |
Confusing, and usually not needed. Build hooks run in top-to-bottom order of declaration |