New SimBricks Configuration Abstractions

Along with our new architecture we also re-designed the configuration abstractions (part of the API) that users rely on to configure virtual prototypes through our programmatic frontend. Why did we take this step and what does the result look like?

The Original Orchestration Framework

Our original orchestration framework was initially simply a means to run SimBricks simulations. Given that we combine multiple instances of different simulators, even for small simulations doing this manually is prohibitively complex. The orchestration framework generate configurations for individual simulators, in the corresponding formats, connects up different simulators according to the specification, and then starts up and runs simulators in the correct order, and finally terminates simulators and cleans up. This works very well, and enables users to run SimBricks simulations completely automatically.

Moving from Simulations to Virtual Prototypes

However, while using SimBricks over time in practice, we frequently found the need to set up different simulator configurations for the same system. Examples include testing out different simulators or parameters to find the ideal trade-off of performance and accuracy, or simulating a systems differently for different use-cases, e.g. functional testing vs. performance evaluation. Additionally, even after choosing a concrete simulation configuration, there are still multiple different ways of instantiating and running this simulation, such as assigning different simulators to different physical machines in distributed simulations, or running different simulators on different processor cores in a NUMA system.

Unfortunately, this generally requires substantially changing or re-creating simulation configurations from scratch while repeating similar information. This complicates maintaining these configurations as systems evolve. More generally, we found that this forces users to start by thinking about simulator details from the beginning, and also makes it difficult to hide unnecessary complexity from users through automation. This is a growing challenge as we aimed to move towards enabling and simplifying new virtual prototyping use-cases.

Virtual Prototype Configuration Abstractions

Thus we decided to fundamentally re-design our virtual prototype configuration abstractions and thereby streamline and simplify configuration for a broad range of use-cases. The key idea is a clear separation of user concerns also in our configuration abstractions. Concretely, with the new configuration abstraction users configure their virtual prototype simulations in three steps: system, simulation, instantiation, each building on the previous step.

Step 1: System Configuration

Users start off by writing a system configuration that contains all information about the virtual prototype system, such as the components involved and their properties (e.g. for a server this includes the core count, memory capacity as well as the software configuration to run on it). The system configuration is generic and these properties are specified independent of any simulators. More generally, the system configuration controls what the virtual prototype looks like, not how it should be simulated.

Here is a concrete example of a simple 2-machine system setup with two network cards connected through a switch with an iperf client and server application running:

syst = system.System()

# Add a pre-configured server host and network card
server = system.I40ELinuxHost(syst)
nic0 = system.IntelI40eNIC(syst)
nic0.add_ipv4("10.0.0.1")
server.connect_pcie_dev(nic0)

server_app = system.IperfTCPServer(h=server)
server.add_app(server_app)


# Add a pre-configured client host
client = system.I40ELinuxHost(syst)
nic1 = system.IntelI40eNIC(syst)
nic1.add_ipv4("10.0.0.2")
client.connect_pcie_dev(nic1)

client_app = system.IperfTCPClient(h=client, server_ip=nic0._ip)
client_app.wait = True
client.add_app(client_app)


# create switch and connect NICs to switch
switch = system.EthSwitch(syst)
switch.connect_eth_peer_if(nic0._eth_if)
switch.connect_eth_peer_if(nic1._eth_if)

Step 2: Simulation Configuration

In the second step, the user then specify how to simulate the previously specified system. This involves creating concrete simulator instances and assigning system components to them. If supported by the simulator, users can even configure a simulator instance to simulate multiple system components, e.g. using an ns-3 network simulator instance to simulate a whole network comprising many interconnected switches. Users can also easily create multiple different simulation configurations for the same system configuration for different uses.

Here is an example of a simulation configuration for the system syst above. We use a helper function to automatically instantiate one simulator of the specified type for each corresponding system component type:

sim = sim_helpers.simple_simulation(syst, compmap={
        system.FullSystemHost: sim.QemuSim,
        system.IntelI40eNIC: sim.I40eNicSim,
        system.EthSwitch: sim.SwitchNet,
    })

Step 3: Instantiation Configuration

Finally, the user specifies details for how and where to run the simulators. For simulations with timing and synchronization enabled, changes in this last step do not affect the functionality or performance of the simulated system, but may affect simulation time. For simple simulation that completely run on a single host this is typically a one-liner:

inst = instantiation.Instantiation(sim)

However, for running a virtual prototype simulation distributed across multiple runners, this step defines which simulators should run together as a fragment and can place constraints about which runners each fragment should run on through tags:

inst = inst_helpers.simple_dist(sim,
    mapper=lambda s: 1 if s.components()[0] in [client, nic1] else 0)
inst.get_fragment(1).tags = ['lab1-runner']

At this point the user can run the virtual prototype instantiation, as before, with the simbricks CLI tool. Alternatively, the user can use the SimBricks API to automate and customize this process.

Increased Complexity?

While this separation conceptually makes sense, we have now turned a simple one-step configuration process into 3 steps. How do we justify this (seemingly) increased complexity? First off, for the simple case, the last step is always a one-liner, and the second step typically takes 3-5 lines. Moreover, even with the old version of the configuration abstraction, we typically found ourselves parametrizing simulator types in our configuration, as changing simulators was a frequent operation. This took similar effort, but resulted in configurations that were more difficult to understand. As a result, we find that the new configurations are typically of a similar length but easier to understand and modify.

Towards Increased Automation

More importantly, the clear separation of concerns in the configuration provides a natural way towards automating parts of this virtual prototype configuration, also with a future graphical UI in mind. Users typically think primarily about the system configuration. For less technical virtual prototyping use-cases, such as interactive sales demos, users ideally should not have to think about choices of simulators or instantiations at all. And even in more technical use-cases users typically switch between a small set of different simulation and instantiation strategies, e.g. one for fast functional tests, and another one for more detailed performance evaluation. Our new configuration abstractions enable developers to separate exactly these choices from the concrete system configuration that typically changes more frequently, and externalize these strategies.

Our configuration scripts are regular python code, and as such users can leverage language features such as functions and modules, to separate out this code. Here is a simple example of two strategies:

# Module: mycompany.vp.strategies
def fast_functional(syst: System) -> Instantiation:
    sim = sim_helpers.simple_simulation(syst, compmap={
            system.FullSystemHost: sim.QemuSim,
            system.IntelI40eNIC: sim.I40eNicSim,
            system.EthSwitch: sim.SwitchNet,
        })
    return instantiation.Instantiation(sim)

def detailed(syst: System) -> Instantiation:
    sim = sim_helpers.simple_simulation(syst, compmap={
            system.FullSystemHost: sim.Gem5TimingSim,
            system.IntelI40eNIC: sim.I40eNicSim,
            system.EthSwitch: sim.NS3Sim,
        })
    return instantiation.Instantiation(sim)

Overall, we believe that this re-design of our configuration abstractions makes using SimBricks simpler and opens the way towards enabling new virtual prototyping use-cases and features. We are looking forward to talk more about these in future posts. Until then, as usual: