Interrupt and syscall handling, redesigned

Posted on Sat 20 June 2020 in MUPS16

I'm going to just call interrupts, syscalls, page faults, privilege errors etc. exceptions from now on to save typing. Since the current design of exceptions has a few flaws, we need a new design. I'm not keen on redesigning the existing hardware I've made if possible, so that means no changes to the register unit. Fortunately, I think it's possible to get a cleaner implementation without needing any extra registers.

The current implementation basically only takes care of saving the current process's SP register, and restores the system TSP stack pointer before jumping to address 0x2. We can repurpose the TSP register to be a pointer to a saved process info structure in memory, which the exception-handling microcode can populate when an exception occurs. In addition, we can change the RFE instruction to restore the state from the same structure.

In addition to cleaning up the implementation, I think we need to add a few new exception types:

  • invalid instruction
  • breakpoint interrupt? Would be useful for debugging
  • bus error (invalid alignment on memory op)

As well as the simple cases (a single exception triggers, and the handler runs uninterrupted to conclusion), there are a few more complicated cases to consider:

  1. a syscall handler is executing, and a pagefault occurs
  2. a pagefault handler is executing, and an interrupt occurs
  3. a pagefault handler triggers another page fault
  4. an interrupt handler is running, and another interrupt occurs
  5. an exception handler triggers an invalid instruction or bus error

I think we can simply not allow case 4 (all interrupts are equal, and only one can be handled at a time, which can be enforced in hardware). Cases 3 and 5 could be handled by just deciding that they aren't allowed. If we add an extra exception type (double_fault?) we could trigger that if any exception except an interrupt occurs during execution of a handler. The double-fault handler will probably just have to halt the system (in which case we may want to add a HALT instruction).

With these restrictions, we have a maximum nested depth of 3 exceptions:

  • process invokes a syscall
  • syscall triggers a page fault
  • interrupt occurs during page-fault handling

This means that we can make do with four process state structures in memory, though we may want to have one structure per process, plus three extra for the worst-case scenario above. In order to simplify nested handling we could add a pointer to the next free struct in each proc_info structure, which would be automatically loaded into TSP when a handler is invoked.

Process state

The minimal state that we need to save before invoking a trap handler is the stack pointer, one user register, and the flags register. Technically that's enough for the exception handling code to explicitly save each of the remaining registers. Given that each memory write instruction takes five cycles, that would make exception handling quite expensive. Under the current design of the control unit there's nothing to stop us saving all the user registers into a memory structure automatically, which can be done in around 20 cycles, which is about half the time of the explicit implementation. Note that this would be a terrible idea if we ever changed the control unit to be pipelined, but that's a problem for the future.

So, we can change the exception handler to save all user registers and the flags register automatically. In addition, we need to pass along some extra info to the handler, depending on the type of exception (for a page fault, bus error, invalid instruction or privilege error the handler will need the address that caused the fault, for example; for a syscall, it will need the syscall number, etc.). Since we've already saved the user registers, we can safely pass extra arguments in registers, ready for the exception handler.

If we separate this into state that might need to be restored, which goes into the process state structure, and arguments to the exception handler, which go into registers, we end up with a proc_state structure that looks like:

Offset Desc
0x00 R1
0x02 R2
0x04 R3
0x06 R4
0x08 SP
0x0A PC
0x0C flags
0x0D tblidx
0x0E prev
0x0E next

The prev and next fields aren't strictly necessary, but could be used by exception handlers to set up a new process info struct prior to re-enabling interrupts, for example.

Starting a new process

Creating a new process would require a few steps:

  • find or allocate a new proc_state struct
  • populate the flags struct appropriately
  • set next to point to the first trap-handler state, which would in turn have a pointer to the second, etc.
  • load the address of the proc_info struct into S1
  • issue an RFE instruction

The RFE instruction could then consist of:

  1. output S2 to the A bus, output 0x2 to the B bus, set MMU to map, set ALU to add, set inreg to S2 (this maps proc_state.r1, and simultaneously updates S1 to point to proc_state.r2 on the next cycle)
  2. set the MMU to output the mapped address, set memory to read, and set inreg to r1

These steps would be repeated 7 times to restore all the registers (the only difference is that setting the flags uses slightly different control lines, as it's not an addressable register, but it's still a single μ-op). After this we need to set the page-table index, and finally restore PC.

Unfortunately, this uses more than 16 μops. The current design of the microcode addressing circuit allows 16 entries for each instruction. Each instruction starts with two ops that fetch the instruction from memory, leaving 14 free. Restoring each register uses two μops, and there are 7 registers, leaving nothing for setting the page-table index. Note that this isn't a problem for the exception handling code as that uses a slightly different circuit that executes up to 32 μ-ops starting at a fixed address in the μcode ROM.

To work around this, at the cost of a bit of inconvenience, we could require the user code to read some of the fields into system registers before the RFE instruction. Moving a value from one register to another can be done in a single μop, so this should let us squeeze under the 14 μop limit. Alternatively, we could simply dictate that restoring the user registers is the responsibility of the OS, as it is something that doesn't require hardware support, and we continue passing the values of PC, flags and tblidx to RFE in system registers. I'll probably start with this, as it involves minimal changes, though it ruins the symmetry of the exception trigger saving all the user registers and the rfe instruction restoring them.

Starting a new process would therefore look something like (assuming that the address of the new process's proc_state struct is in r1, and the struct's fields have been populated appropriately):

lw   sp, 0x8[r1]    ; load sp
lw   r2, 0xa[r1]    ; load pc
lb   r3, 0xc[r1]    ; load flags
lb   r4, 0xd[r1]    ; load page-table index
mts  spc, r2        ; spc gets the proc's starting pc
mts  s1, r3         ; s1 gets the proc's flags
mts  s2, r4         ; s2 gets the page-table index
mts  s3, r1         ; s3 gets the address to store in tsp
rfe

Exception handling

Assuming we've set up the exception handling μcode to save all user registers and the flags register, load r1 (and maybe others) with info about the exception that triggered the handler, disable interrupts and jump to address 0x2 then exception handling in the OS is fairly straightforward.

; Interrupt table, mapped to address 0x2
jmp .handle_priverr         ; privilege error
jmp .handle_pageflt         ; page fault
jmp .handle_buserr          ; alignment error
jmp .handle_invinstr        ; invalid instruction
jmp .handle_userexcept      ; system call or other user exception
jmp .unknown_exception      ; 5-8 aren't used at the moment
jmp .unknown_exception
jmp .unknown_exception
jmp .handle_interrupt       ; hardware interrupts. We could
jmp .handle_interrupt       ; direct these directly to different
jmp .handle_interrupt       ; handlers, or jump to a single handler
jmp .handle_interrupt       ; for each.
jmp .handle_interrupt       ;
jmp .handle_interrupt       ;
jmp .handle_interrupt       ;
jmp .handle_interrupt       ; final hardware interrupt
...

The code at address 0x2 will most likely just be a jump to the address of the actual exception handler, although the handling code could be inlined there, too, if that makes sense.

Then, what we do in software depends on the trap type:

Interrupt

If we simply don't allow nested interrupts, an interrupt handler can simply service the interrupt (copy pending data to a fixed OS buffer, for example). To return from the interrupt handler, we would need something like

mfs sp, tsp     ; get current value of TSP
lw  r1, [0a]sp  ; get saved PC
mts spc, r1     ; move saved PC to SPC, ready for RFE instruction
lw  r1, [0]sp   ; restore saved registers
lw  r2, [2]sp
lw  r3, [4]sp
lw  r4, [6]sp
lw  sp, [8]sp
rfe              ;

This would return to the previous running code, and leave tsp pointing to the same structure, ready for any further traps. One alternative would be to change the rfe instruction to handle restoring state for us, so it's a mirror of the trap invocation, which might be cleaner.

Page fault

A page fault starts with interrupts disabled, but we're going to want to enable them at some point, as page faults can be slow to service. So, we have to be prepared for an interrupt to happen during a page fault.

If we add a restriction that there will only be one page-fault handler executing at any one time, and that page-fault handlers will never execute a syscall (which they should never need to, since they're executing as privileged processes anyway), then we could get away with