This chapter records the Stage 0 scheduler questions. Stage 4 later measured a dirty-CPU burn against a normal-scheduler control in a one-scheduler VM: 18 heartbeats during 200 ms of dirty work, and 0 during the same burn with flags 0. See stages/s4.md. The sentence below that says this repository never launched a GPU kernel is the Stage 0 fact. Stage 8 ran one.
Learning objectives
- Decide whether
addbelongs on a normal scheduler. - Say what
initialize_runtimedid and did not prove. - Read the overhead numbers as a loop-including comparison, not as a pure NIF cost.
Why this matters
Mojo's selling point is native code and a runtime with a thread pool. A BEAM node already has schedulers. Loading another thread pool into every NIF call can oversubscribe the machine and can run code on threads the VM does not know about. The integer proof avoids that pool on purpose.
Mental model
normal scheduler thread
|
+-- NIF add (flags = 0) -- enif_get_long, add, enif_make_long
|
+-- other Erlang processes wait while this NIF runs
dirty CPU scheduler only if flags include 1
dirty IO scheduler only if flags include 2
Mojo runtime threads only if initialize_runtime / parallelize ranTheory
A normal NIF should do a bounded amount of work. It does not preempt. Reductions are how the VM accounts for Erlang work; a NIF that cares calls enif_consume_timeslice. add does not. The body is three external calls and a checked add.
Dirty flags are the unsigned at offset 24 of ErlNifFunc. Values 1 and 2 come from erl_drv_nif.h via erl_nif.h. They are not inferred from Mojo annotations because there are no Mojo annotations in this project. Leaving the field 0 keeps add on the normal scheduler.
A GPU launch and a GPU wait are different costs. A launch that only queues work can be short. A wait that spins or sleeps holds the scheduler for the whole wait. This repository never launched a GPU kernel. The compiler rejected std.gpu and pointed at the max package. No claim is made about GPU code inside a NIF.
initialize_runtime prepares the process-global Mojo runtime. The manual says it is idempotent and that one call covers the process. The experiment in experiments/abi/05_runtime/rtnif.mojo calls it twice inside nif_init and then returns the term 7. The fresh BEAM printed runtime ping 7. So two calls during init did not crash, and a later NIF call on that library worked. The add library does not call it, and mix test still passes. Idempotence under concurrent scheduler threads was not tested. Thread-pool work was not tested.
Mojo traps and memory faults are not catchable as Elixir exceptions. The IR contains llvm.trap. If that runs inside the NIF, the OS process dies. scripts/test-isolated.sh puts mix test under an alarm so a dead or stuck VM does not hang the shell. It does not turn a trap into ArgumentError.
Hands-on exercise
Rebuild and load the runtime experiment:
.venv/bin/mojo build experiments/abi/05_runtime/rtnif.mojo \
--emit shared-lib -o /tmp/librt.so \
-Xlinker -undefined -Xlinker dynamic_lookup
Then load_nif the path without the suffix from a new elixir -e process, as in chapter 01's module shape, and call ping. The recorded result is the integer 7. A crashpad line on stderr can appear during mojo build and is separate from the BEAM result.
Do not point this experiment at priv/native/libpoc.so. That file is the add NIF and does not export ping.
Line-by-line walkthrough
The benchmark that was actually run, via mix run --no-start -e:
- warmup 10_000 calls of
Mojoler.POC.add(20, 22)and 10_000 evaluations of20 + 22 - 5 trials of 100_000 iterations each
:timer.tcaround the wholeforcomprehension, so the loop is inside the number
Recorded microseconds, NIF: 38188, 37250, 36213, 34686, 36222. Elixir +: 24655, 24961, 23204, 23876, 23806. Divided by 100_000 and converted to nanoseconds, that is about 347–382 ns per NIF iteration and about 232–250 ns per Elixir iteration, loop included.
The Elixir addition of two small integers is a VM instruction. The NIF pays for entering native code and two enif_* calls to do the same arithmetic. The NIF is the slower one. Speed is not the result of Stage 0.
Under the hood
libKGENCompilerRTShared.dylib is mapped when the .so loads, even though add never calls initialize_runtime. Its presence means the process is running vendor runtime code at load time. What that code does on dlopen was not traced instruction by instruction. The observable fact is that load_nif returns :ok and later calls work.
Oversubscription: if a future NIF called parallelize across all cores from a normal scheduler, it would compete with the other BEAM schedulers. The dirty CPU pool is the VM's tool for "this native work may use more CPU". Combining it with a Mojo thread pool would still need a measured cap. Not done here.
Failure modes
- Interpreting the nanosecond figures as the cost of
nif_initor as a pure call boundary. Both measurements include the Elixirforloop, and they were taken in one OS process with no statistical model beyond five samples. - Calling
initialize_runtimeon everyadd"to be safe". The passing NIF shows it is not required for this body, and doing it per call would be extra shared-state traffic that was not measured. - Expecting a Mojo panic to show up as
{:error, ...}in Elixir. It will not.
Rustler and Zigler
Rustler documents dirty schedulers as an annotation because Rust NIFs often do real work, and it treats panics as fatal. Zig has no separate runtime thread pool in the usual Zigler NIF, so the oversubscription problem is mostly "did you start threads yourself". Mojo is closer to a runtime language than Zig is. The honest Stage 0 stance is: the runtime dylib is loaded, the thread pool is not started by add, and starting it inside the BEAM was only tested as "call initialize_runtime twice, then return 7".
Checkpoint
- What flag value keeps a NIF on a normal scheduler? 0.
- Did
addcallinitialize_runtime? No. Did a different NIF call it twice and survive? Yes. - Stretch: repeat the five trials in a freshly booted VM three times and keep the spread. Do not drop the Elixir baseline.