How are you planning to enforce safety?

[kFYatek]

Other language-based operating systems, which are based on e.g. JVM or CLR, get away with not having CPU-level user mode or memory protection by having a virtual machine, which is known to take safety precautions (including verifying safety of the user-mode executable bytecode).

But Rust is compiled to native code. rustc invoked on RustOS will probably also just generate x86 (or whatever target architecture) code. How do you plan to distinguish between Rust-compiled, safe (I bet you plan to ban the unsafe keyword) binaries and some malicious code that one may put together using assembly or C or whatever and put on a RustOS machine?

Or are you planning to take some form of bytecode and compile upon exec() (or equivalent)?

[ryanra]

Good question. The original idea I had is to use a trusted compiler that would only compile "safe" code (and yes, for rustc that would essentially mean disallowing unsafe). Of course, the other option of compiling (or interpreting) verified bytecode on the fly would work but it seems harder to do.

[kFYatek]

How would you determine that a user-provided binary was compiled using a "safe" compiler? I can't think of any safe solution other than compiling at exec() time. Or using some cryptographic signature on binaries, but then the compiler would need to have access to a private key, and retrieving it from an installed system shouldn't be too hard...

Personally I can't see how a native code-based system can both be safe without memory protection and accept third-party binaries.

[ryanra]

Ah okay, that's a good point. My original idea was to only run code compiled on that machine. When the binary is outputted by the compiler, it could be marked as "safe" so you don't have to compile every exec().

I remember also reading about compilers that attach some sort of safety proof to the binary, which could be checked when you obtain it. I'm not sure if this is actually used in practice or just a research thing.

Re running third-party binaries directly, you're right. If the binary is running unmediated on hardware, then you need to rely on that same hardware for safety. But that's not the idea for RustOS.