I have a network stack

[Ericson2314]

Please excuse the frivolity of this pull request. Without a public issue tracker, this was the only way I could send you a message.

I have written (for a class) a network stack in Rust, with the intention of adding it to an exokernel in Rust someday. You have the most actively-developed exokernel and, if the commit history is any indication, are currently adding networking support to it.

I would need to get permission from the Professor and my assignment partner to open source it, as it is a school assignment. But other than that I could start working on integrating the two in about a month, as that is when school gets out.

[ryanra]

Awesome! Yeah, I'd welcome contributions, especially for a networking stack.

Pull request is fine too. As you probably noticed, I haven't used much of github's features yet -- my bad that you had to go this indirect route to contact me. Also, I think my email should be in the commit log if you cloned the repo -- you can use that to contact me too.

[Ericson2314]

Great! Now that I made this pull request the issues tab appears, so perhaps it didn't show up simply because there were no issues yet, not because it was private.

[ryanra]

Okay, I also enabled issue tracking for the repo.

Before you start, it would make sense to talk about how you're planning to interface with the rest of the kernel (i.e., the specifics of the dependencies you'll have and that of the interface you're planning to expose). Feel free to ping me here or by email when you're ready.

[ryanra]

Also: was the pull request something you wanted to merge in (it's fine if it's something you'd potentially like/would want to eventually work on) or was it just a way to get in contact?

[Ericson2314]

Sure. So right now, I have it broken up into MANY (cargo) libraries. Off the top of my head there is:

1. Network layer interface (some types and a trait)
2. Network layer implementation on top of libstd UDP (reads packets in a loop and hands them to a callback)
3. IPv4 on top of network layer interface (not UDP impl!).
4. Psuedo-RIP on top of IPv4 (it is registered as a protocol on top of IP, not UDP).
5. TCP on top pf IPv4 (incomplete).

Notably, outside of timeouts for TCP and RIP, none of this code spawns any threads--everything is callbacks based (with unboxed closures). I did this for a number for reasons.

First, I felt that this was the abstraction upon which everything else could best be built on--spawning a thread is from a callback is near free, while spinning up a thread to just run a callback isn't. As proof, the part of TCP that currently works has both an aync API, and a capabilities/sockets/however-you-want-to-view-it synchronous API on top.

Second, this means it should better mirror the workings of a network driver. Truth be told, other than tinkering a bit with https://github.com/pczarn/rustboot I have no kernelspace experience. I read a bunch on DMA drivers and whatnot, and it seemed that the card interrupting the kernel, which would in turn pass off the packet to a callback registered via the network-layer interface, would be a simple design to implement.

The code as written then uses mostly libcollections and below, with the exception that the calbacks are stored in very global hashmaps riddled with locks. It seemed to be a nasty tradeoff that by forgoing, let's say, a thread per IP protocol or TCP connection, I had to maintain this global state--in essence, avoiding scheduling led me to write my own bare-bones coopertive scheduler.

Here would be plan for integrating it:

I like that my net stack is not only is the code modular, but that its modularity is enforced with many Cargo packages. I wonder if might be possible to get RustOS to be built with Cargo now that there are build scripts, and the flexible target specification. I imagine/hope that eventually the standard library and rustc will be built with cargo, so that whatever could be hacked together now will get better with time.

After RustOS is built with cargo, I'd need it to use a more recent version of Rust, as my code makes heavy use of unboxed closures. The current work removing librustrt and consolidating the platform-dependent parts of libstd should help with this, and maintenance going forward.

Then, my net stack can be added as a Cargo dependency. I'm a bit confused on what parts of libstd are supported ATM, it looks like a libstd is linked, but it might be stripped down from the real thing. I should probably right some sot of concurrent map trait anways, and between that, and a mechanism to register callbacks on a timer, I think I could limit myself to libcollections and below. I can forgo the need for the latter by ignoring RIP and TCP initially, which I would do.

Finally, once everything can be built together (dependencies are met, etc), it should be very easy to actually hook it all up.

Future work (not really specific to integrating with RustOS) would be making a network-layer trait analogous to my link-layer trait for IPv4 and IPv6. With Associated types I think the traits could be sub-traits of one trait with associated types for packets, addresses, etc. And making UDP.

Also, once the consolidation of the os-specific parts of libstd is complete, implementing the synchronous sockets part of libstd should be totally doable.

[Ericson2314]

Haha, I figure running Servo is a fine long term goal, but one that's very far off and that I (or anybody else) couldn't work on anytime soon. Do what you want with it. :D

[ryanra]

Cool re the features and implementation.

Yes, std is stripped down. The big things gone from it are threading and IO. collections is there though. I have a mental todo item to actually remove use of std and instead go directly import the implementing library with an extern crate X (because the goal is to implement std). It would be great if you could do that; it would also formalize the dependencies that you have.

Regarding Cargo, I did some investigation awhile ago and found it too hard to do the custom rust lib build that I'm currently doing. There are also assembly and custom linker scripts that need special handling. This could be something look into. Newer rust version should be fine.

For dependencies: it seems like your changes will require threading support, which I'm planning to work on next. Good to see that OS dependent parts are being consolidated -- could you point me to somewhere that describes the plan to do so in more detail?

So, if you limited yourself to core, collections, and rustrt (or whatever equivalent is now), I think things should be good to go by the time you're ready to integrate. Also, let me know when you've open-sourced it so I can take a look.

[Ericson2314]

I have a mental todo item to actually remove use of std and instead go directly import the implementing library with an extern crate X (because the goal is to implement std). It would be great if you could do that; it would also formalize the dependencies that you have.

Regarding using the crates "behind the facade", and not std itself, I couldn't agree more. I would have written it that way originally had I more time.

Regarding Cargo, I did some investigation awhile ago and found it too hard to do the custom rust lib build that I'm currently doing. There are also assembly and custom linker scripts that need special handling.

Yeah, using Cargo would definitely be cumbersome in the short term. I'd hope the devs would agree this sort of usecase ought to be accounted for, so we'd be in a good position to make feature requests.

Newer rust version should be fine.

I checked, and right now they actually temporarily got rid of libsync in preparation for getting rid of librustrt. So while upgrading Rust would thus be a bad idea at the moment, it should be a really good idea soon.

For dependencies: it seems like your changes will require threading support, which I'm planning to work on next

I actually think I won't require threading support at all. Some sort of Cell should suffice instead of a lock in the single-threaded case. Long term, it might make the most sense to have the network stack and scheduler work in tandem -- e.g. the scheduler could implement the various traits meant to store the callbacks for protocol handlers and connections. That said, since I won't have time until school is out in a month to do much, so you could have threading ready once I am.

Good to see that OS dependent parts are being consolidated -- could you point me to somewhere that describes the plan to do so in more detail?

https://github.com/rust-lang/rfcs/blob/master/text/0230-remove-runtime.md lays out the plan for consolidating system specific stuff. The thing to do seems to be to implement libnative directly once it exists -- Avoiding libc (the interface or implementations) seems very much in the spirit of a language based system.

So, if you limited yourself to core, collections, and rustrt (or whatever equivalent is now), I think things should be good to go by the time you're ready to integrate. Also, let me know when you've open-sourced it so I can take a look.

Yeah, with the locks factored out into traits + impl with Cells, the IP part should just use collections and below + some sort of Cell (that I can just hack in for the time being). I have class tomorrow so I will talk to the professor re open-sourcing then.

[Ericson2314]

Voilà https://github.com/Ericson2314/rust-net

[ryanra]

Cool! Won't have much free time for a couple of weeks though -- will be able to do a deeper inspection and get to threading after that (and maybe wait for the upstream rust lib refactor to be complete too).

[Ericson2314]

Sounds like we will both have more free time around the same time---should work out nicely.

[Ericson2314]

Ok, I'm ready to work on this a lot! Great job starting to integrate cargo. I figured with that in place, it might make sense to separate the source from the bundled deps, and also git lazy-static from github. (that is what is now in those three on my branch-but I need to test so don't merge yet). What do you think?

[ryanra]

Cool! I still to make a more thorough review of your net stack (should be able to get to it in the next few days). My preliminary thoughts on interfacing with a network card:

1. For sending packets, it seems like all we'll need is something like a send_frame(bytes: &[u8]) function that will put bits directly on the wire.
2. For receiving packets, the net driver will somehow make the signal pack to the stack that it picked up a frame.
   • The simplest interface I could think of for this is a blocking call where a net stack thread is just constantly polling and will wake up whenever traffic comes. (This could be as simple as a blocking receive_frame() call or more abstract with something like a blocking queue)

The interface on the other side is complicated by demultiplexing with UDP/TCP ports. I'll look more into your code here.

Also, I fixed up your last commit in the pull request (Makefile's rust path needed to change and I squashed the last 2 commits into 1), and I've put it in master.

[Ericson2314]

Excellent, I'll update my repo. Do you go on the rust IRC (or another)? Might be easier to make plans in real time.

[ryanra]

Yeah, that works -- I'm using the nick ryan1 atm.

[Ericson2314]

Hmm, for some reason i can't built it again. Even old commits. lib_context.a in particular won't build. Also, rust-lang/rust#20237 -- we may not need to patch liballoc soon!

[Ericson2314]

Nevermind, had the wrong version of rust checked out.

[Ericson2314]

FYI I started porting to rust master, if you want to work on that at all.

[ryanra]

Is there a new feature that you need from master? (b/c waiting for next official release would have a few benefits)

[Ericson2314]

Well unboxed closures and associated types (assuming the latter is in decent shape) are generally nice to have. Also between both of our wanting to not use libstd (since we are implementing it), and the changes to liballoc, I believe we wouldn't need to patch rust at all.

[Ericson2314]

1.0 alpha is supposed to be January 9. So on one hand, that's coming very soon, on the other should be less churn after.

[ryanra]

Jan 9 is pretty close so I would do an upgrade before then only if it's blocking something now.

[Ericson2314]

I guess think of it as an upgrade in anticipation of Jan 9 :). E.g. it would be useful to make sure vanilla liballoc really works with those PRs merged.